Open Data

Habitat III: A Once-In-A-Generation Civic Experience


Photo: John Paul Farmer

It’s hard to catch your breath in Quito, Ecuador. Whether it’s the thin air of its 10,000 foot elevation, the natural beauty of its volcanic mountains, or the built beauty of its colonial-era architecture, Quito is a city that leaves you breathless.

Last month, 30,000 people came together in the scenic Ecuadorian capital to discuss the future of cities at Habitat III. Hosted by the United Nations, this once-every-20-years convening marked just the third of its kind, following in the footsteps of Habitat I in Vancouver in 1976 and Habitat II in Istanbul in 1996. UN-hosted World Urban Forums have been held every couple of years in recent decades, although none has reached the scale of Habitat.

At Habitat III, a wide range of individuals and organizations – including governments, companies, non-profits, and academic institutions – gathered to share best practices, to celebrate successes, and to approve a New Urban Agenda that marks the culmination of years of negotiations among United Nations member states.

Gatherings ranged from formal (including official delegate discussions in the National Theater), to participatory (such as the youth assemblies) to informal (like the lightning talks that electrified the expo hall). Some of the most interesting highlights were the following:

The Global Municipal Database – Lourdes German, Director of International and Institute-wide Initiatives at the Lincoln Institute, showcased a dashboard for cities that is built upon Microsoft technologies such as Azure, Power Map and Power BI. Working with cities in Africa, Asia, and Latin America, the Global Municipal Database tracks key fiscal indicators including expenditures, revenue, and borrowing and gives communities the tools to visualize the data and create actionable insights. What’s so powerful about these technologies is that many of their functionalities are Excel-based, meaning millions of people could use them tomorrow to make their cities more transparent and accountable, with no further training necessary.

Water and Resilience – It has been said that everyone has a water problem: either too polluted, too much, or not enough. For example, fully one-third of the Netherlands – a country built on its shipping and ports – lies below sea level. The country’s strength – water – is also its greatest vulnerability. With years of such experience living with water, the Netherlands was especially well qualified to host a conversation on the subject, which included viewpoints from Rotterdam and The Hague as well as a framework shared by 100 Resilient Cities’ Andy Salkin. One insight from The Hague was that resilience is not only physical, but must also be social and digital. Every aspect of a city must be able to bounce back. And while the cities of the Netherlands are especially advanced in learning how to live with water, most cities around the world are just getting started.

Public Spaces – Public spaces also played a key role, with planners asking whether placemaking will be at the heart of cities in the future. With a discussion of Eastern and Western traditions in terms of public spaces, the room erupted into a lively debate, during which an audience member noted that urban planners are increasingly using Microsoft’s Minecraft to engage people – particularly the young – in co-designing their own public spaces.

Housing – Housing was a major focus at Habitat III, for developed cities such as New York and for developing cities such as Lagos alike. With the majority of humanity living in cities for the first time in history, the influx of newcomers creates new stresses. Safe, accessible, and affordable housing is a priority.

Accessibility – A theme that was more woven into the conference experience than something explicitly called out was the need for more accessible communities. Microsoft is increasingly collaborating with cities to use technology to improve accessibility to services, information, and opportunity. “Eliminate the unnecessary barriers that limit our potential,” implored Dr. Victor Santiago Pineda of the University of California at Berkeley, who also served as co-chair for accessibility at Habitat III.

Youth – A particularly interesting aspect of Habitat III was the prevalence of young people everywhere you went. While most delegates were more senior, accomplished professionals, the conference grounds also teamed with young people of high school and college age. Many of those youth were local Ecuadorians engaging with this once-in-a-lifetime event that was on their home soil. Others were young people from around the world who journeyed to Quito to serve as agents of change. A middle-aged delegate at one youth-run session exclaimed “I’ve been going to sessions back-to-back for two days and this is the first one that is participatory. I think we need more of this.”

After several incredible days in Quito, the big question on everyone’s lips was, “What happens next?” How does the New Urban Agenda get implemented? To what extent will cities be prioritized by the UN? What role will technology play in forging solutions to our hardest problems? Will upcoming World Urban Forums be effectively leveraged to ensure steady progress on such audacious goals? Will the assumptions and priorities of Habitat III stand the test of time? Only time will tell.

Habitat III brought together planners, policymakers, technologists, and young people who care about the future of cities. Technology was there and will be an increasingly ubiquitous part of our lives. These new cross-sector connections have the potential to pay dividends between now and Habitat IV in 2036 – but that potential requires action by us to be fulfilled.

Source: Habitat III

RECAP: White House Open Data Innovation Summit (#WHOpenData)

This week, we were honored to join and support the White House Open Data Innovation Summit. Leaders in open data, including White House leaders in data U.S. Chief Information Officer Tony Scott and U.S. Chief Technology Officer Megan Smith, spent the summit championing the use of Federal open data across all sectors.

Read more about the growth of open data with the Center for Open Data Enterprise‘s new Open Data Best Practices Report. You can also watch the livestream of the summit here.

Some top tweets from the Open Data Innovation Summit:

Data science for safer streets: DataKind Vision Zero project expands to three new cities


Last August, Microsoft announced its partnership with DataKind to support the Vision Zero movement in the U.S., which aims to reduce traffic-related deaths and severe injuries to zero in cities around the world. Today Microsoft and DataKind said that San Jose, Seattle and New Orleans will join New York City as the lead cities working on this initiative.

“The DataKind Vision Zero project is a demonstration of the possibilities created by bringing diverse sources of data and expertise together,” writes Elizabeth L. Grossman, Microsoft Technology & Civic Engagement director of civic projects.

“New data science analyses, using a combination of public and private data, will be designed to help local decision makers identify and evaluate which engineering, education and enforcement interventions can most effectively address each city’s local efforts to increase traffic safety for all.”

Read more on Microsoft on the Issues.

DataViz for good: How to ethically communicate data in a visual manner: #RDFviz


Catherine D’Ignazio brainstorms around data inclusion

Last Friday I participated in my second Responsible Data Forum. Last year’s workshop on private sector data sharing (data philanthropy, if you like) inspired some of our thinking and collaborations over the past year, and today’s event about data visualization for social impact did not disappoint. You can see what people posted at #RDFviz, on the wiki, and in a great collection of related resources here.


Mushon Zer-Aviv facilitates the Responsible Data Forum

At the top of the day, we did the classic Post-It note brainstorm to inventory all of the potential avenues for working groups. Given the incredible experience of the people in the room, there was a lot to work with. To give you a sense of the conversation and work coming out of this event, I’ve attempted to capture a sample of the questions and prompts the participants asked:

  • Non-screen data visualizations
    • Experiential data visualization, sonification, physical experiences, and installations
    • Data viz for the blind
    • Sand mandalas
    • Getting data offline
    • Translating data visualizations across various forms of media
    • Low-bandwidth visuals for inclusivity
  • Communicating uncertainty
    • How do we communicate uncertainty in data?
    • In metadata?
    • How do we represent gaps in the data?
    • What if our knowledge of the uncertainty in the data is anecdotal?
    • How can visuals show “no answer”?
    • How can data visualization promote ambiguity?
  • Literacy
    • How do we improve everyone’s data visualization literacy, as creators and as viewers?
    • How do we educate people about the data they create?
    • Which people / sectors / fields most need data literacy?
    • Can we provide interactive tools that let viewers adjust data visualizations in real time as a means of improving literacy?
    • How can we support grassroots groups to create better data visualization?
    • Is there a need for basic design principles and data viz 101 resources for human rights activists?
    • How do we navigate a fear of numbers?
  • Perspective
    • How do we visualize when there’s a dispute or a problem with the “facts”?
    • How do we show different perspectives on the same data?
    • How do we establish trust with our audience?
  • Data Visualization Theory (one of the less popular categories in this very practical group)
    • Let’s connect #RDFViz with the academic visualization community
    • How do we create a data visualization of data visualization?
    • Is data visualization abstracted thought?
  • Power and Data Visualization
    • Is persuasive data visualization
      • good?
      • bad?
      • necessary?
    • The relationship between big data and advocacy visualization
    • If we don’t amplify what we don’t know, visualization will amplify the most powerful voices
    • What does good adversarial data visualization look like?
  • BAD data viz
    • Is meaningless data visualization worth anything?
    • What about when people make decisions based on bad data viz?
    • If raw data is unrepresentative, will visualizations on it be bad?
    • We should collect examples of unethical data visualization
  • Data Visualization Tools
    • Let’s consider the limits of software and the tools we use
    • The trade-off between ease of use and privacy
    • Data visualization does not immediately create data storytelling
    • We should be more open about the true cost of doing a data visualization
    • We need tools that allow us to share our process as well as the data source and output
    • “Proprietary viz companies will die” vs. “Open source communities are Kafkaesque nightmares”
    • There’s a distinct lack of non-English data viz tools
    • What are some reasonable principles or guidelines to provide designers creating software tools for use by the general public and specialists?
    • Which types of interactivity are most useful in enhancing analytical inspiration?
  • Data Visualization Methodology
    • We should discuss methodologies when we discuss visualizing data
      • How do we choose what we visualize?
      • How do we represent data quality?
      • How do we visualize metadata?
    • What’s the lifespan of an infographic? Can we design continuously updated visuals, or include expiration dates for stale graphics?
    • How do we encourage consideration of ethics in the creation process of data visualizations?
  • Collaboration
    • Let’s connect the data producers and the visualizers with a tighter feedback loop. The producers will see how their data’s been applied in the world, and visualizers will get a better sense of the contours of the data.
    • How do we encourage more collaboration between human rights activists and data visualizers?
  • Engagement and Participation
  • Audience
    • How do we involve the audience?
      Who is the audience, and why?
    • How do we create community ownership of a data viz?
    • How do we allow a data viz to speak to multiple disparate audiences?
  • Transparency and openness
    • Expose methodologies
    • Replicability of a data viz
    • Making the data viz process transparent
    • What assumptions are there in that data visualization?
    • How do design and aesthetic decisions bias a data viz?
  • Simplicity
    • How can we be succinct without over-simplifying the content?
    • Nuanced vs. bombastic
    • Can we build a language for the critique of data visualizations’ ethics?
    • Are there ethical ways to avoid nuance?
    • Presenting individual data points vs. an overview
  • Objectivity vs. subjectivity
    • Data as expression vs. data as fact
    • Is objectivity desired?
    • How do we use empathy without creating compassion fatigue?
    • The difference between invoking sympathy vs empathy
  • Honesty
    • When is a data viz most true?
    • When is a data viz most honest?
    • What about high-stakes data visualizations, like when there are life and death risks for participating subjects?
    • How do we incorporate criticism and critique into the visualization?
    • Data visualization is rooted in an Enlightenment fallacy that “the truth”, presented just so, will change things
  • Motivation and goals
  • Responsibility
    • Anonymizing data
    • Fact-checking data
    • Transparency vs. protection of subject
    • Marginal populations
    • Whose data is it, and is there consent?
    • Responsibly visualizing video / images
    • Does reliance on data de-humanize subjects?
    • How do we responsibly reduce complexity to convey points?
    • How do we make the creators of data visualizations
  • Culture
  • Risk & danger
  • The future…
    • Is visualization always stuck in the past?
    • Time travel strategies for slowing down time
    • Holodeck data visualization

A constellation of Post-its

This is only a partial list, as I wasn’t able to type quickly enough for the fast-moving Post-It notes. You can view the original Post-It constellation over here and keep up with the conversation and the creative outputs over at

Let Me Get That Data For You: The Bing-Powered Data Inventory Tool

Let Me Get That Data For You: The Bing-Powered Data Inventory Tool

Our friends at the US Open Data Institute work to make it easier for governments and others to open their data. One of the first things a government agency must do before launching an open data repository is conduct an inventory of the data they’re already publishing. It lets you get everything in one place. This is a relatively minor step to creating an open data policy and repository, but it still takes work. We were thrilled to learn that the US Open Data Institute team, including Waldo Jaquith, Ted Han, and Dan Schultz, used the Bing Search API to create a tool that radically streamlines the data inventory process.

It’s called Let Me Get That Data For You. All you have to do is enter in a website URL, and it will search that domain for common data formats and return a machine-readable list for you to use. And then you’re on step closer to sharing your data with the world.

As of today, the service will return up to 2,000 datasets per search. This should cover many of the intended use cases, but if you’re working with an extreme case, you can head over to Github and run the open source code yourself.

This Weekend, Hack Away With #CodeAcross and Open Data Day

This weekend, New York City is all about open data. On February 21 and 22, the New York tech community will be celebrating two benchmark events: #CodeAcross and International Open Data Day. Birthed from the same passion to realize the potential of open data and civic technology, this weekend’s events seek to initiate newcomers into the world of data and bring practitioners and community members together to create new value from our community’s data sets.

We’ll kick off the weekend at Microsoft’s Times Square headquarters in a conversation with New York City’s Chief Analytics Officer, Dr. Amen Ra Mashariki. BetaNYC’s Noel Hidalgo will deliver questions to Amen that have been crowdsourced from the community. Ask Amen your question here.

This Weekend, Hack Away With #CodeAcross and Open Data Day

Then, on Saturday and Sunday, we’ll celebrate Open Data Day and CodeAcross at our partner Civic Hall. #CodeAcross NYC is an open event aimed at directing the incredible energy and talent of the tech community toward some of our community’s most pressing challenges and opportunities. The two-day festival brings together governments, community groups, academic organizations, and individuals passionate about data to create impactful solutions.

Cities around the world will band together to gain insights from datasets, make new applications, drive forward existing projects, and altogether use open data to show how sharing information can transform how governments operate and how society solves problems. Through the use of data platforms like SQL, HDInsight, Microsoft Azure, Excel, and Power BI, citizens can partake in civic engagement that promotes openness through all facets of government.

We’re proud that NYC has more open data sets than any other city. There is a plethora of data to be used for public good,particularly when you include BetaNYC’s community-maintained open data sets.  With hackathons, unconference sessions, a workshop on mapping open data, and a series of themed challenges designed to improve the City of New York’s data and its usage, the next few days will be especially busy. Importantly, these events are open to all – technical and non-technical, governmental and non-governmental – and aimed toward teaching about open data, in addition to building new tools and apps. Because realizing the impact of civic technology, in general, and open data, in particular, requires an all hands on deck approach that will continue long past this weekend. We hope you’ll join us.

Register for #CodeAcross NYC at

Want to work toward public good through civic tech past #CodeAcross? Here’s how to hack for a cause.

[Op-Ed] Full Spectrum Open Data

[Op-Ed] Full Spectrum Open Data

This post was originally published as an op-ed in TechPresident.

Transparency and open government advocates have been successful in convincing governments around the world to share some of their data with society at large. (And thanks to the Sunlight Foundation, we’ll soon know which data they’re not sharing, as well). But there is plenty of important civic information that isn’t collected or maintained by governments. We need to supplement open government data with data from others to give nonprofits, governments, and researchers a more holistic understanding of reality.

Here are a few projects working to augment open government data with open data from other sources:

Open Data Philly, first launched in 2011, is an early manifestation of the idea that official city data can live in the same open repository as other community datasets. Hosting a variety of datasets on the same repository suggests not only that we can bring them together, but also that we expect groups other than governments to contribute to the commons.

Last December, Singapore’s Infocomm agency shared its Federated Dataset Registry to help businesses discover the public- and private-sector open data available to them. This data-as-a-service platform, built on the open-source CKAN platform, is designed to help users discover private sector datasets that have recently been made available. They’ve offered the first 25 data providers $3,000 in web hosting credits to help encourage dataset contributions. And last week, the team hosted a Data Discovery Challenge competition to encourage “mashing” of the private and public data into new solutions, and, they hope, spur commerce.

In the crisis mapping space, online volunteers parse official data as well as social media and traditional news reports to improve emergency responders’ situational awareness. That situational awareness, or sense of what’s taking place on the ground, has traditionally been limited to formal data sources, and can now be augmented by a wider range of available information for a higher resolution picture.

Last summer in New York City, UN Global Pulse, the engine room, and the Data & Society Institute convened a forum on the responsible use of private sector data for public good. There’s a lot of work to be done on “the responsible use of data” side of things, and there’s no better host of that conversation than Data & Society.

Inspired by the workshop, participants from Microsoft, Data & Society, UN Global Pulse, and the Rockefeller Foundation are putting together a guide to help private sector actors (companies) consider opening up their data to the public sector (governments, nonprofits, and researchers). It is no longer possible to talk about opening up data without also considering the potential surveillance and marketing applications, which we begin to do in the roadmap. Please contribute your thoughts and links to this evergreen resource once we publish. The conversation around how to use private data for public good is increasingly nuanced, and we welcome it, because there’s still huge societal value in gaining a clearer picture of our world.

Video: NYC’s first Chief Technology Officer Minerva Tantoco

We were recently lucky enough to host NYC’s first-ever CTO, Minerva Tantoco (@minervatweet), in conversation with BetaNYC‘s Noel Hidalgo. The civic tech community showed up strong to the sold-out event at Microsoft Research. In the video below, Minerva shares her personal path to becoming Chief Technology Officer, as well as her personal thoughts on punk culture, science fiction, and her plans for keeping New York City on the cutting edge of civic tech. We were able to capture the conversation for your viewing pleasure thanks to Joly MacFie of the Internet Society’s New York chapter.

Unpacking open data: power, politics and the influence of infrastructures

Liveblog of a #Berkman lunch written with Erhardt Graeff.

Tim Davies (@timdavies) is a social researcher with interests in civic participation and civic technologies. He has spent the last five years focussing on the development of the open government data landscape around the world, from his MSc work at the Oxford Internet Institute on Data and Democracy, the first major study of, through to leading a 12-country study on the Emerging Impacts of Open Data in Developing Countries for the World Wide Web Foundation.

A broad coalition of companies, governments, and other entities have come together to open data. This work is based on the belief that opening data creates myriad benefits to society, for transparency, for economic value, and other benefits.

Does open data reconfigure power relationships in the political space? The past, promise, and reality of open data reminds wide. Read more >

Mayor de Blasio Announces BigApps Winners


Last night in Brooklyn, Mayor Bill de Blasio, newly-named NYC Chief Technology Officer Minerva Tantoco, Economic Development Corporation Chief Kyle Kimball, and hundreds of civic-minded community members gathered to celebrate the capstone event of BigApps NYC.  Now in its fifth year, the 2014 edition of BigApps NYC consisted of a series of workshops in the categories of Live, Work, Learn, and Play, held throughout the city over several months. New York City’s world-renowned open data initiative does more than just inspire, it produces impactful innovations from regular folks using data and technology to create meaningful solutions for their communities.

In a sign of Microsoft’s support for continued innovation in using data for good, I joined Mayor de Blasio on stage to present the award for best app in the Live category to Heat Seek NYC. The Heat Seek NYC team consists of seven forward-thinking doers, a number of whom developed their tech skills through the well-respected Flatiron School coding bootcamp. Combining these skills with insights into a real-world problem, the team produced a shining solution in a matter of months. The problem: too many New York City apartment buildings provide insufficient heat during cold winter days and nights, failing to meet their legal obligations to tenants and presenting a health risk for city residents. Heat Seek NYC devised and created mesh networks of low-cost, tamper-proof sensor boxes to relay real-time temperature data. This solution empowers residents with hard data on the temperature in their apartments and gives landlords the opportunity to quickly fix any heating outages. The Mayor was so impressed by the potential, that as soon as he learned about Heat Seek NYC he arranged a meeting with the Commissioner of the NYC Housing and Preservation Department.

Other winners announced by the Mayor included Coursekicker (walked away with first place in the Learn category), NYCHired (won in the Work category), and Explore NYC Parks (honored in the Play category).

Technology can empower people to change the world for the better. BigApps NYC is a great example of that. Civic innovation doesn’t come from some other people in some other place. It comes from us, right here. New York City has an incredible history of innovation and it has an exciting present, in which entrepreneurship, technology, and community are combining in powerful new ways.

The future? That’s up to us.