[Op-Ed] Full Spectrum Open Data - Microsoft New York

This post was originally published as an op-ed in TechPresident.

Transparency and open government advocates have been successful in convincing governments around the world to share some of their data with society at large. (And thanks to the Sunlight Foundation, we’ll soon know which data they’re not sharing, as well). But there is plenty of important civic information that isn’t collected or maintained by governments. We need to supplement open government data with data from others to give nonprofits, governments, and researchers a more holistic understanding of reality.

Here are a few projects working to augment open government data with open data from other sources:

Open Data Philly, first launched in 2011, is an early manifestation of the idea that official city data can live in the same open repository as other community datasets. Hosting a variety of datasets on the same repository suggests not only that we can bring them together, but also that we expect groups other than governments to contribute to the commons.

Last December, Singapore’s Infocomm agency shared its Federated Dataset Registry to help businesses discover the public- and private-sector open data available to them. This data-as-a-service platform, built on the open-source CKAN platform, is designed to help users discover private sector datasets that have recently been made available. They’ve offered the first 25 data providers $3,000 in web hosting credits to help encourage dataset contributions. And last week, the team hosted a Data Discovery Challenge competition to encourage “mashing” of the private and public data into new solutions, and, they hope, spur commerce.

In the crisis mapping space, online volunteers parse official data as well as social media and traditional news reports to improve emergency responders’ situational awareness. That situational awareness, or sense of what’s taking place on the ground, has traditionally been limited to formal data sources, and can now be augmented by a wider range of available information for a higher resolution picture.

Last summer in New York City, UN Global Pulse, the engine room, and the Data & Society Institute convened a forum on the responsible use of private sector data for public good. There’s a lot of work to be done on “the responsible use of data” side of things, and there’s no better host of that conversation than Data & Society.

Inspired by the workshop, participants from Microsoft, Data & Society, UN Global Pulse, and the Rockefeller Foundation are putting together a guide to help private sector actors (companies) consider opening up their data to the public sector (governments, nonprofits, and researchers). It is no longer possible to talk about opening up data without also considering the potential surveillance and marketing applications, which we begin to do in the roadmap. Please contribute your thoughts and links to this evergreen resource once we publish. The conversation around how to use private data for public good is increasingly nuanced, and we welcome it, because there’s still huge societal value in gaining a clearer picture of our world.