Open Data

Celebrate Open Data Day in New York This Weekend

Every March, we’re excited to join data enthusiasts worldwide to celebrate International Data Day, a worldwide event that promotes awareness and use of open data. Through a series of events around the globe, people of all skill levels converge to create new projects, analyze data, and find new ways to visualize data.

We believe open data is a priority for civic tech enthusiasts — and we invite you to join us as we kick off the open data celebration this weekend. Here are some highlights of this weekend’s schedule — we hope to see you there:

March 3-5

Giving Tuesday DataDive, Presented by 92Y, DataKind, and the Bill & Melinda Gates Foundation

  • Friday 3/3 6:30pm-8pm EST: discuss goals for the DataDive and dive into the data!
  • Saturday 3/4 9am-9pm EST: choose a team and get to work!
  • Sunday 3/5 9am-3pm EST: final presentations and networking
    Note: You can attend one or all days.

We’re thrilled to be hosting a DataDive March 3-5 and are looking for data pros of all backgrounds to roll up their sleeves and work side by side with experts from the 92Y, the Bill & Melinda Gates Foundation, and Facebook to help use data to unravel tough questions and prototype new solutions.

March 4

International Data Day

Open Data Day is an annual celebration of open data all over the world. For the fifth time in history, groups from around the world will create local events on the day where they will use open data in their communities. It is an opportunity to show the benefits of open data and encourage the adoption of open data policies in government, business and civil society. View activities happening around the world here.

NYC School of Data (SOLD OUT)

New York City School of Data is a community conference showcasing NYC’s civic design, civic/government technology, and open data ecosystem.

March 6

Civic Hall Presents: Open Data, Mapping Global Security & the Department of Defense

How can we get national security data into the open? The National Geospatial-Intelligence Agency (NGA) will demo its geospatial data portals for the Arctic, for combating wildlife trafficking in Africa, and for Hurricane Matthew.

March 7

Five Year Anniversary of New York City’s Open Data Law, Local Law 11 of 2012

In many countries, states and cities Open Data is a policy – here in New York City it is a law, which ensures that Open Data is here to stay.

NYC Chief Analytics Officer Dr. Amen Ra Mashariki speaking at Socrata’s Connect 2017 Conference in DC

10 – 10:25am on the Main Stage. Livestream details coming soon.

NYC Big Apps Workshop – NYC Open Data Portal & Department of City Planning Facilities Explorer Tutorials

Join members of the NYC Open Data team and Department of City Planning for a demo of the NYC Open Data Portal and new Facilities Explorer tool (launching soon) followed by a breakout session at the Tuesday March 7th NYC Big Apps Workshop. You’ll learn the basics about how to access NYC data (1600+ datasets!) and get an overview of other tools such as the Facilities Explorer powered by NYC Open Data that you can use to support your research and work for the Big Apps competition as well.

March 8

Made in NY Media Center + Fabernovel Data & Media: Open Data Breakfast

Whether you are a developer, agency or civil service non-profit having access to data drives business, improves services, and promotes free public access.

Together with FaberNovel we are hosting and interactive breakfast and conversation on March 8th to learn more about the City of New York’s Free Open Data Portal and how you can use it to build products, conduct research and analysis or create new applications.

Department of Small Business Services: 2017 Smart Districts Summit

Inaugural NYC Smart Districts Summit, where community and technology leaders will collaboratively explore how emerging technologies are being leveraged to address the most pressing district-level challenges.

College of Staten Island (CSI) Tech Incubator + Vizalytics: Data – A Driving Force of Innovation

Connect with us to discover how organizations and entrepreneurs are utilizing data to drive innovation within our local community. Learn the practices, technologies, and patterns the experts use to fuel their enterprises by way of big data.

March 9

Reaktor Open Data Studio

The goal of this evening is to share some ideas about how Open Data could be utilized in new ways, especially in New York. We have a happy hour with benchmarks from Helsinki, where open data catalogues have been advanced for a while, and companies and developers alike are used to creating cool applications for it.

Join us to hear examples of applying open data in a user-friendly way, and let’s come up with new ways to use open data to create new tools.

General Assembly Panel Discussion: Data and…Health

Big Data is continuing to significantly impact the way in which organizations operate and make informed business decisions. Emerging technologies are now paving the way to innovative medical developments, and it looks as though data is beginning to transform the entire healthcare industry! In collaboration with the first annual NYC Open Data Week, GA is bringing together influencers from the health and wellness spaces to discuss how data is impacting their organizations.

March 11

NYC Parks Computer Resource Centers Open Data for All: TreesCount! Workshop

This free workshop, presented by NYC Parks and the NYC Open Data team, offers a broad introduction to the NYC Open Data Portal along with the concept of data literacy and analysis.

Using NYC TreesCount! 2015 data, the most accurate map of NYC’s street trees ever created, you will learn how to identify, download, manipulate, and visualize NYC Open Data with a focus on community engagement and awareness. Using tools such as Google Sheets and CARTO, you will be able to create your own graphs and maps from NYC Open Data.

Habitat III: A Once-In-A-Generation Civic Experience

habitat3

Photo: John Paul Farmer

It’s hard to catch your breath in Quito, Ecuador. Whether it’s the thin air of its 10,000 foot elevation, the natural beauty of its volcanic mountains, or the built beauty of its colonial-era architecture, Quito is a city that leaves you breathless.

Last month, 30,000 people came together in the scenic Ecuadorian capital to discuss the future of cities at Habitat III. Hosted by the United Nations, this once-every-20-years convening marked just the third of its kind, following in the footsteps of Habitat I in Vancouver in 1976 and Habitat II in Istanbul in 1996. UN-hosted World Urban Forums have been held every couple of years in recent decades, although none has reached the scale of Habitat.

At Habitat III, a wide range of individuals and organizations – including governments, companies, non-profits, and academic institutions – gathered to share best practices, to celebrate successes, and to approve a New Urban Agenda that marks the culmination of years of negotiations among United Nations member states.

Gatherings ranged from formal (including official delegate discussions in the National Theater), to participatory (such as the youth assemblies) to informal (like the lightning talks that electrified the expo hall). Some of the most interesting highlights were the following:

The Global Municipal Database – Lourdes German, Director of International and Institute-wide Initiatives at the Lincoln Institute, showcased a dashboard for cities that is built upon Microsoft technologies such as Azure, Power Map and Power BI. Working with cities in Africa, Asia, and Latin America, the Global Municipal Database tracks key fiscal indicators including expenditures, revenue, and borrowing and gives communities the tools to visualize the data and create actionable insights. What’s so powerful about these technologies is that many of their functionalities are Excel-based, meaning millions of people could use them tomorrow to make their cities more transparent and accountable, with no further training necessary.

Water and Resilience – It has been said that everyone has a water problem: either too polluted, too much, or not enough. For example, fully one-third of the Netherlands – a country built on its shipping and ports – lies below sea level. The country’s strength – water – is also its greatest vulnerability. With years of such experience living with water, the Netherlands was especially well qualified to host a conversation on the subject, which included viewpoints from Rotterdam and The Hague as well as a framework shared by 100 Resilient Cities’ Andy Salkin. One insight from The Hague was that resilience is not only physical, but must also be social and digital. Every aspect of a city must be able to bounce back. And while the cities of the Netherlands are especially advanced in learning how to live with water, most cities around the world are just getting started.

Public Spaces – Public spaces also played a key role, with planners asking whether placemaking will be at the heart of cities in the future. With a discussion of Eastern and Western traditions in terms of public spaces, the room erupted into a lively debate, during which an audience member noted that urban planners are increasingly using Microsoft’s Minecraft to engage people – particularly the young – in co-designing their own public spaces.

Housing – Housing was a major focus at Habitat III, for developed cities such as New York and for developing cities such as Lagos alike. With the majority of humanity living in cities for the first time in history, the influx of newcomers creates new stresses. Safe, accessible, and affordable housing is a priority.

Accessibility – A theme that was more woven into the conference experience than something explicitly called out was the need for more accessible communities. Microsoft is increasingly collaborating with cities to use technology to improve accessibility to services, information, and opportunity. “Eliminate the unnecessary barriers that limit our potential,” implored Dr. Victor Santiago Pineda of the University of California at Berkeley, who also served as co-chair for accessibility at Habitat III.

Youth – A particularly interesting aspect of Habitat III was the prevalence of young people everywhere you went. While most delegates were more senior, accomplished professionals, the conference grounds also teamed with young people of high school and college age. Many of those youth were local Ecuadorians engaging with this once-in-a-lifetime event that was on their home soil. Others were young people from around the world who journeyed to Quito to serve as agents of change. A middle-aged delegate at one youth-run session exclaimed “I’ve been going to sessions back-to-back for two days and this is the first one that is participatory. I think we need more of this.”

After several incredible days in Quito, the big question on everyone’s lips was, “What happens next?” How does the New Urban Agenda get implemented? To what extent will cities be prioritized by the UN? What role will technology play in forging solutions to our hardest problems? Will upcoming World Urban Forums be effectively leveraged to ensure steady progress on such audacious goals? Will the assumptions and priorities of Habitat III stand the test of time? Only time will tell.

Habitat III brought together planners, policymakers, technologists, and young people who care about the future of cities. Technology was there and will be an increasingly ubiquitous part of our lives. These new cross-sector connections have the potential to pay dividends between now and Habitat IV in 2036 – but that potential requires action by us to be fulfilled.

Source: Habitat III

RECAP: White House Open Data Innovation Summit (#WHOpenData)

This week, we were honored to join and support the White House Open Data Innovation Summit. Leaders in open data, including White House leaders in data U.S. Chief Information Officer Tony Scott and U.S. Chief Technology Officer Megan Smith, spent the summit championing the use of Federal open data across all sectors.

Read more about the growth of open data with the Center for Open Data Enterprise‘s new Open Data Best Practices Report. You can also watch the livestream of the summit here.

Some top tweets from the Open Data Innovation Summit:

Data science for safer streets: DataKind Vision Zero project expands to three new cities

intersection-from-above21

Last August, Microsoft announced its partnership with DataKind to support the Vision Zero movement in the U.S., which aims to reduce traffic-related deaths and severe injuries to zero in cities around the world. Today Microsoft and DataKind said that San Jose, Seattle and New Orleans will join New York City as the lead cities working on this initiative.

“The DataKind Vision Zero project is a demonstration of the possibilities created by bringing diverse sources of data and expertise together,” writes Elizabeth L. Grossman, Microsoft Technology & Civic Engagement director of civic projects.

“New data science analyses, using a combination of public and private data, will be designed to help local decision makers identify and evaluate which engineering, education and enforcement interventions can most effectively address each city’s local efforts to increase traffic safety for all.”

Read more on Microsoft on the Issues.

DataViz for good: How to ethically communicate data in a visual manner: #RDFviz

IMG_20160115_155638

Catherine D’Ignazio brainstorms around data inclusion

Last Friday I participated in my second Responsible Data Forum. Last year’s workshop on private sector data sharing (data philanthropy, if you like) inspired some of our thinking and collaborations over the past year, and today’s event about data visualization for social impact did not disappoint. You can see what people posted at #RDFviz, on the wiki, and in a great collection of related resources here.

IMG_20160115_113307

Mushon Zer-Aviv facilitates the Responsible Data Forum

At the top of the day, we did the classic Post-It note brainstorm to inventory all of the potential avenues for working groups. Given the incredible experience of the people in the room, there was a lot to work with. To give you a sense of the conversation and work coming out of this event, I’ve attempted to capture a sample of the questions and prompts the participants asked:

  • Non-screen data visualizations
    • Experiential data visualization, sonification, physical experiences, and installations
    • Data viz for the blind
    • Sand mandalas
    • Getting data offline
    • Translating data visualizations across various forms of media
    • Low-bandwidth visuals for inclusivity
  • Communicating uncertainty
    • How do we communicate uncertainty in data?
    • In metadata?
    • How do we represent gaps in the data?
    • What if our knowledge of the uncertainty in the data is anecdotal?
    • How can visuals show “no answer”?
    • How can data visualization promote ambiguity?
  • Literacy
    • How do we improve everyone’s data visualization literacy, as creators and as viewers?
    • How do we educate people about the data they create?
    • Which people / sectors / fields most need data literacy?
    • Can we provide interactive tools that let viewers adjust data visualizations in real time as a means of improving literacy?
    • How can we support grassroots groups to create better data visualization?
    • Is there a need for basic design principles and data viz 101 resources for human rights activists?
    • How do we navigate a fear of numbers?
  • Perspective
    • How do we visualize when there’s a dispute or a problem with the “facts”?
    • How do we show different perspectives on the same data?
    • How do we establish trust with our audience?
  • Data Visualization Theory (one of the less popular categories in this very practical group)
    • Let’s connect #RDFViz with the academic visualization community
    • How do we create a data visualization of data visualization?
    • Is data visualization abstracted thought?
  • Power and Data Visualization
    • Is persuasive data visualization
      • good?
      • bad?
      • necessary?
    • The relationship between big data and advocacy visualization
    • If we don’t amplify what we don’t know, visualization will amplify the most powerful voices
    • What does good adversarial data visualization look like?
  • BAD data viz
    • Is meaningless data visualization worth anything?
    • What about when people make decisions based on bad data viz?
    • If raw data is unrepresentative, will visualizations on it be bad?
    • We should collect examples of unethical data visualization
  • Data Visualization Tools
    • Let’s consider the limits of software and the tools we use
    • The trade-off between ease of use and privacy
    • Data visualization does not immediately create data storytelling
    • We should be more open about the true cost of doing a data visualization
    • We need tools that allow us to share our process as well as the data source and output
    • “Proprietary viz companies will die” vs. “Open source communities are Kafkaesque nightmares”
    • There’s a distinct lack of non-English data viz tools
    • What are some reasonable principles or guidelines to provide designers creating software tools for use by the general public and specialists?
    • Which types of interactivity are most useful in enhancing analytical inspiration?
  • Data Visualization Methodology
    • We should discuss methodologies when we discuss visualizing data
      • How do we choose what we visualize?
      • How do we represent data quality?
      • How do we visualize metadata?
    • What’s the lifespan of an infographic? Can we design continuously updated visuals, or include expiration dates for stale graphics?
    • How do we encourage consideration of ethics in the creation process of data visualizations?
  • Collaboration
    • Let’s connect the data producers and the visualizers with a tighter feedback loop. The producers will see how their data’s been applied in the world, and visualizers will get a better sense of the contours of the data.
    • How do we encourage more collaboration between human rights activists and data visualizers?
  • Engagement and Participation
  • Audience
    • How do we involve the audience?
      Who is the audience, and why?
    • How do we create community ownership of a data viz?
    • How do we allow a data viz to speak to multiple disparate audiences?
  • Transparency and openness
    • Expose methodologies
    • Replicability of a data viz
    • Making the data viz process transparent
    • What assumptions are there in that data visualization?
    • How do design and aesthetic decisions bias a data viz?
  • Simplicity
    • How can we be succinct without over-simplifying the content?
    • Nuanced vs. bombastic
    • Can we build a language for the critique of data visualizations’ ethics?
    • Are there ethical ways to avoid nuance?
    • Presenting individual data points vs. an overview
  • Objectivity vs. subjectivity
    • Data as expression vs. data as fact
    • Is objectivity desired?
    • How do we use empathy without creating compassion fatigue?
    • The difference between invoking sympathy vs empathy
  • Honesty
    • When is a data viz most true?
    • When is a data viz most honest?
    • What about high-stakes data visualizations, like when there are life and death risks for participating subjects?
    • How do we incorporate criticism and critique into the visualization?
    • Data visualization is rooted in an Enlightenment fallacy that “the truth”, presented just so, will change things
  • Motivation and goals
  • Responsibility
    • Anonymizing data
    • Fact-checking data
    • Transparency vs. protection of subject
    • Marginal populations
    • Whose data is it, and is there consent?
    • Responsibly visualizing video / images
    • Does reliance on data de-humanize subjects?
    • How do we responsibly reduce complexity to convey points?
    • How do we make the creators of data visualizations
  • Culture
  • Risk & danger
  • The future…
    • Is visualization always stuck in the past?
    • Time travel strategies for slowing down time
    • Holodeck data visualization
IMG_20160115_113237

A constellation of Post-its

This is only a partial list, as I wasn’t able to type quickly enough for the fast-moving Post-It notes. You can view the original Post-It constellation over here and keep up with the conversation and the creative outputs over at responsibledata.io.

Let Me Get That Data For You: The Bing-Powered Data Inventory Tool

Let Me Get That Data For You: The Bing-Powered Data Inventory Tool

Our friends at the US Open Data Institute work to make it easier for governments and others to open their data. One of the first things a government agency must do before launching an open data repository is conduct an inventory of the data they’re already publishing. It lets you get everything in one place. This is a relatively minor step to creating an open data policy and repository, but it still takes work. We were thrilled to learn that the US Open Data Institute team, including Waldo Jaquith, Ted Han, and Dan Schultz, used the Bing Search API to create a tool that radically streamlines the data inventory process.

It’s called Let Me Get That Data For You. All you have to do is enter in a website URL, and it will search that domain for common data formats and return a machine-readable list for you to use. And then you’re on step closer to sharing your data with the world.

As of today, the service will return up to 2,000 datasets per search. This should cover many of the intended use cases, but if you’re working with an extreme case, you can head over to Github and run the open source code yourself.

This Weekend, Hack Away With #CodeAcross and Open Data Day

This weekend, New York City is all about open data. On February 21 and 22, the New York tech community will be celebrating two benchmark events: #CodeAcross and International Open Data Day. Birthed from the same passion to realize the potential of open data and civic technology, this weekend’s events seek to initiate newcomers into the world of data and bring practitioners and community members together to create new value from our community’s data sets.

We’ll kick off the weekend at Microsoft’s Times Square headquarters in a conversation with New York City’s Chief Analytics Officer, Dr. Amen Ra Mashariki. BetaNYC’s Noel Hidalgo will deliver questions to Amen that have been crowdsourced from the community. Ask Amen your question here.

This Weekend, Hack Away With #CodeAcross and Open Data Day

Then, on Saturday and Sunday, we’ll celebrate Open Data Day and CodeAcross at our partner Civic Hall. #CodeAcross NYC is an open event aimed at directing the incredible energy and talent of the tech community toward some of our community’s most pressing challenges and opportunities. The two-day festival brings together governments, community groups, academic organizations, and individuals passionate about data to create impactful solutions.

Cities around the world will band together to gain insights from datasets, make new applications, drive forward existing projects, and altogether use open data to show how sharing information can transform how governments operate and how society solves problems. Through the use of data platforms like SQL, HDInsight, Microsoft Azure, Excel, and Power BI, citizens can partake in civic engagement that promotes openness through all facets of government.

We’re proud that NYC has more open data sets than any other city. There is a plethora of data to be used for public good,particularly when you include BetaNYC’s community-maintained open data sets.  With hackathons, unconference sessions, a workshop on mapping open data, and a series of themed challenges designed to improve the City of New York’s data and its usage, the next few days will be especially busy. Importantly, these events are open to all – technical and non-technical, governmental and non-governmental – and aimed toward teaching about open data, in addition to building new tools and apps. Because realizing the impact of civic technology, in general, and open data, in particular, requires an all hands on deck approach that will continue long past this weekend. We hope you’ll join us.

Register for #CodeAcross NYC at https://codeacrossnyc2015.eventbrite.com.

Want to work toward public good through civic tech past #CodeAcross? Here’s how to hack for a cause.

[Op-Ed] Full Spectrum Open Data

[Op-Ed] Full Spectrum Open Data

This post was originally published as an op-ed in TechPresident.

Transparency and open government advocates have been successful in convincing governments around the world to share some of their data with society at large. (And thanks to the Sunlight Foundation, we’ll soon know which data they’re not sharing, as well). But there is plenty of important civic information that isn’t collected or maintained by governments. We need to supplement open government data with data from others to give nonprofits, governments, and researchers a more holistic understanding of reality.

Here are a few projects working to augment open government data with open data from other sources:

Open Data Philly, first launched in 2011, is an early manifestation of the idea that official city data can live in the same open repository as other community datasets. Hosting a variety of datasets on the same repository suggests not only that we can bring them together, but also that we expect groups other than governments to contribute to the commons.

Last December, Singapore’s Infocomm agency shared its Federated Dataset Registry to help businesses discover the public- and private-sector open data available to them. This data-as-a-service platform, built on the open-source CKAN platform, is designed to help users discover private sector datasets that have recently been made available. They’ve offered the first 25 data providers $3,000 in web hosting credits to help encourage dataset contributions. And last week, the team hosted a Data Discovery Challenge competition to encourage “mashing” of the private and public data into new solutions, and, they hope, spur commerce.

In the crisis mapping space, online volunteers parse official data as well as social media and traditional news reports to improve emergency responders’ situational awareness. That situational awareness, or sense of what’s taking place on the ground, has traditionally been limited to formal data sources, and can now be augmented by a wider range of available information for a higher resolution picture.

Last summer in New York City, UN Global Pulse, the engine room, and the Data & Society Institute convened a forum on the responsible use of private sector data for public good. There’s a lot of work to be done on “the responsible use of data” side of things, and there’s no better host of that conversation than Data & Society.

Inspired by the workshop, participants from Microsoft, Data & Society, UN Global Pulse, and the Rockefeller Foundation are putting together a guide to help private sector actors (companies) consider opening up their data to the public sector (governments, nonprofits, and researchers). It is no longer possible to talk about opening up data without also considering the potential surveillance and marketing applications, which we begin to do in the roadmap. Please contribute your thoughts and links to this evergreen resource once we publish. The conversation around how to use private data for public good is increasingly nuanced, and we welcome it, because there’s still huge societal value in gaining a clearer picture of our world.

Video: NYC’s first Chief Technology Officer Minerva Tantoco

We were recently lucky enough to host NYC’s first-ever CTO, Minerva Tantoco (@minervatweet), in conversation with BetaNYC‘s Noel Hidalgo. The civic tech community showed up strong to the sold-out event at Microsoft Research. In the video below, Minerva shares her personal path to becoming Chief Technology Officer, as well as her personal thoughts on punk culture, science fiction, and her plans for keeping New York City on the cutting edge of civic tech. We were able to capture the conversation for your viewing pleasure thanks to Joly MacFie of the Internet Society’s New York chapter.

Unpacking open data: power, politics and the influence of infrastructures

Liveblog of a #Berkman lunch written with Erhardt Graeff.

Tim Davies (@timdavies) is a social researcher with interests in civic participation and civic technologies. He has spent the last five years focussing on the development of the open government data landscape around the world, from his MSc work at the Oxford Internet Institute on Data and Democracy, the first major study of data.gov.uk, through to leading a 12-country study on the Emerging Impacts of Open Data in Developing Countries for the World Wide Web Foundation.

A broad coalition of companies, governments, and other entities have come together to open data. This work is based on the belief that opening data creates myriad benefits to society, for transparency, for economic value, and other benefits.

Does open data reconfigure power relationships in the political space? The past, promise, and reality of open data reminds wide. Read more >