I’m excited to announce that we rolled out a new open data portal for the Philadelphia region today, OpenDataPhilly.org. Open data and government transparency have been increasingly visible concerns over the past few years. The City of Philadelphia was once a leader in this respect. The municipal government made its GIS data available to the public at no charge almost 10 years ago, and, at the time, was one of the first and largest municipalities in the world to do so. In order to do this, City staff worked through a number of challenging issues that included liability, homeland security and development of a common standard and process for vetting and releasing new data sets. That data has been available on PASDA, the state spatial data clearinghouse for Pennsylvania, ever since.

In the past few years, many municipal governments have been making a public and concerted effort to improve the transparency of their government operations by releasing significant and useful data sets. Washington DC deserves credit for playing a leadership role in this respect. DC was arguably the first major city to not only release downloadable data sets but create real-time streams of data from operational databases. Today the District provides access to 475 datasets from multiple agencies and in a variety of formats, CSV, RSS, KML, XML and shapefiles. In 2008, they doubled-down. To increase exposure and expand usage, the government sponsored a contest, Apps for Democracy, to encourage software developers to create useful applications that consumed this data. The leader of that effort, Vivek Kundra went on to become CIO under President Obama. In May 2009, the federal government launched Data.gov with just 47 data sets. Today there are 380,000 data sets (of which more than 376,000 are geospatial).

Many other cities have followed suit. A few of the most significant include:

• London – http://data.london.gov.uk/
• New York City – http://www.nyc.gov/html/datamine/
• San Francisco – http://datasf.org/
• Seattle – http://data.seattle.gov/
• Toronto – http://www.toronto.ca/open/
• Washington DC – http://data.dc.gov/

And other organizations are getting into the act. The UK launched data.gov.uk in January 2010. The World Bank not only has a great data site, they’ve also sponsored a contest to encourage the development of new applications that use that data. The FCC has an open data site as well as a set of developer APIs. And the app contests have become sufficiently numerous that they are even starting to feel passé.

Philadelphia has been missing from the list. While the City was an early and unsung leader 10 years ago for releasing its GIS data, these recent efforts by other governments have left it far behind. There is no Philadelphia Open Data web site. But there are a lot of people who want to see that change. A BarCamp in late 2009, RefreshPhilly.org, Philly Startup Leaders, Young Involved Philadelphia and other groups have pushed repeatedly for this type of government transparency through publication of operational data. So why is Azavea building this? Well, we really have Roz Duffy to thank. She encouraged me to get involved with the Open Access Philly task force. I attended my first meeting in January and was impressed by the range and diversity of the people who have been attending these meetings. After the first meeting, I felt like Azavea was actually in a good position to create something that would both serve to bring the various City data sets together in a single catalog as well as extend the catalog to other resources.

While the Open Access Philly task force advocated for an online catalog of data, OpenDataPhilly.org is not a City project. The City government doesn’t have the resources to build something right now. I’m proud that Azavea is building this initial version, but, that said, this is not a typical project for us. That’s good and bad. We don’t build open data portals – we build spatial data analysis and visualization tools. And when I ask my colleagues to work on something that isn’t our main focus, it’s distracting and makes us all less productive. And we are a small company that can only afford to do a certain amount of pro bono work in a given year. And, in the long run, I’m not sure it’s actually a good idea for an open data catalog to be operated by a private firm.

Nonetheless, I felt this was important for a number of reasons. First, I kept hearing other technology people in the region lamenting how we were being left in the dust. That’s sad because there’s actually far more data available than most people realize. Second, much of Azavea’s work depends on open standards and the broad availability of useful data sets. By making it easier to find data, we are supporting the ecosystem that supports our business. Third, I buy into the idea that open government encourages both better government and a more engaged citizenry.

Because Azavea is not the City, OpenDataPhilly.org is different from other open government data portals. We have taken a look at a lot of these web sites, and we’ve done our best to incorporate what we thought were the best parts. But we’ve decided to try some different ideas that we hope will make the catalog more useful. First, the catalog is not limited to data from the municipal government – we have also incorporated data from non-profits, universities and commercial organizations. Second, this catalog is not just about downloadable data sets; we’ve also included data-centric web and mobile applications as well as developer-oriented APIs and other structured data feeds. Third, we realize that data for its own sake is not really all that helpful. To be useful, the data needs to actually be put to use in new applications, visualizations and stories. So the OpenDataPhilly.org site includes an Idea Gallery a feature similar to London’s Inspirational Uses page.

These departures from the usual government-sponsored open data catalog has created opportunities, but it has made our task somewhat more difficult. Since we didn’t limit ourselves to government data sources, we needed to both track down these other data sets and develop a series of guidelines to determine what goes in and what doesn’t. I’m sure we missed a lot, and I don’t know if we got the guidelines right. We also didn’t have a lot of material for the Idea Gallery to start out, so we needed to develop some placeholder material. And, as I mentioned above, in the long run, I’m not sure Azavea is the best home for such a project. I think the best home might be a non-profit organization for which transparency and citizen engagement is part of their mission – perhaps a non-profit news organization or a similar entity.

What’s in it?

As our starting point, we took the extensive set of geospatial data sets that were already available on PASDA. We didn’t limit ourselves to City sources; we also added material from DVRPC, the USGS and other organizations when that data was specific to Philadelphia. We added several data-centric applications deployed at the City as well as some applications developed by local universities that use government data. We also included some of the resources we had discovered while working on a data inventory for the WHYY Newsworks web site last summer. OpenDataPhilly is not only a catalog of existing data sets, applications and APIs, it also includes a series of new geodata APIs that the City has implemented over the last few weeks. So the act of constructing the catalog has inspired the City to release some data sets in a new and useful way. That’s pretty exciting. From our perspective, that means the effort is already a success.

How did we build it?

This is not really a geospatial data application, so our usual tools were not going to be appropriate. Since OpenDataPhilly.org will primarily direct people to other data sets, it doesn’t need a lot of processing power. But we’re going to be maintaining this for at least the next few months, so we needed some simple and straightforward content management features. We settled on the following technology mix:

• Ubuntu Linux (hosting from Linode)
• Python
• Django – Python framework with a simple, clean design and a nice template system
• PostgreSQL

Why now?

Sometimes it’s good to have a deadline. Today’s rollout was timed to coincide with Philly Tech Week, a week-long celebration of technology and innovation in Philadelphia organized by TechnicallyPhilly. Open data serves as bookends for the week. Azavea is rolling out OpenDataPhilly.org today. On Saturday as part of the BarCamp NewsInnovation at Temple University, Tropo is organizing an Open Government Hackathon. The Hackathon will aim to build new applications that use the data listed in the catalog. We’ll be involved in some other events this week. There’s a full summary in a blog from last week.

Acknowledgments

While the City didn’t pay for the development of OpenDataPhilly, that doesn’t mean they didn’t make important and significant contributions. Jeff Friedman (City OIT) and Paul Wright (Fuzebox) have been organizing the Open Access Philly meetings for more than a year, and these meetings were the catalyst that got us moving. Several staff at the City’s Office of Information Technology, including Stuart Alter, Paul Wright, Jim Querry, Brian Ivey, Walter Svekla and others have supported the OpenDataPhilly rollout and development through both encouragement, suggestions and the hard work required to roll out these new geodata APIs. The vast majority of the data sets are ones to which a legion of City employees and residents have contributed over the course of many years. The William Penn Foundation has recently awarded a grant to NPower PA to both encourage use of the data catalog as well as the implementation of the OpenDataPhilly features related to developing a community around the web site. And a large community of people have also contributed advice, encouragement, feedback and data sets to the effort. An incomplete list includes: Johnny Bilotta (developed early version of OpenDataPhilly logo); Roz Duffy; Mark Headd (Tropo); John Mertens, Mjumbe Poe and Aaron Ogle (Code for America fellows); Chris Wink (Technically Philly) and Deb Boyer, Carissa Brittain, Brian Jacobs, Rachel Cheetham-Richard, Claire Connelly, Abby Fretz, Jamal Alsarraj, Dana Bauer and Tamara Manik-Perlman (some of the Azavea folks who worked on the project).

Where do we go from here?

So OpenDataPhilly.org is released. What happens now? That depends on you. A catalog won’t be much use without people using and contributing to it. Want to get involved? Here are a few ways:

• Show up on Saturday for the Hackathon and join a team.
• Got data? We know we probably missed a bunch of useful data sets. There is a page for organizations to submit information about their data sets for inclusion in the catalog.
• Is a critical data set missing? We also have a way for you to ask for missing data sets and vote on other people’s requests.
• Write to your city, state and federal legislators and ask them to support open government data policies. [We can help you with that too. Check out Azavea’s Cicero API.
• Say something with the data. Download some data and develop a beautiful visualization that tells a story. Then submit it to the Idea Gallery.
• If you are a developer, build some apps that use the data. Or, better yet, apply for Code for America, an innovative approach to public service where you can apply your skills to making government work better for everyone.
• OpenDataPhilly.org needs a home. We’ve created it, but we don’t think we should own it in the long run. We’re ready to give it away. We estimate it’s going to be a few hours a week to maintain this. If you think you have a good home for it, we’d like to hear from you.