Azavea Atlas!

Maps, geography and the web

Five New GIS Tools in 2014

Last year saw the rise of the #geohipster hashtag, #maptime meetups and continued expansion and adoption of OpenStreetMap. Here are five exciting geospatial software apps and tools released in 2014 and sure to grow in 2015:

Morgan Herlocker at Mapbox left us with perhaps the greatest Christmas present of all in late December, geoprocessing in the browser. Turf is a javascript library for performing common geospatial functions such as buffering, merging or calculating centroids. One of the nice features of Turf is that since it runs in the browser, it can run completely client-side and offline so it doesn’t need to connect to an external API. It’s modular too, so you include only what you need in your web app. Coming soon: the end of the spatial database as we know it?

mapbox_studio_screenshot

Also from Mapbox, 2014 saw the release of Mapbox Studio, a major upgrade to the desktop based Tilemill software. Like Tilemill, Mapbox Studio allows you to add data (such as shapefles, geojson, csv) and style it using CartoCSS. But it also makes connecting to and styling Mapbox’s streets, terrain and satellite data extremely easy – it’s streamed right into the application. Source data can also be streamed from data stored in your Mapbox account. One of the nice features is the color picker, which essentially eliminates the need to figure out what hex code to add, you can simply choose the color and it will do the rest. Along the same lines, typography took a big step forward in Mapbox Studio with over 300 fonts built-in, selected for their use with digital cartography. As always, you can add your own custom fonts as well.

Taking story maps the next level, Odyssey.js was released by CartoDB in the summer. Inspired by experiments in interactive storytelling like the excellent New York Times Snow Fall project, Odyssey is a javascript library for developing interactive map-based stories. There’s also a convenient web-based editor and a number of templates, which allows anyone to craft a professional interactive story in minutes. But of course it’s also an open source javascript library, so developers can build their own custom interaction with ease.

arcgispro_screenshotYou can’t spell ArcGIS without GIS. In 2014, Esri released a brand new desktop software application called ArcGIS Pro. It’s a major overhaul to the Esri desktop ecosystem, and while not meant to replace ArcGIS Desktop (merely supposed to work alongside it), I can’t see how it wouldn’t in time. It’s a new multi-threaded 64-bit application with a fresh user interface for visualization and analysis. It’s available in beta now and the official release is anticipated for this month. The beta is fairly robust, with fast rendering and much-improved color palettes and symbology options, and though the ribbon interface (similar to Microsoft Office) is simpler it takes some time getting used to. Pro has a streamlined interface for working with data stored in ArcGIS Online. I’m hoping to see some features released soon that will make it easier to work with open data and formats.

Unmanned aerial drones were in the news for a lot of reasons in 2014, but mostly because of Amazon. However, with the launch of OpenDroneMap, there will soon be an open source toolkit to process civilian drone products. As of now, it can only process point clouds, but the project is active on GitHub and expects to have tools to process and upload high-resolution imagery, digital elevation models and other data.

Though not a new GIS tool in 2014, the Humanitarian OpenStreetMap Team did some amazing work mapping the Ebola outbreak in West Africa and improving map coverage of the Philippines after Typhoon Haiyan. 2014 was a big year for spatial with open geo taking off and satellite data becoming more attainable than ever. With a track record like that, 2015 should be an exciting year.

 

Summer of Maps 2015: Now Accepting Applications for Nonprofits

Summer of Maps logo

Applications are now open for Nonprofits seeking pro-bono GIS analysis through the Summer of Maps program.  Summer of Maps offers fellowships to student GIS analysts to perform geographic data analysis for non-profit organizations.  The program matches non-profit organizations that have spatial analysis and visualization needs with talented students of GIS analysis to implement projects over a three-month period during the summer.  Below is the timeline for the 2015 program:

    • Jan 5 – Feb 8: Non-profit organizations can submit brief proposals for spatial analysis projects to Azavea
    • Feb 9 – Feb 26: Azavea program administrators review organizations and narrows the list to finalists
    • Feb 27 – Mar 15: Students submit applications including proposals to work on finalist projects
    • Mar 17 – Mar 31: Student candidate reviews and interviews
    • Apr 13: Successful Summer of Maps fellows are notified
    • May 1: Public announcement of fellows and organizations
    • June – August: Summer of Maps fellows work on spatial analysis projects
    • For the most up to date schedule, please consult the Summer of Maps site.

What benefits do non-profit orgs receive?

    • Pro-bono services from a talented student GIS analyst to geographically analyze and visualize your data
    • Visualization of data in new ways and combination of data with other demographic and geographic data to draw new observations
    • High quality maps that can be used to make a case to funders or support new initiatives

What benefits do students receive?

    • Opportunity to work spatial analysis projects that support the social missions of a non-profit organizations
    • Work directly with Azavea mentors to improve GIS skills
    • Receive a monthly stipend
    • Gain work experience implementing and managing a GIS project

If you are a non-profit organization and have a project you would like to see implemented, please submit an application.  The deadline is Sunday Feb 8th, 2015 11pm EST.  Nonprofits can check out the finalist organization proposals from 2014 for inspiration.  Keep in mind that students will be selecting from the finalist projects so identifying a project that is interesting and engaging is key in having your project be selected.  If you are a student, stay tuned – applications will open Feb 27th, 2015.

To learn more about Azavea Summer of Maps check out the web site.  The Summer of Maps website has additional information on:

Fellowship Sponsors

We’d like to expand Summer of Maps and we’re looking for sponsors.  If you are interested in sponsoring a fellow or a mentor, please be in touch.

When Mapping Quantities, Choices Matter

An article just came up last Friday on Technically Philly that mentioned one of the winning projects from Azavea’s Open Data Philly Visualization Contest, a bike theft study by Greg Kaminsky. Greg chose to look at bicycle theft in Philadelphia, which was similar to a 2013 Summer of Maps project I completed for the Bicycle Coalition of Greater Philadelphia. Greg’s takeaway was that the highest number of bicycles stolen from one single location was 15 and this occurred right outside of City Hall. While that might be true based on the geocoding and visualization techniques used, it seems that more complexity regarding clustering of thefts exist in the data. Perhaps there are more significant clusters (though not falling on an exact point) located elsewhere. Based on my previous work in the area, I knew several areas other than City Hall also had high rates of theft, so I decided to explore some other metrics for analyzing data clusters.

Greg wasn’t incorrect, but his results demonstrate how the choices cartographers make during map creation can greatly affect the results. The most bike thefts in a geographic location varies across the city of Philadelphia depending on a number of factors, such as geographic scale or the normalization used. How we define geographic location is very important. For example, we could decide to use small buffers around theft incidents, look at street corners, blocks, city council or police districts, census tracts, and on and on.

I wanted to explore how the choice of geographic level by which to aggregate thefts affects where apparent clusters of theft exist. Let’s take a look at the full set 10,747 reported thefts over six years:

Thefts from 2007-2012.

Thefts from 2007-2012.

It’s not immediately very easy to understand exactly where the highest amounts of thefts are. The huge number of points ends up making everything far too busy. Now let’s see which locations have had the most bikes stolen from them based on the type of geographic clustering or aggregation we use.

Clustering by geographic boundary

1. Clustering based on City Blocks.

For this method, I identified the city block each theft fell within or was nearest, and summed the thefts per block. What we have is a map of thefts over a six-year period from January 1, 2007 to December 31, 2012 by block across Philadelphia. It appears that high-theft areas have been narrowed down considerably. The thefts per block range from two to twelve, with darker colored blocks having higher amounts of theft. Areas of high theft are immediately apparent near Temple University, University City, Center City, and in far south Philadelphia. Keep in mind, some blocks are larger in size than others, so this should not be considered an approximation for density.

Thefts aggregated to each block.

Thefts aggregated to each block.

The only blocks with more than ten thefts over the time period (January 2007 – December 2012) are located in University City:

Highest bike theft blocks in Philadelphia.

Highest bike theft blocks in Philadelphia.

 

2. Clustering based on Census Tracts

Census tracts are areas delineated by the Census Bureau to optimally contain about 3,000 – 8,000 people each, across most of the United States. Tracts are about 1/3 the size of a typical Philadelphia neighborhood, and are extremely useful for visualizing and interpreting thousands of Census Bureau variables for demographic analysis. Here we see the highest-theft areas by census tract.

Bike thefts aggregated by census tract.

Bike thefts aggregated by census tract.

The Census tract that makes up most of University City is yet again the highest theft area. This is the only tract with more than 300 reported thefts over the 2007-2012 period.

The highest bike theft Census Tracts in Philadelphia.

The highest bike theft Census Tract in Philadelphia, with more than 300 thefts.

 

3. Clustering based on Neighborhoods

Neighborhoods are fun to use for this kind of analysis because they’re recognizable places with names and characters we can identify easily. This method again repeats the aggregation methodologies from before, just at a larger level. The highest theft neighborhoods are Washington Square West (530 thefts), Rittenhouse (790 thefts), and University City (816 thefts).

The three neighborhoods with the highest reported amounts of bike theft in Philadelphia.

The three neighborhoods with the highest reported amounts of bike theft in Philadelphia.

It’s interesting to see how the top theft location changes based on what kind of geography we choose to summarize the data by. Here’s another wrinkle: some of these geographies have much larger perimeters or areas, sometimes because they include parks, cemeteries, or have to cover a larger geographic area to include a sufficient number of people to fit the Census requirements.

Clustering by Proximity

Maybe aggregation based on blocks, tracts, or neighborhoods isn’t the best way to measure “a location”. Perhaps the reason a certain area is hit more often by bike thieves is because of values present only in the immediate area, such as a vacant building, poor lighting, or proximity to an easy exit. One common operation in GIS is buffering, where a circle or shape is drawn around a point at a given thickness. If we draw buffers of a given distance around every single theft, and then count how many thefts fall inside each circle, we can find the circles or areas that have the most thefts. Let’s try this out at a few different distances.

100 Foot Buffers

When 100 foot buffers are used, only a few areas in the city show up with buffers that contain more than eight thefts in a 100 foot radius over the six year period.

 

Theft aggregate buffers at 100 foot distances. Highest amount of thefts per buffer is 9.

Theft aggregate buffers at 100 foot distances. Highest amount of thefts per buffer is 9.

300 Foot Buffers

Now let’s bump up the buffer radius around each theft to 300 feet and then see how many fall inside each buffer. Interestingly, it seems that the high theft areas have shifted. The only buffers with more than 15 thefts each are located at Walnut and Broad, 9th and South St, and 13th and Walnut:

300 Foot buffers with more than 15 reported bicycle thefts from 2007 - 2012.

300 Foot buffers with more than 15 reported bicycle thefts from 2007 – 2012.

500 Foot Buffers

When we enlarge the buffers to 500 feet so they contain about one square block, the high theft areas shift yet again, with the largest buffers all containing 25-32 thefts. Most of the thefts are now centered around the blocks of Walnut Street on either side of Broad Street, with another high area still at 9th and South Street:

500 foot buffers with more than 25 bicycle thefts from 2007-2012.

500 foot buffers with more than 25 bicycle thefts from 2007-2012.

 

Advanced Clustering

What if the location with the highest amount of thefts isn’t a discrete, arbitrary circle somewhere in the city, but rather a contiguous area with similar charactersistics? Cluster analysis looks for statistically significant and contiguous clusters of areas with similar values. When we run run cluster analysis on the theft-by-block file, using inverse manhattan distance, we get this:

This map shows three kinds of clusters: Clusters where high amounts of thefts happened in a block surrounded by low-theft blocks (HL), high amounts of thefts happened in a block surrounded by other high-theft blocks (HH), and low amounts of thefts happened in areas surrounded by high-theft blocks.

Welp. Better lock up your bikes, Philadelphians.

Additional Considerations

Other things to consider: Time: All of these operations used all six years’ worth of data. What if I’d just used 2010? 2012? The clusters, hotspots, and prime locations would probably be entirely different! All of this map data was calculated using a projected coordinate system, North American Datum 1983 State Plane Pennsylvania South. In plain English, we chose to use a warped measurement system that would reduce distortion and preserve certain attributes (distance, area) at the local level. The default Tableau map projection is Web Mercator, which is a projection system used to make 256×256 pixel tiles, and it distorts accuracy the further away from the equator the measurements are being taken. This might account for Greg’s cluster of 15 points at a small location that I couldn’t replicate. Another thing to consider is geocoding inaccuracy.  If many addresses were unable to be geocoded to the exact address, they might have defaulted to city name (Philadelphia), which could also account for multiple locations falling at the exact same point near City Hall (perhaps a commonly used location for geocoding to “Philadelphia”).

Cartography is both an art and a science. The decisions the cartographer makes hugely inform the results. With the popularity of easy web-mapping tools like CartoDB, and other tools that provide light mapping capabilities like Tableau, it’s more important than ever to be a discerning producer and consumer of cartography.

The Geographic Context of #MediaInContext

Closing ceremony of the inaugural Canvas hackathon. Photo credit Al Jazeera PR.

Closing ceremony of the inaugural Canvas hackathon. Photo credit Al Jazeera PR.

If you don’t know me personally, you probably don’t know that Azavea and I were involved in the planning for the recent Canvas media innovation hackathon convened this November by Al Jazeera in Doha, Qatar, themed around “Media in Context.” The organizing credit is due to Al Jazeera Innovation and Research – a new group that began this year at Al Jazeera that is focusing on media and technology innovation – which is why we’ve been quiet about the event ourselves. Al Jazeera brought on SecondMuse, our partner, for operational support and we have been collaborating together on the event since this summer.

At the end of the day, Azavea is a geospatial civic software development firm, not an international news agency like Al Jazeera or a collaboration strategy and hackathon organizing company like SecondMuse. Still, civic hackathons are useful and complementary to our technology work and community-building efforts in the civic tech field, so we’ve developed some experience in running them over the past few years – often in parallel with SecondMuse. They were one of the sponsors for Hacks for Democracy, one of Azavea’s first large public hackathons. Also, SecondMuse and Azavea sat side-by-side in organizing the Philadelphia-local (us) and global (them) components of last year’s International Space Apps Challenge. Until now though, our firms had never worked as full partners on an event, despite each of us having a major presence in Philadelphia and similar focuses in civic and social impact technology. So I was glad to contribute some of Azavea’s expertise to this effort as a partner.

Though many have lauded #MediaInContext as a successful event now that it’s complete, including our nearly 100 participants, mentors, and judges, the fact is the organizing team pulled the bulk of this large event off in just a hair under 3 months! This level of success was not a foregone conclusion back when we started in September.

Lots and lots of participants and many other commentators have already written about most of the solutions that were developed at Canvas and other aspects of the event. SecondMuse has also written its own blog about the event. We’re geography nerds at Azavea though, so for my “reporting” I wanted to dive deeper into the geographic context of Al Jazeera’s Media in Context hackathon!

Hackers of Arabia

2013 International Space Apps Challenge locations map

2013 International Space Apps Challenge locations map

It’s hard to believe how far the humble “civic hackathon” has come in the past few years. In 2011, when I was writing my undergraduate thesis on the open government movement and decided to include a chapter on civic hackathons, it was hard to even find solid examples and sources for my research. Only 3 years later, Azavea’s been concretely involved in more civic hackathons just around Philadelphia than I could have found proof of existence of anywhere in the country back then.

However, the rise of our civic hacking community in these years has not been universal, geographically speaking. North America, South America, Europe, and even pockets of Asia and Africa each have solid communities of civic technologists, but as evidenced by the map above of the 2013 Space Apps hackathon locations, there are large swathes of the world where the civic hacking wave has yet to crest. The Middle East and North Africa region (MENA) is one such area, which made this Al Jazeera hackathon all the more exciting. Journalism and civic hacking can both help promote participatory democracies and engaged communities – it would be thrilling to see more of each of these things in every part of the world.

Around the World in 80 Hacks

The Media in Context hackathon was unique not only because it was happening in a part of the world where hackathons are uncommon, but also because we were bringing the world to the hackathon. The application process was global, and we received over 1,400 applications from talented journalists, data storytellers, and civic hackers all over the world. Our final set of 85 participants that were selected and were able to attend the event represented 37 countries and traveled from every inhabited continent on Earth.

This also may be one of the only hackathons where substantial jetlag may have impacted each team’s competitive advantage. Our furthest-traveled participants flew to Qatar all the way from Mexico City, over 14,000 kilometers!

(999 Less Than) 1001 Arabian Nights of Hacking

Within the span of a few hours on Saturday morning, November 29th, these 85 participants debated, discussed, and whiteboarded their way through 12 example challenges as well as a 13th participant-submitted challenge, finally coalescing into 19 hackathon teams ranging from 3-7 people. Coming from Azavea, one of my personal favorite challenges was #4: Mapping an Understanding. Many news outlets already employ web maps in their reporting, but the use of GIS and spatial analysis tools to tell journalism stories has tremendous potential that has yet to be fully explored in my opinion. So I was excited to see what teams came up with in response to this challenge.

Two teams took on this challenge, MapCake and MapFour, and came up with completely different tools. Approaching from the “Consumption” phase of media, MapFour sought to make it easier for media consumers to browse the news geographically. MapFour sifts through the Al Jazeera and New York Times APIs for locations in articles, and plots them on a map the user can browse. The team also incorporated a time-slider, for viewing different stories that erupt in the same places over time.

MapCake app showing stories related to "Michael Brown" near Ferguson, MO.

MapCake app showing stories related to “Michael Brown” near Ferguson, MO.

MapCake, on the other hand, took a “Production” angle on the challenge. Understanding it can be extra work for journalists to create web maps for their stories, MapCake employs an intuitive “highlight-and-geocode” workflow enabling the mapping and tagging of locations straight from an article’s text. The geocoding is pretty accurate even without much additional context than just a city name, too.

I was really surprised with teams that addressed totally different challenges like “Looking at the Numbers” and “Fully Immersed in Media” and “Media on the Move”, yet also incorporated maps and geospatial thinking into their projects.

Narrata embedded in an Al Jazeera story

Narrata embedded in an Al Jazeera story

 

I was amazed with the Narrata team’s easy uploader that effortlessly blends a story narrative and temporal geodata. The result from just two CSVs is an embeddable interactive map and chart allowing media consumers to dive into time-series geodata to explore weekly or monthly trends, as in their South African protests example. Narrata saves time for journalists, who don’t need to code-up a new visualization for every article with time and place data, and provides an immersive interactive for consumers – in a way addressing both the “production” and “consumption” phases of media in one app!

Street Stories Ferguson example

Street Stories Ferguson example

Street stories is another team that created both a compelling example and a great framework that’s already being re-used by other media organizations. The project turns Google Streetview into a storytelling tool, by embedding tweets, YouTube videos, narratives, and even background sounds directly into the streetview “sphere”. What emerges is an immersive experience that allows one to imagine themselves – visually and audibly – at the real site of an event as it’s happening.

Other projects like Context, TinyFM, and Near, while not map based, still have geographic implications for contextual media access in our cities and while we’re in transit.

I’ve been to and organized a lot of hackathons but the Canvas Media in Context hackathon was truly awe-inspiring. Tremendous credit is due to Al Jazeera Innovation and Research for coming up with the vision for the event and having the leadership to make it all happen. The projects that emerged and the exotic venue in Qatar were also magical. But by far the part I’m missing most two weeks later is the people. Our 100 participants and mentors were grinning the entire way through despite the jetlaggy, intense environment. The fact that the #mediaincontext hashtag is still going and some projects have already been re-used tells me this passionate journalism and technology community is just getting started. Even if you weren’t there with us for the inception, sign up for the Canvas newsletter, vote for your favorite projects by January 15th to win the People’s Choice Award, and stay tuned – much of the story has yet to be written!

Announcing: Winners of OpenDataPhilly Visualization Contest

Azavea and OpenDataPhilly are pleased to announce the winners of the OpenDataPhilly Visualization Contest.  Visualizations that represent creative and visually impactful uses of Philadelphia’s Open data were selected to be featured on the new OpenDataPhilly website (soon to be launched).  OpenDataPhilly.org has undergone a redesign thanks in part to grant funding from the Knight Foundation.  In anticipation of the relaunch of Philadelphia’s newly designed data portal, Azavea invited submissions of visualizations of Philadelphia’s open data.  The OpenDataPhilly Visualization Contest collected submissions from designers, data scientists, developers and anyone who enjoys exploring and visualizing open data. Submitted data visualizations utilized data sets shared on OpenDataPhilly.org and were both static and interactive including maps, infographics, charts and more. Prizes included $500 in Amazon gift cards divided among winners.

Winners include:

Name: Chris Alfano and Lauren Ancona
Project: School District of Philadelphia Budget

Visualization by C.Alfano

Name: Gregory Kaminski
Project: Bike Thefts in Philadelphia

Visualization by G.Kaminsky

Name:  Ken Steif
Project: Visualizing Philadelphia’s neighborhood change process

Visualization by K.Steif

Name: Wayne Dunn
Project: History of Philadelphia Police Citizen Complaints 2009-12

Visualisation by W.Dunn

Name: Rachel Weeden
Project: Mapping Residential Burglaries in Philadelphia

Visualization by R.Weeden

Name: Joseph Calamia
Project: CenterCityScience –Visualizing Scientific Scale from a Philadelphia Perspective

Visualization by J.Calamia

Name: Kenneth Elder and Weilin Meng
Project: Energy Consumption in Philadelphia

Visualization by K.Elder

Name: Angela Minster
Project: Roof Deck Permits as a Gauge of Neighborhood Development

Visualization by A.Minster

Name: Chris Whong
Project: Visualizing Philly’s 2012 Budget by Department and Category

Visualization by C.Whong