Azavea Atlas

Maps, geography and the web

Mapping Substandard Housing with Philadelphia’s License and Inspection Data

Summer of Maps logo

Now in its fourth year, Azavea’s Summer of Maps Program has become an important resource for non-profits and student GIS analysts alike.  Non-profits receive pro bono spatial analysis work that can enhance their organizational decision-making processes and programmatic activities, while students benefit from Azavea mentors’ expertise.   This year, three fellows worked on projects for six organizations that spanned a variety of topics and geographic regions.  This blog series documents some of their accomplishments and challenges during their fellowship.  Our 2015 sponsors Esri, Temple University Geography and Urban Professional Masters Studies Program and Betsy and Jesse Fink helped make this program possible.  For more information about the program, please visit the Summer of Maps website.

As part of Azavea’s Summer of Maps program, I worked with the Legal Clinic for the Disabled, a pro bono legal organization that serves low-income individuals with disabilities in the greater Philadelphia area. The Legal Clinic for the Disabled (LCD) wanted to compare their client intake records with publicly-available geographic data in ways that would inform their future intake and services.

LCD was particularly interested in the relationship between poverty, disability, and unsafe housing conditions. They had every right to be: a 2014 report shows that people with disabilities face enormous challenges in the search for affordable and accessible housing. Individuals living on disability payments alone are priced out of the rental market in almost all major cities. This  means that if they do find an affordable place to live, they may have less leverage against a negligent landlord. Poor housing conditions may also increase the likelihood of developing a disability: researchers at the Cincinnati Children’s Hospital Medical Center found a positive correlation between the density of housing violations and the number of childhood asthma cases in an area. If LCD could identify links between clusters of substandard housing and clusters of disability in Philadelphia, they could advocate for improved housing conditions in those specific clusters or even use the links as evidence in a legal case.

Philadelphia is fortunate to have an enormous source of publicly-available housing data at its fingertips: the Licenses and Inspections data set, which covers all housing permits and code violations within the city, can be viewed for free online. Currently, this data can be accessed by using the Licenses and Inspections API, or through the friendlier License to Inspect tool. However, the API (application programming interface) only allows users to view a maximum of 1,000 records at a time, while License to Inspect offers no way to directly view housing violations. I needed all of the violations and their details in my own database for statistical analysis.

I built a web scraper using the Python programming language to extract all of the violations from the Licenses and Inspections API into a single comma-separated values (CSV) document. In all, I extracted 1,079,788 violation records. I identified 2,165 records erroneous and placed them in a separate file. I placed the rest of the file on GitHub for anyone to download.

This map shows the density of all L&I violations in Philadelphia over a 10-year period, using a neighborhood radius of 500 meters:

Map of L+I Violations

Compare that to the following map, which shows the population density of city residents living under 150% of the poverty line:

Comparison of Violations and Poverty

Each record represents a single violation, which can be grouped by License and Inspection case or by location. The 1,077,623 valid records that I surveyed were grouped into 368,574 different cases and 157,350 different locations. That works out to an average of 7 violations per location, but there were some outliers: the top 0.1% of locations had an average of 141 violations each!

Violations and cases could also be broken down by a number of descriptive categories. For example, based on their threat to human safety, violations are given a priority from “Non-Hazardous” (less severe) to “Imminently Dangerous” (the structure might actually fall). Here is the breakdown for the violations I extracted:

Non-Hazardous                       81.0%
Construction Services            5.2%
Unsafe                                    2.9%
Hazard                                    9.4%
Imminently Dangerous           1.2%

And finally, in case you’re concerned about the “Imminently Dangerous” designation, here’s a density map showing only those violations:

Map of L+I Dangerous Violations

Housing justice is vital not just to LCD’s mission, but to many nonprofits and advocacy groups in Philadelphia. Hopefully, this collection Licenses and Inspections data will allow more groups to better understand housing conditions throughout the city.

GIS, Mapping and Cartography Tools to Get You Started on Your Next Project

If you are new to GIS and mapping or just want to find out what tools are available, it can be confusing hunting around for the right ones. I’ve put together a list of tools and resources for various applications. I hope this guide will let you zero in on the the right tools you need and get started on your next project post haste!

  1. Process Geographic Raster and Vector Data QGis_Logo
    QGIS is a fantastic tool to get started with data processing such as manipulating vector data (shapefiles) or working with raster data. It is a free and open source desktop GIS program and has great documentation and training materials. There is a full range of user-built tools and add ons, but these sometimes can have issues or bugs, so tread carefully. 

    If you need a full suite of GIS analysis tools, ArcMap for home use is a great option.  The cost is only $100/yr for all tools and extensions. This is useful if you are already familiar with the interface of ArcMap or need to utilize their popular extensions: Spatial Analyst, Network Analyst or Geostatistical Analyst and you donhave access to a copy through school or work.

  2. Build a Beautiful Basemap MapBox Logo
    If you need to create a beautiful thematic basemap for an interactive map, Mapbox Studio (for desktop) or Mapbox Editor (for browser) are fantastic options.  These tools are great for styling basemaps using OpenStreetMap data, your own data and cartocss. Your basemap can then be published to vector tiles and hosted in your Mapbox account or extracted and published as raster tiles (see this blog post as a guide).

  3. Publish an Interactive Map on the Web
    If you want to do substantial data analysis and editing of data before you publish to the web, then I’d recommend working in QGIS and then publishing with Leaflet using this plug-in.  This guide is a great resource and will help you get up and running quickly. CartoDB Logo

    CartoDB is another great option for some really clean and attractive web mapping. Sign up for a free account, drag in a zipped shapefile or csv and start styling. It allows you to build time-lapse animations and other sophisticated visualizations while still being very easy to use. Check their map gallery for inspiration.

  4. Quickly Analyze Over 1 Million RecordsPostGIS Logo
    If you need to crunch a ton of data, PostGIS is the way to go. Check out this guide which has fantastic step by step instructions for getting data into a PostGIS database. PostGIS data can be used within QGIS as well. CartoDB, mentioned above, is built on PostGIS so you can do the same queries within that web mapping application, but if you are working with very large data, you might encounter limits on file size within CartoDB.

  5. Perform Advanced Spatial StatisticsGeoda Logo
    GeoDa is an awesome desktop tool capable of advanced spatial statistical analysis such as spatial regression, Morans I, Local Indicators of Spatial Association (LISA), histograms, thiessen polygons, scatterplots and boxplots. This is a free application with some serious statistical chops.

  6. Design a Beautiful and Accessible MapColorbrewer Image
    Check out Colorbrewer to discover attractive and intuitive color palettes for your project and don’t forget to consider accessibility to ensure your
    map can be enjoyed by individuals with color blindness or low vision.


header_logo_2x_2Want to hear what others are doing in the industry? NACIS is probably the most cutting edge GIS/Cartography conference running with a schedule featuring dozens of talks that address innovative and exciting techniques.  Many of these are truly expanding the scope and advancing the state of the art for the industry. Additionally, FOSS4G NA features numerous technical talks, many of which hosted by the creators of some of the above tools.  Interested in learning more? Visit a Maptime chapter near you and build your skills at their hands-on project nights.

How to Get Census Data for Maps in 5 Steps

Census Data Doesn’t Have to Be Difficult

One of the largest and best free sources of data in the US is the Census Bureau’s American Fact Finder. It’s a searchable database of all of the nation’s Census counts and American Community Survey (ACS). ACS data is collected on a rolling basis, with 1, 3, and 5-year release cycles with diminishing margins of error. This data spans sea to sea from every locality in the US, and includes tens of thousands of combinations of social and demographic information. The data is also collected by many different geographic summary levels, which you can learn about here.

Finding and using Census data is also difficult. This is why we often turn to CensusReporter.org to rapidly explore and acquire Census Data. A small team of Knight Foundation-funded developers created CensusReporter to facilitate data exploration and download. CensusReporter always uses the most recent year of complete Census or ACS data, depending on the geography collected. You can see an overview of the tables that CensusReporter provides here.

An Intro To CensusReporter

Head to censusreporter.org You’ll be confronted by two large options: Profile, and Explore.

Let’s start with Profile. Type Minneapolis, MN into the Profile search box. You’ll see multiple results come up. CensusReporter will return each geography that contains a place. So we see one result for Minneapolis the place and one for Hennepin County. Choose Minneapolis, MN the place. Check out the map near the top after you’ve made your selection:

CensusReporter.org map displaying Minneapolis, MN
Now scroll down the page and see data grouped by:

  • Demographics
  • Economics
  • Families
  • Housing
  • Social

Digestible charts and graphs break down each subject. These graphics may be embedded on a webpage, like so:



Click “show data” under the pie chart for Age. A table will expand below that breaks down the age information. From here you could also click “View Table” to see the source data. Go ahead and click through to the table.

Tables, Maps, and Distributions

After clicking through to the source data table, you should see a table that displays the age breakdown for Minneapolis. You should notice a graphic above that says “Table Map Distribution”:

 

CensusReporter.org Menu options. Table, Map, Distribution.

CensusReporter.org Menu options

If you click distribution, you’ll be able to see some basic statistics about each of the different variables. You may also click on the Map button, and see Minneapolis, MN on a map. It’s not very informative to see just a big blob of a city with no population breakdown, so click on Table again.

Dividing Up Geographies

What we want to see is population by Census Tract. On the Table page, look at the lefthand tab. Near the bottom, there is a heading entitled “Divide Minneapolis, MN into…”

 

Shows the geographic areas Minneapolis, MN can be divided into, including census tracts, block groups, and zip codes.

The options for dividing up geographic areas

and shows the geographies that the city can be broken into below, including census tracts, block groups, and zip codes. Click census tracts. The main table will have changed a bit. Go back to the top of the page and click Map. Now we can actually see the differences in the age of the population across the city.

 

Minneapolis Census Tracts

Minneapolis Census Tracts

 

Downloading the Geographic Data

Now that we’ve divided the data up by census tract, we can download the data for our own use. On the Map screen, just to the right of the Table/Map/Distribution buttons, you’ll see “Download Data”. Hover over that and you’ll see the various text data formats and geographic formats you can download. CSV and Excel are great for joining to pre-existing geographic data. Geojson is great for web maps. And KML and Shapefile are both standard geographic data formats that can be used in Google Earth or ArcGIS, respectively.

The 5 Steps to Download Geographic Census Data

  1. Go to CensusReporter.org
  2. Search for a place
  3. Search for Data
  4. Click “Divide Into” a given geography
  5. Click Download Data

Now you know how to get geographically-enabled Shapefiles, Excel Files, CSVs, and GeoJSONs full of Census Data in about thirty seconds. While CensusReporter doesn’t allow users to grab a stack of different data sets like the Census Bureau’s own American Fact Finder, it’s a great tool for rapidly finding and downloading data for smaller projects.

Preparing for a Spatial Career: Tips for Students

Summer of Maps logo

This past spring, my colleagues and I reviewed over 170 applications for our fellowship program, Summer of Maps. The applications were truly fantastic and we are overjoyed to now be working with our selected fellows on six very exciting projects.

I often get asked what makes a strong fellowship application? This is an important question, but instead of focusing so narrowly on our program, I thought I’d expand this to tackle the question: how can students prepare themselves for a career in spatial analysis?

I entered the job market not very long ago and, even in a short period of time, positions and desired skill sets have changed drastically. In many ways, the minimum bar for applicants has been raised. But there are a handful of things students can do to distinguish themselves and increase their opportunities in their careers ahead.

Take the Initiative to Expand Your Skills

One of the questions we ask our fellowship applicants is how do you go about acquiring new skills or learning new technologies?  More so than the actual techniques students use (MOOCs, web forums, workshops, educational books, etc), we want to find out if the students have a desire to build their skills beyond the standard curriculum at their university.

Not very long ago, the landscape of tools available was more limited. For reference, in 2008, QGIS had yet to release version 1.0 and web mapping tools ArcGIS Online, CartoDB and Mapbox Tilemill had all yet to be developed. Now, we are privileged to have a plethora of amazing new tools at our disposal, many of them free and open source. Often these come with fantastic guides and resources to help us get started.

leaflet js logo

So, students, I implore you, even if your academic program did not offer any technical training beyond Esri ArcGIS, take some of your free time to build new skills.  I’d recommend starting with QGIS, CartoDB, GeoDa, Leaflet, Mapbox Studio, and PostGIS. You don’t have to learn them all at once or become an expert. But during your summer break, pick one tool a week and get familiar with something new. This shows employers that you’re comfortable learning new skills independently.

QGIS Logo

Similarly, explore some data sets beyond what’s included in your homework tutorials or American FactFinder.  Check out a few open data portals and dig into new and interesting data. You never know what you’ll find and you might even come to some uncharted conclusions! I recommend checking out OpenDataPhilly, DataSF, London Datastore, Data.gov, City of Chicago Data Portal and NYC Open Data. Many of these portals offer data in spatial formats (shp, kmz, geoJSON, etc).

Showcase your Work

Creating an online portfolio of work is my number one recommendation to students.  It is relatively easy to state on a resume what skills you have or courses you’ve taken. But highlighting those skills in a few well designed maps communicates with much more clarity.  These portfolios do not need to be flashy or high-tech, but merely a showcase of your best work.  I’d recommend you include maps (either static or interactive) that highlight the best analysis, visualization and communication skills you can muster.  One tip is to solicit feedback on this content from both technical and nontechnical individuals before publishing to make sure it is easy to understand and visually appealing.  To get started, check out tools about.me, weebly and WordPress to make a basic portfolio website.

Make your work stand out!

Often times, I find that students rely heavily on the step by step instructions of a homework assignment, and the resulting maps are full of default color schemes and layouts.  After a while, even if the content changes, these maps all seem to look the same.  John Nelson detailed numerous cartography tips in his 2014 NACIS talk and subsequent blog posts, but I’d encourage you to pay special attention to tip #3, Defaults are Evil.  Just for fun, peruse some map disasters (and common mistakes to avoid) at Cartastrophe.Interactive Map of Hurricane Sandy by Chris Henrick

Explore unique and appealing layouts (browse the NACIS Atlas of Design for inspiration), pick new color palettes using ColorBrewer or find some inspiration from archival maps. And lastly, make sure the message of your map is clear and concise.

Here are some student map portfolios that are especially impressive: Chris Henrick, Aaron Dennis, and Carolyn Fish.  To the right, you’ll see an interactive map featured in the portfolio of Chris Henrick.

Build a Network

My last suggestion involves engagement with the broader geo community. They are an incredibly collaborative and supportive group, especially with regard to individuals working with open source or open data. The community is comprised of (but not limited to) a robust network of geo meetups, over 70 worldwide maptime chapters, a great GIS stack exchange network (over 30,000 members), energetic OpenStreetMap contributors, numerous high quality conferences and a strong and thoughtful Twitter presence.

I’ve found that nearly everyone I’ve met in this community is supportive, thoughtful and kindhearted.  These are the type of folks that will offer to help solve a problem, contribute to maptime_logo-finalOpenStreetMap for fun or volunteer their GIS skills for disaster relief. Understandably, not everyone has substantial amounts of free time to commit, but a few ways to get started are: join your local geo meetup (like GeoPhilly in Philadelphia), attend (or better yet, teach) a local maptime session, join the twitter conversation, start your own spatial blog or attend the next geo conference (Foss4GNA, NACIS, JS.Geo, LocationTech are all good options). Long story short: be active in the community and  when you ask for help be sure to offer help next time you can.

Final Thoughts

The market for spatial jobs is competitive, especially for those at the entry level.  Take the initiative to build your skills, showcase your work and contribute to the community. These efforts will result in lasting benefits beyond finding a solid career.

Adding State Districts to Your Salesforce Nonprofit Account

Cicero’s integration with the Salesforce Nonprofit Starter Pack has been expanded to include State Upper and Lower Districts. Back in November, we announced the ability to verify household addresses with Cicero and match them to Congressional districts. Now, it is possible to add state legislative districts as well!

 

How Does it Work?

The Nonprofit Starter Pack uses the Cicero API to geocode addresses associated with Household records. It then matches the address to Congressional, State Upper, and State Lower districts, and appends them to the address record. In addition, it updates the record with the Cass-certified standardization, meaning your addresses get verified in the process!

If you haven’t yet used Cicero with Salesforce, you can follow the steps outlined here to get started. Now, instead of only seeing Congressional Districts appended to your household address, State Upper and Lower Districts will be populated as well! Important note: Before starting, make sure you are running version 3.43 of the NPSP or higher. Versions below 3.43 will not support this function.

If you are already utilizing the Cicero integration with the Nonprofit Starter Pack, and you have not upgraded to 3.43, you can manually add the districts to your address detail. Here’s how:

 

1) First, edit your address page layout to include State Upper and State Lower:

NPSP Address Layout 2

 

2) Next, add the new fields to your profile:

Profile Address Object 2A

 

3) You’re done!

NPSP Address Detail Page 2A

Cicero hearts Nonprofits

We offer excellent discounts to nonprofits wishing to use Cicero. Through our partnership with TechSoup, all nonprofits are eligible for 5,000 credits (that’s good for 5,000 addresses matched to legislative districts) for only $30. We also offer a 10% discount overall to any organization qualifying as a nonprofit, Government, or educational institution.

Need more reasons to sign up? Drop us a line or tweet us; we’ll be happy to help. And don’t forget to tune in to the next post in this series, which will cover tips on customizing your Address Layout in the NPSP.