Monthly Archives: December 2009

Dec 24, 2009

Philadelphia Civic Hackathon creates a Gang Survey App

By Robert Cheetham

SunLight Labs recently held it’s Great American Hackathon, an event that encourages groups in each region of the United States to gather together on one weekend and create software that will make government more open. Two Azavea employees, David Middlecamp and yours truly, participated in the Philadelphia version and also hosted the event in our offices. Josh Tauberer, a PhD candidate at U-Penn, and developer of GovTrack.us, organized the event.

njgangsurvey3

Seven of us came together to create a web-based visualization and display tool based on data from the New Jersey Gang Survey 2007. The NJ State Police have been conducting these surveys every three years since 2001. Using Django, MySQL, OpenLayers, OpenStreetMap and ArcGIS Desktop, we put together a full-blown app in two days. Two analysts from the New Jersey State Police joined us on Saturday, explained the background on the data set, wrote up the text and other content for the site and answered questions on how the data was structured.

New Jersey Gang Survey Viewer

New Jersey Gang Survey Viewer

The result is The New Jersey Gang Survey Viewer. Check it out. I was amazed by how much a small group could accomplish in such a short time frame, particularly when most of the participants neither knew each other nor knew many of the technology tools when they started. The players were:

Print This Print This Comments Off
Dec 15, 2009

Scaling Walkshed.org with Varnish and Amazon Web Services

By Jeremy Heffner

We’re excited for voting to open today for our entry into the NYC Big Apps Contest — Walkshed NYC.

Walkshed is very CPU intensive since we generate heatmaps for users’ custom walkability factors on the fly.  Building on the work we did with using Amazon’s content delivery network for RedistrictingTheNation.com, we decided to expand our use of Amazon Web Services (AWS) for Walkshed as well as incorporate technology from the open source Varnish project.

Varnish for hardening (and an easier life)

Varnish is a HTTP accelerator that runs on Linux (and other Unix style OSes).  We experimented with Varnish to solve a few goals:

  • Caching frequently requested files and heatmaps tiles (i.e. the default walkability heatmap tiles)
  • Scaling by letting Varnish load balance between multiple servers
  • Improving reliability by allowing Varnish to resubmit failed requests and monitor server health

By pointing Walkshed.org directly to Varnish, we are able to adjust server configurations on the fly without bringing down our application.   Currently, Varnish provides load balancing between 4 server instances which generate tiles  using Walkshed’s DecisionTree engine.  About 50% of the HTTP requests running through Varnish are cache hits, which helps eliminate unnecessary traffic clogging up our application servers.

One instance is hosted on our private server and is often able to meet demand, but adding 3 High-CPU Extra Large Instances from Amazon lets us improve fault tolerance and handle larger bursts in traffic.  Varnish also monitors the health of our servers and removes them from the cluster if they become unresponsive.

Amazon EC2 Instances (bigger is better)

Our Amazon instances are using the new EBS-based images to improve boot speed.   We’ve found that it takes about 7 minutes from when we launch an instance until it is successfully added to our Varnish pool, which certainly isn’t bad.   By combining Varnish with Amazon’s on-demand resources, we should theoretically be able to scale as much as necessary.  For this demo application, scaling is a manual process, but we are looking toward a future where the cluster would scale automatically based on demand.

We also experimented with a few EC2 instance sizes.   Since our application is CPU intensive we really found we had to go with the High-CPU Extra Large Instance to get decent performance.   The instances still don’t meet the performance we get on our private VMware-based server, but our hunch is that this is due to layers of virtualization causing memory allocation to be slow.

Technologies Used:

Print This Print This Comments Off
Dec 15, 2009

Ignite: Spatial, Boston

By David Zwarg

I got the opportunity to present at Ignite: Spatial, Boston a couple weeks ago.  I was fortunate to present Sourcemap.org in the company of other Boston area techies doing some cool work in laser scanning, CityML, social media and more.

All the videos are on YouTube. The presentation summaries are also online in this Google Doc.

Enjoy your spatial ignition this morning.

Print This Print This Comments Off
Dec 04, 2009

Echos of the Browser Wars

By David Zwarg

I caught this link in my feeds today: http://radar.oreilly.com/2009/12/google-android-on-inevitabilit.html

A good read on where mobile devices are, and why it is a non-trivial thing to gain market share in the mobile market.  Specifically, the article discusses the hurdles that Google is trying to jump with its investment into Android, and how Apple is setting the bar high with its i* products.

One of the things that jumps out at me is that the technical challenges of mobile development are nearly synonymous with those in web application development.  Mark Sigal points out that development in the mobile realm is essentially heterogenous.  I had a conversation with a team lead at uLocate a few weeks ago that explained the matrix that characterized this heterogeny.  It’s nuts.  It’s a 4 dimensional matrix, where the dimensions are: Device, Carrier, Platform, and OS.

I’m comparing it to the browser wars because when I test KIF (Kaleidocade Indicators Framework ) I look at the application across a 3 dimensional matrix, where the dimensions are: Browser, Version, and OS.

I can see how that similarity may make it easy for a developer to switch between developing a mobile application and developing a web application, since the testing strategy would be very similar.  I would like to see that transition be a smooth one (as a web developer and someone with a recreational interest in developing tools/toys for mobile devices), so that warms my heart.

However, what I see as a dangerous element to that matrix is how it can get so big so fast.  In the browser market, the matrix is limited to only a handful of items in each dimension.  In the mobile market, however, the number of handsets is always growing — so much so that it’s hard for developers to keep up.  Russell Beattie (Nokia employee) puts it this way (full article):

Multiply the number of models [Nokia puts out] per year (10-20) by the number of years Symbian’s been around by the various custom carrier modifications, and you get complete developer and consumer confusion.

From the chatter I’ve seen, it seems like it’s going to be a teething process by Google, then all out mobile platform wars after that.  The end result?  Probably the same as where we are today, in terms of browsers: supporting about 4 major browsers, with minor differences between them. That provides support to about 97% (as of 12/4/2009) of all browsers out there.  Not bad, but it’ll take mobile a while to get there, and I suspect there will be some corporate blood letting before it’s all over.

Print This Print This Comments Off
  • Azavea Journal
  • Azavea Atlas
  • Azavea Labs
  • Azavea.com