The application is divided into two areas: modeling location (determining where to plant trees) and modeling growth (extrapolating future benefits).

Modeling Location

To create heat maps that would indicate the best planting locations based on selected factors, we used a two-level weighted overlay algorithm that takes the following parameters:

Using these parameters, we define a decision raster as any function where represents the “no data” element of the raster. Operations involving as a parameter always return . We also define other convenience operations:

returns maximum value in the raster,

returns the minimum value in the raster,

normalizes the input raster such that

 
The weighted overlay algorithm has two phases. The first phase calculates the weighted average of each group as a new raster and then normalizes the result to between 0 and 100.

Then we calculate the final weighted raster:

For Philadelphia, we created rasters from the data shown in the last equation. Our general process was to rasterize each dataset (if not already rasterized) and resample to a 10 meter resolution. To allow for comparability between disjointed datasets, we chose to normalize data to a 0-100 range and picked 100 to mean “an ideal place to plant trees.”

The goal the decision model is to allow the end user to prioritize the factors that they would like to affect by planting trees. Perhaps an organization wants to impact crime and heat island effects. Figure 1 shows the base crime raster. Green indicates good places to plant according to this raster, meaning areas of high crime rates that could be effected by the existence of more trees. As studies (by Kuo and Sullivan (2001) among others) have shown that additional tree planting can result in reduced crime, we would plant in the green areas of this raster in order to potentially reduce crime.

Figure 1: Crime Raster for Philadelphia

Figure 2 shows the scaled temperature of the city based on average summer temperatures calculated using Landsat data. Green areas indicate high temperatures and thus ideal places to plant trees in order to reduce those temperatures. Planting trees can result in a general decrease of temperature, helping to reduce the heat island effect (Akbari et al. (1990)).

Figure 2: Heat Raster for Philadelphia

Figure 3 shows both crime and heat equally weighted. Areas highlighted in green are recommended for planting and represent the areas that contribute the most, on average, to both high crime rates and high temperatures. Planting trees in these locations would thus have the greatest impact on both factors.

Figure 3: Combined Crime and Heat Raster for Philadelphia

After identifying key planting locations, the modeling growth component of the system enables users to determine the long-term economic and ecosystem benefits of trees planted in a certain location. We’ll walk through the algorithms for that feature in another blog entry.

The USDA grant funded the creation of a prototype application. While we won’t be releasing a public version of the prototype, you can sign up for a free demo or contact Deb Boyer at dboyer@azavea.com or 215-701-7506.

The development of the Urban Forestry Modeling and Prioritization Toolkit was funded by a Small Business Innovation Research grant (2011-33610-30511) from the U.S. Department of Agriculture National Institute of Food and Agriculture. 

*deep sigh* Okay, now that that’s out of the way, lets talk about writing good code.

As javascript apps mature, the need for organization and modularity becomes more important. The days of simply inlining your code in the HTML, or keeping all your code in one giant file, or rolling your own class system are fading. If you’re experienced with Javascript, you’ve probably written your own class system at some point, or code loading solution, or had to stitch together plugins for one framework all while your code is written in another. These are all simple solutions to a complex problem, and as is a common theme in software, you probably missed some edge cases. Inlining your code is hard to maintain, and isn’t modular, developing with a team in a single giant file is a nightmare, and rolling your own class loader means lack of compatibility and dealing with cross browser issues. The truth is until recently your new jQuery plugin was written differently than your nice custom layer for Openlayers, was different from your custom Dojo widget, was… well you get the point. You couldn’t expect to write them in the same way, or expect them to arrive at the browser in the same file at the same time. (If you’ve used Dojo, Ext, OpenLayers, jQuery, or Node.JS then I hope you know what I mean)

So, what’s a Javascript developer to do? The language is flexible and forgiving, and there hasn’t been something to standardize your code around, until recently. You might have noticed that the internet is pretty good at coming up with standards for sharing information and software. That’s why I was excited to see the proposal and development of the AMD API. AMD in this case stands for Asynchronous Module Definition. Basically it’s a consistent way to package your code, and it makes it significantly easier to reuse later down the road. A number of major Javascript frameworks have already embraced AMD, including jQuery, Node.JS, Dojo, and others, and I’d like to give you a preview here to hopefully inspire developers to give it a try. More importantly modules, and module loaders are proposed to be part of the spec for the next version of Javascript named “Harmony“, see Modules, and Module Loaders.

So, you like jQuery, modules, and you’re ready to open up an example and tinker around? RequireJS has you covered with a great walkthrough here: http://requirejs.org/docs/download.html#samplejquery . I’d enjoy walking through it on this blog, but it might be beyond the scope of this article. Let me know in the comments if you’d like to see more about this.

So, I wanted to provide a more complicated example, which I hope will be helpful to those looking to make nicely namespaced, properly constructable objects in Javascript. Lets pretend we had the following spec:

1.) Our class should use a private static dictionary of globally unique names that have been booped.
2.) These should be kept long-term, in localStorage when possible
3.) Our class should accept verbs other than boop, and should be extendable

So, lets get fancy for a second, and design something that is re-usable and modular. I’m picturing… A storage class, a smart globally static list class, and our main verb class that is configurable. Lets get to it:

Lets define a basic reusable storage module, something like:

define('storage', [ ], function( ) {
    //private static in-memory storage
    var _data = {};

    var that = {
        hasLocalStorage: function() { ... }
        set: function(key, value) { ... },
        get: function(key) { ... }
    };

    var ClassConstructor = function(name) {
        if (!name || name == "") { throw "Name not provided"; }
        this.prefix = name;
    };
    $.extend(ClassConstructor.prototype, that);
    return ClassConstructor;
});

Full Source for storage.js

and… lets define a class that uses that storage to maintain a globally unique list:

define('unique', ['storage'], function(Storage) {
    //private _globally static_ storage object -- will be shared with other instances of "Unique"
    //you probably wouldn't want this for this implementation, but hey, it's an example, so I wanted
    //to show how to do it.
    var _storage = null;

    var that = {
        setIfUnique: function(key) { ... }
    };

    var ClassConstructor = function(name) {
        if (!name || name == "") { throw "Name not provided"; }
        _storage = new Storage(name);
    };
    $.extend(ClassConstructor.prototype, that);
    return ClassConstructor;
});

Full Source for unique.js

and… lets define a class that uses the unique list, and takes a verb to ‘boop.’

define('booper', ['unique'], function(Unique) {
    var that = {
        //private storage, but not static, local to this object
        _unique: null,
        _verb: 'unset',

        verbObject: function(name) { ...  }
    };

    var ClassConstructor = function(verb) {
        this._verb =  verb || "boop";
        this._unique = new Unique(this._verb);
        this[this._verb] = this.verbObject;
    };
    $.extend(ClassConstructor.prototype, that);
    return ClassConstructor;
});

Full Source for booper.js

so, now we just need a basic web page to test this with:

<html>
    <head>
        <script data-main="js/index-main.js" src="js/require-jquery.js"></script>
    </head>
    <body>
        <h3>Boop em!</h3>
        <input type="text" id="name" name="name"
            style="width:300px"
            placeholder="Enter name to boop, limit once per name"
        />
    </body>
</html>

and how about some basic page code:

require(["jquery", "booper"], function($, Booper) {

    //prep
    var b = new Booper();

    //events
    var doBoop = function(evt) {
        b.boop( $("#name").val() );
    };

    $("#name").on('change', doBoop);
    $("#btnBoop").on('click', doBoop);
});

Full Source for index-main.js

Putting it all together with a working example:
Working Example!

Obviously, this example didn’t need to be this complicated, but we gained a lot from a little bit of extra work. We’ve built three easy modules, that can be easily maintained, extended, and reused. We have a hugely open and flexible page, and when we’re ready for production, we can compact and minify all those files into a single optimized build.

So, you’ll notice I added a somewhat generic three line chunk at the end of each class module. This is just some basic constructor logic so you get the nice internal reference to ‘this’, and you can initialize new objects with “new Foo()”, instead of only having one copy, etc, etc.

I hope you enjoyed this example, and if you’re interested in Azavea’s other projects, checkout our GitHub!

The actual date-time standard is published as ISO 8601 and talks about what date-time data should look like, but programming languages, databases and other programs implement this standard with sometimes vastly different interfaces and rules.

PostgreSQL’s handling of date-time data takes the form of 6 different field types, all suited to different data needs. For the OpenTreeMap project, we’re using the timestamp data type which allows us to store both dates and times for events down to the microsecond.

Geoserver accepts a single date-time type (also called timestamp) that stores dates and times at the same resolution as PostgreSQL’s data type, so it reads PostgreSQL’s date-time data easily. Adding a date filter to Geoserver’s SLD files is also fairly easy so long as you know how Geoserver wants date-time data to be formatted. The various formats that Geoserver knows how to interpret are located here. So, if we have a field in a database table called “last_updated”, then an example SLD filter might look like this:

<Filter>
  <PropertyIsLessThan>
     <PropertyName>last_updated</PropertyName>
     <Literal>2012-01-01 00:00:00</Literal>
  </PropertyIsLessThan>
</Filter>

This filter would display any updates made before midnight on January 01, 2012. Another example (from the OGC 1.0 encoding specification’s examples) catches updates between certain dates:

<ogc:Filter>
  <ogc:PropertyIsBetween>
    <ogc:PropertyName>last_updated</ogc:PropertyName>
    <ogc:LowerBoundary>
      <ogc:Literal>2011-12-01 00:00:00</ogc:Literal>
    </ogc:LowerBoundary>
    <ogc:UpperBoundary>
      <ogc:Literal>2011-12-31 23:59:59</ogc:Literal>
    </ogc:UpperBoundary>
  </ogc:PropertyIsBetween>
</ogc:Filter>

This filter would display any updates made during the month of December, 2011. Anything that was updated outside the specified date and time parameters would bypass these filters and go on to check any other rules in the SLD file. This works great if we’re only interested in static date comparisons. But what if we want to see updates less than a week old? Or objects that haven’t been updated for more than three months? This kind of dynamic filtering is a little harder to do.

After digging through the Geoserver documentation and a lot of googling, we decided to shift the dynamic part of the filter into a PostgreSQL view. Our test view included an id field, a geometry field and a field that calculates the number of days since the last update to that object. The view sql looks like this:

CREATE OR REPLACE VIEW timestamp_test AS
  SELECT
    treemap_tree.id,
    date_part('days'::text, now() - treemap_tree.last_updated) AS days,
    treemap_tree.geometry
  FROM treemap_tree;

So now we can add this view to Geoserver as a source layer and it will see the new days field as a static number field. We can use any of the property filters on this field and style recently added data in a dynamic fashion.

python-sld is a simple python library that enables some basic manipulation of StyledLayerDescriptor (SLD) documents.

What are SLD documents?  SLD is a standard defined by the Open Geospatial Consortium, or OGC. In their words:

The OpenGIS® Styled Layer Descriptor (SLD) Profile of the OpenGIS® Web Map Service (WMS) Encoding Standard defines an extends the WMS standard to allow user-defined symbolization and coloring of geographic feature and coverage data.

In layman’s terms, SLD is a common way to style your own maps that come from any map server that speaks WMS (another standard by OGC). Of all the GIS tools available, the WMS server ecosystem is exceptionally rich and diverse. There are many proprietary choices, as well as a plethora of open source options.

State of the Art

Recently in the course of developing new features for DistrictBuilder, we arrived at a point where we needed to generate SLDs dynamically. Looking around at the existing python libraries, we examined:

• pySLD
• qGIS plugin “Save as SLD”
• Geoscript

What we were looking for was a pure object model access to components in the SLD, as well as XML validation, with very few dependencies. None of the above projects really fit the bill, so we started working on our own.

Introducing python-sld

python-sld in an open source (Apache 2.0) library for dynamic SLD creation and manipulation. The project is hosted over on github, and the packages are in pypi (including generated inline documentation).

Features

Width python-sld, creating new SLD documents is as easy as creating a new instance of a StyledLayerDescriptor object:

>>> from sld import *
>>> sld_doc = StyledLayerDescriptor()

With this SLD document, all descendants are accessed as properties, and most child objects are created off the parent with “create_xxx()” methods:

>>> sld_doc.NamedLayer is None
True
>>> nl = sld_doc.create_namedlayer('My Layer')
>>> nl.Name
'My Layer'

For most complex types, the parent’s property is an instance of the class. In our example:

>>> isinstance(nl, NamedLayer)
True
>>> us = nl.create_userstyle()
>>> us.Title = 'Style Title'
>>> us.Title
'Style Title'
>>> isinstance(us, UserStyle)
True

A couple pythonic classes break up the monotony, too. For elements that contain collections of items (a FeatureTypeStyle element may contain many Rule elements, and Fill, Stroke, and Font elements may contain many CssParameter elements), they behave as pythonic lists.

>>> fts = us.create_featuretypestyle()
>>> len(fts.Rules)
0
>>> r1 = fts.create_rule('Criteria 1')
>>> len(fts.Rules)
1
>>> fts.Rules[0].Title == r1.Title
True

Another bit of pythonic syntactic sugar is the combination of Filters. By constructing filters (with the Rule as a parent) and combining them with “+” or “|”, they create logical “AND” and “OR” filters, respectively.

>>> f1 = Filter(r1)
>>> f1.PropertyIsGreaterThan = PropertyCriterion(f1, 'PropertyIsGreaterThan')
>>> f1.PropertyIsGreaterThan.PropertyName = 'number'
>>> f1.PropertyIsGreaterThan.Literal = '-10'
>>>
>>> f2 = Filter(r1)
>>> f2.PropertyIsLessThanOrEqualTo = PropertyCriterion(f2, 'PropertyIsLessThanOrEqualTo')
>>> f2.PropertyIsLessThanOrEqualTo.PropertyName = 'number'
>>> f2.PropertyIsLessThanOrEqualTo.Literal = '10'
>>>
>>> r1.Filter = f1 + f2

When the SLD object is serialized, it will render an “ogc:And” element that contains both property comparisons. You may have noticed that both the “PropertyIsGreaterThan” and “PropertyIsLessThanOrEqualTo” properties are assigned an instance of a PropertyCriterion class. This is the common class for all property comparitors. The name of the comparitor determines it’s logical comparison (less than, greater than, equal to, etc.), and the class has a PropertyName and Literal property, to control which property gets compared, and which value it is compared against.

Finally, serialization is performed on the main StyledLayerDescriptor object, with options to ‘prettify’ the output:

>>> content = sld_doc.as_sld(pretty_print=True)

Dependencies

The lxml library is required by python-sld. This is the library that provides the underlying parsing and serializing of the XML document, as well as the validation steps against the canonical SLD schema.

Limitations

At the current time, only a subset of the entire SLD specification is implemented. All SLD elements are parsed and stored, but only the following elements may be manipulated as objects in python-sld:

• StyledLayerDescriptor
• NamedLayer
• Name (of NamedLayer)
• UserStyle
• Title (of UserStyle and Rule)
• Abstract
• FeatureTypeStyle
• Rule
• ogc:Filter (implicit ogc:And and ogc:Or)
• ogc:PropertyIsNotEqualTo
• ogc:PropertyIsLessThan
• ogc:PropertyIsLessThanOrEqualTo
• ogc:PropertyIsEqualTo
• ogc:PropertyIsGreaterThanOrEqualTo
• ogc:PropertyIsGreaterThan
• ogc:PropertyIsLike
• ogc:PropertyName
• ogc:Literal
• PointSymbolizer
• LineSymbolizer
• PolygonSymbolizer
• TextSymbolizer
• Mark
• Graphic
• Fill
• Stroke
• Font
• CssParameter

All other SLD elements cannot be directly manipulated in python-sld, but are accessible (from a parsed SLD that is perhaps more complex) via the parent object’s _node property. This is the lxml.Element that the python-sld class represents.

django-sld

django-sld builds upon the capabilities in python-sld by enabling quick SLD generation from geographic models. This library is separate from the python-sld library because of the dependencies on django and pysal, the Python Spatial Analysis Library.

Primer on Geographic Models

I gave a quick background to geographic models in django to the Boston django meetup last week, and the slides of my presentation are available online as a presentation in Google Docs. The slides are embedded here for your convenience:

Introducing django-sld

django-sld is an open source (Apache 2.0) library for generating SLD documents from geographic querysets. The project is hosted over on github, and the packages are in pypi (including generated inline documentation).

Features

django-sld enables quick classification of geographic querysets by passing the data distribution of an individual model field into the classification algorithms built into pysal. Not all classification methods in pysal are available, however. At the current version (1.0.3), the following classification algorithms are supported:

• Equal Interval
• Fisher Jenks
• Jenks Caspall
• Jenks Caspall Forced
• Jenks Caspall Sampled
• Max P Classifier
• Maximum Breaks
• Natural Breaks
• Quantiles

To classify a django queryset, use any of the as_xxx() methods in the djsld.generator module.

>>> from djsld import generator
>>> qs = MySpatialModel.objects.all()
>>> sld = generator.as_quantiles(qs, 'population', 10)

The above example assumes that you have a model named “MySpatialModel” in django’s models.py file. The result is a sld.StyledLayerDescriptor object, which may be serialized to a string with “as_sld()”

>>> sld_content = sld.as_sld(pretty_print=True)

The “pretty_print” option is available to format the SLD in a fashion that is more readable by us humans.

In addition to simple models, django’s support for related fields really shines, as it’s possible to classify the distribution on any related field, using the “__” (double underscore) format preferred by django:

>>> sld = generator.as_quantiles(qs, 'city__population', 10)

The one caveat is that the PropertyName in the criteria will be set to this field name (which is not the way most mapping packages refer to related fields). To accommodate this difference, you may use the ‘propertyname’ keyword to control the output PropertyName:

>>> sld = generator.as_quantiles(qs, 'city__population', 10,
... propertyname='population')

Dependencies

django-sld requires python-sld and the pysal library.

Node.js

I was able to attend some of the workshops — “You’ve got Javascript in your backend” with Node.js and Polymaps was a great beginner workshop, introducing lightweight servers and client mapping libraries. I was amazed that a basic web server in node.js is only 5 lines of code. Equally amazing was seeing what capabilities Polymaps had when it weighted in at only 32K (minified) vs. OpenLayers at 1.2M (minified default build).

i2maps + pico

Some exciting visualization tools are coming out of the National Center for Geocomputation at the National University of Ireland, in the form of i2maps. While it’s relatively immature (not much in the form of documentation), most the basic functionality builds off of OpenLayers.  Since I’ve already learned the OpenLayers library, I has a short learning curve, and was able to get up to speed pretty quickly.  Their library incorporates some awesome features like dynamically loading and evaluating rasters via canvas (this only works on modern browsers), and even agent-based modeling. I could have stayed in that workshop for a week.

A byproduct of the i2maps project is pico. Pico is a bridge between Python and Javascript, enabling you to call native Python methods directly from Javascript. It performs all the plumbing for you, allowing you to write a simple callback to handle your method’s return value. It also takes care of converting Python objects into Javascript objects, allowing you to pass all sort of data back and forth (including rasters!).

mod-geocache

Another new project from a contributor to the MapServer project is mod-geocache, a tile caching service as an Apache module. This skips a lot of overhead (no proxying, no interpreters, no CGI), and is very fast. In addition, the C implementation has excellent speed and performance. You can perform on the fly tile merging, quantization, and recompression. I’m excited about this module, and the promise of caching with an Apache server (looks like it has more features than mod_tile).

Geoserver

Geoserver‘s next release is also going to include some great features. The ones that really jumped out at me:

• Time and elevation filters — e.g. storm tracking, where you can limit the features by a time field.
• Styling SLDs in data units — e.g. “road is 5m wide”, and changes dynamically with scale. This greatly simplifies scale-dependent renderers.
• Georeferencing of layers can be done in the admin interface.
• Layers can be view definitions — you don’t have to roll your own views prior to creating the layer.
• Virtual Services — partition the data layers by workspace.

These aren’t all the new features; take a look at the laundry list yourself, and prepare to be impressed.

Mapnik 2

I think the reason for calling it Mapnik2 is that it is literally twice as awesome as it was before. I learned about the new features in Mapnik2 in the lightning talks at SotM, and I think this was one of the few talks that made you feel like you were actually struck by lightning. I can’t remember half the slides in the talk, but the supported formats, reprojection, styling, and speed improvements left me with my head spinning.

Django has a few modules for rating or voting on content, one of which we’re using for the nomination and comments systems. The inner-workings of the module boil down to the following rules:

1. A user must be logged in to rate/vote
2. A user can rate/vote for any number of items
3. A user can only rate/vote for any particular item once (though they may change their rating/vote later)

Compare this with the rules we wanted to enforce for the contest:

1. A user must be logged in to vote
2. A user can only vote once per 7 day period
3. A user can vote for an item multiple times, so long as rule 2 is preserved

Aside from the first rule, we were trying to do almost exactly the opposite of what our rating module enforced. Rather than retrofit the existing module to allow additional and sometimes contradictory behaviour, we decided to write a very small voting module of our own.

The code revolves around two decision points: is voting allowed and can a specific user vote now. The first question is answered by the contest object itself. A contest knows when it’s starting and ending date are, so if today is after the start date and before the end date, then voting is allowed.

The second question is a bit more complicated, but not by much. Because of rule 2 above, we need to know when a user last voted to know if they’re currently allowed to vote. The database storage for a vote contains a datetime object, a foreign key to the user object and a foreign key to the contest entry so if we sort a user’s votes by time we can retrieve their latest vote.

def user_can_vote(self, user):
    increment = datetime.timedelta(days=7)
    votes = user.vote_set.order_by('-timestamp') #latest on top
    if votes:
        next_date = votes[0].timestamp + increment
        if datetime.datetime.today() < next_date and dt.today() < contest.end_date:
            return False
    return True

The above code gets a user’s votes and orders them by time with the most recent first. If a user has ever voted, we need to check if they’re allowed to vote again yet or if they have to wait. We calculate the earliest time that a user can vote next and check it against the date and time now. We also check the end of the contest against the date and time now. If “now” is before the next time the user can vote or “now” is after the contest’s end date, we return false; the user can’t vote now. If a user has never voted before, or the dates are all ok then the user can vote. This check is done after a user clicks the “vote” button but before a vote is saved to the database. We also display a message saying why this check failed and when a user will be able to vote again.

So we’re taking advantage of all of the spam protection built into Django’s user registration process and running a contest on surprisingly little code: 3 database tables, 200 lines of python (blank lines included) and a few templates is all we needed!

We set out to support several base map options as well as any combination of options, including:

• Bing Maps (satellite, roads and hybrid)
• GoogleMaps (satellite, roads and hybrid)
• ArcGIS Online (any of several maps)
• OpenStreetMap

Since DistrictBuilder needs to be flexible enough to meet the needs of users and administrators in a variety of situations, we decided on a two step approach to basemap configuration. First, the administrator specifies, in the configuration file, which of the combinations of map providers and map types are allowed to be selected. Then DistrictBuilder presents all of the configured options to the user, where they can be toggled among at any time while a plan is being viewed or edited.

Here’s an example of an instance configured with an OpenStreetMap road layer, a Bing hybrid layer, and a Google satellite layer:

Here’s another example with only road layers — one for each of the three configured providers:

DistrictBuilder currently allows the configuration of basemaps using permutations of each of the three vendors and three map types described above. Adding more options is a relatively easy task, however. With the launch of Fix Philly Districts, we wanted the basemap colors to be slightly more muted than the above options, and ended up adding support for the ArcGIS Online World Topographic Map. We also experimented with the Google Maps V3 custom styling API, which looked great, but introduced performance problems when panning and zooming (animations).

There were, of course, some hoops that needed to be jumped through in order to get all of these basemaps behaving correctly on the same map, which will be discussed below. I’ve extracted the logic required to do so into a small demo that can be viewed/downloaded here. The demo has also been embedded into this post, and can be interacted with without going anywhere:

Zoom Levels

Many of the challenges that needed to be overcome to get this working correctly were brought about because we needed to restrict the zoom levels to the area at hand. We wanted to eliminate superfluous zoom levels to ensure the user was always operating within the appropriate boundaries (note: it is not done in this demo, but in DistrictBuilder we also restrict the extent with the ‘restrictedExtent’ map parameter, so users can’t even pan outside of the area).

One difficulty with setting zoom levels on the different layers is that the layers don’t use zoom parameters consistently. In Bing (the VirtualEarth layer), minZoomLevel and maxZoomLevel are needed. In Google, minZoomLevel is needed, but it requires numZoomLevels instead of maxZoomLevel. And in OpenStreetMap (the OSM layer), well…no combination of those seem to work — we needed to slightly modify the XYZ layer (OSM’s base class) to allow maxResolution to be changed based on the minZoomLevel. To see how this is done, view the demo source. With that change in place, the list of required layer parameters is as follows:

• Bing – minZoomLevel, maxZoomLevel, projection, sphericalMercator, maxExtent
• Google – numZoomLevel, minZoomLevel, projection, sphericalMercator, maxExtent
• OpenStreetMap - numZoomLevel, minZoomLevel, projection

Coordinate Systems

We also faced some problems related to coordinate systems. DistrictBuilder uses GeoServer and GeoWebCache to serve up WMS layers. The coordinate system of our data is one version of the the ever-changing “Popular Visualization CRS / Mercator” projection. We needed to match up the OpenLayers projection to the one used on our data, or else we were seeing slight offsets on our overlays. Unfortunately, the ‘projection’ layer parameter isn’t always used within the layers correctly. For example, any layer using the SphericalMercator class gets its projection automatically hardcoded to 900913. We needed to make a slight modification to the SphericalMercator class to allow the ‘projection’ parameter to carry through. This can be seen by viewing the demo source.

Bonus: Math Time!

One interesting part about implementing zoom restriction was that we needed it to work in any instance of DistrictBuilder — from large states to small towns, which may have vastly different extents. Instead of having an administrator figure out the proper minimum zoom level, we calculate it automatically based on the extent, which requires a little bit of basic algebra.

For Philadelphia, the extent of our area is:

[-8397913.926216, 4842467.609439, -8329120.600772, 4895973.529229]

In DistrictBuilder, we calculate this dynamically on the server side (using Django) by filtering all of the geounits in the database and calling the ‘extent’ function on the query set. For the demo, this is hardcoded. Here’s how to transform this extent into a Spherical Mercator minZoomLevel:

• Find the width of the area in meters.

var studyWidthMeters = extent[2] - extent[0];

• Find the width of the map in pixels. In the demo, this is hardcoded, because we are setting the div size of the map. In DistrictBuilder, the map takes up the whole screen, and this value is calculated on the fly based on the size of the div in which the map occupies.

var mapWidthPixels = 450;

• Find the map resolution, or meters per pixel.

var resolution = studyWidthMeters / mapWidthPixels;

• Find the maximum map resolution. In Spherical Mercator, the maximum resolution is one 256×256 tile taking up the entire circumference Earth. So dividing the circumference of the earth (~40,000km) by 256 gives us the maximum meters per pixel, which is a constant.

var maxResolution = 156543.033928;

• Spherical Mercator zoom levels work like a pyramid. Each zoom breaks the current tile up into a 2×2 group of 256×256 tiles, essentially halving the resolution each time. Therefore, finding the resolution at a given zoom level looks like this:

maxresolution / 2^zoom = resolution

• We know the resolution and max resolution already, and need to find the zoom:

zoom = log(maxresolution/resolution)/log(2)

• Or in javascript:

var minZoom = Math.log(maxResolution / resolution) / Math.LN2;

Most recently, the Politics, Redistricting and Elections team has been working closely with the

We spent quite a bit of time making the application easy to use and responsive in modern desktop web browsers.  The “easy to use” part was tackled by our excellent UI/UX design team. The “responsive” part was the domain of  our engineers.  That’s where the fun began for me.

DistrictBuilder is designed to use any polygon shapefile, transform it into an internal data model, then make that accessible via map tiles and geometric features.  When serving map tiles, we use GeoServer and GeoWebCache to generate the tiles and cache them, respectfully. This performance is great — pre-generated map tiles are the best we can aim for with respect to the base map tiles. Serving geometric features at full resolution, however, introduces a slew of problems. A few that stood out right away:

• Web Browser Limitations — 9 out of 10 experts agree: too many map features has a significant performance impact on web browsers, with the greatest impact on the Microsoft Internet Explorer browser.
• Excessive Coordinates — delivering lots of polygon coordinate pairs that the user may never see consumes valuable bandwidth and rendering time.
• Server Processing Time — recalculating state-wide geometric features consumes valuable CPU time.

Web Browser Limitations

First, we tackled the browser performance issues. A sluggish browser is the kiss of death in the web world, and we had to make the application experience as fast as possible before looking at the server processing time.

We originally gave users the power to create highly detailed districts at the statewide level, but realized that no modern web browser could handle the volume of polygon features that would need to be served to represent an entire state.  In order to mitigate this limitation, we limited the size and number of features sent to the browser. With some scale-dependent logic, a user zoomed in to a detail of a district can finely tune the boundary by moving smaller geographic features (e.g. census blocks), and a user zoomed out to the state-wide level can manipulate the districts by moving large geographic features only (e.g. counties). In addition, when editing the finest details, we limit the number of features a user can move in a single edit.

Excessive Coordinates

The next thing to go was the set of full resolution geometries. In DistrictBuilder, users never actually see the full geometries, but an adaptively simplified (sometimes called generalized) geometry; depending on the scale of the map view, the server will deliver geometries with appropriate coordinate resolutions. Simply put: as you zoom in on the map, you get more detail in the geometries.

By simplifying counties, the geometries are reduced from 166,958 points to 4,821. When a user is zoomed out, there is no noticeable difference between these geometries!  However, as the user is interacting with higher resolution maps, DistrictBuilder loads in higher-resolution geometries on demand. The following images demonstrate the difference in the geometry detail:

The zoomed in County layer, with a low resolution district overlay (orange line). There are currently 1,414 coordinates in this view of the district overlay.

The zoomed in VTD layer, with a high resolution district overlay (orange line). There are currently 3,253 coordinates in this view of the district overlay.

You can notice the differences in the district detail if you look closely at the orange district boundary. This transition happens seamlessly in the application, loading in the higher resolution geometries as web users zoom in to areas of interest.

We also eliminated coordinates that you never see.  It made no sense to serve  coordinates that were located in the opposite side of the state where a user was editing, just like you wouldn’t expect to get an encyclopedia in the mail when releasing an RFI. With the OpenLayers library, Strategies came in handy here, particularly BBOX.

Server Processing Time

After we had optimized the performance of the user interface, we shifted our focus to the server-side processing.  One of the features that makes DistrictBuilder such a powerful tool is the accuracy of the underlying data and constant feedback of important district statistics. In order to calculate all these statistics on the fly, it is necessary to leverage some tricks already mentioned with respect to map tiles: caching and generalizing.

Computation of the district statistics must happen every time a district boundary is changed. A naive solution to this problem would be to aggregate the values within the boundary every time a change is made.  This approach results in horrible performance. Instead, we just determine what has changed — which areas were added, which areas were removed — and recompute the delta, or change, on the previous district value.

Another trick to optimizing performance is in the way we determine the changing boundaries.  I’ll describe the problem using the census geographies of counties, tracts, and blocks. The structure and detail of the underlying data yielded computationally expensive queries against the block geometries.  We came up with a method of searching for the geographies in a hierarchical fashion — searching the counties first, then continuing to the next smallest-scale geography only if there was any remaining geometry left in the query.  We did the same for the tracts, and took a shortcut at the block level to exclude the block geometries.  This increased server side performance considerably.

King William county is comprised of 22 Voter Tabulation Districts and 1,527 Census Blocks.

Consider the following scenario: a user wants to move King William County (highlighted in yellow) from District 1, which is over populated, to District 3, which is under populated. Changing the boundaries with all the blocks in King William County would require testing at least 4,000 blocks for spatial intersections, then aggregating 1,527 data values, and recomputing the spatial aggregate (union) of those 1,527 geometries. With our hierarchical approach, we can change the boundary of the district with the county boundary, and change the population totals by the county’s population. A few orders of magnitude fewer operations to perform, and much faster from the user’s perspective.

Lessons Learned

Throughout the DistrictBuilder development process, the same core performance challenge has arisen: the volume of data must be reduced. This applies to all aspects of the application:

• Map Tiles: pre-render tiles to keep the number of rendered tiles to a minimum at runtime.
• Map Features: deliver to the browser only as much information as you can see (perhaps even less).
• Database Queries: do anything possible to ensure that geometric operations are performed on simplified geometries.
• Aggregating Statistics: cache whatever you can, and only compute the difference from the last cache state.

The above steps reduced the sheer number of operations and volume of processing that both the server and browser need to complete when creating new districts. These are lessons that translate well to any “big data” problem, and are crucial in bringing sophisticated GIS operations to the web.

Lets look at how this changes the user story first:

A logged-in user makes an edit to a tree. The system needs to decide if these changes are applied to the tree or placed in a pending queue. If this is a publicly-entered tree, the changes are applied to the tree. (Start new requirements) If this is an inventory tree, and the user isn’t a member of a management group, add the change to the pending queue. Display any pending changes reasonably near the appropriate current value. (End new requirements)

Most of this happens behind the scenes in the saving logic. I added a bit of code to the top of our tree updater to check if the pending system is active, the user’s permissions and the tree’s origins. If everything checks out, the change goes straight into the updater code. For changes that go into the pending queue, the path to becoming an official change is a little more tortuous.

Since we’re storing these changes for later review, they have to go into the database. I created a new table to hold onto the original tree’s id, the field being changed and the new value as well as the user who submitted it, a date/time stamp and a status field. Each pending change is stored separately; even if the user makes more than one change to the tree, each ‘pend’ can be applied individually.

The rest of the pending system is eye-candy and a bit of slightly tedious templating. Almost every field on a tree’s detail page now needs to check two new things: are there any pending changes for this field, and does this user have permission to approve/disapprove pending edits. If there are pending edits, the new values are added below the current official value. When a managing user views the page, small approve and disapprove buttons also appear next to each pending change. Throw in a management-access-only page for some bulk evaluation and the system is complete!

A post today, also by Bern, explained a new feature within ArcGIS.com to allow the user to mute the basemap by adjusting it’s display transparency.    The HunchLab team stumbled upon this idea a few months back and it’s been a great way to use the existing topographic basemap.    In our demo instance of HunchLab, we are using the ArcGIS.com Topographic tiles set to a transparency of 60%.   You can see what it looks like below.

Kudos to Esri — keep the basemap options coming!