python-sld is a simple python library that enables some basic manipulation of StyledLayerDescriptor (SLD) documents.

What are SLD documents?  SLD is a standard defined by the Open Geospatial Consortium, or OGC. In their words:

The OpenGIS® Styled Layer Descriptor (SLD) Profile of the OpenGIS® Web Map Service (WMS) Encoding Standard defines an extends the WMS standard to allow user-defined symbolization and coloring of geographic feature and coverage data.

In layman’s terms, SLD is a common way to style your own maps that come from any map server that speaks WMS (another standard by OGC). Of all the GIS tools available, the WMS server ecosystem is exceptionally rich and diverse. There are many proprietary choices, as well as a plethora of open source options.

State of the Art

Recently in the course of developing new features for DistrictBuilder, we arrived at a point where we needed to generate SLDs dynamically. Looking around at the existing python libraries, we examined:

• pySLD
• qGIS plugin “Save as SLD”
• Geoscript

What we were looking for was a pure object model access to components in the SLD, as well as XML validation, with very few dependencies. None of the above projects really fit the bill, so we started working on our own.

Introducing python-sld

python-sld in an open source (Apache 2.0) library for dynamic SLD creation and manipulation. The project is hosted over on github, and the packages are in pypi (including generated inline documentation).

Features

Width python-sld, creating new SLD documents is as easy as creating a new instance of a StyledLayerDescriptor object:

>>> from sld import *
>>> sld_doc = StyledLayerDescriptor()

With this SLD document, all descendants are accessed as properties, and most child objects are created off the parent with “create_xxx()” methods:

>>> sld_doc.NamedLayer is None
True
>>> nl = sld_doc.create_namedlayer('My Layer')
>>> nl.Name
'My Layer'

For most complex types, the parent’s property is an instance of the class. In our example:

>>> isinstance(nl, NamedLayer)
True
>>> us = nl.create_userstyle()
>>> us.Title = 'Style Title'
>>> us.Title
'Style Title'
>>> isinstance(us, UserStyle)
True

A couple pythonic classes break up the monotony, too. For elements that contain collections of items (a FeatureTypeStyle element may contain many Rule elements, and Fill, Stroke, and Font elements may contain many CssParameter elements), they behave as pythonic lists.

>>> fts = us.create_featuretypestyle()
>>> len(fts.Rules)
0
>>> r1 = fts.create_rule('Criteria 1')
>>> len(fts.Rules)
1
>>> fts.Rules[0].Title == r1.Title
True

Another bit of pythonic syntactic sugar is the combination of Filters. By constructing filters (with the Rule as a parent) and combining them with “+” or “|”, they create logical “AND” and “OR” filters, respectively.

>>> f1 = Filter(r1)
>>> f1.PropertyIsGreaterThan = PropertyCriterion(f1, 'PropertyIsGreaterThan')
>>> f1.PropertyIsGreaterThan.PropertyName = 'number'
>>> f1.PropertyIsGreaterThan.Literal = '-10'
>>>
>>> f2 = Filter(r1)
>>> f2.PropertyIsLessThanOrEqualTo = PropertyCriterion(f2, 'PropertyIsLessThanOrEqualTo')
>>> f2.PropertyIsLessThanOrEqualTo.PropertyName = 'number'
>>> f2.PropertyIsLessThanOrEqualTo.Literal = '10'
>>>
>>> r1.Filter = f1 + f2

When the SLD object is serialized, it will render an “ogc:And” element that contains both property comparisons. You may have noticed that both the “PropertyIsGreaterThan” and “PropertyIsLessThanOrEqualTo” properties are assigned an instance of a PropertyCriterion class. This is the common class for all property comparitors. The name of the comparitor determines it’s logical comparison (less than, greater than, equal to, etc.), and the class has a PropertyName and Literal property, to control which property gets compared, and which value it is compared against.

Finally, serialization is performed on the main StyledLayerDescriptor object, with options to ‘prettify’ the output:

>>> content = sld_doc.as_sld(pretty_print=True)

Dependencies

The lxml library is required by python-sld. This is the library that provides the underlying parsing and serializing of the XML document, as well as the validation steps against the canonical SLD schema.

Limitations

At the current time, only a subset of the entire SLD specification is implemented. All SLD elements are parsed and stored, but only the following elements may be manipulated as objects in python-sld:

• StyledLayerDescriptor
• NamedLayer
• Name (of NamedLayer)
• UserStyle
• Title (of UserStyle and Rule)
• Abstract
• FeatureTypeStyle
• Rule
• ogc:Filter (implicit ogc:And and ogc:Or)
• ogc:PropertyIsNotEqualTo
• ogc:PropertyIsLessThan
• ogc:PropertyIsLessThanOrEqualTo
• ogc:PropertyIsEqualTo
• ogc:PropertyIsGreaterThanOrEqualTo
• ogc:PropertyIsGreaterThan
• ogc:PropertyIsLike
• ogc:PropertyName
• ogc:Literal
• PointSymbolizer
• LineSymbolizer
• PolygonSymbolizer
• TextSymbolizer
• Mark
• Graphic
• Fill
• Stroke
• Font
• CssParameter

All other SLD elements cannot be directly manipulated in python-sld, but are accessible (from a parsed SLD that is perhaps more complex) via the parent object’s _node property. This is the lxml.Element that the python-sld class represents.

django-sld

django-sld builds upon the capabilities in python-sld by enabling quick SLD generation from geographic models. This library is separate from the python-sld library because of the dependencies on django and pysal, the Python Spatial Analysis Library.

Primer on Geographic Models

I gave a quick background to geographic models in django to the Boston django meetup last week, and the slides of my presentation are available online as a presentation in Google Docs. The slides are embedded here for your convenience:

Introducing django-sld

django-sld is an open source (Apache 2.0) library for generating SLD documents from geographic querysets. The project is hosted over on github, and the packages are in pypi (including generated inline documentation).

Features

django-sld enables quick classification of geographic querysets by passing the data distribution of an individual model field into the classification algorithms built into pysal. Not all classification methods in pysal are available, however. At the current version (1.0.3), the following classification algorithms are supported:

• Equal Interval
• Fisher Jenks
• Jenks Caspall
• Jenks Caspall Forced
• Jenks Caspall Sampled
• Max P Classifier
• Maximum Breaks
• Natural Breaks
• Quantiles

To classify a django queryset, use any of the as_xxx() methods in the djsld.generator module.

>>> from djsld import generator
>>> qs = MySpatialModel.objects.all()
>>> sld = generator.as_quantiles(qs, 'population', 10)

The above example assumes that you have a model named “MySpatialModel” in django’s models.py file. The result is a sld.StyledLayerDescriptor object, which may be serialized to a string with “as_sld()”

>>> sld_content = sld.as_sld(pretty_print=True)

The “pretty_print” option is available to format the SLD in a fashion that is more readable by us humans.

In addition to simple models, django’s support for related fields really shines, as it’s possible to classify the distribution on any related field, using the “__” (double underscore) format preferred by django:

>>> sld = generator.as_quantiles(qs, 'city__population', 10)

The one caveat is that the PropertyName in the criteria will be set to this field name (which is not the way most mapping packages refer to related fields). To accommodate this difference, you may use the ‘propertyname’ keyword to control the output PropertyName:

>>> sld = generator.as_quantiles(qs, 'city__population', 10,
... propertyname='population')

Dependencies

django-sld requires python-sld and the pysal library.

Node.js

I was able to attend some of the workshops — “You’ve got Javascript in your backend” with Node.js and Polymaps was a great beginner workshop, introducing lightweight servers and client mapping libraries. I was amazed that a basic web server in node.js is only 5 lines of code. Equally amazing was seeing what capabilities Polymaps had when it weighted in at only 32K (minified) vs. OpenLayers at 1.2M (minified default build).

i2maps + pico

Some exciting visualization tools are coming out of the National Center for Geocomputation at the National University of Ireland, in the form of i2maps. While it’s relatively immature (not much in the form of documentation), most the basic functionality builds off of OpenLayers.  Since I’ve already learned the OpenLayers library, I has a short learning curve, and was able to get up to speed pretty quickly.  Their library incorporates some awesome features like dynamically loading and evaluating rasters via canvas (this only works on modern browsers), and even agent-based modeling. I could have stayed in that workshop for a week.

A byproduct of the i2maps project is pico. Pico is a bridge between Python and Javascript, enabling you to call native Python methods directly from Javascript. It performs all the plumbing for you, allowing you to write a simple callback to handle your method’s return value. It also takes care of converting Python objects into Javascript objects, allowing you to pass all sort of data back and forth (including rasters!).

mod-geocache

Another new project from a contributor to the MapServer project is mod-geocache, a tile caching service as an Apache module. This skips a lot of overhead (no proxying, no interpreters, no CGI), and is very fast. In addition, the C implementation has excellent speed and performance. You can perform on the fly tile merging, quantization, and recompression. I’m excited about this module, and the promise of caching with an Apache server (looks like it has more features than mod_tile).

Geoserver

Geoserver‘s next release is also going to include some great features. The ones that really jumped out at me:

• Time and elevation filters — e.g. storm tracking, where you can limit the features by a time field.
• Styling SLDs in data units — e.g. “road is 5m wide”, and changes dynamically with scale. This greatly simplifies scale-dependent renderers.
• Georeferencing of layers can be done in the admin interface.
• Layers can be view definitions — you don’t have to roll your own views prior to creating the layer.
• Virtual Services — partition the data layers by workspace.

These aren’t all the new features; take a look at the laundry list yourself, and prepare to be impressed.

Mapnik 2

I think the reason for calling it Mapnik2 is that it is literally twice as awesome as it was before. I learned about the new features in Mapnik2 in the lightning talks at SotM, and I think this was one of the few talks that made you feel like you were actually struck by lightning. I can’t remember half the slides in the talk, but the supported formats, reprojection, styling, and speed improvements left me with my head spinning.

Django has a few modules for rating or voting on content, one of which we’re using for the nomination and comments systems. The inner-workings of the module boil down to the following rules:

1. A user must be logged in to rate/vote
2. A user can rate/vote for any number of items
3. A user can only rate/vote for any particular item once (though they may change their rating/vote later)

Compare this with the rules we wanted to enforce for the contest:

1. A user must be logged in to vote
2. A user can only vote once per 7 day period
3. A user can vote for an item multiple times, so long as rule 2 is preserved

Aside from the first rule, we were trying to do almost exactly the opposite of what our rating module enforced. Rather than retrofit the existing module to allow additional and sometimes contradictory behaviour, we decided to write a very small voting module of our own.

The code revolves around two decision points: is voting allowed and can a specific user vote now. The first question is answered by the contest object itself. A contest knows when it’s starting and ending date are, so if today is after the start date and before the end date, then voting is allowed.

The second question is a bit more complicated, but not by much. Because of rule 2 above, we need to know when a user last voted to know if they’re currently allowed to vote. The database storage for a vote contains a datetime object, a foreign key to the user object and a foreign key to the contest entry so if we sort a user’s votes by time we can retrieve their latest vote.

def user_can_vote(self, user):
    increment = datetime.timedelta(days=7)
    votes = user.vote_set.order_by('-timestamp') #latest on top
    if votes:
        next_date = votes[0].timestamp + increment
        if datetime.datetime.today() < next_date and dt.today() < contest.end_date:
            return False
    return True

The above code gets a user’s votes and orders them by time with the most recent first. If a user has ever voted, we need to check if they’re allowed to vote again yet or if they have to wait. We calculate the earliest time that a user can vote next and check it against the date and time now. We also check the end of the contest against the date and time now. If “now” is before the next time the user can vote or “now” is after the contest’s end date, we return false; the user can’t vote now. If a user has never voted before, or the dates are all ok then the user can vote. This check is done after a user clicks the “vote” button but before a vote is saved to the database. We also display a message saying why this check failed and when a user will be able to vote again.

So we’re taking advantage of all of the spam protection built into Django’s user registration process and running a contest on surprisingly little code: 3 database tables, 200 lines of python (blank lines included) and a few templates is all we needed!

We set out to support several base map options as well as any combination of options, including:

• Bing Maps (satellite, roads and hybrid)
• GoogleMaps (satellite, roads and hybrid)
• ArcGIS Online (any of several maps)
• OpenStreetMap

Since DistrictBuilder needs to be flexible enough to meet the needs of users and administrators in a variety of situations, we decided on a two step approach to basemap configuration. First, the administrator specifies, in the configuration file, which of the combinations of map providers and map types are allowed to be selected. Then DistrictBuilder presents all of the configured options to the user, where they can be toggled among at any time while a plan is being viewed or edited.

Here’s an example of an instance configured with an OpenStreetMap road layer, a Bing hybrid layer, and a Google satellite layer:

Here’s another example with only road layers — one for each of the three configured providers:

DistrictBuilder currently allows the configuration of basemaps using permutations of each of the three vendors and three map types described above. Adding more options is a relatively easy task, however. With the launch of Fix Philly Districts, we wanted the basemap colors to be slightly more muted than the above options, and ended up adding support for the ArcGIS Online World Topographic Map. We also experimented with the Google Maps V3 custom styling API, which looked great, but introduced performance problems when panning and zooming (animations).

There were, of course, some hoops that needed to be jumped through in order to get all of these basemaps behaving correctly on the same map, which will be discussed below. I’ve extracted the logic required to do so into a small demo that can be viewed/downloaded here. The demo has also been embedded into this post, and can be interacted with without going anywhere:

Zoom Levels

Many of the challenges that needed to be overcome to get this working correctly were brought about because we needed to restrict the zoom levels to the area at hand. We wanted to eliminate superfluous zoom levels to ensure the user was always operating within the appropriate boundaries (note: it is not done in this demo, but in DistrictBuilder we also restrict the extent with the ‘restrictedExtent’ map parameter, so users can’t even pan outside of the area).

One difficulty with setting zoom levels on the different layers is that the layers don’t use zoom parameters consistently. In Bing (the VirtualEarth layer), minZoomLevel and maxZoomLevel are needed. In Google, minZoomLevel is needed, but it requires numZoomLevels instead of maxZoomLevel. And in OpenStreetMap (the OSM layer), well…no combination of those seem to work — we needed to slightly modify the XYZ layer (OSM’s base class) to allow maxResolution to be changed based on the minZoomLevel. To see how this is done, view the demo source. With that change in place, the list of required layer parameters is as follows:

• Bing – minZoomLevel, maxZoomLevel, projection, sphericalMercator, maxExtent
• Google – numZoomLevel, minZoomLevel, projection, sphericalMercator, maxExtent
• OpenStreetMap - numZoomLevel, minZoomLevel, projection

Coordinate Systems

We also faced some problems related to coordinate systems. DistrictBuilder uses GeoServer and GeoWebCache to serve up WMS layers. The coordinate system of our data is one version of the the ever-changing “Popular Visualization CRS / Mercator” projection. We needed to match up the OpenLayers projection to the one used on our data, or else we were seeing slight offsets on our overlays. Unfortunately, the ‘projection’ layer parameter isn’t always used within the layers correctly. For example, any layer using the SphericalMercator class gets its projection automatically hardcoded to 900913. We needed to make a slight modification to the SphericalMercator class to allow the ‘projection’ parameter to carry through. This can be seen by viewing the demo source.

Bonus: Math Time!

One interesting part about implementing zoom restriction was that we needed it to work in any instance of DistrictBuilder — from large states to small towns, which may have vastly different extents. Instead of having an administrator figure out the proper minimum zoom level, we calculate it automatically based on the extent, which requires a little bit of basic algebra.

For Philadelphia, the extent of our area is:

[-8397913.926216, 4842467.609439, -8329120.600772, 4895973.529229]

In DistrictBuilder, we calculate this dynamically on the server side (using Django) by filtering all of the geounits in the database and calling the ‘extent’ function on the query set. For the demo, this is hardcoded. Here’s how to transform this extent into a Spherical Mercator minZoomLevel:

• Find the width of the area in meters.

var studyWidthMeters = extent[2] - extent[0];

• Find the width of the map in pixels. In the demo, this is hardcoded, because we are setting the div size of the map. In DistrictBuilder, the map takes up the whole screen, and this value is calculated on the fly based on the size of the div in which the map occupies.

var mapWidthPixels = 450;

• Find the map resolution, or meters per pixel.

var resolution = studyWidthMeters / mapWidthPixels;

• Find the maximum map resolution. In Spherical Mercator, the maximum resolution is one 256×256 tile taking up the entire circumference Earth. So dividing the circumference of the earth (~40,000km) by 256 gives us the maximum meters per pixel, which is a constant.

var maxResolution = 156543.033928;

• Spherical Mercator zoom levels work like a pyramid. Each zoom breaks the current tile up into a 2×2 group of 256×256 tiles, essentially halving the resolution each time. Therefore, finding the resolution at a given zoom level looks like this:

maxresolution / 2^zoom = resolution

• We know the resolution and max resolution already, and need to find the zoom:

zoom = log(maxresolution/resolution)/log(2)

• Or in javascript:

var minZoom = Math.log(maxResolution / resolution) / Math.LN2;

Most recently, the Politics, Redistricting and Elections team has been working closely with the

We spent quite a bit of time making the application easy to use and responsive in modern desktop web browsers.  The “easy to use” part was tackled by our excellent UI/UX design team. The “responsive” part was the domain of  our engineers.  That’s where the fun began for me.

DistrictBuilder is designed to use any polygon shapefile, transform it into an internal data model, then make that accessible via map tiles and geometric features.  When serving map tiles, we use GeoServer and GeoWebCache to generate the tiles and cache them, respectfully. This performance is great — pre-generated map tiles are the best we can aim for with respect to the base map tiles. Serving geometric features at full resolution, however, introduces a slew of problems. A few that stood out right away:

• Web Browser Limitations — 9 out of 10 experts agree: too many map features has a significant performance impact on web browsers, with the greatest impact on the Microsoft Internet Explorer browser.
• Excessive Coordinates — delivering lots of polygon coordinate pairs that the user may never see consumes valuable bandwidth and rendering time.
• Server Processing Time — recalculating state-wide geometric features consumes valuable CPU time.

Web Browser Limitations

First, we tackled the browser performance issues. A sluggish browser is the kiss of death in the web world, and we had to make the application experience as fast as possible before looking at the server processing time.

We originally gave users the power to create highly detailed districts at the statewide level, but realized that no modern web browser could handle the volume of polygon features that would need to be served to represent an entire state.  In order to mitigate this limitation, we limited the size and number of features sent to the browser. With some scale-dependent logic, a user zoomed in to a detail of a district can finely tune the boundary by moving smaller geographic features (e.g. census blocks), and a user zoomed out to the state-wide level can manipulate the districts by moving large geographic features only (e.g. counties). In addition, when editing the finest details, we limit the number of features a user can move in a single edit.

Excessive Coordinates

The next thing to go was the set of full resolution geometries. In DistrictBuilder, users never actually see the full geometries, but an adaptively simplified (sometimes called generalized) geometry; depending on the scale of the map view, the server will deliver geometries with appropriate coordinate resolutions. Simply put: as you zoom in on the map, you get more detail in the geometries.

By simplifying counties, the geometries are reduced from 166,958 points to 4,821. When a user is zoomed out, there is no noticeable difference between these geometries!  However, as the user is interacting with higher resolution maps, DistrictBuilder loads in higher-resolution geometries on demand. The following images demonstrate the difference in the geometry detail:

The zoomed in County layer, with a low resolution district overlay (orange line). There are currently 1,414 coordinates in this view of the district overlay.

The zoomed in VTD layer, with a high resolution district overlay (orange line). There are currently 3,253 coordinates in this view of the district overlay.

You can notice the differences in the district detail if you look closely at the orange district boundary. This transition happens seamlessly in the application, loading in the higher resolution geometries as web users zoom in to areas of interest.

We also eliminated coordinates that you never see.  It made no sense to serve  coordinates that were located in the opposite side of the state where a user was editing, just like you wouldn’t expect to get an encyclopedia in the mail when releasing an RFI. With the OpenLayers library, Strategies came in handy here, particularly BBOX.

Server Processing Time

After we had optimized the performance of the user interface, we shifted our focus to the server-side processing.  One of the features that makes DistrictBuilder such a powerful tool is the accuracy of the underlying data and constant feedback of important district statistics. In order to calculate all these statistics on the fly, it is necessary to leverage some tricks already mentioned with respect to map tiles: caching and generalizing.

Computation of the district statistics must happen every time a district boundary is changed. A naive solution to this problem would be to aggregate the values within the boundary every time a change is made.  This approach results in horrible performance. Instead, we just determine what has changed — which areas were added, which areas were removed — and recompute the delta, or change, on the previous district value.

Another trick to optimizing performance is in the way we determine the changing boundaries.  I’ll describe the problem using the census geographies of counties, tracts, and blocks. The structure and detail of the underlying data yielded computationally expensive queries against the block geometries.  We came up with a method of searching for the geographies in a hierarchical fashion — searching the counties first, then continuing to the next smallest-scale geography only if there was any remaining geometry left in the query.  We did the same for the tracts, and took a shortcut at the block level to exclude the block geometries.  This increased server side performance considerably.

King William county is comprised of 22 Voter Tabulation Districts and 1,527 Census Blocks.

Consider the following scenario: a user wants to move King William County (highlighted in yellow) from District 1, which is over populated, to District 3, which is under populated. Changing the boundaries with all the blocks in King William County would require testing at least 4,000 blocks for spatial intersections, then aggregating 1,527 data values, and recomputing the spatial aggregate (union) of those 1,527 geometries. With our hierarchical approach, we can change the boundary of the district with the county boundary, and change the population totals by the county’s population. A few orders of magnitude fewer operations to perform, and much faster from the user’s perspective.

Lessons Learned

Throughout the DistrictBuilder development process, the same core performance challenge has arisen: the volume of data must be reduced. This applies to all aspects of the application:

• Map Tiles: pre-render tiles to keep the number of rendered tiles to a minimum at runtime.
• Map Features: deliver to the browser only as much information as you can see (perhaps even less).
• Database Queries: do anything possible to ensure that geometric operations are performed on simplified geometries.
• Aggregating Statistics: cache whatever you can, and only compute the difference from the last cache state.

The above steps reduced the sheer number of operations and volume of processing that both the server and browser need to complete when creating new districts. These are lessons that translate well to any “big data” problem, and are crucial in bringing sophisticated GIS operations to the web.

Lets look at how this changes the user story first:

A logged-in user makes an edit to a tree. The system needs to decide if these changes are applied to the tree or placed in a pending queue. If this is a publicly-entered tree, the changes are applied to the tree. (Start new requirements) If this is an inventory tree, and the user isn’t a member of a management group, add the change to the pending queue. Display any pending changes reasonably near the appropriate current value. (End new requirements)

Most of this happens behind the scenes in the saving logic. I added a bit of code to the top of our tree updater to check if the pending system is active, the user’s permissions and the tree’s origins. If everything checks out, the change goes straight into the updater code. For changes that go into the pending queue, the path to becoming an official change is a little more tortuous.

Since we’re storing these changes for later review, they have to go into the database. I created a new table to hold onto the original tree’s id, the field being changed and the new value as well as the user who submitted it, a date/time stamp and a status field. Each pending change is stored separately; even if the user makes more than one change to the tree, each ‘pend’ can be applied individually.

The rest of the pending system is eye-candy and a bit of slightly tedious templating. Almost every field on a tree’s detail page now needs to check two new things: are there any pending changes for this field, and does this user have permission to approve/disapprove pending edits. If there are pending edits, the new values are added below the current official value. When a managing user views the page, small approve and disapprove buttons also appear next to each pending change. Throw in a management-access-only page for some bulk evaluation and the system is complete!

A post today, also by Bern, explained a new feature within ArcGIS.com to allow the user to mute the basemap by adjusting it’s display transparency.    The HunchLab team stumbled upon this idea a few months back and it’s been a great way to use the existing topographic basemap.    In our demo instance of HunchLab, we are using the ArcGIS.com Topographic tiles set to a transparency of 60%.   You can see what it looks like below.

Kudos to Esri — keep the basemap options coming!

Icky Syntax

The most noticeable problem when using Numeric (at least in 2.8) is the syntax. As Kevin Wright alluded to in a comment on the last post, you can include the Numeric type class in two ways: via a context bound on your type parameter (A) or as an implicit argument.

In my earlier examples I used the implicit argument because I thought it was easier to follow (especially for those who are new to implicits) and it bound the instance of the type class to a named variable (n). But you can also use the type bound syntax, which makes the function declaration a bit cleaner looking.

// this makes it obvious that n is an instance of Numeric[A]
// and gives us a reference to use.
def square[A](a:A)(implicit n:Numeric[A]) = {
  n.times(a, a)
}
 
// this definition is a bit cleaner looking.
// implicitly[] searches for an implicit Numeric[A] instance.
// unfortunately the body of the function is less clean.
def square[A:Numeric](a:A) = {
  implicitly[Numeric[A]].times(a, a)
}

Either way, you’re still writing things like n.times(a, a) rather than a * a. Fortunately, Scala 2.9 introduced some nice new implicit parameters that allow us to use infix operators:

import Numeric.Implicits._
def square[A:Numeric](a:A) = a * a

Confusing Conversions

The implicits providing infix operators are definitely a big improvement! But while they are nice for simple cases like the previous one they can end up confusing the issue a little bit:

def twice[A:Numeric](a:A) = a * 2
//error: could not find implicit value for parameter num:
// scala.math.Numeric[Any]

What’s going on here? The problem is that * is mapped to Numeric.times(a:A, b:A) but the second argument we provided isn’t an A, it’s an Int. In Java (and Scala) we are used to being able to mix numeric types like double and int without worrying: the result will be promoted to double (the type with the widest range). Unfortunately, as written Numeric isn’t able to do these kinds of things and instead types must be converted explicitly. Numeric does provide two special methods, zero and one which allow you to get those values for all its types.

// use Numeric.one to get the correct type
def once[A:Numeric](a:A):A = a * implicitly[Numeric[A]].one
 
// convert 2 to be the correct type
def twice[A:Numeric](a:A):A = a * implicitly[Numeric[A]].fromInt(2)
 
// convert a to be an Int
def thrice[A:Numeric](a:A):Int = a.toInt * 3

There’s also no good way to combine two different generic Numeric types, other than converting to a common concrete type (e.g. Double). You could imagine Numeric providing something like fromType[B] (as well as things like fromDouble) but currently it doesn’t.

def combine[A:Numeric, B:Numeric](a:A, b:B) = a + b
//error: could not find implicit value for parameter num:
// scala.math.Numeric[Any]
 
// this works
def combine[A:Numeric, B:Numeric](a:A, b:B) = a.toDouble + b.toDouble
 
// this would be nice (for some definition of nice)
// but it doesn't work
def combine[A:Numeric, B:Numeric](a:A, b:B):A = {
  a + implicitly[Numeric[A]].fromType[B](b)
}

Performance

The most disappointing thing (in my opinion) about using scala.math.Numeric is its performance. In some cases it performs adequately, but in most cases it is very slow: actual algorithms written with Numeric tend to be 6-30 times slower than their direct counterparts (depending on what exactly they do). Some rough observations are that the comparison operators (e.g. >) seem to be slower than the math operators (e.g. *), and performance does depend on which type you end up using: Numeric[Double] does relatively well (compared to the same code written for Double). Numeric[Int] does very badly compared to code written for Int.

Specialization to the Rescue!

Fortunately, Scala itself contains the tool to fix Numeric’s performance problems: specialization. Specialization was first described in the paper Compiling Generics Through User-Directed Type Specialization by Iulian Dragos and Martin Odersky and was introduced in Scala 2.8 via the @specialized annotation. Other articles have been written explaining specialization, but I will give another brief one here.

When writing generic functions in Scala (as well as Java), the compiler generates one implementation that supports all possible argument types (e.g. java.lang.Object). This fine for Java [1] since all the valid types you could use with your generic function are guaranteed to be subclasses of java.lang.Object. If you remember my first post, you’ll recall how we had to create boxed instances of java.lang.Integer to hold int values when using generics.

Unfortunately for Scala it means that using generic functions with value-types (e.g.Int) is much slower than functions implemented directly in terms of the value-type itself, since there is a single implementation which requires its arguments to be objects (e.g. boxed).

The @specialized annotation allows the Scala compiler to generate separate implementations of the generic function: one for all the reference types (inheriting fromAnyRef) and one for each of the value types we choose to specialize over (e.g. Char, Double, Int). So if we have a generic function foo[A](a:A): A the compiler will generate other functions on primitive types, for instance fooInt(a:Int): Int, fooDouble(a:Double): Double, etc. And when we write foo(33) the compiler will translate that code into fooInt(33).

Since Scala hides boxing/unboxing, specialization doesn’t make the code look nicer. But it makes a huge difference in performance: Dragos and Odersky report a 22-40x speed up in micro-benchmarks!

Specializing Numeric

Soon after specialization was added to Scala 2.8, people began discussing specializing Numeric. Searching Google I find names like Jason Zaugg, Iulian Dragos, and others talking about it. Andreas Flierl emailed the scala-language mailing list on January 3, 2011 with some early tests which got me interested in working on this. So I don’t want to claim any credit for the amazing work done on implementing @specialized or even the idea of specializing Numeric.

Starting with Andreas’ code I began working on building a specialized Numeric, with the goal of making all its operations as fast as direct implementations. There were a bunch of ways that the code needed to be restructured (which I may discuss in a later article) and I learned a ton about implicits, typeclasses, and profiling Scala code. The code is available at: https://github.com/azavea/numeric along with the tests I’m running. The results follow, but the short story is that with the exception of implicit infix operators all com.azavea.math.Numeric code runs just as fast as the direct implementations.

The Numbers

Discussion

Profiling this was a little bit annoying–I had to write 4 direct implementations (one for each of the types I was interested in) plus two generic implementations (one using the old scala.math.Numeric and one using the new com.azavea.math.Numeric). I used scala.testing.Benchmark although Yuvi Masory recently suggested using https://github.com/sirthias/scala-benchmarking-template (which I still need to go check out).

Some things to note:

1. Old Numeric does much better on floating-point numbers than on integral ones. Why?
2. For some reason old Numeric is able to beat the direct implementation of the array-allocator test.
3. Since scala.util.Sorting[T] uses scala.math.Ordering[T] (which isn’t specialized) it’s not possible to make Numeric any faster in this test.
4. scala.util.Sorting[Long] is over 4x slower than Int, Float and Double in the direct implementation.
5. Infix operators are still slow enough that I still have mixed feelings endorsing them.

One additional gotcha which I haven’t discussed at length is what writing your own specialized numeric code (currently) looks like. Unfortunately, there’s a lot of boiler-plate. While users of your library will just say doSomethingComplicated(Array(0, 1, 2), 18, 99) you will be stuck writing:

def doSomethingComplicated[@specialized A:Numeric:Manifest]
(ns:Array[A], x:A, y:A) = {
  // implementation here
}

And that’s only one type parameter; God forbid you want A and B to both be Numeric!

I don’t know of any easy way around this. You have to say @specialized if you want your code to be specialized. If you fail to do this anywhere in the call chain you will use generic implementations from that point down, losing any benefit to specialization. You need to specify Numeric because that’s the point, right? And when writing code that needs to use the type parameter A to allocate arrays (among other things) you have to provide a Manifest. This gets ugly really fast.

Given my ignorance of the Scala compiler, I can imagine various magical things that might fix this. For instance, I could imagine some kind of synthetic type bound that combines others, e.g. FastNumeric = @specialized Numeric:Manifest or something. Is this possible (or even a good idea)? I don’t know.

Changes

This section is a bit rushed–I may expand it into another post later, but I just wanted to get the information out there.

One of the biggest changes I made to Numeric was tearing out scala.math.Ordering–I didn’t want to try to specialize “everything at once” but scala.math.Numeric inherits all of its (incredibly slow) comparison methods from Ordering. Specializing Ordering would solve this problem.

I also created a specialized typeclass with the humorous name “Convertable” which applies to all the value types which represent numbers (Byte, Short, Int, Long, Float and Double). The great thing about this is that you can avoid doing boxing/unboxing during conversion between types. It also allows you to implement the fromType[T] method I discussed earlier.

I left out Short and Byte after Paul Phillips pointed out that neither classes operations behave as the direct implementations do. In Scala (as in Java) Byte + Byte -> Int, whereas in Numeric A + A -> A (so Byte + Byte -> Byte). This causes all kinds of bugs. Since Short and Byte (and Char) are converted to Int when you actually do things with them, I decided it was unlikely users wanted to be able to use them in generic numeric functions.

A (potentially) controversial change I made was not including scala.math.Integral and scala.math.Fractional, and unifying support for the division and mod operators. Coming from dynamically-typed languages, I expected to be able to do the following:

// this does not work with the built-in Numeric
import Numeric.Implicits._
def scale[A:Numeric](n:A, num:A, denom:A) = (n * num) / denom
 
// this *does* work
import scala.math.Integral.Implicits._
def scale[A:Integral](n:A, num:A, denom:A) = (n * num) / denom

I understand that for someone who’s excited about creating user-defined numeric types this split isn’t a big deal. But from my perspective I want a generic numeric type that abstracts across all the useful value-types in Scala. I’m not opposed to having Integral and Fractional, but I would rather that Numeric not be defined in terms of them.

Finally I did some restructuring to avoid using inner traits and classes, since I had heard that those don’t specialize well.

Conclusion

While I still think there’s some room to make Numeric better, at this point it’s possible to write fast generic Numeric code in Scala. Stepping back, I think it’s incredibly exciting that I can avoid code duplication *and* maintain the speed that I’m used to from direct implementations.

I’m not sure what the best path is to add this capability to scala.math.Numeric. I think the trait lacks some important features (more type conversions, division, etc) and some of the restructuring is currently necessary to get these numbers. Since binary compatibility is important moving forward for Scala and the Typesafe team it’s not entirely clear what the best plan is. Hopefully some discussion and work at Scalathon can clarify things a bit.

Please respond with any questions or comments. Do let me know if you find problems in the profiling, or try it yourself and get dramatically different numbers. I spent some time trying not to fall into obvious micro-benchmark pitfalls, but there’s still a lot I don’t know about the JVM. Thanks for reading!

If you look through the CQL and ECQL page in GeoServer’s documentation, there are several examples but they don’t cover everything you can do with CQL. Basically, a CQL filter is a list of phrases similar to where clauses in SQL, separated by the combiner words AND or OR. You can use the following operators in a CQL phrase:

• Comparison operations: =, <, >, and combinations
• Math expressions: +, -, *, /
• NOT
• IS, EXISTS
• BETWEEN, BEFORE, AFTER
• IN
• LIKE
• Geometric operators: CONTAINS, BBOX, DWITHIN, CROSS(ES), INTERSECT(S)

Some of these operations have examples in the GeoServer documentation, and others can be inferred from the GeoTools documentation (the stuff in their CQL.toFilter() calls). CQL can also call any of GeoServer’s filter functions.You can add parenthesis as needed to affect the order the filters are evaluated in, just like in SQL.

CQL has a lot of power for such a short spec, but it has a one very large deficiency that requires some database designing to avoid: the utter lack of join support. This makes sense when you consider that GeoServer doesn’t know about joins either. Ultimately, you’re using CQL against the GeoServer layer, not the underlying database structure. Building views or adding reference columns to the table GeoServer is accessing can help get around this.

In PhillyTreeMap.org, we use 4 types of CQL filters: id lookups using =, null checks using IS, date and integer ranges using BETWEEN and text searches using LIKE. Here are examples of those uses along with some array joining to get a valid CQL filter string at the end:

filter_list = []
filter_list.append("species_id = 212")
filter_list.append("height IS NULL")
filter_list.append("dbh BETWEEN 10 and 20")
filter_list.append("neighborhood_id_list LIKE '%42%' ")
cql = ' AND '.join(filter_list)
# should look like this: "species_id = 212 AND height IS NULL AND dbh BETWEEN 10 and 20 AND neighborhood_id_list LIKE '%42%' "


The above CQL filter would locate trees in a specific species, have no height value, only have dbh values between 10 and 20 inches and are in a particular neighborhood. The neighborhood_id_list filter would have been a join if this were written in standard SQL since neighborhoods and trees have a many-to-many relationship in our database. Since we can’t do joins, any time a tree is added or the location is updated, all of it’s related geographies’ ids are added to a reference column on the tree, and used specifically for this type of query.

CQL is passed to GeoServer in the same way as any other WMS variable. We’re using openlayers, so most of the WMS configuration variables are already set when we create the layer. The WMS layer has a little method called mergeNewParams that lets us change those parameters after the layer has been initialized. It also automatically redraws the layer, so the changes take place immediately. To add CQL to the WMS call, just add the CQL_FILTER variable to the layer’s parameters and the layer should update.

wms_layer.mergeNewParams({'CQL_FILTER': "species_id = 212 AND height IS NULL AND dbh BETWEEN 10 and 20 AND neighborhood_id_list LIKE '%42%' "});


You can remove any filters by deleting the parameter from the layer as if it was a normal javascript object. You’ll need to redraw the layer yourself before the change will be visible.

delete wms_layer.params.CQL_FILTER;
wms_layer.redraw();


These posts will assume you know a bit of Scala, but I will try to keep the discussion as general as I can.

The Goal

The goal seems simple: to implement an algorithm but without choosing in advance which numeric type (e.g. int) to use, and without paying a performance penalty over a “direct” implementation.

Most dynamically-typed languages get this “for free” (in the sense that they pay a performance penalty regardless) and some statically-typed languages can accomplish this with macros, but I will focus on another approach: type classes. I will explain how Scala is able to transcend its Java roots and accomplish this task using the Numeric type class.

Java background

If you work with Java, you probably never write functions which can operate on both integral and floating point values.

Or at least, you probably never support both with one function–in that case you’d have to use java.lang.Number, which is implemented by Java’s boxed primitive types (java.lang.Integer, java.lang.Double) and which only gives you accessors for the various primitive types. It’s a mess:

public class Foo {
    public static Number add(Number t, Number u) {
        return new Double(t.doubleValue() + u.doubleValue());
    }
 
    public static void main(String[] args) {
        Number a = Foo.add(new Integer(3), new Integer(4));
        System.out.println(a.intValue());
    }
}

This code won’t work with some long values, and it basically forces the implementation to choose a primitive type that it hopes will work with the values it will be given (or to use java.math.BigDecimal for everything and incur even more slowness).

You might just support double and call it a day. Or you might write one version to deal with int and then copy-paste the same code but change all the int types to double, etc.. No matter what, the whole exercise feels icky.

Scala background

Scala tries to cover up some of Java warts by combining the notion of primitive types (int) and boxed types (java.lang.Integer) into a single unified type (Int). Scala’s Int type uses int when possible, but will transform them into java.lang.Integer instances behind the scenes if necessary. This makes boxing less annoying to do, but doesn’t make it any faster.

It also doesn’t directly help us accomplish our goal. Int and Double don’t share an interface which exposes things like + so we can’t usefully abstract across their types. In fact, they don’t share any superclass short of AnyVal so we can’t even implement something like our Java solution. (This isn’t totally correct–we can use Java’s types from Scala–but you get the idea.) So what gives? Is Scala even worse than Java when it comes to generic numbers?

No. Scala has more tools at its disposal than Java, and we can get around this problem using one of them: type classes.

Type classes in Scala

In Java, if you want a class to satisfy an interface, the class must be written with knowledge of the interface. When you create new interfaces, you have to go back and modify any classes which should implement them. If you’re using someone else’s classes and they don’t know about your interface, too bad!

This makes using other people’s code harder, and also locks you into someone else’s (possibly poorly-thought-out) type hierarchies. While I am not claiming that the type hierarchy for Scala’s numeric types is poorly-thought-out, it is certainly inconvenient for what we want to accomplish.

Type classes (introduced in the Haskell programming language) offer an alternative approach. Type classes are not interfaces which classes must implement. Instead a type class requires an instance to “glue” each member to the type class. When I say “glue” the member classes to the type class, I mean that each instance will provide the methods that the type class requires in terms of a particular member class. For example, if I have a Flammable type class and a House class, I can create a HouseIsFlammable instance that specifies how the House class satisfies Flammable.

We can implement type classes in Scala using implicit parameters. The basic idea is that we create a trait (Flammable[T]), along with implicit objects which “glue” the supported classes to new our type class (e.g. WoodIsFlammable). Each object corresponds to a single supported class (e.g. Wood). It implements the trait in terms of that class (e.g. Flammable[Wood]), and implements the trait’s methods (e.g. burn) in terms of the supported class.

Then when we want to use the type class, we make sure the instances are in scope and Scala’s compiler will automatically pick up those instances for us.

case class Alcohol(liters:Double)
case class Water(liters:Double)
 
case class Fire(heat:Double)
trait Flammable[A] {
  def burn(fuel:A): Fire
}
 
implicit object AlcoholIsFlammable extends Flammable[Alcohol] {
  def burn(fuel:Alcohol) = Fire(120.0)
}
 
def setFire[T](fuel:T)(implicit f:Flammable[T]) = f.burn(fuel)
 
setFire(Alcohol(1.0)) // ok
setFire(Water(1.0)) // FAIL

This is great! Notice that our Alcohol class doesn’t have to know that it is flammable; we just need to make sure that AlcoholIsFlammable is in scope whenever we want to pass Alcohol to setFire().

This is the exact approach that Scala takes: it defines a Numeric type class along with implicit objects that glue various types (like Int) to the type class.

For instance in Scala 2.8 (and later) this function will square the given number:

def square[T](a:T)(implicit n:Numeric[T]) = n.times(a, a)

Notice that we call the methods we’re interested in (e.g. times()) via the instance of our type class (n) and not the actual values that belong to the typeclass (a). Also, we don’t use a traditional type bound on T, but rather, requiring that an implicit object of type Numeric[T] can be found.

Numeric is part of the Standard Scala Library but unfortunately it does not seem to be widely used. I believe there are several reasons for this:

1. You can’t do this stuff in Java so people are used to working around the problem.
2. Using type classes in Scala can be a little bit confusing and cumbersome.
3. Scala’s Numeric lacks important features (e.g. division).
4. Scala’s Numeric is slow.

In my next blog post, I will explain how to use the Numeric type class in more detail, point out the problems I’m alluding to, and present our improved version along with some benchmarking numbers.