Tag Archives: Django

Geoserver Timestamp Styling and PostgreSQL DateTime Fields

Think for a moment about how many ways we talk about dates and time. In regular language, we can say things like “last Tuesday” or “week after the week after next” or “Friday the 12th” and people generally can figure out when we’re talking about. Add time into the mix and phrases start getting strange: “noon-thirty” and “half-past midnight” are my favorite oddities.

The actual date-time standard is published as ISO 8601 and talks about what date-time data should look like, but programming languages, databases and other programs implement this standard with sometimes vastly different interfaces and rules.

PostgreSQL’s handling of date-time data takes the form of 6 different field types, all suited to different data needs. For the OpenTreeMap project, we’re using the timestamp data type which allows us to store both dates and times for events down to the microsecond.

Geoserver accepts a single date-time type (also called timestamp) that stores dates and times at the same resolution as PostgreSQL’s data type, so it reads PostgreSQL’s date-time data easily. Adding a date filter to Geoserver’s SLD files is also fairly easy so long as you know how Geoserver wants date-time data to be formatted. The various formats that Geoserver knows how to interpret are located here. So, if we have a field in a database table called “last_updated”, then an example SLD filter might look like this:

<Filter>
  <PropertyIsLessThan>
     <PropertyName>last_updated</PropertyName>
     <Literal>2012-01-01 00:00:00</Literal>
  </PropertyIsLessThan>
</Filter>

This filter would display any updates made before midnight on January 01, 2012. Another example (from the OGC 1.0 encoding specification’s examples) catches updates between certain dates:

<ogc:Filter>
  <ogc:PropertyIsBetween>
    <ogc:PropertyName>last_updated</ogc:PropertyName>
    <ogc:LowerBoundary>
      <ogc:Literal>2011-12-01 00:00:00</ogc:Literal>
    </ogc:LowerBoundary>
    <ogc:UpperBoundary>
      <ogc:Literal>2011-12-31 23:59:59</ogc:Literal>
    </ogc:UpperBoundary>
  </ogc:PropertyIsBetween>
</ogc:Filter>

This filter would display any updates made during the month of December, 2011. Anything that was updated outside the specified date and time parameters would bypass these filters and go on to check any other rules in the SLD file. This works great if we’re only interested in static date comparisons. But what if we want to see updates less than a week old? Or objects that haven’t been updated for more than three months? This kind of dynamic filtering is a little harder to do.

After digging through the Geoserver documentation and a lot of googling, we decided to shift the dynamic part of the filter into a PostgreSQL view. Our test view included an id field, a geometry field and a field that calculates the number of days since the last update to that object. The view sql looks like this:

CREATE OR REPLACE VIEW timestamp_test AS
  SELECT
    treemap_tree.id,
    date_part('days'::text, now() - treemap_tree.last_updated) AS days,
    treemap_tree.geometry
  FROM treemap_tree;

So now we can add this view to Geoserver as a source layer and it will see the new days field as a static number field. We can use any of the property filters on this field and style recently added data in a dynamic fashion.

Django, contests and weekly voting

I’ve written before about how OpenDataPhilly uses a ratings module to drive a nomination system. Recently, we added a contest to the site to determine what kinds of data local non-profits and the public would like to see made available. Contests generally have a winner and, in this case, we’re letting the public vote on data sets nominated by non-profits. At first glance this isn’t much different from our current nomination system, but there’s one catch; we wanted users to be able to vote for one entry once a week. Turns out this was more novel than it sounds.

Django has a few modules for rating or voting on content, one of which we’re using for the nomination and comments systems. The inner-workings of the module boil down to the following rules:

  1. A user must be logged in to rate/vote
  2. A user can rate/vote for any number of items
  3. A user can only rate/vote for any particular item once (though they may change their rating/vote later)

Compare this with the rules we wanted to enforce for the contest:

  1. A user must be logged in to vote
  2. A user can only vote once per 7 day period
  3. A user can vote for an item multiple times, so long as rule 2 is preserved

Aside from the first rule, we were trying to do almost exactly the opposite of what our rating module enforced. Rather than retrofit the existing module to allow additional and sometimes contradictory behaviour, we decided to write a very small voting module of our own.

The code revolves around two decision points: is voting allowed and can a specific user vote now. The first question is answered by the contest object itself. A contest knows when it’s starting and ending date are, so if today is after the start date and before the end date, then voting is allowed.

The second question is a bit more complicated, but not by much. Because of rule 2 above, we need to know when a user last voted to know if they’re currently allowed to vote. The database storage for a vote contains a datetime object, a foreign key to the user object and a foreign key to the contest entry so if we sort a user’s votes by time we can retrieve their latest vote.

def user_can_vote(self, user):
    increment = datetime.timedelta(days=7)
    votes = user.vote_set.order_by('-timestamp') #latest on top
    if votes:
        next_date = votes[0].timestamp + increment
        if datetime.datetime.today() < next_date and dt.today() < contest.end_date:
            return False
    return True

The above code gets a user’s votes and orders them by time with the most recent first. If a user has ever voted, we need to check if they’re allowed to vote again yet or if they have to wait. We calculate the earliest time that a user can vote next and check it against the date and time now. We also check the end of the contest against the date and time now. If “now” is before the next time the user can vote or “now” is after the contest’s end date, we return false; the user can’t vote now. If a user has never voted before, or the dates are all ok then the user can vote. This check is done after a user clicks the “vote” button but before a vote is saved to the database. We also display a message saying why this check failed and when a user will be able to vote again.

So we’re taking advantage of all of the spam protection built into Django’s user registration process and running a contest on surprisingly little code: 3 database tables, 200 lines of python (blank lines included) and a few templates is all we needed!

Restricting Zoom with Multiple OL Basemaps

DistrictBuilder logoAs David recently posted, our team has been hard at work implementing DistrictBuilder, where we’ve been investing a great deal of effort on both performance and usability. One feature we added in the spring was the addition of basemaps to the user interface. Before this addition, users labored over drawing the perfect district configurations without a whole lot of context of the surrounding environment (e.g. roads, water boundaries, etc.). When the time came to add a basemap to the application, it didn’t feel right restricting it to a single type of map, or even a single provider. We wanted to allow for users to have the choice to select the best map for the task at hand. Could an application promoting democracy really have it any other way?

We set out to support several base map options as well as any combination of options, including:

  • Bing Maps (satellite, roads and hybrid)
  • GoogleMaps (satellite, roads and hybrid)
  • ArcGIS Online (any of several maps)
  • OpenStreetMap

Since DistrictBuilder needs to be flexible enough to meet the needs of users and administrators in a variety of situations, we decided on a two step approach to basemap configuration. First, the administrator specifies, in the configuration file, which of the combinations of map providers and map types are allowed to be selected. Then DistrictBuilder presents all of the configured options to the user, where they can be toggled among at any time while a plan is being viewed or edited.

Here’s an example of an instance configured with an OpenStreetMap road layer, a Bing hybrid layer, and a Google satellite layer:

Road, Hybrid, and Satellite

Here’s another example with only road layers — one for each of the three configured providers:

Roads for three vendors

DistrictBuilder currently allows the configuration of basemaps using permutations of each of the three vendors and three map types described above. Adding more options is a relatively easy task, however. With the launch of Fix Philly Districts, we wanted the basemap colors to be slightly more muted than the above options, and ended up adding support for the ArcGIS Online World Topographic Map. We also experimented with the Google Maps V3 custom styling API, which looked great, but introduced performance problems when panning and zooming (animations).

There were, of course, some hoops that needed to be jumped through in order to get all of these basemaps behaving correctly on the same map, which will be discussed below. I’ve extracted the logic required to do so into a small demo that can be viewed/downloaded here. The demo has also been embedded into this post, and can be interacted with without going anywhere:

Zoom Levels

Many of the challenges that needed to be overcome to get this working correctly were brought about because we needed to restrict the zoom levels to the area at hand. We wanted to eliminate superfluous zoom levels to ensure the user was always operating within the appropriate boundaries (note: it is not done in this demo, but in DistrictBuilder we also restrict the extent with the ‘restrictedExtent’ map parameter, so users can’t even pan outside of the area).

One difficulty with setting zoom levels on the different layers is that the layers don’t use zoom parameters consistently. In Bing (the VirtualEarth layer), minZoomLevel and maxZoomLevel are needed. In Google, minZoomLevel is needed, but it requires numZoomLevels instead of maxZoomLevel. And in OpenStreetMap (the OSM layer), well…no combination of those seem to work — we needed to slightly modify the XYZ layer (OSM’s base class) to allow maxResolution to be changed based on the minZoomLevel. To see how this is done, view the demo source. With that change in place, the list of required layer parameters is as follows:

  • Bing – minZoomLevel, maxZoomLevel, projection, sphericalMercator, maxExtent
  • Google – numZoomLevel, minZoomLevel, projection, sphericalMercator, maxExtent
  • OpenStreetMap - numZoomLevel, minZoomLevel, projection

Coordinate Systems

We also faced some problems related to coordinate systems. DistrictBuilder uses GeoServer and GeoWebCache to serve up WMS layers. The coordinate system of our data is one version of the the ever-changing “Popular Visualization CRS / Mercator” projection. We needed to match up the OpenLayers projection to the one used on our data, or else we were seeing slight offsets on our overlays. Unfortunately, the ‘projection’ layer parameter isn’t always used within the layers correctly. For example, any layer using the SphericalMercator class gets its projection automatically hardcoded to 900913. We needed to make a slight modification to the SphericalMercator class to allow the ‘projection’ parameter to carry through. This can be seen by viewing the demo source.

Bonus: Math Time!

One interesting part about implementing zoom restriction was that we needed it to work in any instance of DistrictBuilder — from large states to small towns, which may have vastly different extents. Instead of having an administrator figure out the proper minimum zoom level, we calculate it automatically based on the extent, which requires a little bit of basic algebra.

For Philadelphia, the extent of our area is:

[-8397913.926216, 4842467.609439, -8329120.600772, 4895973.529229]

In DistrictBuilder, we calculate this dynamically on the server side (using Django) by filtering all of the geounits in the database and calling the ‘extent’ function on the query set. For the demo, this is hardcoded. Here’s how to transform this extent into a Spherical Mercator minZoomLevel:

  • Find the width of the area in meters.
var studyWidthMeters = extent[2] - extent[0];
  • Find the width of the map in pixels. In the demo, this is hardcoded, because we are setting the div size of the map. In DistrictBuilder, the map takes up the whole screen, and this value is calculated on the fly based on the size of the div in which the map occupies.
var mapWidthPixels = 450;
  • Find the map resolution, or meters per pixel.
var resolution = studyWidthMeters / mapWidthPixels;
  • Find the maximum map resolution. In Spherical Mercator, the maximum resolution is one 256×256 tile taking up the entire circumference Earth. So dividing the circumference of the earth (~40,000km) by 256 gives us the maximum meters per pixel, which is a constant.
var maxResolution = 156543.033928;
  • Spherical Mercator zoom levels work like a pyramid. Each zoom breaks the current tile up into a 2×2 group of 256×256 tiles, essentially halving the resolution each time. Therefore, finding the resolution at a given zoom level looks like this:
maxresolution / 2^zoom = resolution
  • We know the resolution and max resolution already, and need to find the zoom:
zoom = log(maxresolution/resolution)/log(2)
  • Or in javascript:
var minZoom = Math.log(maxResolution / resolution) / Math.LN2;

Pending edit system using Django

A common concern when we talk to people about OpenTreeMap is how much to trust the public with an organisation’s tree inventory. Every implementation of this open source system has a different answer. The original site, UrbanForestMap.org, allows a logged-in user to edit almost every bit of information they gather about a tree. PhillyTreeMap.org requires a certain level of reputation before a user can edit everything, but even a new user has considerable edit capabilities. The most recent implementation (still a work-in-progress) introduces a bit of oversight to public edits. The managing group wanted to double-check changes to officially inventoried trees, but didn’t want to get in the way of people adding and editing their own trees.

Lets look at how this changes the user story first:

A logged-in user makes an edit to a tree. The system needs to decide if these changes are applied to the tree or placed in a pending queue. If this is a publicly-entered tree, the changes are applied to the tree. (Start new requirements) If this is an inventory tree, and the user isn’t a member of a management group, add the change to the pending queue. Display any pending changes reasonably near the appropriate current value. (End new requirements)

Most of this happens behind the scenes in the saving logic. I added a bit of code to the top of our tree updater to check if the pending system is active, the user’s permissions and the tree’s origins. If everything checks out, the change goes straight into the updater code. For changes that go into the pending queue, the path to becoming an official change is a little more tortuous.

Since we’re storing these changes for later review, they have to go into the database. I created a new table to hold onto the original tree’s id, the field being changed and the new value as well as the user who submitted it, a date/time stamp and a status field. Each pending change is stored separately; even if the user makes more than one change to the tree, each ‘pend’ can be applied individually.

The rest of the pending system is eye-candy and a bit of slightly tedious templating. Almost every field on a tree’s detail page now needs to check two new things: are there any pending changes for this field, and does this user have permission to approve/disapprove pending edits. If there are pending edits, the new values are added below the current official value. When a managing user views the page, small approve and disapprove buttons also appear next to each pending change. Throw in a management-access-only page for some bulk evaluation and the system is complete!

Using django_sorting without text anchors

In creating the OpenDataPhilly website, we knew we needed to pay extra attention to the usability and features available on the search page. After all, what use is a data catalog if you can’t rely on the search results? While designing the page, we decided we wanted to include several ways to sort and filter results along with the standard text search. I’d used directeur’s django_sorting module before, and was highly impressed at how well it integrated with other info already in search parameters, so I decided to use it again. The only thing keeping me from seamlessly dropping in this module was our desire to use images instead of words for the “click on this to sort” link. Django_sorting was built with table headers in mind; so much so that the examples are all about table header tags with a link inside them. We had a slightly different implementation in mind.

The first hurdle that you might think of would be to not require a table structure. Thankfully, django_sorting doesn’t care how you display the data, it only cares about the fields you want to sort on. You can put the sorting links anywhere and it just works. So far, so good.

The second possible hurdle here is that the anchor template tag specification calls for two parameters: the field to sort on, and a string for the link. Since we didn’t want a text link, we really didn’t care about the second parameter. To my surprise, neither did django_sorting! So our anchor template tags look something like this:

<li id="sort_rating_score">{% anchor rating_score %}</li>

This winds up creating a link that still has something for the title attribute and for the inner text:

<li ...><a title="Rating_score" href="/blah/blah/?sort=rating_score">Rating_score</a></li>

So our template tag is nice and clean, but we still have to deal with some text in the link. Using either straight-up javascript, or some smaller and nicer jQuery, removing the innerHTML is easy so long as the dom can uniquely identify our sort links. I just gave the link’s parent container an id and cleared the innerHTML of each link. At the same time, I added a class and some css to define the image and size.

So now we have a django_sorting anchor with an image instead of the default text link. All done, right? Not quite. We didn’t just want to use an image here, we wanted some mouseover and focus sprite action, too. Another chuck of jQuery, and a querystring plugin, helps us out again:

$("#sort .sort_image").each(function () {
    $(this).hover(function() {
        this.style.backgroundPosition="0 -89px"; //the hover image location
    }, function () {
        var filter_split = this.parentNode.id.split('sort_');
        if ($.query.get('sort') && $.query.get('sort') == filter_split[1]) {
            this.style.backgroundPosition="0 -45px";  //the active image location
        } else {
            this.style.backgroundPosition="0 0"; //the non-active image location
        }
    });
 });

 

jQuery Star Rating plugin as a frontend for Django-ratings

My current Django project, OpenDataPhilly, needed a ratings system and, guess what? Django has a ratings module! All done, right? Weeelllll.. the thing is, jQuery also has a beautiful and well-behaved ratings widget that is very easy to re-skin. How do you choose between two stable plugins? In this case there was no need. I decided to use the Django-ratings module for everything but the visual element and the jQuery Star Rating Plugin for some flashy effect.

So why blog about it then? The implementation was not quite as easy as it sounds.

First thing’s first: download and install the Django-ratings module the same way you would any other module. Follow their instructions to add a ratings field to your model and fix any import or database issues. You should wind up with something like this in your models file:

from djangoratings.fields import RatingField

class MyClass(models.Model):
    ....
    rating = RatingField(range=5)
    ...

Since we’re using jQuery for the front end, that’s all we need to do in the models file.

Next, download the jQuery plugin and put the files somewhere accessible by your app. I put them in my project’s static folder, but I think anywhere is fine. The plugin worked right out of the box for me, no tweaking necessary.

On to the slightly hard part! If you look in the jQuery plugin’s documentation, it expects a pile of radio buttons to skin. No divs or lists or other tricks of nice formatting, because the plugin does all that. Unfortunately, Django’s radio button widgets think they need to do all that, too. Someone has to be told ‘No‘ very firmly. Since the whole point of using the jQuery plugin is to have a pretty and easily skinable front end, we’re going to tell Django to stop rendering radio buttons quite as nicely as it does. And that’s going to take some doing.

Somewhere in your project, create a widgets.py file and hunt down the django.forms.widget.py file so you can copy a few things. The two classes we’re interested in are RadioInput and RadioFieldRenderer. Copy those two classes from Django’s file into your new one and close the Django file. We don’t want to accidentally mess something up in there.  First, we’ll need a few imports at the top of our file:

from django.forms.util import flatatt

from django.utils.encoding import StrAndUnicode, force_unicode
from django.utils.html import conditional_escape
from django.utils.safestring import mark_safe

Next, rename the classes to something not in Django. I chose StarsRadioInput and StarsRadioFieldRenderer. Also the StarsRadioFieldRenderer makes a few references to the RadioInput class. Change these to StarsRadioInput, too. Now that the classes are safely renamed, there are two methods in charge of the html coming out of our custom widget: StarsRadioInput.tag and StarsRadioFieldRenderer.render. We basically need to strip out all of the labels and list tags so that the jQuery plugin can do its css magic.

Change the StarsRadioInput.tag method to:

def tag(self):
  if 'id' in self.attrs:
    self.attrs['id'] = '%s_%s' % (self.attrs['id'], self.index)
    final_attrs = dict(self.attrs, type='radio', name=self.name, value=self.choice_value)
  if self.is_checked():
    final_attrs['checked'] = 'checked'
  return mark_safe(u'<input%s />' % flatatt(final_attrs))

And change the StarsRadioFieldRenderer.render method to

def render(self):
  return mark_safe(u'\n%s\n' % u'\n'.join([u'%s'
                   % force_unicode(w) for w in self]))

Now, all you have to do is make your form add our special renderer to the field and push the rating value into the right data model when the form is submitted.

In the forms.py file, add a field to your form. Don’t forget to create the rating choices tuple and add the ‘star’ class to the field.

RATING_CHOICES = ((1,1), (2,2), (3,3), (4,4), (5,5),)
forms.CharField(widget=forms.RadioSelect(renderer=StarsRadioFieldRenderer, attrs={'class':'star'}, choices=RATING_CHOICES))

In your form’s model, add this to the save method to connect the form data to the actual model’s data (if you need to):

def save(self, *args, **kwargs):
  myobject.rating.add(score=self.rating, user=self.user, ip_address=self.ip_address)
  super(CommentWithRating, self).save(*args, **kwargs)

Here’s what it looks like attached to a comments form:

django-rating and jquery star rating

django-rating and jquery star rating together