Tag Archives: tips

May 09, 2011

Using django_sorting without text anchors

By

In creating the OpenDataPhilly website, we knew we needed to pay extra attention to the usability and features available on the search page. After all, what use is a data catalog if you can’t rely on the search results? While designing the page, we decided we wanted to include several ways to sort and filter results along with the standard text search. I’d used directeur’s django_sorting module before, and was highly impressed at how well it integrated with other info already in search parameters, so I decided to use it again. The only thing keeping me from seamlessly dropping in this module was our desire to use images instead of words for the “click on this to sort” link. Django_sorting was built with table headers in mind; so much so that the examples are all about table header tags with a link inside them. We had a slightly different implementation in mind.

The first hurdle that you might think of would be to not require a table structure. Thankfully, django_sorting doesn’t care how you display the data, it only cares about the fields you want to sort on. You can put the sorting links anywhere and it just works. So far, so good.

The second possible hurdle here is that the anchor template tag specification calls for two parameters: the field to sort on, and a string for the link. Since we didn’t want a text link, we really didn’t care about the second parameter. To my surprise, neither did django_sorting! So our anchor template tags look something like this:

<li id="sort_rating_score">{% anchor rating_score %}</li>

This winds up creating a link that still has something for the title attribute and for the inner text:

<li ...><a title="Rating_score" href="/blah/blah/?sort=rating_score">Rating_score</a></li>

So our template tag is nice and clean, but we still have to deal with some text in the link. Using either straight-up javascript, or some smaller and nicer jQuery, removing the innerHTML is easy so long as the dom can uniquely identify our sort links. I just gave the link’s parent container an id and cleared the innerHTML of each link. At the same time, I added a class and some css to define the image and size.

So now we have a django_sorting anchor with an image instead of the default text link. All done, right? Not quite. We didn’t just want to use an image here, we wanted some mouseover and focus sprite action, too. Another chuck of jQuery, and a querystring plugin, helps us out again:

$("#sort .sort_image").each(function () {
    $(this).hover(function() {
        this.style.backgroundPosition="0 -89px"; //the hover image location
    }, function () {
        var filter_split = this.parentNode.id.split('sort_');
        if ($.query.get('sort') && $.query.get('sort') == filter_split[1]) {
            this.style.backgroundPosition="0 -45px";  //the active image location
        } else {
            this.style.backgroundPosition="0 0"; //the non-active image location
        }
    });
 });

 

Comments Off
Apr 08, 2011

jQuery Star Rating plugin as a frontend for Django-ratings

By

My current Django project, OpenDataPhilly, needed a ratings system and, guess what? Django has a ratings module! All done, right? Weeelllll.. the thing is, jQuery also has a beautiful and well-behaved ratings widget that is very easy to re-skin. How do you choose between two stable plugins? In this case there was no need. I decided to use the Django-ratings module for everything but the visual element and the jQuery Star Rating Plugin for some flashy effect.

So why blog about it then? The implementation was not quite as easy as it sounds.

First thing’s first: download and install the Django-ratings module the same way you would any other module. Follow their instructions to add a ratings field to your model and fix any import or database issues. You should wind up with something like this in your models file:

from djangoratings.fields import RatingField

class MyClass(models.Model):
    ....
    rating = RatingField(range=5)
    ...

Since we’re using jQuery for the front end, that’s all we need to do in the models file.

Next, download the jQuery plugin and put the files somewhere accessible by your app. I put them in my project’s static folder, but I think anywhere is fine. The plugin worked right out of the box for me, no tweaking necessary.

On to the slightly hard part! If you look in the jQuery plugin’s documentation, it expects a pile of radio buttons to skin. No divs or lists or other tricks of nice formatting, because the plugin does all that. Unfortunately, Django’s radio button widgets think they need to do all that, too. Someone has to be told ‘No‘ very firmly. Since the whole point of using the jQuery plugin is to have a pretty and easily skinable front end, we’re going to tell Django to stop rendering radio buttons quite as nicely as it does. And that’s going to take some doing.

Somewhere in your project, create a widgets.py file and hunt down the django.forms.widget.py file so you can copy a few things. The two classes we’re interested in are RadioInput and RadioFieldRenderer. Copy those two classes from Django’s file into your new one and close the Django file. We don’t want to accidentally mess something up in there.  First, we’ll need a few imports at the top of our file:

from django.forms.util import flatatt

from django.utils.encoding import StrAndUnicode, force_unicode
from django.utils.html import conditional_escape
from django.utils.safestring import mark_safe

Next, rename the classes to something not in Django. I chose StarsRadioInput and StarsRadioFieldRenderer. Also the StarsRadioFieldRenderer makes a few references to the RadioInput class. Change these to StarsRadioInput, too. Now that the classes are safely renamed, there are two methods in charge of the html coming out of our custom widget: StarsRadioInput.tag and StarsRadioFieldRenderer.render. We basically need to strip out all of the labels and list tags so that the jQuery plugin can do its css magic.

Change the StarsRadioInput.tag method to:

def tag(self):
  if 'id' in self.attrs:
    self.attrs['id'] = '%s_%s' % (self.attrs['id'], self.index)
    final_attrs = dict(self.attrs, type='radio', name=self.name, value=self.choice_value)
  if self.is_checked():
    final_attrs['checked'] = 'checked'
  return mark_safe(u'<input%s />' % flatatt(final_attrs))

And change the StarsRadioFieldRenderer.render method to

def render(self):
  return mark_safe(u'\n%s\n' % u'\n'.join([u'%s'
                   % force_unicode(w) for w in self]))

Now, all you have to do is make your form add our special renderer to the field and push the rating value into the right data model when the form is submitted.

In the forms.py file, add a field to your form. Don’t forget to create the rating choices tuple and add the ‘star’ class to the field.

RATING_CHOICES = ((1,1), (2,2), (3,3), (4,4), (5,5),)
forms.CharField(widget=forms.RadioSelect(renderer=StarsRadioFieldRenderer, attrs={'class':'star'}, choices=RATING_CHOICES))

In your form’s model, add this to the save method to connect the form data to the actual model’s data (if you need to):

def save(self, *args, **kwargs):
  myobject.rating.add(score=self.rating, user=self.user, ip_address=self.ip_address)
  super(CommentWithRating, self).save(*args, **kwargs)

Here’s what it looks like attached to a comments form:

django-rating and jquery star rating

django-rating and jquery star rating together

 

Comments (2)
Apr 05, 2011

PhillyHistory Augmented Reality Journal 2: Building Data Services

By

As I talked about in a previous journal, we’re exploring two different approaches to putting together an augmented reality application:  rolling our own client and using an existing framework and client. But regardless of what kind of client you’re using, the data (and images) have to come from somewhere, and that’s where data services come in to play. To support different client-side augmented reality viewers, we wanted to build an architecture that separated the data services from the client technology (the actual application that runs on a phone to provide the AR experience). This means that there could be a single source of digital asset information for any number of augmented reality clients.

Augmented Reality out my window (from our AR app!)

After reviewing the available technology and standards, we decided to build web services that conformed to the Layar standards, a mobile augmented reality platform developed in The Netherlands. Launched in 2009, Layar has quickly become ubiquitous as a platform for augmented reality applications. To implement an augmented reality layer in Layar, one publishes the “augmentations,” the points of interest that are visible in the augmented reality application, by creating a web service that client applications can query for information about what’s around them. A web service is simply a term for a standard method of allowing two computer programs to request and communicate data in a structured way. For example, a request to this “augmentation” database might request all of the points of interest within 200 meters of a given latitude and longitude. While the Layar webservice format has some limitations, it is relatively simple to implement both server-and client-side support for it. While not strictly what is often called a “RESTful” web service, which is a lightweight style of web service that in many ways is similar to loading a web page, the service can be implemented with a simple web application that can read in POST variables. As there is no independent standard for requesting and publishing augmented reality points of interest, the Layar service is as close as we could find. It had the additional advantage that we could directly test the result in the Layar clients available for both the Android and iPhone platforms.

Layar API Architecture (courtesy of Layar)

 

The API documentation for Layar’s REST services are on their documentation wiki.  Overall they have done a great job, but there are two big gotchas here.  First of all, they are in heavy development and their platform doesn’t yet seem fully stable and mature — we often felt like we were developing against a moving target, as what was supported overall and from device to device kept changing.  Secondly, their documentation wiki is somewhere between a really fantastic wiki and relatively chaotic and poorly organized documentation.  The details are spread out across various pages with various comments (some of which are totally critical) thrown in the mix.  We would have loved stable, versioned API documentation that was separate from the wiki and had all of the specifications in one place.   But my overall feeling is a bit like the Churchill quote: “Democracy is the worst form of Gov­ern­ment except for all those other forms that have been tried from time to time.”  While there are problems, it’s a relatively complete API and it’s a defacto standard for augmented reality data services.  That said, I’d love to see an open source standard for augmented reality data services  – and some open source clients that support it!

Winston Churchill at age 7 (thanks wikipedia.org)

While there are many advantages to this architecture, it is important to keep in mind that it means that all imagery is being transmitted to the mobile device while the user is using it. This creates a number of issues. If the user is has no or poor connectivity, the application will not be able to load photos – and even under good circumstances there will be a noticable delay, and there are restrictions to how many photos can be sent in a short amount of time over a network. An alternate approach would be to package the asset images with the application, and install those images along with the application. But while this approach might work well for a custom built application with 100 photos, the storage requirements would make this impractical for a large collection of 100k photos.

So … data munging (transforming from one format to another) projects  are almost always kind of sticky, but we hit some particular challenges in working with data for augmented reality. As discussed earlier, we used Google Street View as a tool to identify and select the desired angle in 3D space that we wished to place each photograph. However, Google Street View and Layar specify this angle differently. Both Layar and Google Street View represent a viewer facing north with a value of 0 degrees (in Layar this is the “angle” parameter and in Google Street View it is called “yaw”). However, in Google Street View rotations go clockwise (so 90 degrees is East and -90 degrees is West) whereas in Layar rotations go counter-clockwise (so 90 degrees is West and -90 degrees is East). However, the effort required to make these transformations was worthwhile. By January 2011, the PhillyHistory.org database management team had “pinned” more than 10,850 images to their Google Street View coordinates, providing a large subset of materials with which to test the 3D space options.

We chose to use to use a spatial database (PostgreSQL database with the PostGIS spatial extensions) to store our assets. A spatial database is designed to store and reason about objects in space – for example, it is possible to ask a spatial database to find the assets that are within a specific distance from the viewer.   Check out this newsletter article from Robert about PostGIS to learn more.   It is possible to add a stored procedure to a non-spatial database to make the same query (for example, the Layar wiki documentation provides such a function) but we found that with large numbers of points, the optimizations found in a spatial database were necessary for reasonable performance. The creation of a “spatial index” allows the database to limit its searches very quickly to likely candidates found within the database, instead of needing to search through all of the assets in the database.    That said, however, the overall performance of an AR application is limited by the network transmission time far more significantly than the backend server performance — but with 90k points it sure doesn’t hurt.

The ImageMagick wizard

Some image processing also needs to occur before images can be displayed on the small screen of a mobile device. This is extremely important because we found that clients (such as Layar’s client) will silently drop images that did not fit their specification. Because the client will not include the photo for a number of distinct reasons but there is no feedback for the developer explaining why the photo was not included, the process of diagnosing missing photos can be tricky. For Layar, the file size of all images must be smaller than 75 kB, and there are specific resolution limitations (e.g. full images in Layar must be less than 640×480). Given that mobile device screens have significantly smaller resolution than 640×480, that resolution is probably much higher than necessary. Additionally, some clients (like Layar) do not support making images transparent. It is therefore necessary to set the alpha channel of the photos in a pre-processing step. For example, using the open source ImageMagick package, the following command line invocation could perform the necessary scaling and transparency conversion on an incoming image stream (on a linux box): “cat input_image.jpg | convert – PNG32:- | convert -scale 240×180 -channel Alpha -evaluate Multiply 0.9 output.png”.

 

There are already a number of open source platforms for publishing data services compatable with the Layar API, most notably PorPOISe (PHP), django-layar (Python) from our open data crush Sunlight Labs and LayarDotNet (C#), with PorPOISe being the most fully featured of the platforms we reviewed. However, PorPOISE lacked some crucial 3D features at the time of our review. The beta release of an online service called Hoppala Augmentation does support 3D layers, but we were unable to get the 3D service to work and found the documentation and usability to be underdeveloped. It is certainly necessary to have a full understanding of the Layar protocol to use the Hoppala service (at this point) as the API allows developers to set a range of settings without explanation or checks on invalid or conflicting settings. Given these limitations and our desire to implement our own interactive capabilities and user settings, we decided to develop our own data web services in Python, which turned out to be a great choice for us because it let us prioritize, shape, and alter the results in a variety of ways.   Our next journal (from Erik) will feature some of the ways that we needed to change the 3D placement of photos and how we went about it.

Want to know more about our AR work?

Comments (1)
Mar 23, 2011

When is seven days not a week?

By

I’ve been using Date.js on a project, and it’s an awesome library. I especially love when someone else handles dates for me (am-i-right fellow developers? ;) ), and everything was going great until we discovered a rare time bug. I had forgotten something very important regarding time: it’s relative.

We’re not writing guidance software for rockets, so we don’t have to deal with relativity in the Einstein sense, but relative in a more day-to-day and somewhat artificial sense. Although I’m not exactly sure how Benjamin Franklin really felt about daylight savings time, it certainly makes programming a little trickier for us Philadelphians.

Here is an example for those following along in firebug:

var start = new Date('2010/01/01');
var end = new Date('2011/01/01');
var diff = (new TimeSpan(end - start)).days;

for(var i=0;i<diff;i++) {
    var today = new Date(start).add(i).days();
    var plusSeven = today.clone().add(7).days();
    var oneWeek = (new TimeSpan(plusSeven - today));

    if ((oneWeek.days != 7) || (oneWeek.hours != 0)) {
        console.debug("Time Warp: ", today);
    }
}

Say you made some bit of code that generated a date range by adding or subtracting some number of days off a date, you’d have a bug from March 8th through March 14th, and then it’d be gone until November 1st through the 7th! Not only is this sneaky, it’s hard to predict! Although this isn’t really a bug in Date.js, it requires us to change how we think about time.

The recommended Googling didn’t provide a straightforward, and rock-solid technique for identifying and adjusting for this effect in any given time zone.

So I made one:

First, a quick example:
(my apologies if this is obvious or full of mistakes)

//Lets say you wanted to setup some DateTime controls
//with a rolling date range, lets say using an offset in days
//with Date.js you'd probably do something like this:

var offsetDays = 7;
var start = new Date('2010/03/09');
var end = start.clone().add(offsetDays).days();

//so far so good, we got the intended date:
//Tue, March 16th, 2010
//but we want to update and store the rolling date range,
//so lets get that range back out:

// ... snip! ...

var diff = (new TimeSpan(end - start));
if (diff.days != offsetDays) {
    //daylight savings time caused time to 'spring ahead',
    //and our 7 days is instead 6 days 23 hours
    console.debug('user changed date');
}

We’re going to need a simple function that gets a TimeSpan that is adjusted for the daylight savings time adjustment:

// gets the difference between two dates,
// adjusting for daylight savings time.
var getDSTAdjustedDiff = function(startDate, endDate) {
    var startOffset = startDate.getUTCOffset();
    var endingOffset = endDate.getUTCOffset();

    if (startOffset > endingOffset) {
        startDate = startDate.clone().add({hours: -1 });
    }
    if (endingOffset > startOffset) {
        endDate = endDate.clone().add({hours: -1 });
    }
    return (new TimeSpan(endDate - startDate));
};

If we revisit our example from before:

var start = new Date('2010/01/01');
var end = new Date('2011/01/01');
var diff = (new TimeSpan(end - start)).days;

for(var i=0;i<diff;i++) {
    var today = new Date(start).add(i).days();
    var plusSeven = today.clone().add(7).days();
    var oneWeek = (new TimeSpan(plusSeven - today));
    var adjustedDiff = getDSTAdjustedDiff(today, plusSeven);

    if ((oneWeek.days != 7) || (oneWeek.hours != 0)) {
        console.debug("Time Warp: ", today);
    }
    if ((adjustedDiff.days != 7) || (adjustedDiff.hours != 0)) {
        console.debug("Adjustment fail!: ", today, adjustedDiff );
    }
}

If everything went well, this code block should run and run without the second block ever firing. I hope this helps someone else out there avoid this pitfall, now if I can just figure out time zones…

Comments Off
Feb 28, 2011

jQuery, OpenLayers.Layer.Vector and IE8

By

One of the very first things you learn about OpenLayers is the importance if initializing maps in either the body tag’s onload event or in script tags at the bottom of the page. The basic reason for this is that many of the OpenLayers components need a page’s dom to be complete before it starts adding it’s own dom structure. The onload event is thrown after the browser is finished loading everything else, and the bottom of the page naturally gets rendered last, so these two places tend to work the best. Makes sense right?

If you’ve been using additional frameworks, like  jQuery ( or EXT or Django or Drupal or whatever) you’ve learned to rely on that framework’s onready event, or its rough equivalent.  This event tends to be thrown before the onload event so that the frameworks don’t need to wait for every last image and tag to be loaded. In theory, this could be a very problematic place to put OpenLayers map initialization code. The dom elements that OpenLayers is looking for may or may not be there, and there’s no way to tell. In practice, most browsers, and most OpenLayers components, work fine when initialized in a framework’s onready event. One layer in particular (OpenLayers.Layer.Vector) will not play nice with one particular browser (IE8) if initialized in a framework’s onready event. The vector layer depends on the document object’s namespace attribute, and this attribute simply isn’t always available before IE’s onload event (which is called after the onready event). As you can imagine, this was one headache I wish I could have avoided.

Thankfully, there is a fairly simple fix. Instead of the usual convention of :

$.ready(function() { init_my_map(); });

Add a stand-alone script tag to the extreme end of the body content on the pages where you need a vector layer:

<script>
    init_my_map();
</script>
Comments Off
Aug 18, 2010

How We Launched a New CMS (301 redirects and counting)

By

When we decided to give our website a brand new look, we also evaluated whether to change Content Management Systems and decided to migrate our content management system to Concrete 5 from DotNetNuke.  DotNetNuke had served us well, but we wanted to move to a more modern and lightweight framework with a better administrative interface.   This migration presented us with a few challenges to overcome.

While we redesigned our company website, we didn’t want to wait to launch it until we had migrated all of our product websites to Concrete 5.   To combine the two systems under one domain name, we leveraged Varnish as a proxy to split requests between our LAMP stack hosting Concrete 5 and our Windows server hosting DotNetNuke.    We’ve found Varnish to be a robust and yet lightweight tool for proxying, caching, and load balancing.  You can read more about how we used Varnish to host Walkshed on Amazon Web Services in this related blog post.

Another concern when migrating CMS’s is making sure that the old URLs continue to function and issue the right redirects to users and search engines.   Andrew Jennings took this project on and whipped up a URL redirection system for us using Apache’s mod_rewrite module.

  • As requests come into Varnish, we first test to see if Concrete5 can respond to the URL.
  • If Concrete5 isn’t managing the content for the URL, the request bounces into our redirection system that maps old URLs to new URLs using a binary hashtable and issues 301 permanent redirects to the client.   We found that this approach is faster than simply relying on Apache .htaccess files since we have several thousand legacy URLs to support.
  • If the URL request is not in the mapping file, we simply forward the request to the Windows server to be served by DotNetNuke.
  • If DotNetNuke returns a 404 error, we redirect the request back to Concrete 5 to respond with a 404 error so that our redirection is invisible to the user.

A final goal of the CMS migration was to improve page load times for our visitors.   Having pages that load quickly not only improves the experience for our web visitors, but also has an impact on search engine rankings.    Concrete5 suggested we setup Alternative PHP Cache (APC) to speed up the pages it is serving, which has definitely improved performance on our new pages.

We found a great WordPress plugin (which runs our blogs) that can also leverage APC called W3 Total Cache.   It provides a host of other functionality that we’re exploring including HTML, JS, and CSS minification as well as pushing design assets to Amazon’s CDN.

I always find it amusing how something as straight forward as a new website launch has so much happening behind the scenes.   To find out more about the content that went into the new Azavea website read Rachel’s related blog post.

Comments (2)
Feb 04, 2010

Getting an ArcGIS Server Map Cache in S3

By

When deciding how to best handle the air photos in the new Philadelphia Water Department Stormwater Map Viewer, we kicked around a few ideas. We decided to put the cache in Amazon’s Simple Storage Service to offload some of the local disk requirements and leverage their fast data storage and delivery infrastructure. In moving the process, we learned a few things:

Tune Your Cache

Make sure you spend time planning the cache. Not only will the cache look better in the final application, but it will also load to S3 faster and cost less in the long run.

  • Set the extents in the MXD or MSD before publishing to a map service. The overhead of transferring the 254 byte empty tiles caused a lot of unnecessary burden on the upload process as well as the fact that you are paying for them to be stored in the cloud. If it doesn’t need to be there, don’t build it.
  • Choose the correct image format for the cache. If you are caching a base map and do not need to support transparency, make it a JPEG. If it needs to support background transparency, use PNG. ESRI’s suggestions for planning a map cache can be found here.

Get a Good Tool to Transfer the Files

I started using the free version of Cloudberry Labs S3 explorer. But I had to move over 90 Gbs worth of data to my S3 bucket. The CloudBerry S3 Explorer – Pro supported multithreading which allowed for up to 5 threads to either enumerate through the folders, copy the files or apply the ACL. It is a low cost application that more than pays for itself when moving a lot of files up to a bucket.

When transferring the files up, I was working in blocks of directories, not the whole scale level. It was quicker for me to work in 20 to 30 subdirectories than grabbing a whole scale level. It did require a little bit more management on my end, but more steady progress was made.

Accessing the Tiles

ArcGIS Server does not support cloud hosted caches at the 9.3.1 release. The ESRI Javascript API and Flex API can be extended to use caches hosted in the cloud (Flex example from Mansour Raad), so you’ll have to roll your own. For the Philly Storm Water project, we were using the Open Layers and someone has rolled one for us. There is a patch that can be used to access the cache without communicating through ArcGIS Server straight from the client-side library. The one thing to note is that the Tile Origin is pretty touchy, we had to make some adjustments to the origin values to make sure everything lined up correctly.

Summary

Now that the site is up there and we are starting to get some traffic hitting it, putting the tiles in S3 was the right decision. There is no reason for ArcGIS Server to waste any cycles moving tiles around, let it do the heavy lifting with the vector layers and queries. Hopefully the rumors are true, and the ArcGIS Server 10 release will be more aligned with cloud computing. Until then, there are still plenty of ways to take advantage of the benefits.

Comments (1)
  • Azavea Journal
  • Azavea Atlas
  • Azavea Labs
  • Azavea.com