The first hurdle that you might think of would be to not require a table structure. Thankfully, django_sorting doesn’t care how you display the data, it only cares about the fields you want to sort on. You can put the sorting links anywhere and it just works. So far, so good.

The second possible hurdle here is that the anchor template tag specification calls for two parameters: the field to sort on, and a string for the link. Since we didn’t want a text link, we really didn’t care about the second parameter. To my surprise, neither did django_sorting! So our anchor template tags look something like this:

<li id="sort_rating_score">{% anchor rating_score %}</li>

This winds up creating a link that still has something for the title attribute and for the inner text:

<li ...><a title="Rating_score" href="/blah/blah/?sort=rating_score">Rating_score</a></li>

So our template tag is nice and clean, but we still have to deal with some text in the link. Using either straight-up javascript, or some smaller and nicer jQuery, removing the innerHTML is easy so long as the dom can uniquely identify our sort links. I just gave the link’s parent container an id and cleared the innerHTML of each link. At the same time, I added a class and some css to define the image and size.

So now we have a django_sorting anchor with an image instead of the default text link. All done, right? Not quite. We didn’t just want to use an image here, we wanted some mouseover and focus sprite action, too. Another chuck of jQuery, and a querystring plugin, helps us out again:

$("#sort .sort_image").each(function () {
    $(this).hover(function() {
        this.style.backgroundPosition="0 -89px"; //the hover image location
    }, function () {
        var filter_split = this.parentNode.id.split('sort_');
        if ($.query.get('sort') && $.query.get('sort') == filter_split[1]) {
            this.style.backgroundPosition="0 -45px";  //the active image location
        } else {
            this.style.backgroundPosition="0 0"; //the non-active image location
        }
    });
 });

So why blog about it then? The implementation was not quite as easy as it sounds.

First thing’s first: download and install the Django-ratings module the same way you would any other module. Follow their instructions to add a ratings field to your model and fix any import or database issues. You should wind up with something like this in your models file:

from djangoratings.fields import RatingField

class MyClass(models.Model):
    ....
    rating = RatingField(range=5)
    ...

Since we’re using jQuery for the front end, that’s all we need to do in the models file.

Next, download the jQuery plugin and put the files somewhere accessible by your app. I put them in my project’s static folder, but I think anywhere is fine. The plugin worked right out of the box for me, no tweaking necessary.

On to the slightly hard part! If you look in the jQuery plugin’s documentation, it expects a pile of radio buttons to skin. No divs or lists or other tricks of nice formatting, because the plugin does all that. Unfortunately, Django’s radio button widgets think they need to do all that, too. Someone has to be told ‘No‘ very firmly. Since the whole point of using the jQuery plugin is to have a pretty and easily skinable front end, we’re going to tell Django to stop rendering radio buttons quite as nicely as it does. And that’s going to take some doing.

Somewhere in your project, create a widgets.py file and hunt down the django.forms.widget.py file so you can copy a few things. The two classes we’re interested in are RadioInput and RadioFieldRenderer. Copy those two classes from Django’s file into your new one and close the Django file. We don’t want to accidentally mess something up in there.  First, we’ll need a few imports at the top of our file:

from django.forms.util import flatatt

from django.utils.encoding import StrAndUnicode, force_unicode
from django.utils.html import conditional_escape
from django.utils.safestring import mark_safe

Next, rename the classes to something not in Django. I chose StarsRadioInput and StarsRadioFieldRenderer. Also the StarsRadioFieldRenderer makes a few references to the RadioInput class. Change these to StarsRadioInput, too. Now that the classes are safely renamed, there are two methods in charge of the html coming out of our custom widget: StarsRadioInput.tag and StarsRadioFieldRenderer.render. We basically need to strip out all of the labels and list tags so that the jQuery plugin can do its css magic.

Change the StarsRadioInput.tag method to:

def tag(self):
  if 'id' in self.attrs:
    self.attrs['id'] = '%s_%s' % (self.attrs['id'], self.index)
    final_attrs = dict(self.attrs, type='radio', name=self.name, value=self.choice_value)
  if self.is_checked():
    final_attrs['checked'] = 'checked'
  return mark_safe(u'<input%s />' % flatatt(final_attrs))

And change the StarsRadioFieldRenderer.render method to

def render(self):
  return mark_safe(u'\n%s\n' % u'\n'.join([u'%s'
                   % force_unicode(w) for w in self]))

Now, all you have to do is make your form add our special renderer to the field and push the rating value into the right data model when the form is submitted.

In the forms.py file, add a field to your form. Don’t forget to create the rating choices tuple and add the ‘star’ class to the field.

RATING_CHOICES = ((1,1), (2,2), (3,3), (4,4), (5,5),)
forms.CharField(widget=forms.RadioSelect(renderer=StarsRadioFieldRenderer, attrs={'class':'star'}, choices=RATING_CHOICES))

In your form’s model, add this to the save method to connect the form data to the actual model’s data (if you need to):

def save(self, *args, **kwargs):
  myobject.rating.add(score=self.rating, user=self.user, ip_address=self.ip_address)
  super(CommentWithRating, self).save(*args, **kwargs)

Here’s what it looks like attached to a comments form:

django-rating and jquery star rating together

Augmented Reality out my window (from our AR app!)

After reviewing the available technology and standards, we decided to build web services that conformed to the Layar standards, a mobile augmented reality platform developed in The Netherlands. Launched in 2009, Layar has quickly become ubiquitous as a platform for augmented reality applications. To implement an augmented reality layer in Layar, one publishes the “augmentations,” the points of interest that are visible in the augmented reality application, by creating a web service that client applications can query for information about what’s around them. A web service is simply a term for a standard method of allowing two computer programs to request and communicate data in a structured way. For example, a request to this “augmentation” database might request all of the points of interest within 200 meters of a given latitude and longitude. While the Layar webservice format has some limitations, it is relatively simple to implement both server-and client-side support for it. While not strictly what is often called a “RESTful” web service, which is a lightweight style of web service that in many ways is similar to loading a web page, the service can be implemented with a simple web application that can read in POST variables. As there is no independent standard for requesting and publishing augmented reality points of interest, the Layar service is as close as we could find. It had the additional advantage that we could directly test the result in the Layar clients available for both the Android and iPhone platforms.

Layar API Architecture (courtesy of Layar)

The API documentation for Layar’s REST services are on their documentation wiki.  Overall they have done a great job, but there are two big gotchas here.  First of all, they are in heavy development and their platform doesn’t yet seem fully stable and mature — we often felt like we were developing against a moving target, as what was supported overall and from device to device kept changing.  Secondly, their documentation wiki is somewhere between a really fantastic wiki and relatively chaotic and poorly organized documentation.  The details are spread out across various pages with various comments (some of which are totally critical) thrown in the mix.  We would have loved stable, versioned API documentation that was separate from the wiki and had all of the specifications in one place.   But my overall feeling is a bit like the Churchill quote: “Democracy is the worst form of Gov­ern­ment except for all those other forms that have been tried from time to time.”  While there are problems, it’s a relatively complete API and it’s a defacto standard for augmented reality data services.  That said, I’d love to see an open source standard for augmented reality data services  – and some open source clients that support it!

Winston Churchill at age 7 (thanks wikipedia.org)

While there are many advantages to this architecture, it is important to keep in mind that it means that all imagery is being transmitted to the mobile device while the user is using it. This creates a number of issues. If the user is has no or poor connectivity, the application will not be able to load photos – and even under good circumstances there will be a noticable delay, and there are restrictions to how many photos can be sent in a short amount of time over a network. An alternate approach would be to package the asset images with the application, and install those images along with the application. But while this approach might work well for a custom built application with 100 photos, the storage requirements would make this impractical for a large collection of 100k photos.

So … data munging (transforming from one format to another) projects  are almost always kind of sticky, but we hit some particular challenges in working with data for augmented reality. As discussed earlier, we used Google Street View as a tool to identify and select the desired angle in 3D space that we wished to place each photograph. However, Google Street View and Layar specify this angle differently. Both Layar and Google Street View represent a viewer facing north with a value of 0 degrees (in Layar this is the “angle” parameter and in Google Street View it is called “yaw”). However, in Google Street View rotations go clockwise (so 90 degrees is East and -90 degrees is West) whereas in Layar rotations go counter-clockwise (so 90 degrees is West and -90 degrees is East). However, the effort required to make these transformations was worthwhile. By January 2011, the PhillyHistory.org database management team had “pinned” more than 10,850 images to their Google Street View coordinates, providing a large subset of materials with which to test the 3D space options.

We chose to use to use a spatial database (PostgreSQL database with the PostGIS spatial extensions) to store our assets. A spatial database is designed to store and reason about objects in space – for example, it is possible to ask a spatial database to find the assets that are within a specific distance from the viewer.   Check out this newsletter article from Robert about PostGIS to learn more.   It is possible to add a stored procedure to a non-spatial database to make the same query (for example, the Layar wiki documentation provides such a function) but we found that with large numbers of points, the optimizations found in a spatial database were necessary for reasonable performance. The creation of a “spatial index” allows the database to limit its searches very quickly to likely candidates found within the database, instead of needing to search through all of the assets in the database.    That said, however, the overall performance of an AR application is limited by the network transmission time far more significantly than the backend server performance — but with 90k points it sure doesn’t hurt.

The ImageMagick wizard

Some image processing also needs to occur before images can be displayed on the small screen of a mobile device. This is extremely important because we found that clients (such as Layar’s client) will silently drop images that did not fit their specification. Because the client will not include the photo for a number of distinct reasons but there is no feedback for the developer explaining why the photo was not included, the process of diagnosing missing photos can be tricky. For Layar, the file size of all images must be smaller than 75 kB, and there are specific resolution limitations (e.g. full images in Layar must be less than 640×480). Given that mobile device screens have significantly smaller resolution than 640×480, that resolution is probably much higher than necessary. Additionally, some clients (like Layar) do not support making images transparent. It is therefore necessary to set the alpha channel of the photos in a pre-processing step. For example, using the open source ImageMagick package, the following command line invocation could perform the necessary scaling and transparency conversion on an incoming image stream (on a linux box): “cat input_image.jpg | convert – PNG32:- | convert -scale 240×180 -channel Alpha -evaluate Multiply 0.9 output.png”.

There are already a number of open source platforms for publishing data services compatable with the Layar API, most notably PorPOISe (PHP), django-layar (Python) from our open data crush Sunlight Labs and LayarDotNet (C#), with PorPOISe being the most fully featured of the platforms we reviewed. However, PorPOISE lacked some crucial 3D features at the time of our review. The beta release of an online service called Hoppala Augmentation does support 3D layers, but we were unable to get the 3D service to work and found the documentation and usability to be underdeveloped. It is certainly necessary to have a full understanding of the Layar protocol to use the Hoppala service (at this point) as the API allows developers to set a range of settings without explanation or checks on invalid or conflicting settings. Given these limitations and our desire to implement our own interactive capabilities and user settings, we decided to develop our own data web services in Python, which turned out to be a great choice for us because it let us prioritize, shape, and alter the results in a variety of ways.   Our next journal (from Erik) will feature some of the ways that we needed to change the 3D placement of photos and how we went about it.

We’re not writing guidance software for rockets, so we don’t have to deal with relativity in the Einstein sense, but relative in a more day-to-day and somewhat artificial sense. Although I’m not exactly sure how Benjamin Franklin really felt about daylight savings time, it certainly makes programming a little trickier for us Philadelphians.

Here is an example for those following along in firebug:

var start = new Date('2010/01/01');
var end = new Date('2011/01/01');
var diff = (new TimeSpan(end - start)).days;

for(var i=0;i<diff;i++) {
    var today = new Date(start).add(i).days();
    var plusSeven = today.clone().add(7).days();
    var oneWeek = (new TimeSpan(plusSeven - today));

    if ((oneWeek.days != 7) || (oneWeek.hours != 0)) {
        console.debug("Time Warp: ", today);
    }
}

Say you made some bit of code that generated a date range by adding or subtracting some number of days off a date, you’d have a bug from March 8th through March 14th, and then it’d be gone until November 1st through the 7th! Not only is this sneaky, it’s hard to predict! Although this isn’t really a bug in Date.js, it requires us to change how we think about time.

The recommended Googling didn’t provide a straightforward, and rock-solid technique for identifying and adjusting for this effect in any given time zone.

So I made one:

First, a quick example:
(my apologies if this is obvious or full of mistakes)

//Lets say you wanted to setup some DateTime controls
//with a rolling date range, lets say using an offset in days
//with Date.js you'd probably do something like this:

var offsetDays = 7;
var start = new Date('2010/03/09');
var end = start.clone().add(offsetDays).days();

//so far so good, we got the intended date:
//Tue, March 16th, 2010
//but we want to update and store the rolling date range,
//so lets get that range back out:

// ... snip! ...

var diff = (new TimeSpan(end - start));
if (diff.days != offsetDays) {
    //daylight savings time caused time to 'spring ahead',
    //and our 7 days is instead 6 days 23 hours
    console.debug('user changed date');
}

We’re going to need a simple function that gets a TimeSpan that is adjusted for the daylight savings time adjustment:

// gets the difference between two dates,
// adjusting for daylight savings time.
var getDSTAdjustedDiff = function(startDate, endDate) {
    var startOffset = startDate.getUTCOffset();
    var endingOffset = endDate.getUTCOffset();

    if (startOffset > endingOffset) {
        startDate = startDate.clone().add({hours: -1 });
    }
    if (endingOffset > startOffset) {
        endDate = endDate.clone().add({hours: -1 });
    }
    return (new TimeSpan(endDate - startDate));
};

If we revisit our example from before:

var start = new Date('2010/01/01');
var end = new Date('2011/01/01');
var diff = (new TimeSpan(end - start)).days;

for(var i=0;i<diff;i++) {
    var today = new Date(start).add(i).days();
    var plusSeven = today.clone().add(7).days();
    var oneWeek = (new TimeSpan(plusSeven - today));
    var adjustedDiff = getDSTAdjustedDiff(today, plusSeven);

    if ((oneWeek.days != 7) || (oneWeek.hours != 0)) {
        console.debug("Time Warp: ", today);
    }
    if ((adjustedDiff.days != 7) || (adjustedDiff.hours != 0)) {
        console.debug("Adjustment fail!: ", today, adjustedDiff );
    }
}

If everything went well, this code block should run and run without the second block ever firing. I hope this helps someone else out there avoid this pitfall, now if I can just figure out time zones…

If you’ve been using additional frameworks, like  jQuery ( or EXT or Django or Drupal or whatever) you’ve learned to rely on that framework’s onready event, or its rough equivalent.  This event tends to be thrown before the onload event so that the frameworks don’t need to wait for every last image and tag to be loaded. In theory, this could be a very problematic place to put OpenLayers map initialization code. The dom elements that OpenLayers is looking for may or may not be there, and there’s no way to tell. In practice, most browsers, and most OpenLayers components, work fine when initialized in a framework’s onready event. One layer in particular (OpenLayers.Layer.Vector) will not play nice with one particular browser (IE8) if initialized in a framework’s onready event. The vector layer depends on the document object’s namespace attribute, and this attribute simply isn’t always available before IE’s onload event (which is called after the onready event). As you can imagine, this was one headache I wish I could have avoided.

Thankfully, there is a fairly simple fix. Instead of the usual convention of :

$.ready(function() { init_my_map(); });

Add a stand-alone script tag to the extreme end of the body content on the pages where you need a vector layer:

<script>
    init_my_map();
</script>

While we redesigned our company website, we didn’t want to wait to launch it until we had migrated all of our product websites to Concrete 5.   To combine the two systems under one domain name, we leveraged Varnish as a proxy to split requests between our LAMP stack hosting Concrete 5 and our Windows server hosting DotNetNuke.    We’ve found Varnish to be a robust and yet lightweight tool for proxying, caching, and load balancing.  You can read more about how we used Varnish to host Walkshed on Amazon Web Services in this related blog post.

Another concern when migrating CMS’s is making sure that the old URLs continue to function and issue the right redirects to users and search engines.   Andrew Jennings took this project on and whipped up a URL redirection system for us using Apache’s mod_rewrite module.

• As requests come into Varnish, we first test to see if Concrete5 can respond to the URL.
• If Concrete5 isn’t managing the content for the URL, the request bounces into our redirection system that maps old URLs to new URLs using a binary hashtable and issues 301 permanent redirects to the client.   We found that this approach is faster than simply relying on Apache .htaccess files since we have several thousand legacy URLs to support.
• If the URL request is not in the mapping file, we simply forward the request to the Windows server to be served by DotNetNuke.
• If DotNetNuke returns a 404 error, we redirect the request back to Concrete 5 to respond with a 404 error so that our redirection is invisible to the user.

A final goal of the CMS migration was to improve page load times for our visitors.   Having pages that load quickly not only improves the experience for our web visitors, but also has an impact on search engine rankings.    Concrete5 suggested we setup Alternative PHP Cache (APC) to speed up the pages it is serving, which has definitely improved performance on our new pages.

We found a great WordPress plugin (which runs our blogs) that can also leverage APC called W3 Total Cache.   It provides a host of other functionality that we’re exploring including HTML, JS, and CSS minification as well as pushing design assets to Amazon’s CDN.

I always find it amusing how something as straight forward as a new website launch has so much happening behind the scenes.   To find out more about the content that went into the new Azavea website read Rachel’s related blog post.

Tune Your Cache

Make sure you spend time planning the cache. Not only will the cache look better in the final application, but it will also load to S3 faster and cost less in the long run.

• Set the extents in the MXD or MSD before publishing to a map service. The overhead of transferring the 254 byte empty tiles caused a lot of unnecessary burden on the upload process as well as the fact that you are paying for them to be stored in the cloud. If it doesn’t need to be there, don’t build it.
• Choose the correct image format for the cache. If you are caching a base map and do not need to support transparency, make it a JPEG. If it needs to support background transparency, use PNG. ESRI’s suggestions for planning a map cache can be found here.

Get a Good Tool to Transfer the Files

I started using the free version of Cloudberry Labs S3 explorer. But I had to move over 90 Gbs worth of data to my S3 bucket. The CloudBerry S3 Explorer – Pro supported multithreading which allowed for up to 5 threads to either enumerate through the folders, copy the files or apply the ACL. It is a low cost application that more than pays for itself when moving a lot of files up to a bucket.

When transferring the files up, I was working in blocks of directories, not the whole scale level. It was quicker for me to work in 20 to 30 subdirectories than grabbing a whole scale level. It did require a little bit more management on my end, but more steady progress was made.

Accessing the Tiles

ArcGIS Server does not support cloud hosted caches at the 9.3.1 release. The ESRI Javascript API and Flex API can be extended to use caches hosted in the cloud (Flex example from Mansour Raad), so you’ll have to roll your own. For the Philly Storm Water project, we were using the Open Layers and someone has rolled one for us. There is a patch that can be used to access the cache without communicating through ArcGIS Server straight from the client-side library. The one thing to note is that the Tile Origin is pretty touchy, we had to make some adjustments to the origin values to make sure everything lined up correctly.

Summary

Now that the site is up there and we are starting to get some traffic hitting it, putting the tiles in S3 was the right decision. There is no reason for ArcGIS Server to waste any cycles moving tiles around, let it do the heavy lifting with the vector layers and queries. Hopefully the rumors are true, and the ArcGIS Server 10 release will be more aligned with cloud computing. Until then, there are still plenty of ways to take advantage of the benefits.

Our strategy is broken down into a few components:

• Improving download speed using Amazon’s Content Delivery Network, Cloudfront
• Reducing the size of the page components to improve loads times
• Measuring results with Yahoo’s YSlow toolkit

This post covers the Cloudfront CDN.

Previously, we had experimented with Amazon’s Web Services stack to host applications, but we hadn’t experimented with their Cloudfront CDN product.   Pricing for the CDN is quite similar to Amazon S3 and allows organizations to build scalable applications without the upfront cost of most CDNs.  We decided to use the CDN to host some large Javascript assets as well as our image components.

Cloudfront is quite easy to setup.   We simply created an Amazon S3 bucket called s3.azavea.com and pointed a CNAME record for s3.azavea.com to the full bucket domain — s3.azavea.com.s3.amazonaws.com.    Then, we enabled a Cloudfront distribution for the s3.azavea.com bucket using the free tool Cloudberry.   Finally, we setup a CNAME record for cdn.azavea.com to the Cloudfront distribution domain d17ib0dlm1q8qa.cloudfront.net and we were rolling.

Since the CDN is heavily cached, it was easiest to use s3.azavea.com links during development to reduce the amount of file versioning that was necessary.   Once we were settled on our assets, we switched to cdn.azavea.com links and started using the CDN.

The speed of the CDN is quite astounding.  Splitting assets across another domain name also improves the browser’s ability to request more files at once improving the user experience.  We were quite pleased with how easily we could offload assets to Cloudfront and realize gains with limited time investment.

A few notes to keep in mind when you are working with a CDN for the first time:

• Since there is no way to flush assets out of Cloudfront’s edge nodes, be sure to use file name versioning.   This was a bit alien to us, but is easy to incorporate once you think it through.   For instance, we decided not to set a far-future expiration header on our PDF assets as they are often directly linked to and we wanted to be able to update them regularly.
• Speaking of PDFs, it seems that while Cloudfront supports byte-range requests for assets, it doesn’t assert the “Accept-Ranges: bytes” HTTP header. This makes our large PDFs fully download before Adobe displays them within the browser.  Unfortunately there is no way to add this header at the moment.
• Cloudberry is great to add HTTP headers to S3 assets.   We decided that most of our assets would have a six month cache lifespan by asserting the “Cache-Control: max-age=15552000″ header.

Let’s address a few data prerequisites before diving into the actual how-to of template nesting. No matter how your comment data gets to your page, it should be in order. For example: say we have 3 comments :

• Root A – Oct 1
  • Nest 1 – Oct 15
• Root B – Oct 4

We seem to be following a chronological paradigm here and everything is in a logical order. In order to do the least processing in your javascript, your data should come to the page in this same order: Root A, Nest 1 then Root B.

Each comment should have some basic information available to the template. Root comments only need some kind of id along with the actual comment information (text, date, all that kind of thing). Nest comments need a bit more information. They should know the id of their parent (whether a root or another nest) and their nesting level or how deeply they’re nested from the top of the conversation. For our example above, a sample data set should look something like this:

{ [id: 100, text: "Root A", date: "Oct 1", nestLevel: 0, parent: 0],
  [id: 350, text: "Nest 1", date: "Oct 15", nestLevel: 1, parent: 100],
  [id: 288, text: "Root B", date: "Oct 4", nestLevel: 0, parent: 0] }

Notice that our root comments have nestLevel and parent values of 0 (zero). This makes sense: Roots can’t have parents by our definition above, and they will always have a nest level of 0. Take a look at the second entry, our nest comment. The parent value is 100 which is the same as the id value for our first root comment, Root A. In effect, we have an internal relationship key for our data.

Now that we have data in the right order and our nests know their parents and how deep they are, we can set up our Ext.XTemplate object to handle the visual component of our nesting.

var tplComments = new Ext.XTemplate(
 '<tpl for=".">',
    '<div id="comment-{id}" style="margin-left:{[values.nestLevel === 0 ? "5" : "15"]}px;
              background-color:{[values.nestLevel % 2 === 0 ? "#fff" : "#f9f9dd"]};
              border:1px #ccc solid;
              padding:5px;">',
        //comment text, date and other info here
    '</div>',
 '</tpl>',
);

Lets pick apart our template and see how it visually nests our comments for us. On the first line, we’re declaring our object, nothing new here. The next line uses a template looping mechanism (for=”.”) to create the internal tag structure for each data element in our comment set.  Now things get interesting.

The id of our eventual div tags will include the id of the comment being templated. This is done using some token replacement where you see {id}. So the html for our first comment will be in the div with the id “comment-100″.

In the style attribute of our comment div are two blocks of javascript surrounded by {[    ]} that the template will treat as javascript instead of just text. Inside those blocks, the “values” object refers to the particular comment object we’re dealing with inside our loop. The attributes on this object are the same as the property names in our data set: id, text, date, nestLevel and parent. When English-a-fied, the first block reads: If the nestLevel is 0 (meaning a root comment) then write 5 here, for any other value write 15. This will give our roots a left margin of 5px and our nests a left margin of 15. While this doesn’t quite get us to an infinitely-nested set of comments, it gets us close and provides a visual cue to the audience. We’ll get closer to a truly nested solution in the javascript utilizing this template later.

The second block of javascript in the style attribute switches the background from white to cream on alternating comments. Its English sentence might read: If the nestLevel is an even number, use a white background, otherwise use cream.

Now for some javascript to bring it all together and make our nesting work properly. After your comment data is available in the page, you can use the XTemplate‘s append function and some basic Ext DOM querying to stack your comments indefinitely.

if(comment.get("nestLevel") === 0)
{
   tplComments.append(commentLocation, comment.data);
}
else
{
    tplComments.append(Ext.get("comment-" + comment.get("parent")), comment.data);
}

This block of javascript splits our comments set into the two definitions from the beginning: roots and nests. Roots just get appended to the end of the commentLocation element on your page. Nothing fancy there. Nests, however, take a bit more thought. A nest knows the id of it’s parent comment. In our template object, each comment is contained in a div with the same id as the comment. In order to nest a comment under its proper parent, use Ext’s built-in DOM searching capabilities to locate the parent, then append the nest.

Each nest will wind up nested 15px in from the left edge of its parent; alternating comments have subtle color changes and each wrapping div is associated to a comment by id for future tinkering. And we’re done!

We use an internal wiki here at the office, using the Dokuwiki framework.  I personally look in the wiki for something at least 5 times a day.  Usually, this involves:

1. Opening a new browser tab
2. Go to our Intranet page
3. Click on the wiki link
4. Search the wiki (let’s say for “configuration”)
5. Look through the list of search hits
6. Maybe repeat step 4

As I was looking at the Ubiquity tutorials, it became apparent that with the right Ubiquity command, the whole workflow could be simplified to:

1. Control-Space
2. Type “dw configuration”
3. Review search hits
4. Maybe repeat step 2

Ok, so not a profound difference in the number of steps, but I can do all of these steps without letting my hands leave the keyboard.  Some Firefox purists out there may argue that I could do the same with the first set of steps, but there are fewer Alts and Controls in the second set — anyway, this is just an experiment, so lighten up, ok?

I won’t paste the whole code here, but I wanted to point out a few neat tricks that are part of the command:

1. XML RPC
   A transport protocol for web services, XML Remote Procedure Calls let you invoke a web service with some arguments, and get a result back.  Dokuwiki has an XML-RPC interface, where one can perform basic searching and editing functionality. Note: this interface is disabled on the main Dokuwiki site, and is something a system administrator needs to manually enable on an installation of Dokuwiki.

   In this chunk of code, I set up the xmlrpc url, and create an xml blob that will contain the remote command that I want to execute (wiki.getAllPages).  This will return a list of all the pages in the Dokuwiki instance.

   ...
   // the command preview
   preview: function(pblock, args) {
     if ( args.object.text != '' ) {
       // set the url for the xmlrpc command
       var url = 'http://www.dokuwiki.org/lib/exe/xmlrpc.php';
       // set up the data to POST
       var data = '<?xml version="1.0"?><methodCall><methodName>' +
         'wiki.getAllPages</methodName></methodCall>';
       // provide some feedback to the user
       pblock.innerHTML = 'Searching for page matches for ' +
         args.object.text + '...';
       // hold on to a reference to the currently-scoped object
       var me = this;
       jQuery.ajax( {
         async: true,
         contentType: 'text/xml',
         data: data,
         dataType: 'xml',
         url: url,
         type: 'POST',
         success: function(x) {
           ...
         }
       });
     }
   },
   ...

2. jQuery
   Ubiquity has a version of jQuery built into it, so you have the full power of jQuery at your disposal.  I used this to do some ajax calls to the XML RPC interface of Dokuwiki.

   You can see this in action in the above code excerpt.

3. XPath
   No mention of XML would be complete without some way to navigate your XML document using XPath.  To get the search results and other information, I used some of Firefox’s built in objects to extract the relevant search hits.

   The object returned from the success method above is an xml document (since I specified the dataType in the jQuery.ajax call). With an xml document, one can perform XPath tricks with the evaluate() method.  This code selects the “name” elements where the page id is the same as the command argument (in this case “configuration”).

   // get the pages with IDs that match the input string
   var pages = x.evaluate( '//struct/member/name[text()="id"]' +
     '/../value/string[contains(text(),"' + args.object.text + '")]',
     x, null, XPathResult.ANY_TYPE, null );
   var nuggets = [];
   // return the top 10 results, if multiple matches are found
   var item = pages.iterateNext();
   while ( item && nuggets.length < 10 ) {
     nuggets.push( item.textContent );
     item = pages.iterateNext();
   }
   // if there are many hits, show a list of matches
   if ( nuggets.length > 1 ) {
     CmdUtils.previewList( pblock, nuggets, function(){
       me.gotoUrl(nuggets[arguments[0]]);
     });
   }
   // if there is one hit, generate an html preview in the
   // ubiquity preview frame
   else {
     pblock.innerHTML = 'Found match, generating preview...';
     ...

That was my first stab at it, and it has a couple quirks, but I find myself using it to find information really quickly.  I think there is a great amount of potential here, especially with the collection of commands that are already built in to Ubiquity.

For the fearless, the full command is available for download here: dokuwiki search.