We set out to support several base map options as well as any combination of options, including:

• Bing Maps (satellite, roads and hybrid)
• GoogleMaps (satellite, roads and hybrid)
• ArcGIS Online (any of several maps)
• OpenStreetMap

Since DistrictBuilder needs to be flexible enough to meet the needs of users and administrators in a variety of situations, we decided on a two step approach to basemap configuration. First, the administrator specifies, in the configuration file, which of the combinations of map providers and map types are allowed to be selected. Then DistrictBuilder presents all of the configured options to the user, where they can be toggled among at any time while a plan is being viewed or edited.

Here’s an example of an instance configured with an OpenStreetMap road layer, a Bing hybrid layer, and a Google satellite layer:

Here’s another example with only road layers — one for each of the three configured providers:

DistrictBuilder currently allows the configuration of basemaps using permutations of each of the three vendors and three map types described above. Adding more options is a relatively easy task, however. With the launch of Fix Philly Districts, we wanted the basemap colors to be slightly more muted than the above options, and ended up adding support for the ArcGIS Online World Topographic Map. We also experimented with the Google Maps V3 custom styling API, which looked great, but introduced performance problems when panning and zooming (animations).

There were, of course, some hoops that needed to be jumped through in order to get all of these basemaps behaving correctly on the same map, which will be discussed below. I’ve extracted the logic required to do so into a small demo that can be viewed/downloaded here. The demo has also been embedded into this post, and can be interacted with without going anywhere:

Zoom Levels

Many of the challenges that needed to be overcome to get this working correctly were brought about because we needed to restrict the zoom levels to the area at hand. We wanted to eliminate superfluous zoom levels to ensure the user was always operating within the appropriate boundaries (note: it is not done in this demo, but in DistrictBuilder we also restrict the extent with the ‘restrictedExtent’ map parameter, so users can’t even pan outside of the area).

One difficulty with setting zoom levels on the different layers is that the layers don’t use zoom parameters consistently. In Bing (the VirtualEarth layer), minZoomLevel and maxZoomLevel are needed. In Google, minZoomLevel is needed, but it requires numZoomLevels instead of maxZoomLevel. And in OpenStreetMap (the OSM layer), well…no combination of those seem to work — we needed to slightly modify the XYZ layer (OSM’s base class) to allow maxResolution to be changed based on the minZoomLevel. To see how this is done, view the demo source. With that change in place, the list of required layer parameters is as follows:

• Bing – minZoomLevel, maxZoomLevel, projection, sphericalMercator, maxExtent
• Google – numZoomLevel, minZoomLevel, projection, sphericalMercator, maxExtent
• OpenStreetMap - numZoomLevel, minZoomLevel, projection

Coordinate Systems

We also faced some problems related to coordinate systems. DistrictBuilder uses GeoServer and GeoWebCache to serve up WMS layers. The coordinate system of our data is one version of the the ever-changing “Popular Visualization CRS / Mercator” projection. We needed to match up the OpenLayers projection to the one used on our data, or else we were seeing slight offsets on our overlays. Unfortunately, the ‘projection’ layer parameter isn’t always used within the layers correctly. For example, any layer using the SphericalMercator class gets its projection automatically hardcoded to 900913. We needed to make a slight modification to the SphericalMercator class to allow the ‘projection’ parameter to carry through. This can be seen by viewing the demo source.

Bonus: Math Time!

One interesting part about implementing zoom restriction was that we needed it to work in any instance of DistrictBuilder — from large states to small towns, which may have vastly different extents. Instead of having an administrator figure out the proper minimum zoom level, we calculate it automatically based on the extent, which requires a little bit of basic algebra.

For Philadelphia, the extent of our area is:

[-8397913.926216, 4842467.609439, -8329120.600772, 4895973.529229]

In DistrictBuilder, we calculate this dynamically on the server side (using Django) by filtering all of the geounits in the database and calling the ‘extent’ function on the query set. For the demo, this is hardcoded. Here’s how to transform this extent into a Spherical Mercator minZoomLevel:

• Find the width of the area in meters.

var studyWidthMeters = extent[2] - extent[0];

• Find the width of the map in pixels. In the demo, this is hardcoded, because we are setting the div size of the map. In DistrictBuilder, the map takes up the whole screen, and this value is calculated on the fly based on the size of the div in which the map occupies.

var mapWidthPixels = 450;

• Find the map resolution, or meters per pixel.

var resolution = studyWidthMeters / mapWidthPixels;

• Find the maximum map resolution. In Spherical Mercator, the maximum resolution is one 256×256 tile taking up the entire circumference Earth. So dividing the circumference of the earth (~40,000km) by 256 gives us the maximum meters per pixel, which is a constant.

var maxResolution = 156543.033928;

• Spherical Mercator zoom levels work like a pyramid. Each zoom breaks the current tile up into a 2×2 group of 256×256 tiles, essentially halving the resolution each time. Therefore, finding the resolution at a given zoom level looks like this:

maxresolution / 2^zoom = resolution

• We know the resolution and max resolution already, and need to find the zoom:

zoom = log(maxresolution/resolution)/log(2)

• Or in javascript:

var minZoom = Math.log(maxResolution / resolution) / Math.LN2;

Icky Syntax

The most noticeable problem when using Numeric (at least in 2.8) is the syntax. As Kevin Wright alluded to in a comment on the last post, you can include the Numeric type class in two ways: via a context bound on your type parameter (A) or as an implicit argument.

In my earlier examples I used the implicit argument because I thought it was easier to follow (especially for those who are new to implicits) and it bound the instance of the type class to a named variable (n). But you can also use the type bound syntax, which makes the function declaration a bit cleaner looking.

// this makes it obvious that n is an instance of Numeric[A]
// and gives us a reference to use.
def square[A](a:A)(implicit n:Numeric[A]) = {
  n.times(a, a)
}
 
// this definition is a bit cleaner looking.
// implicitly[] searches for an implicit Numeric[A] instance.
// unfortunately the body of the function is less clean.
def square[A:Numeric](a:A) = {
  implicitly[Numeric[A]].times(a, a)
}

Either way, you’re still writing things like n.times(a, a) rather than a * a. Fortunately, Scala 2.9 introduced some nice new implicit parameters that allow us to use infix operators:

import Numeric.Implicits._
def square[A:Numeric](a:A) = a * a

Confusing Conversions

The implicits providing infix operators are definitely a big improvement! But while they are nice for simple cases like the previous one they can end up confusing the issue a little bit:

def twice[A:Numeric](a:A) = a * 2
//error: could not find implicit value for parameter num:
// scala.math.Numeric[Any]

What’s going on here? The problem is that * is mapped to Numeric.times(a:A, b:A) but the second argument we provided isn’t an A, it’s an Int. In Java (and Scala) we are used to being able to mix numeric types like double and int without worrying: the result will be promoted to double (the type with the widest range). Unfortunately, as written Numeric isn’t able to do these kinds of things and instead types must be converted explicitly. Numeric does provide two special methods, zero and one which allow you to get those values for all its types.

// use Numeric.one to get the correct type
def once[A:Numeric](a:A):A = a * implicitly[Numeric[A]].one
 
// convert 2 to be the correct type
def twice[A:Numeric](a:A):A = a * implicitly[Numeric[A]].fromInt(2)
 
// convert a to be an Int
def thrice[A:Numeric](a:A):Int = a.toInt * 3

There’s also no good way to combine two different generic Numeric types, other than converting to a common concrete type (e.g. Double). You could imagine Numeric providing something like fromType[B] (as well as things like fromDouble) but currently it doesn’t.

def combine[A:Numeric, B:Numeric](a:A, b:B) = a + b
//error: could not find implicit value for parameter num:
// scala.math.Numeric[Any]
 
// this works
def combine[A:Numeric, B:Numeric](a:A, b:B) = a.toDouble + b.toDouble
 
// this would be nice (for some definition of nice)
// but it doesn't work
def combine[A:Numeric, B:Numeric](a:A, b:B):A = {
  a + implicitly[Numeric[A]].fromType[B](b)
}

Performance

The most disappointing thing (in my opinion) about using scala.math.Numeric is its performance. In some cases it performs adequately, but in most cases it is very slow: actual algorithms written with Numeric tend to be 6-30 times slower than their direct counterparts (depending on what exactly they do). Some rough observations are that the comparison operators (e.g. >) seem to be slower than the math operators (e.g. *), and performance does depend on which type you end up using: Numeric[Double] does relatively well (compared to the same code written for Double). Numeric[Int] does very badly compared to code written for Int.

Specialization to the Rescue!

Fortunately, Scala itself contains the tool to fix Numeric’s performance problems: specialization. Specialization was first described in the paper Compiling Generics Through User-Directed Type Specialization by Iulian Dragos and Martin Odersky and was introduced in Scala 2.8 via the @specialized annotation. Other articles have been written explaining specialization, but I will give another brief one here.

When writing generic functions in Scala (as well as Java), the compiler generates one implementation that supports all possible argument types (e.g. java.lang.Object). This fine for Java [1] since all the valid types you could use with your generic function are guaranteed to be subclasses of java.lang.Object. If you remember my first post, you’ll recall how we had to create boxed instances of java.lang.Integer to hold int values when using generics.

Unfortunately for Scala it means that using generic functions with value-types (e.g.Int) is much slower than functions implemented directly in terms of the value-type itself, since there is a single implementation which requires its arguments to be objects (e.g. boxed).

The @specialized annotation allows the Scala compiler to generate separate implementations of the generic function: one for all the reference types (inheriting fromAnyRef) and one for each of the value types we choose to specialize over (e.g. Char, Double, Int). So if we have a generic function foo[A](a:A): A the compiler will generate other functions on primitive types, for instance fooInt(a:Int): Int, fooDouble(a:Double): Double, etc. And when we write foo(33) the compiler will translate that code into fooInt(33).

Since Scala hides boxing/unboxing, specialization doesn’t make the code look nicer. But it makes a huge difference in performance: Dragos and Odersky report a 22-40x speed up in micro-benchmarks!

Specializing Numeric

Soon after specialization was added to Scala 2.8, people began discussing specializing Numeric. Searching Google I find names like Jason Zaugg, Iulian Dragos, and others talking about it. Andreas Flierl emailed the scala-language mailing list on January 3, 2011 with some early tests which got me interested in working on this. So I don’t want to claim any credit for the amazing work done on implementing @specialized or even the idea of specializing Numeric.

Starting with Andreas’ code I began working on building a specialized Numeric, with the goal of making all its operations as fast as direct implementations. There were a bunch of ways that the code needed to be restructured (which I may discuss in a later article) and I learned a ton about implicits, typeclasses, and profiling Scala code. The code is available at: https://github.com/azavea/numeric along with the tests I’m running. The results follow, but the short story is that with the exception of implicit infix operators all com.azavea.math.Numeric code runs just as fast as the direct implementations.

The Numbers

Discussion

Profiling this was a little bit annoying–I had to write 4 direct implementations (one for each of the types I was interested in) plus two generic implementations (one using the old scala.math.Numeric and one using the new com.azavea.math.Numeric). I used scala.testing.Benchmark although Yuvi Masory recently suggested using https://github.com/sirthias/scala-benchmarking-template (which I still need to go check out).

Some things to note:

1. Old Numeric does much better on floating-point numbers than on integral ones. Why?
2. For some reason old Numeric is able to beat the direct implementation of the array-allocator test.
3. Since scala.util.Sorting[T] uses scala.math.Ordering[T] (which isn’t specialized) it’s not possible to make Numeric any faster in this test.
4. scala.util.Sorting[Long] is over 4x slower than Int, Float and Double in the direct implementation.
5. Infix operators are still slow enough that I still have mixed feelings endorsing them.

One additional gotcha which I haven’t discussed at length is what writing your own specialized numeric code (currently) looks like. Unfortunately, there’s a lot of boiler-plate. While users of your library will just say doSomethingComplicated(Array(0, 1, 2), 18, 99) you will be stuck writing:

def doSomethingComplicated[@specialized A:Numeric:Manifest]
(ns:Array[A], x:A, y:A) = {
  // implementation here
}

And that’s only one type parameter; God forbid you want A and B to both be Numeric!

I don’t know of any easy way around this. You have to say @specialized if you want your code to be specialized. If you fail to do this anywhere in the call chain you will use generic implementations from that point down, losing any benefit to specialization. You need to specify Numeric because that’s the point, right? And when writing code that needs to use the type parameter A to allocate arrays (among other things) you have to provide a Manifest. This gets ugly really fast.

Given my ignorance of the Scala compiler, I can imagine various magical things that might fix this. For instance, I could imagine some kind of synthetic type bound that combines others, e.g. FastNumeric = @specialized Numeric:Manifest or something. Is this possible (or even a good idea)? I don’t know.

Changes

This section is a bit rushed–I may expand it into another post later, but I just wanted to get the information out there.

One of the biggest changes I made to Numeric was tearing out scala.math.Ordering–I didn’t want to try to specialize “everything at once” but scala.math.Numeric inherits all of its (incredibly slow) comparison methods from Ordering. Specializing Ordering would solve this problem.

I also created a specialized typeclass with the humorous name “Convertable” which applies to all the value types which represent numbers (Byte, Short, Int, Long, Float and Double). The great thing about this is that you can avoid doing boxing/unboxing during conversion between types. It also allows you to implement the fromType[T] method I discussed earlier.

I left out Short and Byte after Paul Phillips pointed out that neither classes operations behave as the direct implementations do. In Scala (as in Java) Byte + Byte -> Int, whereas in Numeric A + A -> A (so Byte + Byte -> Byte). This causes all kinds of bugs. Since Short and Byte (and Char) are converted to Int when you actually do things with them, I decided it was unlikely users wanted to be able to use them in generic numeric functions.

A (potentially) controversial change I made was not including scala.math.Integral and scala.math.Fractional, and unifying support for the division and mod operators. Coming from dynamically-typed languages, I expected to be able to do the following:

// this does not work with the built-in Numeric
import Numeric.Implicits._
def scale[A:Numeric](n:A, num:A, denom:A) = (n * num) / denom
 
// this *does* work
import scala.math.Integral.Implicits._
def scale[A:Integral](n:A, num:A, denom:A) = (n * num) / denom

I understand that for someone who’s excited about creating user-defined numeric types this split isn’t a big deal. But from my perspective I want a generic numeric type that abstracts across all the useful value-types in Scala. I’m not opposed to having Integral and Fractional, but I would rather that Numeric not be defined in terms of them.

Finally I did some restructuring to avoid using inner traits and classes, since I had heard that those don’t specialize well.

Conclusion

While I still think there’s some room to make Numeric better, at this point it’s possible to write fast generic Numeric code in Scala. Stepping back, I think it’s incredibly exciting that I can avoid code duplication *and* maintain the speed that I’m used to from direct implementations.

I’m not sure what the best path is to add this capability to scala.math.Numeric. I think the trait lacks some important features (more type conversions, division, etc) and some of the restructuring is currently necessary to get these numbers. Since binary compatibility is important moving forward for Scala and the Typesafe team it’s not entirely clear what the best plan is. Hopefully some discussion and work at Scalathon can clarify things a bit.

Please respond with any questions or comments. Do let me know if you find problems in the profiling, or try it yourself and get dramatically different numbers. I spent some time trying not to fall into obvious micro-benchmark pitfalls, but there’s still a lot I don’t know about the JVM. Thanks for reading!

Fast forward a few years and bring OpenLayers into the mix. We rewrote the asset editing portion of the software to allow data managers to move asset coordinates using an OpenLayers map. These coordinates were saved with still considerably more precision than we needed, but remember, we’re storing whatever precision we get. So far so good.

Now back to the present and we’re working on a comparison tool for our data managers. Suddenly values in the database are not matching the values coming out of our OpenLayers map. Almost, but not quite. In fact, only the last degrees of precision are different. After a bit of digging, we discovered that OpenLayers was returning numbers with between 1 and 3 fewer decimal places than our stored coordinates. Remember that we’re talking about distance differences smaller than a crack in the sidewalk here, but programing languages don’t know anything about “close enough”. Either two numbers are the same or they aren’t and -39.6827663878 is not the same as -39.682766387 no matter how small the physical difference is. So we started digging for the reason.

OpenLayers has a value tucked away in its utility files that sets the default precision of a floating point number to 14 characters. This limit was added when a user noticed that the edges of certain coordinate systems were not behaving correctly due to some floating-point math precision errors. While the OpenLayers community recognizes that most systems allow floats to have 16 digits,  “14 significant digits are sufficient to represent sub-millimeter accuracy in any coordinate system that anyone is likely to use with OpenLayers“. So OpenLayers’ answer to the accuracy question is to save everything that will fit in a standard float, with a few decimal places pared off just in case.

So the next question is: “So what?” The difference between 14 and 16 decimal places in a meter-based coordinate system is microscopic, and in a degrees-based one it’s not much bigger. So far as storing a saved coordinate in Sajara, we didn’t really care if we had 16 digits or 14 digits; the result wouldn’t look any different to our audience. However, since our initial coordinates had 16 digits and OpenLayers only preserved 14 of them, any programmatic comparison fails! No one likes to deal with false positives, but a 100% false positive rate was unacceptable.

We had a few choices here. First we could reset the default precision value in OpenLayers to zero, which would tell the library to never truncate anything. That’s a fairly simple change but we weren’t sure it wouldn’t have unforeseen data effects. Also, there’s a somewhat vague warning about problems with the Web Mercator projection when this value is zero, which is one of the projections Sajara can use. So that option was out.

Second, we could have told SQLServer to alter the precision of coordinate values to 14, which is a fairly major change. This option was ruled out because of a difference in the definition of “precision” between SQLServer and OpenLayers. I mentioned earlier that SQLServer will store a maximum of 16 numeric digits plus a decimal and a negative sign, so a total of 18 characters. OpenLayers, however, considers the default precision of 14 to mean 14 characters instead of 14 numeric digits. So if a number has a decimal and a negative sign, we’re down to 12 numeric digits.  This little difference reintroduces the possibility of false positives, so it isn’t really a change for the better.

The solution we finally decided to use was to change the OpenLayers default precision value to 18. Why 18? That’s the maximum amount of characters that SQLServer will store for a float, so OpenLayers will always be able to deal with any stored coordinates without having to truncate. Now, if we compare our stored coordinates with OpenLayer coordinates, we only get a change notice when an asset has actually been moved. Which is exactly what we wanted.

Here are some technical details for those interested:

The full variable name is OpenLayers.Util.DEFAULT_PRECISION and can be found in the Util.js file. There are a few good comments preceding the variable in the code, but more background can be found in the OpenLayers ticket #1951. SQLServer information can be found in mdsn. Note that if you wind up changing the OpenLayers precision value, you should do it as soon as possible after loading the library, so you don’t have the possibility of code using different precision values.

In the midst of implementing Google Maps in our up-coming Sajara sample application, I realized that the International Date Line was causing problems. Large problems. Whenever the IDL was in play, I never retrieved the correct results from our database. One large problem was how we defined our bounding box inside Sajara itself. Once that was fixed, however, I realized that something was still wrong. The Nearest-to-Farthest sort was not behaving quite right: it was really doing a Nearest-on-this-side-of-the-IDL-first sort! The IDL was acting like a wall for our distance sort.

Example of linear distance with IDL in play. X = center of map

Here’s what we used before (in sql-speak):

RETURN sqrt(power(@FromX - @ToX,2) + power(@FromY - @ToY,2))

Your standard square-root of difference-squared plus difference-squared. This formula is fast and worked perfectly well so long as I operated under these two criteria:

1. a FLAT projection
2. POSITIVE numbers only

Once either of those things aren’t true anymore, I can’t use the basic distance formula. In my case, both things stopped being true at the same time! The solution? Something fairly math-magical to my mind, but accepted as THE formula to get at the distance between any two global coordinates: the Haversine Formula. Delving into descriptions of the formula brings up concepts such as spherical triangles, the planet’s elipticity, great circles and other things very important to navigation. It’s accurate down to about a meter, which is plenty for anything I’ll be doing with it.

Haversine Formula in Javascript:

var R = 6371; // radius of the earth
var dLat = (lat2-lat1).toRad();
var dLon = (lon2-lon1).toRad();
var a = Math.sin(dLat/2) * Math.sin(dLat/2) +
        Math.cos(lat1.toRad()) * Math.cos(lat2.toRad()) *
        Math.sin(dLon/2) * Math.sin(dLon/2);
var c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
var distance = R * c;

So here is the resulting SQL function that gives me the right distance between any two global coordinates.. basically a direct translation of the above javascript into T-SQL:

CREATE FUNCTION [dbo].[distance] (@FromX float, @FromY float, @ToX float, @ToY float)
RETURNS float AS BEGIN
DECLARE @R AS FLOAT;
SET @R = 6371;
DECLARE @DLAT AS FLOAT;
DECLARE @DLON AS FLOAT;
DECLARE @A AS FLOAT;
DECLARE @C AS FLOAT;
DECLARE @D AS FLOAT;
SET @DLAT = RADIANS(@ToY - @FromY);
SET @DLON = RADIANS(@ToX - @FromX);
SET @A =
SIN(@DLAT/2) * SIN(@DLAT/2) +
COS(RADIANS(@FromY)) * COS(RADIANS(@ToY)) * SIN(@DLON/2) * SIN(@DLON/2);
SET @C = 2 * ATN2(SQRT(@A), SQRT(1-@A));
SET @D = @R * @C;
RETURN @D
END