In the previous article Josh described how we built data services to support augmented reality applications. This article will be a more detailed discussion of our experiences placing 2D photos in 3D space using position and angle information with Layar. We do this in the PhillyHistory Augmented Reality application to let users browse photographs of historic Philadelphia. I will also provide some of the Python code we ended up needing to get things working.
When dealing with 3D objects, Layar supports two kinds of rotation: relative and absolute. Relative rotation is pretty straightforward: the image will always be rotated relative the viewer’s current viewing angle. You will always see the photo or model from the same angle no matter where you stand. This works pretty reliably but obviously breaks the illusion that the billboard is positioned in the world (unless you imagine a billboard rotating to face you no matter where you stand).
Absolute rotation is more interesting (and trickier to get right). Basically, the object or photo will face a certain direction in real space: you might need to move where you’re standing in order to see it clearly. This means you need to figure out what direction (e.g. North) an object is facing in addition to the latitude and longitude of its coordinates (you might also worry about altitude although we ended up not doing this).
We used absolute rotation when displaying 500 select photographs which had data for both the position (stored as latitude/longitude) and angle (stored as a Google Street View angle) and we used relative rotation when dealing with PhillyHistory’s full archive of 87,000 or so photographs (most of which obviously had no angle data).
Initial problems
We knew we wanted to use absolute rotation, but it took us awhile to get it working well enough to include in our layer. Some of the obstacles we encountered: having to remember high school trigonometry, a lack of high-level documentation on how Layar does 2D/3D rendering, and somewhat flaky or confusing error behavior.
One problem is that there doesn’t seem to be a standard way to represent angles out there. Your high school math class has 0 pointing East, and then the values proceeding counter-clockwise around the unit circle (with one rotation being 2π). Layar uses a similar counterclockwise scheme buts starts with 0 at North instead of East, and uses degrees (with one rotation being 360 degrees). Google Street View also starts with 0 at North and uses degrees, but goes clockwise instead of counterclockwise. Confusion!
Another confusing detail is what the rotation angle means. I had assumed that if a photograph’s angle is “South” then that means that the billboard faces South, and the viewer should look North to see it (and be on the South side of the photograph). In fact, Layar takes the opposite view. If a photograph’s angle is “South” it means that the viewer should look South to see the photograph (and be on the North side). We didn’t find any documentation about this so we had to learn it through trial and error.
Fig. 2: viewing at 60° angle
Once we got that worked out, we still noticed that a lot of our points weren’t rendering in the Camera view. They were showing up in the map view and list view but we weren’t seeing images, or icons, or anything. After a bunch of research we figured out that this had to do with their orientation–we were facing the wrong side of the image. If you imagine a billboard, we were seeing the scaffolding and back of it, not the advertisement. It took us awhile to realize that these 2D billboards were “invisible” when seen from behind.
Fig. 5: viewing from behind and to the side
This behavior didn’t work very well for us. We want users to know when they are near a location with historic photographs available, even if the user is on the “wrong side” of the location. And we found it frustrating to have photos that failed to show up, or photos whose viewing angle was so sharp as to completely obscure the image. On the other hand, we liked the 3D effect of seeing photos angled when appropriate, so we didn’t want to just give up on absolute rotations.
Ultimately, we decided to “cheat” and transform the photographs when necessary. Given the general lack of GPS accuracy and the fact that our highest priority is making the photographs available it seemed like a good compromise.
Transformation in Python
What follows is a relatively in-depth description of the kind of processing we ended up needing in order to ensure images were visible when viewed with absolute rotation. Code very similar to this is included in the Layar API endpoint we built (using Python and PostgreSQL).
The first thing we have to do is compute the viewing angle to a point of interest. We can accomplish this by dusting off a little trigonometry. Imagine that viewer (vx, vy) and the point of interest (px, py) form a right triangle. In this case we want to compute the angle at the viewer point, which we can do with atan2 given the lengths of the opposite and adjacent sides (which end up being py – vy and px – vy, respectively). The resulting angle will be 0 when facing East (when px and py are the same), and proceed counter-clockwise (so π/2 is North, π/-π is South and -π/2 is West). We convert this into the form that Layar uses (where 0 is North, -90 is East, 180/-180 is South, and 90 is West).
Fig. 6: calculating the viewing angle
Here’s some Python code that accomplishes this. It’s worth noting that it transforms the angle from the trigonometric form (pi radians counterclockwise from East) to the Layar form (degrees counterclockwise from North).
import math def get_angle(vx, vy, px, py): # find the angle in pi radians (-pi to pi) theta = math.atan2(py - vy, px - vx) # convert from pi radians to degrees (-180 to 180). degrees = (theta * 180.0) / math.pi # return the angle relative to the positive-Y axis return degrees - 90
We also use the standard Euclidian distance function to calculate how close points are to each other.
def get_distance(x1, y1, x2, y2): dx = x2 - x1 dy = y2 - y1 return math.sqrt(dx ** 2 + dy ** 2)
When a photograph is visible at a particular angle, we need to determine how much difference there is between the user’s viewing angle and the ideal angle at which the photograph should be viewed. In this case, a difference of 0 would mean the user is viewing the photograph at the exact angle at which the photograph was taken, 180 would mean that the user is behind the photograph, and 90 would mean that the user is at a right angle to the photograph.
Since there is a point (180/-180) where angles wrap around, it’s important to make sure to handle this correctly. For instance, -160 and 179 are only 21 degrees apart. We can use modular arithmetic to normalize angles to 0-360. Here is an implementation:
def angle_diff(angle1, angle2): # calculate the difference between angle1 and angle2. # this value will range from 0-360. diff = abs(angle1 % 360 - angle2 % 360) # if the difference between angle1 and angle2 is more # than 180 degrees return the different between angle2 # and angle1 (which will be less than 180 degrees). if diff > 180: return 360 - diff else: return diff
When an angle is too close to 90 degrees the image won’t be visible; in these cases we can soften the angle so the user can see the image a bit better. This function will nudge the start angle closer to the goal angle by a given number of degrees (amount):
def nudge_angle(start, goal, amount): # calculate the difference between start and goal. # this value will range from 0-360. diff = abs(start % 360 - goal % 360) # don't nudge further than we need to reach the goal. if diff < amount: amount = diff # figure out whether we need to subtract or add diff. # if start is greater than end then we subtract diff, # and otherwise we will be adding diff. subtract = start % 360 > goal % 360 # if diff is greater than 180 we need to flip our # decision (going the other direction means the # difference will be less than 180). if diff > 180 subtract = not subtract # add or subtract the amount and return the new angle. if subtract: return start - amount else: return start + amount
We can put this all together to implement our strategy for dealing with oblique angles and image flipping. The code could be made more terse but it’s easy to get the math wrong so we try to do things in a well-commented procedural way.
# points are represented as (x, y) tuples in web mercator. # angles are given in degrees counterclockwise from North. def calc_angle(self, viewer_pt, point_pt, img_angle): vx, vy = viewer_pt px, py = point_pt # get the angle the viewer faces when seeing the point. angle = get_angle(vx, vy, px, py) # get the difference between the previous viewing angle # and the direction the photograph should be seen from. diff = angle_diff(angle, img_angle) # if the view is behind the photo, flip it. if abs(diff) = 90 - wiggle: angle = nudge_angle(angle, img_angle, wiggle) # return two things: the viewing angle, # and if the photo was flipped or not. return angle, flipped
Hope this helps! It’s the sort of thing that would have saved us a lot of time if we’d had it!
Layar documentation can be found here.
