RobotPanoramas
From IPRE Wiki
Learn how to use your IPRE robot, myro, and possibly a tool called Hugin to create panoramas.
Contents |
Collecting Images
First, we need to grab some pictures from the Fluke's camera. We can rotate the scribbler approximately 360 degrees and grab our set of images:
from myro import *
import sys
init()
manualCamera() # this turns off auto-gain, auto-exposure, auto-*
for i in range(60):
p = takePicture()
name = "%03d.jpg" % i
savePicture(p, name)
turnLeft(.6, .03)
Here you see the last 4 pictures:
As a Movie
We could, of course, always put those pictures into a list and create an animated gif, like so:
from myro import * movie = [] for i in range(60, 0, -1): this = "%.03d.jpg" % i p1= makePicture(this) movie.append(p1) savePicture(movie, 'movie.gif')
But we want to create one big picture - a panorama!
Hugin Panorama
Let's use an off-the-shelf tool to turn these images into a panorama. We'll use Hugin and supporting tools.
I had trouble using the SIFT features to grab "control points" via the GUI - it didn't find enough features in the images to stitch them all together.
Instead, I first used the command line tool align_image_stack that finds plenty of matching features:
./align_image_stack -p out.pto *.jpg
This creates a Hugin project file called out.pto, let's open that in the Hugin application:
Hugin out.pto (on linux) open out.pto (on mac)
- Then click the "2. Align.." images button.
- Then click the "3. Create panorama..." button.
- Viola!
There are a variety of options in Hugin to play with (e.g. cylindrical, vs. rectilinear projections). You can also manually place or fix image correspondances, etc. As you can see it isn't perfect. There is a giant hole in my chair!
Myro Panorama
Although we leave it as future work to implement SIFT feature detection and the homography optimization in myro ;-) (some of the code (like autopano-sift) is actually written in C# so maybe in Myro 3.0 ...), we do present here a somewhat simpler way to do a panorama in myro.
The following code assumes that the robot was spinning in place to the left when it was collecting images (using the above #Collecting_Images code, for example).
The myro panorama program looks at images in pairs, and searches for the best alignment in the horizontal direction between the two images. It assumes no movement in the vertical direction, this makes the search easier, but if there are bumps in your carpet could result in weird artifacts. You can think of it as sliding the 2nd image across the 1st, until they "click". We slide, or shift, the second image, by one pixel and then compare each overlapping pixel. We try to find the offset with minimum error. We define the error to be the difference in intensity between aligned pixels. We also normalize the sum of errors by the number of pixels in the intersection. That way bigger overlaps aren't penalized for larger errors - since there are more pixels to contribute to the error. The actual program below will hopefully make this clearer.
The only clever thing done is a little bit of divide and conquer. Instead of exhaustively searching each possible alignment, we do this in a hierarchical fashion. We start at a coarse level, and hone in on more fine grained alignments. This assumes the images are relatively smooth, which isn't always the case, but speeds things up a bit.
Also, there isn't any post-processing, like smoothing, which would make the final product look a lot better.
So here is the code. Of course, the standard:
from myro import *
First, let's write a function that converts each pixel to black and white. We'll be comparing each pixel based on intensity rather than RGB so we need to find the intensity of each pixel.
def getIntensity(pixel): """ Returns the intensity of pixel """ return (getRed(pixel) + getGreen(pixel) + getBlue(pixel)) / 3
Next, we'll write a function computerError() that takes 2 images, and an offset
and return the error between the two pictures aligned at the specified offset
(relative to p1). We also provide a resolution parameter that tells the
function if we should look at each pixel or skip some.
def computeError(p1, p2, hshift, resolution):
"""
Computes the sum squared error between the differences of the
images p1 and a shifted p2. 'hshift' determines the shift in
pixels. 'resolution' is the step size for computing the error in
both the horizontal and vertical directions. Returns a tuple: the
sum squared error, the number of pixels in the
intersection, and the total error divided by the number of pixels.
"""
error = 0.0
pixs = 0
for i in range(0, getWidth(p1) - hshift, resolution):
for j in range(0, getHeight(p1), resolution):
px1 = getPixel(p1, i+ hshift, j)
px2 = getPixel(p2, i, j)
error += (getIntensity(px1) - getIntensity(px2))**2
pixs += 1
return error, pixs, error/pixs
findMinError() will, given 2 pictures, try to find the best offset to register the two images. It does it in a hierarchical fashion to speed things up a bit, but might result in sub-optimal alignments.
def findMinError(p1, p2):
"""
Uses a divide and conquer technique (ala image pyramids) to find
the minimum horizontal shift of p2 to line up with p1. Returns
a tuple: offset with minimum error, the normalized error
"""
minx= 0
maxx= 256
for res in [16, 8, 4, 2, 1]:
minerr = 1e7
minidx = 0
for i in range(minx, maxx, res):
terror, pixs, error = computeError(p1, p2, i, res)
if error < minerr:
minidx = i
minerr = error
minx = minidx
maxx = minidx + res
print res, minidx, minerr
#print "new bounds", minx, maxx
return minidx, minerr
The patch() function takes 2 pictures and offsets. It combines the pictures based on the offsets and return the merged result. Because the images have a black bar in the left, we have to ignore those bits, that is what the 'fudge' factor is about.
def patch(p1, p2, offset1, offset2, fudge=8):
"""
Returns a new image p3, composed of p1 and p2 composed such that
up to the offset column p3 is p1, and after p2. There is a fudge
factor so that the left columns of p2 are ignored since that is a
black stripe.
"""
p3 = makePicture(getWidth(p1) + offset1, getHeight(p1))
if fudge > offset1:
fudge = offset1
for i in range(0, getWidth(p3)):
for j in range(0, getHeight(p3)):
if i < (offset2 + fudge):
px = getPixel(p1, i, j)
else:
px = getPixel(p2, i - offset2, j)
px2 = getPixel(p3, i, j)
setColor(px2, getColor(px))
return p3
Finally, we are going through each pair of images, find the offset with minimum error and combine them.
The tricky bit is we are keeping a new image 'panorama' that needs to be augmented after each iteration.
Also, we go through the images in reverse order, since we are patching from left to right. We took the images
right to left ...
panorama = None
for i in range(60, 1, -1):
this = "%.03d.jpg" % i
p1= makePicture(this)
next = "%.03d.jpg" % (i-1)
p2 = makePicture(next)
idx, err = findMinError(p1, p2)
print "error between (", i, i-1, ") was at pos = ", idx, "with error =", err
if (idx >= 255):
print "discarding this image"
continue
if panorama:
panorama = patch(panorama, p2, idx, getWidth(panorama) - getWidth(p1) + idx)
else:
panorama = patch(p1, p2, idx, idx)
print "New panorama size = ", getWidth(panorama)
savePicture(panorama, 'out.jpg')
Misc.
The new manualCamera() disables autogain, autoexposure, autowhitebalance. It takes 3 parameters, the gain, brightness, and exposure. They default to reasonable values, but depending on lighting, need to be tweeked. Also added support so if you have the MYROROBOT env. variable set init() uses that robot serial port instead of prompting ... both changes in SVN.
