Marker Tracking as Visualization Interface

My idea is that I would be able to do an ARToolkit based visualization performance by using a clear table with markers I can slide, rotate, add and remove, and all those movement could correspond to events on screen. Unlike other AR videos the source video wouldn't be incorporated into the output necessarily, the markers provide an almost infinitely expressive set of UI knobs and sliders.

So far I have this:


AR User Interface from binarymillenium on Vimeo.

The lighting is difficult, the markers need to be white and black pixels but the plexiglass tends to produce reflections. Also if the light source itself is visible a marker will not be able to be right on top of it. I need a completely black backdrop under the plexiglass so there are no reflections that will obscure the markers, and also more numerous and softer diffuse lights.

One way to solve the reflection problem is to have the camera looking down at a table, though it's a little harder to get the camera up high enough, and I didn't want my hands or body to obscure the markers- the clear table idea is more elegant and self-contained.

The frame rate isn't very high, I need to work on making it all more real-time and responsive. It may have to be that one computer is capturing video and finding marker positions and sending them to another computer completely free to visualize it. Also more interpolation and position prediction could smooth things out, and cover up gaps if a marker isn't recognized in a frame, but that could produce more lag.

Bundler - the Photosynth core algorithms GPLed

[update- the output of bundler is less misaligned looking than this, I was incorrectly displaying the results here and in the video]

Bundler (http://phototour.cs.washington.edu/bundler) takes photographs and can create 3D point clouds and camera positions derived from them similar to what Photosynth does- this is called structure from motion. It's hard to believe this has been out as long as the publically available Photosynth but I haven't heard about it- it seems to be in stealth mode.


Bundler - GPLed Photosynth - Car from binarymillenium on Vimeo.

From that video it is apparent that highly textured flat surfaces do best. The car is reflective and dull grey and so generates few correspondences, but the hubcaps, license plate, parking strip lines, and grass and trees work well. I wonder if this could be combined with a space carving technique to get a better car out of it.

It's a lot rougher around the edges lacking the Microsoft Live Labs contribution, a few sets I've tried have crashed with messages like "RunBundler.sh: line 60: 2404 Segmentation fault (core dumped) $MATCHKEYS list_keys.txt matches.init.txt" or sometimes individual images throw it with "This application has requested the Runtime to terminate it..." but it appears to plow through (until it reaches that former error).

Images without good EXIF data trip it up, the other day I was trying to search flickr and find only images that have EXIF data and allow full view, but am not successful so far. Some strings supposed limit search results by focal length, which seems like would limit results only to EXIF, but that wasn't the case.

Bundler outputs ply files, which can be read in Meshlab with the modification that these two lines be added to ply header:

element face 0
property list uchar int vertex_index

Without this Meshlab will give an error about there being no faces, and give up.

Also I have some Processing software that is a little less user friendly but doesn't require the editing:

http://code.google.com/p/binarymillenium/source/browse/trunk/processing/bundler/

Bundler can't handle filenames with spaces right now, I think I can fix this myself without too much work, it's mostly a matter of making sure names are passed everywhere with quotes around them.

Multi-megapixel files load up sift significantly until it crashes after taking a couple of gigabytes of memory (and probably not able to get more from windows):

...
[Found in EXIF tags]
[CCD width = 5.720mm]
[Resolution = 3072 x 2304]
[Focal length (pixels) = 3114.965
[Found 18 good images]
[- Extracting keypoints -]

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.


Resizing them to 1600x1200 worked without crashing and took only a few hundred megabytes of memory per image, so more megapixels may work as well.

The most intriguing feature is the incremental option, I haven't tested it yet but it promises to be able to take new images and incorporate them into existing bundles. Unfortunately each new image has a matching time proportional to the number of previous images- maybe it would be possible to incrementally remove images also, or remove found points that are in regions that already have high point densities?

Nested blocks in Django duplication problems

The official instructions and cursory google searches didn't turn up a good explanation, but I've figured it out for myself. I was confused about nesting blocks, sometimes getting no output or getting duplicate output.

In this example the base has a single level of nesting with two sub-blocks.

base.html:

{% block outer %}

{% block inner1 %}
this is inner1
{% endblock inner1 %}


{% block inner2 %}
this is inner2
{% endblock inner2 %}

{% endblock outer %}


This file duplicate the original block structure but adds a comment:
some.html:

{% extends base.html %}

{% block outer %}
{{ block.super }}

new stuff

{% endblock outer %}


The output would be

this is inner1
this is inner 2
new stuff

Moving the 'new stuff' line before the the block.super would swap the order of the output statements. There is no way to interject the new comment inbetween inner1 and inner2 without creating a new block that sits inbetween them in the parent base.html file.



Don't try to do this (which is what I thought to do initially):


{% extends base.html %}

{% block outer %}
{{ block.super }}

new stuff

{% block inner2 %}
new inner2
{% endblock inner2 %}

{% endblock outer %}


It will result in duplication like this:

this is inner1
new inner2
new stuff
new inner2

Instead, the extending file that wants to alter any parent block does it in a non-nested way, don't redefine an inherited block while inside of another inherited block:

{% extends base.html %}

{% block outer %}
{{ block.super }}

new stuff

{% endblock outer %}

{% block inner2 %}
new inner2
{% endblock inner2 %}


And now the output will be without duplication.

this is inner1
new inner2
new stuff

block.super needs to be in there or the redefinition of inner2 won't be applied to anything.

Laser Scanning

The idea is to project laser lines onto a flat surface, image them, and then put objects in front of the surface and compute the displacement made by the object.

Here is the flat base with a line on it:



Here is the line at the same position with objects intersecting:



Finding depth involves figuring out what the 2d projection of the normal line that is perpendicular to the wall at any point along the laser line. I'm working on this but it's also possible to guess an average line for low precision demonstration. The software looks for all points where it believes the laser is shining, and then computes the intersection of the normal line with the original object free laser line, and gets depth.

I had about 8 different images from laser lines, here are the results from two:





The yellow lines are the projected normals from the base line to the found laser line on the backpack and broom. There are some spurious results, and also on the dark woven backpack material the laser was not always reflected strongly enough to register.


The source code is here:

http://code.google.com/p/binarymillenium/source/browse/trunk/processing/laserscan

Multicamera Balloon Imagery

AHAB Tether Test from binarymillenium on Vimeo.

I originally was reminded of the unused source images because another team member recently posted some pictures from their camera, and then I made this, and then Bre posted about it, so it's getting a lot of positive feedback.

This video would have been made earlier but I had assumed that the cameras were screwy and firing at different times and image sequences would not line up at all- turns out they did, they just had wildly different start points.

Also I finished this school project that was kind of a simple occupancy grid inspired thing, this video shows parts of it:


2.5D from binarymillenium on Vimeo.

I might revisit some of this and get the registration code working (and working a lot faster), instead of cheating and using the known camera position and attitude.