2009-01-29

Bundler - the Photosynth core algorithms GPLed

bundler 212009 65922 AM.bmp
[update- the output of bundler is less misaligned looking than this, I was incorrectly displaying the results here and in the video]

Bundler (http://phototour.cs.washington.edu/bundler) takes photographs and can create 3D point clouds and camera positions derived from them similar to what Photosynth does- this is called structure from motion. It's hard to believe this has been out as long as the publically available Photosynth but I haven't heard about it- it seems to be in stealth mode.


Bundler - GPLed Photosynth - Car from binarymillenium on Vimeo.

From that video it is apparent that highly textured flat surfaces do best. The car is reflective and dull grey and so generates few correspondences, but the hubcaps, license plate, parking strip lines, and grass and trees work well. I wonder if this could be combined with a space carving technique to get a better car out of it.

It's a lot rougher around the edges lacking the Microsoft Live Labs contribution, a few sets I've tried have crashed with messages like "RunBundler.sh: line 60: 2404 Segmentation fault (core dumped) $MATCHKEYS list_keys.txt matches.init.txt" or sometimes individual images throw it with "This application has requested the Runtime to terminate it..." but it appears to plow through (until it reaches that former error).

Images without good EXIF data trip it up, the other day I was trying to search flickr and find only images that have EXIF data and allow full view, but am not successful so far. Some strings supposed limit search results by focal length, which seems like would limit results only to EXIF, but that wasn't the case.

Bundler outputs ply files, which can be read in Meshlab with the modification that these two lines be added to ply header:

element face 0
property list uchar int vertex_index

Without this Meshlab will give an error about there being no faces, and give up.

Also I have some Processing software that is a little less user friendly but doesn't require the editing:

http://code.google.com/p/binarymillenium/source/browse/trunk/processing/bundler/


Bundler can't handle filenames with spaces right now, I think I can fix this myself without too much work, it's mostly a matter of making sure names are passed everywhere with quotes around them.

Multi-megapixel files load up sift significantly until it crashes after taking a couple of gigabytes of memory (and probably not able to get more from windows):

...
[Found in EXIF tags]
[CCD width = 5.720mm]
[Resolution = 3072 x 2304]
[Focal length (pixels) = 3114.965
[Found 18 good images]
[- Extracting keypoints -]

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.


Resizing them to 1600x1200 worked without crashing and took only a few hundred megabytes of memory per image, so more megapixels may work as well.

The most intriguing feature is the incremental option, I haven't tested it yet but it promises to be able to take new images and incorporate them into existing bundles. Unfortunately each new image has a matching time proportional to the number of previous images- maybe it would be possible to incrementally remove images also, or remove found points that are in regions that already have high point densities?

21 comments:

Unknown said...

Found this some time ago and find it usefull.

Just one question : in the video, it looks like the points are not correctly aligned. For example, the wheel covers, the front, ... Did I miss something in the explanation or is this a problem in the matching algorithm ?

binarymillenium said...

No I screwed up here, I didn't read the explanation clearly and thought all the different ply files were part of the same set and not intermediate files... I guess I could regenerate the video but I probably will just move on to new ones.

Though if the intermediate files were to be animated it might be interesting to see the evolution/convergence towards the end point cloud.

Unknown said...

ok I understand also why there were so many files.

From a legal point of view, do you think it is possible to use the code inside a commercial software ?

binarymillenium said...

Using Bundler inside commercial software? Bundler is GPLed, therefore it can be sold commercially but the source code has to be made available.

But Bundler needs SIFT to work right, and SIFT is patented (in the US at least) and the implementation Bundler uses is not GPL.

But there might be GPL versions of SIFT (although the patent issue is still there), and there might also be of SURF which I think could replace it- I'm going to see if the output of something like OpenSURF can be formatted so Bundler can read it.

Unknown said...

Nice Job!
probably you could be interested in this other GPL tool for browsing the data output by the bundler.

http://photocloud.sourceforge.net

cheers
p.

Unknown said...

Paolo, I don't see a link for the application.

Paolo I have another question : did you investigate the possibility to rebuild the 3D model based on points cloud ?

Unknown said...

No link to downloadable app currently. :(
Brave users you have to compile photocloud from the sources...

About the issue of rebuiilding meshes from the point clouds, we have worked a lot on that. Inside MeshLab Poisson surface reoconstruction can make a good job. But it is rather difficult becouse these point clouds are really very noisy...

P.

Gaz Davidson said...

fyi, PhotoSynth resizes photos to 2MP before trying to match anything.

Unknown said...

Hi,
I am trying to use the bundler for reconstruction and it seems that all of the generated points are cluttered together or there are very little points.

Any tips on taking the photos ?

and Bundler version 0.3 is released.

binarymillenium said...

indi,

Thanks for the tip on the new Bundler version, I created a changedetection.com alert for it's homepage so I won't miss the next one.

The first thing to try is to upload the same pictures you are working on in Bundler to Photosynth and compare the results. If the points are too close together, you should be able to scale them to spread them out in a viewer.

You also should be able to look at tips for good Photosynths and apply those to Bundler. Also get your pictures into a web album (like picasaweb), provide a link to that or to a photosynth page and you can come back here and post for help, or email me at my gmail account (using the name of this blog).

Unknown said...

mr binarymillenium, i have read all your posts and watched all your vids and i'm a big fan of your work - i have successfully extracted many point clouds from photosynth with your process and had plenty of fun processing them - but my windows drive "vista'd" all over itself and i am now based out of linux ubuntu 9.04. i have tried bundler for the past couple weeks, and had no luck running it...could you please post a quick tutorial on how to use bundler? thank you man keep up the good work!!! you are official bookmarked now :)

binarymillenium said...

the office box,

I haven't tried bundler on Linux yet- I'll give it a shot and post what I find.

Unknown said...

binarymillenium, the office box,

I have managed to get the bundler working in Ubuntu 8.10. Here are the steps.

1. You need to specify the location of libANN_char.so. This can be done by typing export LD_LIBRARY_PATH="location of libANN.char.so" This file is in the lib folder of the bundler application and I usually export the absolute path.

2. As I remember there is a file called libgfortran.so.1 in the bundler lib folder. Just copy this to /usr/lib/ If its not there you can easily find it on the net.

3. You also need to make a few modifications to a few files in the budler, basically set a few paths. I cant remember the files exactly, they are specified in the bundler manual.

4. I had an issue with paths. I usually copy the RunBundler.sh to where the images are and modify the paths in the file to where the required executables/scripts are.

5. Finally ./RunBundler.sh will run the script. If you have a permission issue just do chmod 777 RunBundler.sh and it should be ok.

Read the bundler manual and the FAQs.

Hope this helps.

Good Luck.

Indika said...

Hi,
I am trying to run bundler on Windows using bundler version 0.3.
I seem to be getting an error halfway through the program.

Were you able to get bundler running on Windows ?

binarymillenium said...

Indika,

I have gotten Bundler to work on Windows (in the Cygwin environment). What is the error that you see?

Indika said...

Hi,
Thanks for the reply.
I think I found out what the error is. I am not using the SIFT implementation provided with the bundler, but a different one.

I analyzed the code to see if I can find it. I think its because of the slight differences in values that I get from the implementation that I am using.

If I figure out how to solve this, I will let you know.

Miguel said...

Hi binarymillenium,

I liked the video a lot! I'd like to run the bundler with the car images to try out things myself, but I haven't found them anywhere. You know if they available somewhere?
Thanks!

Josh Harle said...

I'm a PhD candidate working in Architecture visualization and looking at many methods of mapping. In particular I've been using PhotoSynth, and more recently Bundler, to get sparse point clouds. Now I've started using PMVS2 (another Washington Uni project) to convert the sparse point-clouds into dense point-clouds, and then with Poisson reconstruction, into meshes.

I've had some very good results from this, but PhotoSynth still fairs better at EXIF-less images (e.g. from video) than Bundler. PMVS2 needs the camera parameters that Bundler spits out, as well as source images (it uses stereo-image analysis).

Here is an example of reconstruction of example data: http://www.youtube.com/watch?v=7zhxPiAuOV8
and our own images, and bundler output: http://www.youtube.com/watch?v=Ext1PeT_R-U

I'm wondering if you could give any advice on getting Bundler working as well as PhotoSynth, or advice on getting the camera parameters out of the PhotoSynth? I'd be more than happy to help put in the setup of PMVS2 pipeline! :)

As an example of difficulty, http://photosynth.net/view.aspx?cid=00601666-e6fa-4ee7-8bbf-e6dd3a9c2539 worked really well (so many points!) but it was pretty useless at bundler.

Mark said...

Josh, checkout this blog for help with Bundler: http://www.personal.psu.edu/nmc15/blogs/anthspace/2010/01/bundler-installation-procedures.html . It is a real bear to get up and running on a Windows box.

I'll have to look into PMVS2. It might solve a few problems.

Josh Harle said...

@Mark,
Thanks for that link. I actually have Bundler up and working on Linux and Windows now (v0.4 on Linux), but for some imagesets PhotoSynth is far more successful than Bundler. I've played around with different options-file settings, but without improved results.

Josh

kapoorvinny said...

hi, I have start working with bundler and need some assistance to construct the viewing camera positions from the obtained point cloud.