2008-08-27

Exporting Point Clouds From Photosynth


Since my last post about photosynth I've revisited the site and discovered that the pictures can be toggled off with the 'p' key, and the viewing experience is much improved given there is a good point cloud underneath. But what use is a point cloud inside a browser window if it can't be exported to be manipulated into random videos that could look like all the lidar videos I've made, or turned into 3D meshes and used in Maya or any other program?

Supposedly export will be added in the future, but I'm impatient like one of the posters on that thread so I've gone forward and figured out my own export method without any deep hacking that might violate the terms of use.

Using one of those programs to intercept 3D api calls might work, though maybe not with DirectX or however the photosynth browser window is working. What I found with Wireshark is that http requests for a series of points_m_n.bin files are made. The m is the group number, if the photosynth is 100% synthy then there will only be one group labeled 0. The n splits up the point cloud into smaller files, for a small synth there could just be points_0_0.bin.

Inside each bin file is raw binary data. There is a variable length header which I have no idea how to interpret, sometimes it is 15 bytes long and sometimes hundreds or thousands of bytes long (though it seems to be shorter in smaller synths).

But after the header there is a regular set of position and color values each 14 bytes long. The first 3 sets of 4 bytes are the xyz position in floating point values. In python I had to do a byteswap on those bytes (presumably from network order) to get them to be read in right with the readfile command.

The last 2 bytes is the color of the point. It's only 4-bits per color channel, which is strange. The first four bits I don't know about, the last three sets of 4 bits are red, blue, and green. Why not 8-bits per channel, does the photosynth process not produce that level of precision because it is only loosely matching the color of corresponding points in photos? Anyway as the picture above shows I'm doing the color wrong- if I have a pure red or green synth it looks right, but maybe a different color model than standard rgb is at work.

I tried making a photosynth of photos that were masked to be blue only- and zero synthiness resulted - is it ignoring blue because it doesn't want to synth up the sky in photos?

Anyway here is the python script for interpreting the bin files.

The sceneviewer (taken from the Radiohead sceneviewer) in that source dir works well for displaying them also.

Anyway to repeat this for any synth wireshark needs to figure out where the bin files are served from (filter with http.request), and then they can be downloaded in firefox or with wget or curl, and then my script can be run on them, and processing can view them. The TOC doesn't clearly specify how the point clouds are covered so redistribution of point clouds, especially those not from your own synths or someone who didn't CC license it, may not be kosher.

11 comments:

Marvin said...

Hey Binary, nice job.

You might try 5:6:5 (bits) for the color channels. That's what people usually use when they want to store colors in 16bits.

binarymillenium said...

That's exactly it. Thanks!

Marco E. G. Maltese said...

Photosynth doesn't work much well with things he can't recognize as detailed.
This means that large surfaces like sky and sea cannot be interpreted at all. Where is it supposed to place the blue points?

Marco E. G. Maltese said...
This comment has been removed by the author.
sfdgsfdg said...
This comment has been removed by the author.
sfdgsfdg said...

I know nothing about python. I downloaded and installed the program and saved your code as a .py file, but what am I supposed to do next in order to download the point cloud?

Unknown said...

Hello,

I tried your export script and it works great.

Do you know if there a way to extract also the position and orientation of the different pictures ?

Craig Dunn said...

I've given your Python code a try in C#. Not sure it's correct at this stage, but it gives some output...

If anyone has a known-good source .bin file and points output file I'd be interested to see it (alternatively check out the source/results I posted).

Thanks for all your effort in the first place.

Unknown said...

Hey http://www.blogger.com/profile/15249264104814512534,

Did you ever get an answer to your request for the position and orientation data for each frame?

Marco :Polo said...

How in hell do i process the file with the script?

Unknown said...

This is amazing, and I have been able to use Your method in the past.
However, since the direct 3d viewer is officially dead, how can I access the bin files now?