Exporting Point Clouds From Photosynth
Since my last post about photosynth I've revisited the site and discovered that the pictures can be toggled off with the 'p' key, and the viewing experience is much improved given there is a good point cloud underneath. But what use is a point cloud inside a browser window if it can't be exported to be manipulated into random videos that could look like all the lidar videos I've made, or turned into 3D meshes and used in Maya or any other program?
Using one of those programs to intercept 3D api calls might work, though maybe not with DirectX or however the photosynth browser window is working. What I found with Wireshark is that http requests for a series of points_m_n.bin files are made. The m is the group number, if the photosynth is 100% synthy then there will only be one group labeled 0. The n splits up the point cloud into smaller files, for a small synth there could just be points_0_0.bin.
Inside each bin file is raw binary data. There is a variable length header which I have no idea how to interpret, sometimes it is 15 bytes long and sometimes hundreds or thousands of bytes long (though it seems to be shorter in smaller synths).
But after the header there is a regular set of position and color values each 14 bytes long. The first 3 sets of 4 bytes are the xyz position in floating point values. In python I had to do a byteswap on those bytes (presumably from network order) to get them to be read in right with the readfile command.
The last 2 bytes is the color of the point. It's only 4-bits per color channel, which is strange. The first four bits I don't know about, the last three sets of 4 bits are red, blue, and green. Why not 8-bits per channel, does the photosynth process not produce that level of precision because it is only loosely matching the color of corresponding points in photos? Anyway as the picture above shows I'm doing the color wrong- if I have a pure red or green synth it looks right, but maybe a different color model than standard rgb is at work.
I tried making a photosynth of photos that were masked to be blue only- and zero synthiness resulted - is it ignoring blue because it doesn't want to synth up the sky in photos?
Anyway here is the python script for interpreting the bin files.
The sceneviewer (taken from the Radiohead sceneviewer) in that source dir works well for displaying them also.
Anyway to repeat this for any synth wireshark needs to figure out where the bin files are served from (filter with http.request), and then they can be downloaded in firefox or with wget or curl, and then my script can be run on them, and processing can view them. The TOC doesn't clearly specify how the point clouds are covered so redistribution of point clouds, especially those not from your own synths or someone who didn't CC license it, may not be kosher.
Posted by binarymillenium at 10:28 AM
Labels: photosynth, processing, python
Subscribe to: Post Comments (Atom)
Hey Binary, nice job.
You might try 5:6:5 (bits) for the color channels. That's what people usually use when they want to store colors in 16bits.
That's exactly it. Thanks!
Photosynth doesn't work much well with things he can't recognize as detailed.
This means that large surfaces like sky and sea cannot be interpreted at all. Where is it supposed to place the blue points?
I know nothing about python. I downloaded and installed the program and saved your code as a .py file, but what am I supposed to do next in order to download the point cloud?
I tried your export script and it works great.
Do you know if there a way to extract also the position and orientation of the different pictures ?
I've given your Python code a try in C#. Not sure it's correct at this stage, but it gives some output...
If anyone has a known-good source .bin file and points output file I'd be interested to see it (alternatively check out the source/results I posted).
Thanks for all your effort in the first place.
Did you ever get an answer to your request for the position and orientation data for each frame?
How in hell do i process the file with the script?
This is amazing, and I have been able to use Your method in the past.
However, since the direct 3d viewer is officially dead, how can I access the bin files now?
Post a Comment