I have the colors figured out now: I was forgetting to byteswap the two color bytes, and after that the rgb elements line up nicely. And it's 5:6:5 bits per color channel rather than 4 as I thought previously, thanks to Marvin who commented below.
The sphinx above looks right, but earlier the boxer shown below looked so wrong I colored it falsely to make the video:
The Boxer - Photosynth Export from binarymillenium on Vimeo.
But I've fixed the boxer now:
The python script is updated with this code:
red = (bin >> 11) & 0x1f
green = (bin >> 5) & 0x3f
blue = (bin >> 0) & 0x1f