More python pcap with pcapy

After running into pcap files several hundreds of megabytes in size that caused wireshark to crash when loaded, I returned to trying to make python work with the source pcap file:


import pcapy

vel = pcapy.open_offline('unit 46 sample capture velodyne area.pcap')

vel
Reader object at 0xb7e1b308

pkt = vel.next

pkt
built-in method next of Reader object at 0xb7e1b308


What is a Reader object, and a built-in method of it? Why are the addresses the same?

try

pkt = vel.next()

type(vel)
type 'tuple'

vel[1]


'\xff\xff\xff\xff\xff\xff\x00\x00\x00\x00\x00\x00\x08\x00E\x00\x04\xd2
\x00\x01\x00\x00\x80\x11\xad\x9f\xc0\xa8\x03+\xc0\xa8\x03\xff\x01\xbb
\t@\x04\xbe\x00\x00\xff\xddz\x13\xee
...
\x00\x00\x10\xd63Md\xc2\xff\x00\x00\x0fp\x07v25b'


So that's just the problem I was running into before: '\xff' is an ascii representation of binary data that's really 0xff. But I'm given it in ascii- I can't index into this and get a specific 8-bit 0-255 binary value, I get a '\'. Do I have to write something that goes through this and reinterpets the ascii-ized hex back into real hex?

Also I note the ff dd that marks the beginning of the data frame is there but not at the beginning- so there are other parts of the packet here I need to get rid of. Is this where I need Impacket?

import impacket
from impacket.ImpactDecoder import EthDecoder

decoder = EthDecoder()
b = decoder.decode(a)
Traceback (most recent call last):
File "stdin", line 1, in ?
File "/var/lib/python-support/python2.4/impacket/ImpactDecoder.py", line 38, in decode
e = ImpactPacket.Ethernet(aBuffer)
File "/var/lib/python-support/python2.4/impacket/ImpactPacket.py", line 340, in __init__
self.load_header(aBuffer)
File "/var/lib/python-support/python2.4/impacket/ImpactPacket.py", line 255, in load_header
self.set_bytes_from_string(aBuffer)
File "/var/lib/python-support/python2.4/impacket/ImpactPacket.py", line 59, in set_bytes_from_string
self.__bytes = array.array('B', data)
TypeError: an integer is required

oops

b = decoder.decode(a[1])
print b
Ether: 0:0:0:0:0:0 -> ff:ff:ff:ff:ff:ff
IP 192.168.3.43 -> 192.168.3.255
UDP 443 -> 2368

ffdd 7a13 ee09 3cbe 093f 1811 2b1f 1020 ..z.....?..+..
0a0a 63ac 0848 d708 53ea 085b 0000 1bc0 ..c..H..S..[....
0a35 0c09 425b 0936 000d 2e44 0d2f 120b .5..B[.6...D./..
4f5e 0b30 200f 3c4e 0e50 c30b 46c7 0b4d O^.0 .

The decode makes a nice human readable text of the packet, but not what I want.

Here is a different tack- by looking in the Impacket.py source I found how to do this which converts that annoying ascii back to real bytes, which is the only real issue:

mybytes = array.array('B', vel[1])

mybytes is of size 1248, so there appear to be 42 extra bytes of unwanted ethernet wrapper there- why not just index into mbytes like mybytes[42:] and dump that to a binary file?

I don't know about the dumping to binary file (print mybytes prints it in ascii, not binary)- but I could easily pass that array straight into the velodyne parsing code- and this skips the intermediate file step, saving time and precious room on my always nearly full laptop hd.

So here is the final result, which produce good CSVs I was able to load with my 'velosphere' Processing project to create 360 degree panoramas from the lidar data:

Next I need a way to write pngs from python, and I could eliminate the CSVs & Processing step.