[Pytables-users] using pytables for saving realtime video?

[Andrew S. <str...@as...>]

I am trying to save realtime video (640x480x100 fps uint8 grayscale 
data) using PyTables for scientific purposes (lossy compression is 
bad).  So, first off, is using PyTables for this task a reasonable 
idea?  Although I'm no expert, it seems the compression algorithms that 
PyTables offers may be ideal.  It also may be nice to use HDF5 to 
incorporate some data.

At the moment, however, I'm stymied by slow write speeds, and I seek 
suggestions on how to speed this up.  I need to get this working 
approximately 10x faster to be viable.

At first pass, I've started with code like this:

        self.fileh = tables.openFile(filename,
                                     mode="w",
                                     title="Raw camera stream")
        root = self.fileh.root
        a = tables.UInt8Atom((self.cam_height,self.cam_width,0))
        filter_args = dict( complevel = 0,
                            complib = 'ucl',
                            )
        self.hdfarray = self.fileh.createEArray(root,
                                                'images', a,
                                                "Unsigned byte array",
                                                
tables.Filters(**filter_args))

while 1:
                    # other stuff that fills self.grabbed_frames

                    n_frames = len(self.grabbed_frames)
                    # set to rank 3
                    def add_dim(f):
                        f.shape = (self.cam_height,self.cam_width,1)
                    map( add_dim, self.grabbed_frames)
                    frames = na.concatenate( self.grabbed_frames, axis=2)
                    print frames.shape
                    self.hdfarray.append(frames)

Using this code, I get approximately 4 MB/sec with no compression, and 
MB/sec with complevel=1 UCL.  This is with an XFS filesystem on linux 
kernel 2.6.6 using a SerialATA drive which benchmarks writing at 50 
MB/sec using iozone.

So, are there any suggestions for getting this to run faster?

Cheers!
Andrew