Re: [Pytables-users] using pytables for saving realtime video?

[Francesc A. <fa...@py...>]

A Dimarts 22 Juny 2004 13:27, Francesc Alted va escriure:
> An small update: I re-made this C benchmark, but using only the zlib
> compressor (i.e. whitout shuffling) and setting all data to zeros, and I've
> obtained 33 MB/s. Without compression, that figure may perfectly grow up to
> 40 MB/s (whenever the hard disk would support this throughput, of course).

More updates ;). This morning I remembered that Table objects has a much
more efficient interface for writing than EArrays (just because I've spend
more time on optimizing Tables than anything else), and besides, I've
recently reworked the algorithm to compute buffer sizes for Tables, in order
to make them still faster. And the good news is that all of this largely
address to this problem :)

So, if you use the lastest CVS and use a Table instead of an EArray, this
small script would be far more efficient than the equivalent using EArrays:

*****************************************************
import tables
import numarray as na

class Test(tables.IsDescription):
    var1 = tables.UInt8Col(shape=(640,480))

nframes = 200
filename = "data.nobackup/test2.h5"
fileh = tables.openFile(filename, mode="w", title="Raw camera stream")
root = fileh.root
filter_args = dict( complevel = 1, complib = 'lzo', shuffle=0)
hdftable = fileh.createTable(root,'images', Test, "Unsigned byte table",
                             tables.Filters(**filter_args),
                             expectedrows=nframes)

frame = na.zeros(type="UInt8", shape=(640,480))
for i in range(nframes):
    hdftable.row["var1"] = frame
    hdftable.row.append()
fileh.close()
*****************************************************

With it, I was able to save frames at a speed of 46.2 MB/s without
compression (this script generates a 60 MB file, so that it fits well on my
laptop cache). Using ZLIB, I've got 36.4 MB/s, with LZO compression 54.5
MB/s and with UCL it drops down to 8.0 MB/s.

I was curious about how much memory would take a long run, and I made a test
with 20000 frames for a total dataset size of 6 GB. I've used LZO compressor
in order to keep the file size small. With that, the run took 1m23s, for a
total throughput of more than 70 MB/s. And the process took 16 MB of memory
during the run, which is quite reasonable. However, there seems to be a
"small" memory leak that develops at a rate of 3 KB/frame. Whether this is
acceptable or not is up to you.

Cheers,

-- 
Francesc Alted