[Pytables-users] Re: PyTables row size limit

[Francesc A. <fa...@py...>]

Hi Russel,

The 8k limit in pytables 0.8 was mainly a limitation due to my relatively
poor understanding of the chunks and the BTree that HDF5 keeps in memory to
be used to map chunks on disk. A bigger rowsize would make reading access
*much* more slower.

However, PyTables 0.8.1 has implemented a new algorithm to compute both
chunks sizes and buffer sizes. As a consequence, the limit has been raised
to 512 KB without a noticeable impact on performance. I hope that would be
enough for you, but that limit can be further increased in case of
necessity.

Cheers,

A Dijous 12 Agost 2004 20:56, vareu escriure:
>     I have just begun experimenting with PyTables for a project analyzing 
> LC/MS data (from this machine 
> <http://www.ionspec.com/Our%20Products/HiResESI/HiResESI%20fs.htm> ). 
> These data sets consist of 1200 mass spectra each consisting of about 
> 1800000 double-precision values.
>     I created a table with each row consisting of some information about 
> each spectrum and the data points for that spectrum.  This results in a row 
> size substantially larger than the 8192 byte limit imposed by pytables.  I 
> could find no such corresponding limit in the HDF5 code, so I took out the 
> row size test in pytables and recompiled.  I generated some files and 
> everything appears fine.  I can read all the data back and manipulate it 
> just as I hoped.  My question is where does this limit come from?  What 
> would I have to do to safely remove (or substantially increase) the row 
> size limit in pytables?
>      Thanks in advance for any hints.
> Russel Howe
> 
> 

-- 
Francesc Alted