[Pytables-users] Re: slowness w/many groups in pytables

[Francesc A. <fa...@op...>]

Hi John,

A Dimecres 11 Juny 2003 19:45, vareu escriure:
> If I create a trivial set of data w/3000+groups, when opening for
> reading, pytables takes almost 30 seconds, on the command:
>

Well, 3000 groups is not too bad for an start :-)

> fileh=tables.openFile(Hdf_file)
>
> Should it take this long to build the metadata?

My tests (using a P4@2GHz w 256 MB & Linux) show that it takes roughly 4
seconds for 3000 groups and little more that 5 s for 3000 leafs (small
arrays in this case). So, I don't know about your machine, but it seems
like if it is somewhat slow. Anyway, the metainformation reading process can
surely be improved by converting the appropriate python code into Pyrex, but
I don't know if this is worth the effort.

>
> This implies you want to organize w/few groups and many tables.

I think that your best bet is to use as few groups and tables as you can and
create as large tables as possible. This is what pytables (and HDF5) is
optimized for. Even if you have to made some redundancies on table columns,
you can always activate the compression, so that in my opinion, this will
not affect the access perfomance or final file sizes very much (rather the
contrary).

Cheers,

-- 
Francesc Alted