Re: [Pytables-users] Re: read rates...

[Francesc A. <fa...@ca...>]

Hi Damon,

A Divendres 18 Febrer 2005 20:31, damon fasching va escriure:
> I built PyTables from sources.  I've attached the
> stdout and stderr logs from the build script.  These
> error messages aren't causing me any problems, so I

In your build logs I've seen a fatal error that should have prevented
a successful build, namely:

src/hdf5Extension.c:8959: error: too few arguments to function
`H5ARRAYget_info'
error: command 'gcc' failed with exit status 1

I can't understand how you have generated the hdf5Extension with such
an error(??). Besides, I've double checked that this error should not
happen with PyTables 0.9.1. Are you sure that you have taken the
original tarball from pytables web site and that this was unmodified
before the build process?

> Regarding read rates, I was thinking that 4 seconds to
> read 23 items was slower than expected.  But maybe
> not.    There were probably 5 accesses to the index
> and then 23 accesses to the tables, 140 ms per access.

Well, depending on how far are the entries from each other, 140 ms
maybe large or not. You've already said that you have a slow hard
disk, so this may be source of slowness. The new indexing algorithm in
forthcoming pytables Pro may achieve significantly better performance.
If you are curious, I can send you an alpha version of pytables Pro so
that you can check it with your own data pattern.

> What will be the function of the "sorted by" and
> "group by" qualifiers.  Since they are qualifiers on
> search methods it sounds as if they don't alter the
> structure of the underlying table, rather they present
> the results in a particular order.  So search
> performance would not be impacted.

Correct

> What I would like to be able to do is actually
> reorganize my tables so the entries are ordered by a
> particular field. I will normally be accessing data in
> order of this field, so if the table were ordered by
> this field I could just use a cursor to access the
> data...fast fast fast :)  My tables will be small
> enough to fit in memory, so in principle I could load
> the entire array into memory and then sort it, but I
> don't see a sort method on the table in either
> PyTables or numarray.

I see. You can reorder you RecArrays (in-memory tables) very easily in
that way:

r2 =3D records.array(r[r.field('c2').argsort()])  #sort by column 'c2'


Cheers,

=2D-=20
>qo<   Francesc Altet =A0 =A0 http://www.carabos.com/
V =A0V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 ""