Re: [Pytables-users] Question about pytables

[Francesc A. <fa...@op...>]

A Dimecres 02 Juliol 2003 00:15, Vineet Jain va escriure:
> I have decided to go forward with pytables for the time being. Great to
> hear that hd1.6 is planning to implement the updating feature.

Well, it's not exactly a feature on HDF5 1.6, but rather a combination of
HDF5_HL library (which ships with pytables) and HDF5 1.6. But, it seems like
if HDF5 1.6 is the missing factor to achieve that.

Hope you will be happy to use pytables. If in the future you need some
specific need that is not implemented in pytables or just want professional
support, remember that I'm offering commercial support for that kind of
things ;-).

> I'm
> looking to store stock minute bars for around 5000 stocks for several
> years. This will be a lot of data and pytables is very fast so I don't
> hav eto worry about the IO part of things.

That sounds nice. In addition, if your data is compressible, you will find
that you need a fraction of the space of a relational database.

>
> Yes you assumed correctly that I wanted to create a new recarray with
> the total number of rows which would be numrows1+numrows2. Once I read
> the objects from memory are they mutable or immutable. Can I change some
> of the values in place?

The RecArray object is mutable, so you can change this values in-memory.
Besides, if you save this object on the file later-on and delete the
original table you have a rather primitive, yet effective way of upgrading
rows, until a more efficient way would be implemented.

>
> So if read gets all the rows of a table in memory does iterrow only load
> the rows that you requested?

To be exact, read() only reads the rows specified on its start, stop, step
and field arguments, in the same way that iterrows(). The difference between
them is that read() returns a monolithic object (i.e. a RecArray) with all
the info you have requested, while iterrows() is a row iterator, so you get
only a row each time it is invoked.

> Is there any way to get a recarray back and
> not load all the data into memory with out having to go through the
> iterator?

That's possible, as I said before, by using the start, stop and step
parameters of read(). But if you want to read over all the table without
loading all the data in memory, you will need iterrows(), of course.

Cheers,

-- 
Francesc Alted