Re: [Pytables-users] PyTables memory usage expectations

[Francesc A. <fa...@ca...>]

Hi Norm,

Ivan and I are glad that you like PyTables.

As far as I know, PyTables is relatively free of leaks. There is an
issue, though, that exposes when using indexed columns in tables. This
is due to a leak in the HDF5 library (I've checked that it exists in
HDF5 1.6.2, 1.6.3 and 1.6.4), that has been reported to the NCSA. They
claim that this issue will be fixed in the next series of HDF5 1.8.x
and I've checked that this is true, because in the developer versions
of HDF5, i.e. 1.7.x, this leak seems to have disappeared.

So, the thing is, if you are using indexing with PyTables, you have
been hitten by a know leak. If you don't use indexing, I'd ask you to
send me a small program that can reproduce the leak, so that I can
check it more thoughtfully.

Cheers,

A Dijous 21 Abril 2005 17:23, Norm Petterson va escriure:
> Hello all,
>
> I'm designing an industrial data logging application, and hope to use
> PyTables as the data storage mechanism. Thanks, Francesc, Ivan, and
> everybody else for such an attractive product.
>
> I have a question, though, about process memory usage with PyTables:
> specifically, it appears that the virtual memory size of my Python
> process increases constantly as I append rows to my tables, and never
> decreases. Is this a feature or a leak? Does PyTables (and/or the
> underlying HDF5 library) map the full table(s) into my process address
> space at all times? Logging 2500 values per second (new rows to each of
> 10 tables), my process virtual size is over 100MB in an hour, and I run
> out of virtual memory before my expected 86400 rows per day are
> accumulated. It appears to make some difference whether I open/close the
> file each scan, or just leave it open (flushing after each row is
> appended, of course); opening/closing seems to cause virtual size to
> increase faster.
>
> I am running on Windows 2000, with HDF5 1.6.3 and PyTables 0.9.1,
> although I see the same behavior under Cygwin with HDF5 1.6.4 and the
> most-recent PyTables snapshot I downloaded and built yesterday.
>
> I'd appreciate anything you can tell me about memory usage (to save me
> having to dig directly into the sources ;-) ... Thanks again for your
> fine work.
>
> Regards,
>
> Norm Petterson
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: New Crystal Reports XI.
> Version 11 adds new functionality designed to reduce time involved in
> creating, integrating, and deploying reporting solutions. Free runtime
> info, new features, or free trial, at:
> http://www.businessobjects.com/devxi/728
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"