Re: [Pytables-users] PyTables memory usage expectations

[Francesc A. <fa...@ca...>]

Hi Norm,

Please, don't be afraid to write to the PyTables list, as your
problems can provide hints and feedback for other people; also, others
can help you as well. After all, this tries to be an open community ;)

I've tried your test script (ShowLeak.py), and it seems like you have
discovered a real leak. I think the problem arises when you are
filling several tables that have plenty of columns (in your case
256/table). At first sight, I suspect that this can be a problem with
HDF5 1.6.x series. It would be nice to check out with HDF5 1.7.x but,
unfortunately, PyTables does not support the 1.7.x series
right-of-the-box (although it is relatively easy to realize the
necessary changes in PyTables to do that, so you can have a try if you
want to).

Meanwhile, what you can do is to group columns with equal type under a
multidimensional column, so that you can create Tables with much less
number of columns. As an example, I've modified your script
(ShowLeak2.py) to use these multidimensional columns. With that, you
have 10 tables with just 1 column each instead of 10 tables with 256
columns each. Using this new structure, the resulting code does not
leak (or at least, I cannot detect it).

Hope this helps,

A Divendres 22 Abril 2005 16:45, v=E0reu escriure:
> Hi Francesc,
>
> At the risk of bothering you further, it seems that Python (2.3.x)
> gc.get_objects() crashes PyTables. I was curious about  gc objects
> before and after garbage collection, but the following error occurs
> after one iteration of the script I sent you yesterday (modified to
> import gc and do a gc.set_debug(gc.DEBUG_LEAK) before the iteration is
> performed). Maybe this is a clue?
>
> < --- snip --- >
> 11978057371, 'Tag1022': 0.34435703147292973, 'Tag1023':
> 0.49131899838928439}, 'C
> omplexCol': <class tables.IsDescription.ComplexCol at 0x010540F0>,
> 'UInt64Col':
> <class tables.IsDescription.UInt64Col at 0x0104CF30>, 'UInt8Atom':
> <class tables
> .VLArray.UInt8Atom at 0x01054300>, 'myrows': [Traceback (most recent
> call last):
>
>   File "C:\ShowLeak.py", line 53, in ?
>     print gc.get_objects()
>   File "hdf5Extension.pyx", line 2516, in hdf5Extension.Row.__repr__
>   File "hdf5Extension.pyx", line 2510, in hdf5Extension.Row.__str__
> TypeError: unsubscriptable object
>
> I then tried calling gc.collect() before looking at gc.get_objects(),
> and what flashes before my eyes *is* different. It looks like
> gc.collect() succeeds in collecting all the "array" objects, but a whole
> lot of what seem like rows are still in the snipped output before the
> error above occurs again.
>
> One more thing, I ran my script on my Debian (unstable) system after
> installing the latest python-tables package today, and it's up to 67MB
> just in the past hour or so, so we are consistent across Win32, Cygwin,
> and Linux...
>
> Thanks for your attention,
>
> Norm
>
> PS: Please let me know if you'd prefer I just post to the list.

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"