Re: [Pytables-users] MAX_CHILDS_IN_GROUP

[Francesc A. <fa...@op...>]

Hi Dave,

You are lucky because I was already doing that when I received your request.
I've just uploaded the patches you are interested in to the CVS. It seems
that all the unitests passes, so, hopefully, it won't break any existing
code. Please, feel free to checkout the latest CVS version and tell me if it
works for you. Note that a UserWarning is still issued if the user tries to
create more than 4096 childs on a group or 4096 attributes on a node. If you
don't like that, just filter it using the warnings.filterwarings() method.

Also, I would be interested in your feedback when using large amounts of
childs hanging from a group (or large amounts of objects in the whole object
tree, in general). My experience is that when exceeding 8000 childs or so in
groups, the underlying HDF5 starts to show signs of *very slow* data access.
In addition, the pytables object tree takes *lots* of memory. If you have a
machine plenty of memory, that should be not a problem, but I'm mainly
worried with the slow data access. Please, tell me if you are experiencing
such a thing.

By the way, in the CVS you will find the next improvements apart of the
child limit removal:

- A memory leak was located (after a hunting of two weeks!) and fixed (!).
It was occurring during attribute writes. As pytables uses lots of
attributes for storing metadata, that fix is a good relieve on its memory
hungriness. With that, no known memory leaks remains on pytables. However,
I've tracked down some leaks on the underlying numarray library; I've
reported that to Todd Miller (it seems to exist such a leak from numarray
0.5 on), and he should be working on fixing it.

- The opening of files with lots of nodes has been optimized between a
factor 2 and 3. For example, a file with 10 groups and 3000 tables that
takes 9.3 seconds to open in 0.7.1, now takes only 2.8 seconds. 

- The Table.read() method has been refactored and optimized and some parts
of its code has been moved to Pyrex. In particular, in the special case of
step=1, up to a factor 5 of speedup (reaching 160 MB/s on a Pentium4 @ 2
GHz) when reading table contents can be achieved now.

Enjoy!,

A Dimarts 16 Setembre 2003 19:39, DPGrote va escriure:
> I've come across the MAX_CHILDS_IN_GROUP limit and am wondering if this
> limit is there for some fundemental reason. I notice that it only
> affects the size of temporary arrays in Giterate in utils.c. Can that
> routine be rewritten to dynamically allocate those temporary arrays to
> whatever the size of num_obj is? This would remove that limit
> altogether. The is also true for the MAX_ATTRS_IN_NODE limit - it could
> be removed in the same way.
>    Thanks,
>      Dave

-- 
Francesc Alted