Re: [Pytables-users] [Fwd: More than 4096 tables in a group]

[Francesc A. <fa...@ca...>]

El dl 11 de 09 del 2006 a les 20:02 +0200, en/na Francesc Altet va
escriure:
> > -------- Missatge reenviat --------
> > De: ks...@sc...
> > Per a: pyt...@li...
> > Assumpte: More than 4096 tables in a group
> > Data: Mon, 11 Sep 2006 16:34:12 +0200
> >=20
> > Hello,
> >=20
> > I'm currently building a storage backend for so-called interest regions=
=20
> > in images. I have 90,000 images.
> > I started out by having one big group with one table per image (since=20
> > the number of regions per image is variable, this seemed like a nice=20
> > solution to me). All these tables have the exact same structure.
> > Now, I received a PerformanceWarning that having more than 4096 'image=20
> > tables' in a group will increase memory usage.
> >=20
> > My question: should should I keep this big group? Or should I split it=20
> > into groups (which is possible in a natural way) of 200-500 images?=20
> > Which is more memory efficient? Does this make a big difference in=20
> > lookup performance? Or in 'creation/write' performance?

To say the truth, this has been a warning that existed in pre-1.2
versions when all the *metainformation* about nodes was loaded
completely in-memory. After the introduction of a cache for nodes in
1.2, we decided to keep this warning because we felt that keeping too
much leaves on a single tree was not a good way of structuring datasets.

Having said that, I'm not very sure whether this recommendation is still
valid or not (from the performance point of view). You may want to
experiment with spliting and without spliting (you can avoid the warning
by increasing the MAX_GROUP_WIDTH in tables/constants.py) and decide
which schema is better for you. Also, if you do a lot of browsing on the
object tree, you may want to play as well with changing the value of
NODE_CACHE_SIZE (also in tables/constants.py) before to decide the value
you should stick in.

I'm insterested in your findings, so please, report back your experiences.

--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=C3=A1rabos Coop. V.   Enjoy Data
 "-"