[Pytables-users] Re: Dimension Scales and Pytables

[Francesc A. <fa...@ca...>]

Hi R=E9mi,

I'm out for a workshop on HDF in San Francisco. When I go back to
Spain (next week), I'll check your patches more in detail and will
give you my feedback shortly after. Maybe Ivan would like to have a
look at it meanwhile.

Please, remember to attach just your patches, not the entire code.
Also, remember that you should not send attachements bigger than 40 KB
to this list. If you want to make your patches publicly available, you
can submit it to SF:

http://sourceforge.net/tracker/?group_id=3D63486&atid=3D504146

Thank you very much for your contribution!

=46rancesc

A Dilluns 28 Novembre 2005 18:40, rem...@gm... va escriure:
> > Hi,
> > It's been quite a long time since I sent you last news about Dimension
> > Scales. Don't worry, I have not stayed iddle all this time long.
> > I join to this mail an archive of what I've done so far. I've only put =
in
> > this archive (as you may guess) the files from PyTables that I have
> > modified.
> > I shall precise that the files I've modified are the ones from PyTables
> > v1.2 (I've started to work with a release candidate, and then I shifted=
 -
> > without any problems - to the lattest release).
> > If you encounter any problem in trying to make Dimension Scales work,
> > feel free to ask me.
> >
> > Here is a link for the specification for Dimension Scales, from the HDF5
> > team. It 's not fully up to date, but to my mind it's interesting (it c=
an
> > help you understand the "philosophy" of Hdf5 Dimension Scales).
>
> -> http://hdf.ncsa.uiuc.edu/RFC/H5DimScales/
>
> Here is a summary of what  I've done :
> > - I've added the functions of the DS Api into the "hdf5Extension.pyx",
> > and then created new methods for the class hdf5Extension.Array which use
> > these functions. That was rather easy.
> >
> > - Then I've added new methods for the class Array of tables.Array, which
> > call these (inherited) methods and add exception handling stuff.
> > Then I realized that an exception would always be raised when I try to
> > open a hdf5 file which contains a Dimension Scale.
> > Actually the hdf5 class name didn't correspond to the '_c_classId' of t=
he
> > PyTables Python class (it's badly explained but I hope you'll get the
> > idea).
> >
> > - So  I created a new class on the Python side, with a new '_c_classId'=
 :
> > 'DIMENSION_SCALE'. I had to put this class in the same file as
> > tables.Array (ie "Array.py") to avoid a loop in the Python imports :
> > DimensionScale should inherit from Array, and so should import it, but
> > Array should also import DimensionScale.
> > There I use a little trick. I needed some kind of a "cast" to turn an
> > existing Array into a DimensionScale. The trick merely consists in
> > modifying the attribute __class__ . Simple and pretty efficient. It
> > doesn't sound very clean, but I haven't found a way to work around my
> > problem differently.
> >
> > - Then I started tackling the special attributes for DS, which turn out
> > to be the biggest difficulty in my work.
> > I first needed to understand the code related to these attribute in hdf5
> > code, and then add it into hdf5Extension.pyx. I was not sure whether I
> > should declare them as system attributes, so in the end I declared them
> > as public ones (but I protect them).
> >
> > Currently these attributes are read from hdf5 whenever one access them.
> > I'll try to improve this in the coming days.
> >
> > Two of these attributes are "reference list". The "DIMENSION_LIST" is an
> > attribute for an Array, it stores references on the Dimension Scales th=
at
> > are attached to it. Conversely the "REFERENCE_LIST" is an attribute for=
 a
> > Dimension Scale, it stores references on the Arrays it is attached to.
> > These references consist of four digit numbers (and an other number to
> > specify the index of the dimension concerned). The problem is that I've
> > not managed in fully retrieving the target of a reference from its
> > reference...
> >
> > I succeeded in retrieving the number of dimension, the number of points,
> > the type, etc...But the name of the target seems to be lost. I've asked
> > one of the hdf5 team about this point, and he didn't know how I can get
> > it back. Actually from the reference one can retrieve an ID, which is n=
ot
> > the ID of the target but a new ID pointing to it. There are some infos -
> > including the name - that can't be obtained from this new ID...
> >
> > But for example, let's assume "(ref, idx)" is a tuple from the
> > "REFERENCE_LIST" of a DImension Scale "ds" ("ds" is attached to the
> > dimension "idx" of the Array  the reference of which is "ref").
> > From "ref" one can retrieve a new ID for this Array (using
> > 'H5Rdereference'). Then from this new ID one can detach "ds" from the
> > Array (but one can't get the name of this Array !). I've taken advantage
> > of this to create two new methods which allow a Dimension Scale to deta=
ch
> > itself from all the Arrays it was attached to and to allow an Array to
> > detach itself from all the Dimension that were attached to it .
> >
> > In the end, it's maybe not that usefull for an user to see these
> > attributes (look at them and you'll understand ;-)). But they were very
> > usefull for me to test my work, and I spent so much time on them...but =
if
> > you want me to hide them, it's not a problem.
> >
> > - If you don't understand all this stuff about attributes, play a little
> > with DS and you'll understand.
> > I've made two small test files (one for the DS and the other focused on
> > the attributes of DS).
> > I was too lazzy to make test files which look like the ones from
> > PyTables, but I could improve it.
> >
> > - Finally, I've thought a little about how I could extend DS to CArrray=
s,
> > EArrays, etc. It doesn't sound very difficult, I've only done it for
> > CArray - but I haven't tested it so far. Actually I've done the easiest
> > part. Indeed I don't know how to deal with the enlargeable dimension.
> > I've some ideas, but I would be interested in knowing yours (and this
> > mail seems long enough to me :-)).
> >
> > If you have any question about this mail or DS  or whatever, feel free =
to
> > ask me !
> > Cheers
> > --
> > Remi

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"