Re: [Pytables-users] Re: [ pytables-Patches-1192700 ] Multi-dimensional HDF5 attributes

[Antonio V. <val...@co...>]

Alle 14:08, marted=EC 10 maggio 2005, Francesc Altet ha scritto:
> Hola Antonio,

Hi

> Sorry for the late response, but I was very busy lately (PyTables 1.0
> was the responsible of that ;)
>
> A Dijous 05 Maig 2005 22:37, Antonio Valentino va escriure:
> > Hi Francesc,
> > the problem on scalar values seems to be fixed now.
> > In the new patch is included your test suite with some changes.
> > Let me know if it is good for you.
>
> Thanks. It seems cleaner to me indeed. Unfortunately, the patch does
> not apply well against PyTables 1.0. Could you make the necessary
> modifications to this?

I don't have my laptop with me in this moment. I'll send you a pach again=
st=20
PyTables 1.0 as soon as I can.

> > Anyway there are still a couple of things, concerning the attributes
> > handling, I don't like too much in this solution (pytables snapshot
> > 20050504 + mdattr patch).
> >
> > - there is no way to set an HDF5 scalar attribute or an attribute wit=
h
> >   shape (1,) having a data type different from H5T_NATIVE_INT or
> >   H5T_NATIVE_DOUBLE
>
> What about using H5T_ARRAY types for keeping multidimensional
> attributes? That way you can distinguish between a regular
> H5T_NATIVE_INT or H5T_NATIVE_DOUBLE and one with shape (1,). That
> would be an elegant solution, IMO.

I never used H5T_ARRAY. It sounds good anyway :) I will think about..
Unfortunately this should be good solution for PyTables but not so good f=
or=20
me. :(
I'm working to a project related to COSMO SkyMED

http://www.aleniaspazio.it/earth_observation_page.aspx?IdProg=3D23

My problem is that i have to deal hdf5 data that have a fixed structure=20
(specified in a "product specification document"). For each node and leaf=
 it=20
is stated the number of attributes and their name, shape and datatype (an=
d=20
none of them is H5T_ARRAY :((( ).
With the current mdattr-patch (together with another pach for chunked lay=
out=20
datasets that I will submit soon) I can "import" this kind of data in the=
=20
PyTables format.
I still have some problems for the "export" operation.

I think that having to import/export ganeric HDF5 files is a situation no=
t so=20
unusual and I would like a lot that PyTables could have some mechanism to=
=20
allow that.
My doubt is that using exclusively H5T_ARRAY to manage multidimentional=20
attributes could make more difficult the managing of generic HDF5 files.
What is your opinion about?

> > - numeric HDF5 attributes with shape () or (1,) are always "imported"
> >   as python int or python float (the original data type and sign are
> >   lost)
>
> You can prevent this from happening if you use H5T_ARRAY types.
>
> > - scalar attributes are stored as HDF5 attributes with shape (1,)
> >   and not as HDF5 attributes with shape ()
>
> Again, H5T_ARRAY types can save the day. Then scalar attributes can
> still be saved as regular attributes with shape (1,).
>
> > I could try to implement a couple of additional methods for the
> > AttributeSet class that could be used as workaround for the first
> > two points:
> >
> > =09AttributeSet.setNAAttribute(self, name, value)
> > =09AttributeSet.getNAAttribute(self, name)
> >
> > could be used in alternative to
> >
> > =09node._v_attrs.name =3D value
> > =09value =3D node._v_attrs.name
> >
> > to force pytables to preserve datatype and shape into HDF5 numerical
> > attributes. The mapping would be
> >
> > numarray.Int8      : H5T_NATIVE_CHAR
> > numarray.Int16     : H5T_NATIVE_SHORT
> > numarray.Int32     : H5T_NATIVE_INT
> > numarray.Int64     : H5T_NATIVE_LLONG
> > numarray.UInt8     : H5T_NATIVE_UCHAR
> > numarray.UInt16    : H5T_NATIVE_USHORT
> > numarray.UInt32    : H5T_NATIVE_UINT
> > numarray.UInt64    : H5T_NATIVE_ULLONG
> > numarray.Float32   : H5T_NATIVE_FLOAT
> > numarray.Float64   : H5T_NATIVE_DOUBLE
> >
> > This is but an elegant solution in my opinion.
>
> H5T_ARRAY would be more elegant.
>
> > In alternative one could think to completely change the attribute
> > handling mechanism and store all python objects including int and
> > float as HDF5 strings using cpickle, and map only numarray objects
> > on numerical HDF5 attributes.
> > Of course this would break the backward compatibility but, in my
> > opinion, will make the attribute handling mechanism more clear
> > (also the relative pytables code will result a lot simpler).
>
> Did I mentioned the H5T_ARRAY solution? ;)
>
> > P.S. excuse me for my bad english.
>
> Don't worry, mine is not any better actually!.
>
> Cheers,

ciao

--=20
Antonio Valentino
INNOVA - Consorzio per l'Informatica e la Telematica
via della Scienza - Zona Paip I
75100 Matera (MT) Italy