pytables-users — PyTables users discussion list

Re: [Pytables-users] performance question ( with source )

[Francesc A. <fa...@ca...>]

A Dilluns 26 Setembre 2005 16:03, Stefan Kuzminski va escriure:
> The point here is to use the column operators and push the loops into
> numarray, makes sense.  So is there help for a merge situation?
> Suppose I have multiple input tables that I am merging into a single
> output table.  Ideally I would grab the TableExtension.Row from the
> input tables and append this to the output table, but I get this
> error..
>
> ValueError: rows parameter cannot be converted into a recarray object
> compliant with tabl\
> e '/sorted (Table(0,), shuffle, lzo(1)) '''. The error was: <len() of
> unsized object>

I recognize that allowing the feeding of Tables with Row objects would
be a good thing. Perhaps in a future (I think it would be just a
matter of adding a __len__() method to Row), although as you pointed
out, it is not clear that this is going to be very efficient.

> So I guess I can't take the row object from one table and append it to
> another?  It seems like the merge or join code is going to run slow no
> matter what, because it is going to pull arbitrarily ordered records
> from N Record arrays into a new single array, I don't see support for
> that kind of operation in numarray.

Well, you can always do the merge in an efficient way by reading
blocks of your source tables and write them to destination, can't you?

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] performance question ( with source )

[Stefan K. <pon...@ya...>]

a followup..

The point here is to use the column operators and push the loops into
numarray, makes sense.  So is there help for a merge situation? 
Suppose I have multiple input tables that I am merging into a single
output table.  Ideally I would grab the TableExtension.Row from the
input tables and append this to the output table, but I get this
error..

ValueError: rows parameter cannot be converted into a recarray object
compliant with tabl\
e '/sorted (Table(0,), shuffle, lzo(1)) '''. The error was: <len() of
unsized object>

So I guess I can't take the row object from one table and append it to
another?  It seems like the merge or join code is going to run slow no
matter what, because it is going to pull arbitrarily ordered records
from N Record arrays into a new single array, I don't see support for
that kind of operation in numarray.

thanks,
Stefan

--- Francesc Altet <fa...@ca...> wrote:

> El dg 25 de 09 del 2005 a les 07:53 -0700, en/na Stefan Kuzminski va
> escriure:
> > I also tried just indexing the whole RecArray,
> > i.e.
> > 
> > cur_rows = cur_rows[new_order]
> > 
> > but that is slow as well, I'm guessing that is beacuse it's a
> Record
> > structure that Numarray is holding in a collection of seperate
> vectors
> > so it has to jump around to get and set the values for each row (
> > probably creating a Record object along the way )..?
> 
> Yes, that's correct. RecArray objects are built using NumArray
> columns
> so this is why column-wise operations are much faster than row-wise.
> 
> > Anyway the per column recipe seems to work well.
> 
> Glad that this helped :-)
> 
> -- 
> >0,0<   Francesc Altet     http://www.carabos.com/
> V   V   Cárabos Coop. V.   Enjoy Data
>  "-"
> 
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by:
> Tame your development challenges with Apache's Geronimo App Server.
> Download it for free - -and be entered to win a 42" plasma tv or your
> very
> own Sony(tm)PSP.  Click here to play:
> http://sourceforge.net/geronimo.php
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> 



		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

[Pytables-users] =?windows-1252?Q?Re=3A_=5BPytables-users=5D_Re=3A_=5BPytab?= =?windows-1252?Q?les-users=5D_How_to_create_hdf5_elements_wi?= =?windows-1252?Q?th_=E9=2C_=E8=2C_=E0=2C_=2E=2E=2E_caracters?=

[Ivan V. i B. <iv...@ca...>]

En/na phi...@ho... ha escrit::

> Thanks a lot for your answer.
> When i try to view the hdf5 file with hdfView, the element containing
> the accented caracters is not show in the treeview of the file elements=
=2E

Since you can see the node in PyTables, that seems a limitation of
hdfview.  Maybe you should contact the authors to see if it is a bug.

En/na phi...@ho... ha escrit::

> Furthermore, when i use the python shell, effectively, i can access the=

> node. Nevertheless, it is hard to find the caracter which code for '=E9=
'
> in pytable encoding.
>=20
> Here is an example:
>=20
>>>> import tables
>>>> h =3D tables.openFile("bureau\\tutorial545.h5")
>>>> print h
> bureau\tutorial545.h5 (File) 'Test file'
> Last modif.: 'Tue Sep 13 11:41:40 2005'
> Object Tree:
> / (RootGroup) 'Test file'
> /readout (Table(10L,)) 'Readout example'
> /d=DAtecteur (Group) 'D\xe9tector information'
> /d=DAtecteur/rendement (Array(10L,)) ''
>=20
>>>> g =3D h.getNode("/d=E9tecteur")
> Traceback (most recent call last):
>  File "<stdin>", line 1, in ?
>  File "C:\Python23\Lib\site-packages\tables\File.py", line 738, in getN=
ode
>    raise NoSuchNodeError(where)
> tables.exceptions.NoSuchNodeError: /d=E9tecteur
>=20
>>>> g =3D h.getNode("/d=DAtecteur")
>>>> print g
> /d=DAtecteur (Group) 'D\xe9tector information'
>=20
>>>> g =3D h.getNode("/d\xe9tecteur")
>>>> print g
> /d=DAtecteur (Group) 'D\xe9tector information'
>=20
> Is it same for you?

PyTables makes no additional encoding or transformation on the provided
names for their nodes (apart from using ``trMap``).  It seems to me that
the file has been created in a ISO-8859-1 environment (maybe the ``-*-
coding...`` or ``setdefaultencoding()`` thingies had to do with that),
but you are now using an environment with a different encoding.  Running
the following statement should show you the escaped versions of the
names you are using::

>>> print [n._v_name for n in h.root]

You can always use the escaped strings (as you did) on any platform.
Hope it helps,

::

	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
	       C=E1rabos Coop. V.  V  V   Enjoy Data
	                          ""

Re: [Pytables-users] performance question ( with source )

[Francesc A. <fa...@ca...>]

El dg 25 de 09 del 2005 a les 07:53 -0700, en/na Stefan Kuzminski va
escriure:
> I also tried just indexing the whole RecArray,
> i.e.
>=20
> cur_rows =3D cur_rows[new_order]
>=20
> but that is slow as well, I'm guessing that is beacuse it's a Record
> structure that Numarray is holding in a collection of seperate vectors
> so it has to jump around to get and set the values for each row (
> probably creating a Record object along the way )..?

Yes, that's correct. RecArray objects are built using NumArray columns
so this is why column-wise operations are much faster than row-wise.

> Anyway the per column recipe seems to work well.

Glad that this helped :-)

--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=E1rabos Coop. V.   Enjoy Data
 "-"

Re: [Pytables-users] performance question ( with source )

[Stefan K. <pon...@ya...>]

Thanks, that is indeed much faster ( about 3 secs vs almost 18 on my
slow test below ).

I had avoided the columnwise operations because it seemed like that
would slow down as the number of columns increased.  And since the
'new_order' index list is the same for all the columns being accessed,
a row-wise approach would be faster.  But I guess this way the per-row
loop is always in C.  I also tried just indexing the whole RecArray,
i.e.

cur_rows = cur_rows[new_order]

but that is slow as well, I'm guessing that is beacuse it's a Record
structure that Numarray is holding in a collection of seperate vectors
so it has to jump around to get and set the values for each row (
probably creating a Record object along the way )..?  Anyway the per
column recipe seems to work well.

thanks again,
Stefan

--- Francesc Altet <fa...@ca...> wrote:

> El dv 23 de 09 del 2005 a les 12:22 -0700, en/na Stefan Kuzminski va
> escriure:
> > Sorry for the 2 posts, but I thought an example would help..
> 
> Sure, an example is always much better!
> 
> >     # if we dump out the original rows, it's blazing fast, < second
> >     new_table1.append( src_rows )
> >     new_table1.flush()
> >     
> >     # but this operation, with the same number of rows, but in 
> >     # a different order and in a list, takes almost 20 times as
> long!
> >     # is there a better way?
> >     new_table2.append( tmp )
> >     new_table2.flush()
> 
> Yes, that's normal because in your second approach you have converted
> your NestedRecArray object into a python list of records and then
> when
> you save this list, pytables internally convert it again into a
> NestedRecArray. This roundtrip makes the code very slow.
> 
> The trick to get maximum performance is like everything else when one
> is
> working with numbers in Python: try to work as much as possible with
> specialized numerical packages. In this case, NestedRecArray is a
> class
> derived from RecArray (from numarray package). So, for example, if
> you
> want to sort your table by, say, column 'var2', then you can do
> something like:
> 
>     new_order = src_rows.field('var2').argsort()
>     for colname in src_rows._names:
>         src_rows.field(colname)[:] =
> src_rows.field(colname)[new_order]
> 
> (look into numarray manual to know what is happening under the hood)
> 
> and then save the new src_rows NestedRecArray to get fast output
> performance back.
> 
> Cheers,
> 
> -- 
> >0,0<   Francesc Altet     http://www.carabos.com/
> V   V   Cárabos Coop. V.   Enjoy Data
>  "-"
> 
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by:
> Tame your development challenges with Apache's Geronimo App Server.
> Download it for free - -and be entered to win a 42" plasma tv or your
> very
> own Sony(tm)PSP.  Click here to play:
> http://sourceforge.net/geronimo.php
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> 



		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: [Pytables-users] performance question ( with source )

[Francesc A. <fa...@ca...>]

El dv 23 de 09 del 2005 a les 12:22 -0700, en/na Stefan Kuzminski va
escriure:
> Sorry for the 2 posts, but I thought an example would help..

Sure, an example is always much better!

>     # if we dump out the original rows, it's blazing fast, < second
>     new_table1.append( src_rows )
>     new_table1.flush()
>    =20
>     # but this operation, with the same number of rows, but in=20
>     # a different order and in a list, takes almost 20 times as long!
>     # is there a better way?
>     new_table2.append( tmp )
>     new_table2.flush()

Yes, that's normal because in your second approach you have converted
your NestedRecArray object into a python list of records and then when
you save this list, pytables internally convert it again into a
NestedRecArray. This roundtrip makes the code very slow.

The trick to get maximum performance is like everything else when one is
working with numbers in Python: try to work as much as possible with
specialized numerical packages. In this case, NestedRecArray is a class
derived from RecArray (from numarray package). So, for example, if you
want to sort your table by, say, column 'var2', then you can do
something like:

    new_order =3D src_rows.field('var2').argsort()
    for colname in src_rows._names:
        src_rows.field(colname)[:] =3D src_rows.field(colname)[new_order]

(look into numarray manual to know what is happening under the hood)

and then save the new src_rows NestedRecArray to get fast output
performance back.

Cheers,

--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=E1rabos Coop. V.   Enjoy Data
 "-"

Re: [Pytables-users] performance question ( with source )

[Stefan K. <pon...@ya...>]

Sorry for the 2 posts, but I thought an example would help..
    # assume that
    # src_table is source table, filled with data
    # new_table1 and 2 are empty tables, the destination

    # get the rows
    src_rows = src_table.read( start=0, stop=len(src_table))

    # do 'something' to the rows, in reality sort them, but here
    # just put them in a list
    tmp = []
    for row in src_rows:
        tmp.append( row )

    # if we dump out the original rows, it's blazing fast, < second
    new_table1.append( src_rows )
    new_table1.flush()
    
    # but this operation, with the same number of rows, but in 
    # a different order and in a list, takes almost 20 times as long!
    # is there a better way?
    new_table2.append( tmp )
    new_table2.flush()

thanks again,
Stefan

--- Stefan Kuzminski <pon...@ya...> wrote:

> I need an 'external sort' utility for large pytables, since I haven't
> been able to find one I wrote one, but performance is a problem. 
> Specifically if I build up a list of NestedRecords and then append
> them
> to a new table, the write time seems very slow.  If I even use a
> NestedRecArray as a buffer there seems to be the same slowdown just
> setting the new rows in NestedRecArray buffer.  Any pointers would be
> appreciated..
> 
> thanks,
> Stefan
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by:
> Tame your development challenges with Apache's Geronimo App Server. 
> Download it for free - -and be entered to win a 42" plasma tv or your
> very
> own Sony(tm)PSP.  Click here to play:
> http://sourceforge.net/geronimo.php
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

[Pytables-users] performance question

[Stefan K. <pon...@ya...>]

I need an 'external sort' utility for large pytables, since I haven't
been able to find one I wrote one, but performance is a problem. 
Specifically if I build up a list of NestedRecords and then append them
to a new table, the write time seems very slow.  If I even use a
NestedRecArray as a buffer there seems to be the same slowdown just
setting the new rows in NestedRecArray buffer.  Any pointers would be
appreciated..

thanks,
Stefan

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Re Re: [Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

[Francesc A. <fa...@ca...>]

A Dijous 22 Setembre 2005 10:55, Cyril Giraudon va escriure:
> Well, i'll do my best.
> However, dimension scales well be included in hdf5 with the 1.8 version
> (perhaps a long time).

There are snapshots of the development release (1.7.x branch) in:

ftp://ftp.ncsa.uiuc.edu/HDF/pub/outgoing/hdf5/snapshots

It seems that the current implementation of dimension scales are
included there (hl/src directory).

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: Re Re: [Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

[Cyril G. <cyr...@fr...>]

Well, i'll do my best.
However, dimension scales well be included in hdf5 with the 1.8 version 
(perhaps a long time).

http://hdf.ncsa.uiuc.edu/HDF5/Tutor/h5dimscale.html

Does a CVS version exist ?

Cyril.

Re: [Pytables-users] Reading vlen data created by libhdf5 with pytables

[Francesc A. <fa...@ca...>]

A Dijous 22 Setembre 2005 09:36, Cournapeau David va escriure:
> I updated to 1.2b4 version, and the problem remains the same. Actually,
> if I have data which last dimension is declared as H5S_UNLIMITED, I can
> can read it. If I have an vlen array named cell, in the 'root' group,
> the first time I call root.cell, it does not work and generates this
> error. But playing a bit with it in python shell, I realized that
> calling root.cell a second time gives me the data !

Strange. Anyway, I recommend you to stick with pytables 1.2b4, which
is should be quite stable right now.

> I hope being able to track down the problem (I guess errors in hdf5 api
> use is considered as bug, right ?). Is there a simple way to quickly
> test changes in pyrex code ? I would like to avoid building a new deb
> package each time I modify the pyrex file to find and solve the bug :)
> (sorry if this sounds dumb, I've never used pyrex).

If you are going to modify the Pyrex sources, the best to rebuild the
extensions is to use the distutils:

python setup.py build_ext --inplace

If you are making many modifications and need fast rebuilding, try to
disable optimisation for Python extensions:

=2D Edit /usr/lib/python2.3/config/Makefile (or python2.4)

=2D Substitute:

OPT=3D           -DNDEBUG -g -O3 -Wall -Wstrict-prototypes

by something like:

OPT=3D           -DNDEBUG -g

Also, you don't need to re-install the package. Instead, put the root
directory of your pytables development branch in your PYTHONPATH, and
you are done.

Luck!

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] Reading vlen data created by libhdf5 with pytables

[Cournapeau D. <cou...@at...>]

On Fri, 2005-09-16 at 20:22 +0200, Francesc Altet wrote:

> 
> Uh, which version of PyTables are you using, the yesterday snapshot?
> In that case, please, use better a stable version, like 1.1.1 or
> 1.2b3, available in
> 
> http://www.carabos.com/downloads/pytables/preliminary/
I was using a home made debian package using the sources from 1.1.1 (I
had to build
my own package because my distribution, ubuntu, does not include an
up-to-date pytable
package).

I updated to 1.2b4 version, and the problem remains the same. Actually,
if I have data which last dimension is declared as H5S_UNLIMITED, I can
can read it. If I have an vlen array named cell, in the 'root' group,
the first time I call root.cell, it does not work and generates this
error. But playing a bit with it in python shell, I realized that
calling root.cell a second time gives me the data ! 

I hope being able to track down the problem (I guess errors in hdf5 api
use is considered as bug, right ?). Is there a simple way to quickly
test changes in pyrex code ? I would like to avoid building a new deb
package each time I modify the pyrex file to find and solve the bug :)
(sorry if this sounds dumb, I've never used pyrex).

David

Re: Re Re: [Pytables-users] Suggestions for =?iso-8859-1?q?storing=09mutlidimensionnal_data_in_several_pytables?= =?iso-8859-1?q?_arrays?=

[Francesc A. <fa...@ca...>]

A Dimarts 20 Setembre 2005 14:31, v=E0reu escriure:
> Well, it's right. So, can you tell me briefly how you work.
> I 've seen in the source code  you use  pyrex.
> What is you method ? From the hdf5 sources to pytables sources ?

Yes, PyTables is using Pyrex in order to link with HDF5 functions.

What I'd do in order to implement dimension scales is something like:

=2D Study the dimension scales API from HDF5 site:

http://hdf.ncsa.uiuc.edu/HDF5/hdf5_hl/doc/RM_hdf5ds.html

Do some tests in C, if needed

=2D Study the best API for PyTables: How are you going to link with DS?
Are you going to support several DS per dimension? How indexing with
new scales would be accessible?

=2D Look at the hdf5Extension.Array object and try to attach the DS there.

=2D Try to extend DS to EArray and CArray objects (should be
straightforward)

If you have more time and want to make your patches easily integrated
in PyTables.

=2D Add test units

=2D Add docs

All in all, it seems like a lot of work, but like everything else,
that depends on your priorities :-)

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: Re Re: [Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

[Cyril G. <cyr...@fr...>]

Francesc Altet a =E9crit :

>El dl 19 de 09 del 2005 a les 13:02 +0200, en/na Cyril Giraudon va
>escriure:
> =20
>
>>Francesc Altet a =E9crit :
>>What can i do to help you ?
>>However i haven't the time to study pytables code and write the feature=
.
>>I think i can test the cvs or ...
>>   =20
>>
>
>Well, as I said before, we don't have time neither right now. On the
>other hand, I consider this a *nice* thing to have, so it will
>eventually go into PyTables.
>
>The nice thing about PyTables is that it is a completely open source
>project, so anyone can hack it in case of need (i.e. you don't need to
>wait until someone else do what you are interested in ;).
>
>Cheers,
>
> =20
>
Well, it's right. So, can you tell me briefly how you work.
I 've seen in the source code  you use  pyrex.=20
What is you method ? From the hdf5 sources to pytables sources ?

And if i have time ....

Thank you.

Cyril.

Re: Re Re: [Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

[Francesc A. <fa...@ca...>]

El dl 19 de 09 del 2005 a les 13:02 +0200, en/na Cyril Giraudon va
escriure:
> Francesc Altet a =E9crit :
> What can i do to help you ?
> However i haven't the time to study pytables code and write the feature.
> I think i can test the cvs or ...

Well, as I said before, we don't have time neither right now. On the
other hand, I consider this a *nice* thing to have, so it will
eventually go into PyTables.

The nice thing about PyTables is that it is a completely open source
project, so anyone can hack it in case of need (i.e. you don't need to
wait until someone else do what you are interested in ;).

Cheers,

--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=E1rabos Coop. V.   Enjoy Data
 "-"

Re: Re Re: [Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

[Cyril G. <cyr...@fr...>]

Francesc Altet a =E9crit :

>>I can't wait for this feature in pytables.
>>   =20
>>
>
>Well, you know, you can help on this in several ways.
>
> =20
>
Good answer.

What can i do to help you ?
However i haven't the time to study pytables code and write the feature.
I think i can test the cvs or ...

Cyril.

Re: [Pytables-users] Indexing Arrays: a la numarray

[Francesc A. <fa...@ca...>]

El dv 16 de 09 del 2005 a les 20:22 -0400, en/na travlr va escriure:
> Here is the diff patch for pytables-1.1.1

Thanks. I'll see to integrate it shortly.

> BTW... Numeric question if someone knows..
> >>> isinstance(arr, array)
> Traceback (most recent call last):
>   File "<input>", line 1, in ?
> NameError: name 'array' is not defined
>=20
> >>> isinstance(arr, type(Numeric.array([])))
> True
>=20
> Why is type(arr) reported as 'array', and yet isinstance() doesn't recogn=
ize it?

The next seems to work:

In [7]:arr=3DNumeric.arange(2)

In [8]:isinstance(arr, Numeric.ArrayType)
Out[8]:True

I'll have to update PyTables accordingly ;-)

Cheers,

--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=E1rabos Coop. V.   Enjoy Data
 "-"

Re: [Pytables-users] Indexing Arrays: a la numarray

[travlr <vel...@gm...>]

Here is the diff patch for pytables-1.1.1

The utility is the same as what I provided above for 1.0 and 1.1, but
I've also included  using either of the following object types:

Numarray array
Numeric array
List
Tuple

...(to be used as described in my prior post).

I tried once again (unsuccessfully) to enable pytables.Array[key]
functionality (as well as attempting to enable Table/Column/Array[key]
setting functionality) for non-sequential index-array [keys].


BTW... Numeric question if someone knows..

Python 2.4.1 (#1, Jul 29 2005, 03:50:01)=20
[GCC 3.4.4 (Gentoo 3.4.4, ssp-3.4.4-1.0, pie-8.7.8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import Numeric=20
>>> arr =3D Numeric.array([1,2,3,4,5])
>>> type(arr)
<type 'array'>

>>> isinstance(arr, array)
Traceback (most recent call last):
  File "<input>", line 1, in ?
NameError: name 'array' is not defined

>>> isinstance(arr, type(Numeric.array([])))
True

Why is type(arr) reported as 'array', and yet isinstance() doesn't recogniz=
e it?

Re: [Pytables-users] Reading vlen data created by libhdf5 with pytables

[Francesc A. <fa...@ca...>]

A Divendres 16 Setembre 2005 04:57, Cournapeau David va escriure:
> The last dimension is declared H5S_UNLIMITED in pytables, which is
> natural for the purpose of pytable. I haven't studied the architecture

Yes, this is a possible source of problems with PyTables.

> of pytables yet, but would it be difficult to support these kind of
> data ? If it is not too difficult, I would actually be interested in
> doing the support myself.

Well, this is in fact a problem very similar to what lead Antonio
Valentino to come up with the CArray object. EArray also enforces an
enlargeable dimension in array, but he wanted a chunked array (in
order to be able to compress it) with all dimensions to be *fixed*,
and this is what CArray is.

If you want to have a try, please, check carefully the job of Antonio
and compare how the logic of CArray and EArray objects differs. Most
of his approach should be useful to you. Also, I guess we should
choose a new name for the new object. Perhaps CVLArray applies, but it
seems to me a bit complicated. Perhaps a new parameter for VLArray
would be enough (and better).

> To see if the problems goes away by declaring the dimension as
> H5S_UNLIMITED, I modified my C program, but the new file is impossible
> to open using pytables: there are some hdf5 library related problems.
> The error messages are the following:
[trimmed]

Uh, which version of PyTables are you using, the yesterday snapshot?
In that case, please, use better a stable version, like 1.1.1 or
1.2b3, available in

http://www.carabos.com/downloads/pytables/preliminary/

> Thank you for your attention,

Keep us informed on your progress. Good luck!

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] Reading vlen data created by libhdf5 with pytables

[Cournapeau D. <cou...@at...>]

Hi there,

	I would like to use pytables to read hdf5 files created by some C
programs I wrote, mainly wrappers to matlab to use hdf5 with matlab.
Using pytables, I was able to read vectors and matrices with pytables.
Unfortunately, I cannot read vlen arrays: the vlen arrays are pretty
simple, rank 1, fixed number of rows, each row is a variable size vector
of double. 

	To see where the problem is coming from, I created a similar ragged
array with pytables, and compared the pytable generated file with my
file. Not considering pytable metadata, I can see that there is only one
difference between the data I wrote and the pytable file:

pytable:

DATASET "vlarray1" {
      DATATYPE  H5T_VLEN { H5T_IEEE_F64LE}
      DATASPACE  SIMPLE { ( 4 ) / ( H5S_UNLIMITED ) }
      DATA {
      (0): (5, 6), (5, 6, 7), (5, 6, 9, 8), (5, 6, 9, 10, 12)
      }
}

The pytable metadata were omitted.

my file:

 DATASET "cell" {
      DATATYPE  H5T_VLEN { H5T_IEEE_F64LE}
      DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
      DATA {
      (0): (-0.432565, -1.66558, 0.125332), (0.287676, -1.14647)
      }
   }

The last dimension is declared H5S_UNLIMITED in pytables, which is
natural for the purpose of pytable. I haven't studied the architecture
of pytables yet, but would it be difficult to support these kind of
data ? If it is not too difficult, I would actually be interested in
doing the support myself.

To see if the problems goes away by declaring the dimension as
H5S_UNLIMITED, I modified my C program, but the new file is impossible
to open using pytables: there are some hdf5 library related problems.
The error messages are the following:

/usr/lib/python2.4/site-packages/tables/File.py:223: UserWarning: file
``unlimited_mex.h5`` exists and it is an HDF5 file, but it does not have
a PyTables format; I will try to do my best to guess what's there using
HDF5 metadata
  warnings.warn("""\
HDF5-DIAG: Error detected in HDF5 library version: 1.6.4 thread
3085332608.  Back trace follows.
  #000: ../../../src/H5A.c line 457 in H5Aopen_name(): attribute not
found
    major(18): Attribute layer
    minor(05): Bad value
  #001: ../../../src/H5A.c line 404 in H5A_get_index(): attribute not
found
    major(18): Attribute layer
    minor(48): Object not found
/usr/lib/python2.4/site-packages/tables/Group.py:252: UserWarning:
problems loading leaf ``/cell``: flavor of type 'Float64' must be one of
the "NumArray", "Numeric", "Tuple" or "List" values, and you tried to
set it to "unknown".
; it will become an ``UnImplemented`` node
  warnings.warn("""\
Traceback (most recent call last):
  File "./readhd5.py", line 9, in ?
    fileh = openFile("unlimited_mex.h5", mode = "r")
  File "/usr/lib/python2.4/site-packages/tables/File.py", line 231, in
openFile
    return File(path, mode, title, new, trMap, rootUEP, isPTFile,
filters)
  File "/usr/lib/python2.4/site-packages/tables/File.py", line 411, in
__init__
    self.root = self.__getRootGroup(rootUEP)
  File "/usr/lib/python2.4/site-packages/tables/File.py", line 462, in
__getRootGroup
    rootGroup = RootGroup(self, '/', rootUEP, new=self._v_new)
  File "/usr/lib/python2.4/site-packages/tables/Group.py", line 855, in
__init__
    self._g_openFile()
  File "/usr/lib/python2.4/site-packages/tables/Group.py", line 257, in
_g_openFile
    objleaf._g_putUnder(objgroup, name)
  File "/usr/lib/python2.4/site-packages/tables/Leaf.py", line 223, in
_g_putUnder
    super(Leaf, self)._g_putUnder(parent, name)
  File "/usr/lib/python2.4/site-packages/tables/Node.py", line 693, in
_g_putUnder
    parent._g_refNode(self, ptname, validate)
  File "/usr/lib/python2.4/site-packages/tables/Group.py", line 327, in
_g_refNode
    raise NodeError(
tables.exceptions.NodeError: group ``/`` already has a child node named
``cell``

I would appreciate any hint to be able to read

Thank you for your attention,

David.

P.S: I didn't find a link to the mailing list from the pytables webpage.
Have I just missed the link ? Otherwise, it may be useful to put it,
because finding the mailing list through sourceforge is a bit ackward...

Re: [Pytables-users] ANN: PyTables 1.1.1 released

[Francesc A. <fa...@ca...>]

Hi List,

Unfortunately, I've made a mistake and forgot to fix an issue in the
PyTables 1.1.1 released yesterday. I've just uploaded new versions of
files for 1.1.1 in the usual PyTables file repository:

http://sourceforge.net/project/showfiles.php?group_id=3D63486

The problem was related with persistence of properties of indexes for
tables on disk. If you don't have indexed tables, then you are not
affected by this and you don't need to upgrade. Have in mind that if
you do not upgrade, some tests in heavy mode will not pass. This is
not grave anyway.

The new packages has been extensively checked (Linux on Intel Pentium,
Win on Intel Pentium, Linux on Intel Itanium, FreeBSD on AMD Opteron
and MacOSX on IBM PowerPC, all in --heavy mode), so chances are that
this time the 1.1.1 is as stable as it is supposed to be.

Sorry for the inconvenience!

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] Testing PyTables 1.1.1 under Tiger.

[Ivan V. i B. <iv...@ca...>]

Hi all,

We have been reported that heavy tests fail under MacOS X because of the
UCL tests.  We suspect that the problem lies either in the very UCL
library or in the compiler provided with Xcode 1.5 (GCC 3.3).  However,
we are not able to test PyTables under a newer version of Xcode or MacOS
X than 1.5 and Panther, respectively.

Could someone please run the heavy tests of PyTables 1.1.1 under a newer
Xcode or under Tiger?  This would let us know where the problem comes
from.  Please remember to enable UCL using the ``--force-ucl`` option of
``setup.py``.

If someone needs a list of steps to follow to build PyTables under MacOS
X, please tell me.  Anyway, it is quite straightforward.

Thanks a lot!

::

	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
	       C=C3=A1rabos Coop. V.  V  V   Enjoy Data
	                          ""

[Pytables-users] =?windows-1252?Q?Re=3A_=5BPytables-users=5D_How_to_creat?= =?windows-1252?Q?e_hdf5_elements_with_=E9=2C_=E8=2C_=E0=2C_?= =?windows-1252?Q?=2E=2E=2E_caracters?=

[<phi...@ho...>]

Ivan Vilata i Balaguer wrote:

>En/na phi...@ho... ha escrit::
>
> =20
>
>>I have a silly question about creating hdf5 elements with =E9, =E8, =E0
>>caracters.
>>For the moment, the file is created successfully but the element with
>>accented caracters isn't apperaing in the file
>>group =3D h5file.createGroup("/", 'd=E9t=E9cteur', 'D=E9tecteur informa=
tion')
>>[...]
>>   =20
>>
>
>I have had no problem in creating such a node under PyTables 1.1.  Of
>course, a warning is issued because it is not a valid Python identifier,
>so you will not be able to use natural naming (tab completion) on it,
>but you can still use ``h5file.getNode()`` to access the node.  In fact,
>``print repr(h5file)`` would show the node as a child of the root group.
>Could you please be more specific when saying =93isn't appearing in the
>file=94?  Thanks!
>
>import disclaimer
>
>::
>
>	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
>	       C=E1rabos Coop. V.  V  V   Enjoy Data
>	                          ""
>
> =20
>
Thanks a lot for your answer.
When i try to view the hdf5 file with hdfView, the element containing=20
the accented caracters is not show in the treeview of the file elements.

Furthermore, when i use the python shell, effectively, i can access the=20
node. Nevertheless, it is hard to find the caracter which code for '=E9'=20
in pytable encoding.

Here is an example:

 >>> import tables
 >>> h =3D tables.openFile("bureau\\tutorial545.h5")
 >>> print h
bureau\tutorial545.h5 (File) 'Test file'
Last modif.: 'Tue Sep 13 11:41:40 2005'
Object Tree:
/ (RootGroup) 'Test file'
/readout (Table(10L,)) 'Readout example'
/d=DAtecteur (Group) 'D\xe9tector information'
/d=DAtecteur/rendement (Array(10L,)) ''

 >>> g =3D h.getNode("/d=E9tecteur")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\Python23\Lib\site-packages\tables\File.py", line 738, in getNo=
de
    raise NoSuchNodeError(where)
tables.exceptions.NoSuchNodeError: /d=E9tecteur

 >>> g =3D h.getNode("/d=DAtecteur")
 >>> print g
/d=DAtecteur (Group) 'D\xe9tector information'

 >>> g =3D h.getNode("/d\xe9tecteur")
 >>> print g
/d=DAtecteur (Group) 'D\xe9tector information'

Is it same for you?
Thanks,
Philippe Collet

Re: Re Re: [Pytables-users] Suggestions for storing mutlidimensionnal data in several pytables arrays

[Francesc A. <fa...@ca...>]

A Tuesday 13 September 2005 11:08, Cyril Giraudon va escriure:
> Francesc Altet a =E9crit :
> >Yes. I've done some research and found this:
> >
> >http://hdf.ncsa.uiuc.edu/RFC/H5DimScales/
> >
> >So, it seems that what you are asking for are Dimension Scales. Well,
> >the good news is that the HDF group is working on implementing this.
> >Once their job would be done, pytables can use them in a much more
> >portable way (i.e. compatible with all HDF5 apps).
>
> Yes, it seems to be the solution.
> Apparently, there's also an API :
> http://hdf.ncsa.uiuc.edu/HDF5/hdf5_hl/doc/ and perhaps stable.
>
> You know, I agre woth you,  "tables" could be (is) the right way to work
> with linked arrays even with structured grid .

No, perhaps tables is not the best way to solve this problem
(dimension scales seems a much better approach), but they can be a
solution in the interim.

> But, you know,  people of the structured grids world don't use to work
> like this.
> Do they all accept to change their mind ?

Well, these people are supposed to be very experienced in their work,
so I'm not going to change their mind ;)

> I can't wait for this feature in pytables.

Well, you know, you can help on this in several ways.

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] =?windows-1252?Q?Re=3A_=5BPytables-users=5D_How_to_creat?= =?windows-1252?Q?e_hdf5_elements_with_=E9=2C_=E8=2C_=E0=2C_?= =?windows-1252?Q?=2E=2E=2E_caracters?=

[Ivan V. i B. <iv...@ca...>]

En/na phi...@ho... ha escrit::

> I have a silly question about creating hdf5 elements with =E9, =E8, =E0=

> caracters.
> For the moment, the file is created successfully but the element with
> accented caracters isn't apperaing in the file
> group =3D h5file.createGroup("/", 'd=E9t=E9ctor', 'D=E9tector informati=
on')
> [...]

I have had no problem in creating such a node under PyTables 1.1.  Of
course, a warning is issued because it is not a valid Python identifier,
so you will not be able to use natural naming (tab completion) on it,
but you can still use ``h5file.getNode()`` to access the node.  In fact,
``print repr(h5file)`` would show the node as a child of the root group.
Could you please be more specific when saying =93isn't appearing in the
file=94?  Thanks!

import disclaimer

::

	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
	       C=E1rabos Coop. V.  V  V   Enjoy Data
	                          ""