pytables-users — PyTables users discussion list

Re: [Pytables-users] Problem with pytables-1.2 and Row._fillCol on win32

[Francesc A. <fa...@ca...>]

A Divendres 07 Octubre 2005 04:09, Ashley Walsh va escriure:
> The patch didn't work.  I've attached a file with a table that fails
> the test::
>
>      table =3D openFile('field_problem.h5').root.table
>      assert table.cols._f_col('col003')[0] =3D=3D table[0].field('col003')

Sorry, but I'm not able to reproduce your error:

In [2]:table=3Dtables.openFile('/tmp/field_problem.h5').root.table

In [3]:table.cols._f_col('col003')[0] =3D=3D table[0].field('col003')
Out[3]:True

In [4]:table.cols._f_col('col003')[0]
Out[4]:1128178800.0

In [5]:table[0].field('col003')
Out[5]:1128178800.0

using Debian on Intel here:

=2D=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
=2D=3D
PyTables version:  1.2-pre1
HDF5 version:      1.6.4
numarray version:  1.3.3
Zlib version:      1.2.2
LZO version:       1.08 (Jul 12 2002)
UCL version:       1.03 (Jul 20 2004)
BZIP2 version:     1.0.2 (30-Dec-2001)
Python version:    2.3.5 (#2, May  4 2005, 08:51:39)
[GCC 3.3.5 (Debian 1:3.3.5-12)]
Platform:          linux2-i686
Byte-ordering:     little
=2D=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
=2D=3D


Anyone using win32 can reproduce the error? I'm attaching Ashley's
file just if anybody wants to have a try.


> I'm running Tiger (10.4.2 / Darwin-8.2.0).  I've just recompiled and
> rerun the tests on older versions (1.0 and 1.1.1), and they both
> crash with segmentation faults.

Excellent!. Ivan is telling me that in Panther PyTables 1.2 is working
correctly as well, so I'll re-enable UCL for MacOSX for next PyTables
1.2 and remove the flag --force-ucl in setup.py.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] HDF5-DIAG

[Francesc A. <fa...@ca...>]

A Dijous 06 Octubre 2005 17:11, nmi...@jh... va escriure:
> Hi Francesc,
>
> This is how I set up pytables (I'm using version 1.1.1)
>
> I downloaded hdf5-1.6.4 from the NCSA site and configured it normally
> except for disabling shared libraries and enabling static libraries.
> (./configure --prefix=3D$HOME/Desktop/hdf5 --disable-shared --enable-stat=
ic)

I'm afraid that this is the problem. I've recompiled with static HDF5
libraries and I can reproduce it :-/

Sorry, but I'm not sure what is happening here. Anyone on the list can
bring some light?

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] HDF5-DIAG

[<nmi...@jh...>]

Hi Francesc,

This is how I set up pytables (I'm using version 1.1.1)

I downloaded hdf5-1.6.4 from the NCSA site and configured it normally
except for disabling shared libraries and enabling static libraries.
(./configure --prefix=3D$HOME/Desktop/hdf5 --disable-shared --enable-static=
)
Followed by make, make test and make install. The tests all passed.

Then I built pytables with "python setup.py --hdf5=3D$HOME/Desktop/hdf5
build" followed my an install. When I go into the test directory and try
the simplest test (which I think is test_basics.py) I get the output which
I included in the first attachment (since it's so long)

Here's the output for running tutorial1-1.py:

-**-**-**-**-**-**- file creation  -**-**-**-**-**-**-**-
Creating file: tutorial1.h5

-**-**-**-**-**- group and table creation  -**-**-**-**-**-**-**-
Group '/detector' created
HDF5-DIAG: Error detected in HDF5 library version: 1.6.4 thread 0.  Back
trace follows.
  #000: H5D.c line 1072 in H5Dcreate(): not a location ID
    major(01): Function arguments
    minor(03): Inappropriate type
  #001: H5G.c line 2438 in H5G_loc(): invalid group ID
    major(01): Function arguments
    minor(05): Bad value
  #002: H5I.c line 1262 in H5I_find_id(): invalid ID
    major(07): Atom layer
    minor(32): Unable to find atom information (already closed?)
Traceback (most recent call last):
  File "tutorial1-1.py", line 46, in ?
    table =3D h5file.createTable(group, 'readout', Particle, "Readout
example")
  File
"/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/sit=
e-packages/tables/File.py",
line 571, in createTable
    return self._createNode(where, name, Table, doCreateNode, _log)
  File
"/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/sit=
e-packages/tables/File.py",
line 499, in _createNode
    setattr(parent, name, node)
  File
"/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/sit=
e-packages/tables/Group.py",
line 635, in __setattr__
    value._g_putUnder(self, name)
  File
"/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/sit=
e-packages/tables/Leaf.py",
line 232, in _g_putUnder
    self._create()
  File
"/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/sit=
e-packages/tables/Table.py",
line 396, in _create
    self._createTable(self._v_new_title, self.filters.complib, obversion)
  File "TableExtension.pyx", line 265, in
TableExtension.Table._createTable
tables.exceptions.HDF5ExtError: Problems creating the table

If I open the resulting hdf5 file using the interactive python intepreter
all the data seems to be there, so I'm not sure if the errors just signify
some debugging code that got compiled into the hdf5 library.

Naveen

---------------------------------------------------------------------
Naveen Michaud-Agrawal
Program in Molecular Biophysics
Johns Hopkins University
(410) 614 4435

On Thu, 6 Oct 2005, Francesc Altet wrote:

> Hi Naveen,
>
> Can you report all the errors as well as the version of your PyTables
> package? There are quite a few PyTables users on MacOSX, so this should
> be a problem in your local configuration.
>
> El dc 05 de 10 del 2005 a les 14:27 -0400, en/na nmi...@jh... va
> escriure:
> > I'm trying to install PyTables onto Max OS X 10.4, and it compiles and
> > installs fine, but when I run the tests or any examples I always get
> > "HDF5-DIAG: Error detected in HDF5 library version: 1.6.4 thread 0.
> > Back trace follows." errors. The data seems to be saved properly into t=
he
> > hdf5 file, i'm just not sure what the errors mean.
> >
> > Naveen
> >
> > ---------------------------------------------------------------------
> > Naveen Michaud-Agrawal
> > Program in Molecular Biophysics
> > Johns Hopkins University
> > (410) 614 4435
> >
> > - the plural of 'anecdote' is not 'data'
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by:
> > Power Architecture Resource Center: Free content, downloads, discussion=
s,
> > and more. http://solutions.newsforge.com/ibmarch.tmpl
> > _______________________________________________
> > Pytables-users mailing list
> > Pyt...@li...
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> --
> >0,0<   Francesc Altet     http://www.carabos.com/
> V   V   C=E1rabos Coop. V.   Enjoy Data
>  "-"
>
>=

Re: [Pytables-users] Problem with pytables-1.2 and Row._fillCol on win32

[Francesc A. <fa...@ca...>]

A Dijous 06 Octubre 2005 02:55, Ashley Walsh va escriure:
> It looks like something is messing up the _fields dictionary on the
> NestedRecArray -- I didn't delve any deeper trying to find out *why*
> this correction works.

Thanks for the patch. However, I can't detect under which situations
Row._fillCol is working bad. Please, may you provide an small test
case that exposes the problem? That would facilitate the making of a
test unit in order to avoiding this (or similar) problem in the
future.

BTW, try to see if the attached patch also works (although I don't
think so). If don't, I'll apply your patch until more investigation
would be done.

> PS. UCL is working on OSX:

Oh, really? Which OSX version do you have, Panther or Tiger?.

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] HDF5-DIAG

[Francesc A. <fa...@ca...>]

Hi Naveen,

Can you report all the errors as well as the version of your PyTables
package? There are quite a few PyTables users on MacOSX, so this should
be a problem in your local configuration.

El dc 05 de 10 del 2005 a les 14:27 -0400, en/na nmi...@jh... va
escriure:
> I'm trying to install PyTables onto Max OS X 10.4, and it compiles and
> installs fine, but when I run the tests or any examples I always get
> "HDF5-DIAG: Error detected in HDF5 library version: 1.6.4 thread 0.
> Back trace follows." errors. The data seems to be saved properly into the
> hdf5 file, i'm just not sure what the errors mean.
>=20
> Naveen
>=20
> ---------------------------------------------------------------------
> Naveen Michaud-Agrawal
> Program in Molecular Biophysics
> Johns Hopkins University
> (410) 614 4435
>=20
> - the plural of 'anecdote' is not 'data'
>=20
>=20
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Power Architecture Resource Center: Free content, downloads, discussions,
> and more. http://solutions.newsforge.com/ibmarch.tmpl
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=E1rabos Coop. V.   Enjoy Data
 "-"

[Pytables-users] Problem with pytables-1.2 and Row._fillCol on win32

[Ashley W. <ash...@gm...>]

Hi,

While testing pytables-1.2-pre1 (also tested 20051005 snapshot) on  
win32 with some older HDF5 files I encountered a problem with  
retrieving a single column from the table.

Using:

     values = table.cols._f_col('somecolumn')

returned the *next* column after 'somecolumn'.  I tracked the problem  
down to the Row._fillCol method in TableExtension.pyx, and the  
problem was fixed by the patch at the end.

It looks like something is messing up the _fields dictionary on the  
NestedRecArray -- I didn't delve any deeper trying to find out *why*  
this correction works.

I was using:

Pytables-1.2-pre1
numarray-1.3.3
Numeric-23.8
HDF5-1.6.4
Python-2.3.5


This problem did not occur on MacOSX.

The fix:

TableExtension.pyx
984,989c984,990
<     if field:
<       # If field is not None, select it
<       fields = self.rbufRA._fields[field]
<     else:
<       # if don't, select all fields
<       fields = self.rbufRA
---
 >     #if field:
 >     #  # If field is not None, select it
 >     #  fields = self.rbufRA._fields[field]
 >     #else:
 >     #  # if don't, select all fields
 >     #  fields = self.rbufRA
 >     fields = self.rbufRA
1006c1007,1010
<       result[startr:stopr] = fields[istartb:istopb:istep]
---
 >       if field:
 >         result[startr:stopr] = fields.field(field) 
[istartb:istopb:istep]
 >       else:
 >         result[startr:stopr] = fields[istartb:istopb:istep]


Cheers,
Ashley

PS. UCL is working on OSX:

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 
=-=-=-=
PyTables version:  1.2-pre1-1
HDF5 version:      1.6.4
numarray version:  1.3.3
Zlib version:      1.2.2
LZO version:       1.08 (Jul 12 2002)
UCL version:       1.03 (Jul 20 2004)
BZIP2 version:     1.0.2 (30-Dec-2001)
Python version:    2.3.5 (#1, Mar 20 2005, 20:38:20)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1809)]
Platform:          darwin-Power Macintosh
Byte-ordering:     big
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 
=-=-=-=
Performing the complete test suite!
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 
=-=-=-=
Numeric (version 23.3) is present. Adding the Numeric test suite.
test01_readTable (test_backcompat.Table1_0UCL) ... ok
test01_readTable (test_backcompat.Table2_0UCL) ... ok
test01_readTable (test_backcompat.Table2_1UCL) ... ok
... more ok's
test03_setitemCArray (test_carray.UCLShuffleTestCase) ... ok
test01_readVLArray (test_vlarray.UCLComprTestCase) ... ok
test02_appendVLArray (test_vlarray.UCLComprTestCase) ... ok

[Pytables-users] HDF5-DIAG

[<nmi...@jh...>]

I'm trying to install PyTables onto Max OS X 10.4, and it compiles and
installs fine, but when I run the tests or any examples I always get
"HDF5-DIAG: Error detected in HDF5 library version: 1.6.4 thread 0.
Back trace follows." errors. The data seems to be saved properly into the
hdf5 file, i'm just not sure what the errors mean.

Naveen

---------------------------------------------------------------------
Naveen Michaud-Agrawal
Program in Molecular Biophysics
Johns Hopkins University
(410) 614 4435

- the plural of 'anecdote' is not 'data'

Re: [Pytables-users] error creating table with many columns

[Francesc A. <fa...@ca...>]

A Dimecres 05 Octubre 2005 17:23, Stefan Kuzminski va escriure:
> The domain here is analytic ( statistics ) data, so 1000s of variables
> is within the normal usage.  They are currently using SAS datasets.  I
> guess we can have multiple tables look like one, but I fret about a
> performance hit of an extra lookup layer in the Python space.  It
> doesn't look like that header size value is expandable in HDF5.

Well, you can always try doing it and profile and try to optimize the
code in case that you are discontent with the results. The beauty of
python is that it can do complicated things with relatively low effort
;-)

If you end doing something that you think is worth adding in pytables,
do not hesitate to contribute it back to the community.

Luck!

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] error creating table with many columns

[Stefan K. <pon...@ya...>]

The domain here is analytic ( statistics ) data, so 1000s of variables
is within the normal usage.  They are currently using SAS datasets.  I
guess we can have multiple tables look like one, but I fret about a
performance hit of an extra lookup layer in the Python space.  It
doesn't look like that header size value is expandable in HDF5.

S

--- Francesc Altet <fa...@ca...> wrote:

> A Dimarts 04 Octubre 2005 21:14, Stefan Kuzminski va escriure:
> > I can create multiple tables in the same group that are 500
> columns, so
> > I suppose this is a limit in a single table's header?  The overall
> size
> > of the full file does not seem limited. ( I hope )
> 
> Yes, that's right. Header table data (i.e. where the information on
> the table structure is kept) has a limit of 16k, while actual data
> space has not limit (I've created tables with more than 100 GB of
> data
> with not problem).
> 
> > The data is heterogenous so muti-dimensional columns aren't
> applicable.
> >  I suppose we will have to split the big datasets apart into
> multiple
> > tables and iterate them in parallel.
> 
> Well, that's a possibility.
> 
> -- 
> >0,0<   Francesc Altet     http://www.carabos.com/
> V   V   Cárabos Coop. V.   Enjoy Data
>  "-"
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Power Architecture Resource Center: Free content, downloads,
> discussions,
> and more. http://solutions.newsforge.com/ibmarch.tmpl
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> 



		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: [Pytables-users] error creating table with many columns

[Francesc A. <fa...@ca...>]

A Dimarts 04 Octubre 2005 21:14, Stefan Kuzminski va escriure:
> I can create multiple tables in the same group that are 500 columns, so
> I suppose this is a limit in a single table's header?  The overall size
> of the full file does not seem limited. ( I hope )

Yes, that's right. Header table data (i.e. where the information on
the table structure is kept) has a limit of 16k, while actual data
space has not limit (I've created tables with more than 100 GB of data
with not problem).

> The data is heterogenous so muti-dimensional columns aren't applicable.
>  I suppose we will have to split the big datasets apart into multiple
> tables and iterate them in parallel.

Well, that's a possibility.

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] error creating table with many columns

[Francesc A. <fa...@ca...>]

Hi Stefan,

A Dimarts 04 Octubre 2005 20:12, Stefan Kuzminski va escriure:
> I am getting an error creating a table with 1000+ columns, the error is
>
>
>   #005: H5O.c line 2157 in H5O_new_mesg(): object header message is too
> large (16k max)
>
> the full error message is below, is there a limit on the number of
> columns  I can put in a table?

I've never checked the creation of tables so wide, but, looking at the
error, this effectively seems a limitation of the underlying HDF5
library on the size of the headers of datasets (which is 16k, as you
see). I don't know if this limitation can be surmounted in some way,
but I'd recommend you to do not surpass a reasonable limit for columns
(maybe 256 or 512 as maximum). You can use multidimensional columns if
you don't have enough with the recommended maximum.

[Perhaps it would be nice to issue a warning when user is trying to
exceed 256 or 512 columns.]

> There error also implies there is a
> limit to the size of the attributes on a table?

Yes, I think so. Generally speaking you should use attributes to add
metadata information that is small in size. If you try to use metainfo
that is bigger than, say, 1k, you should put it on a dedicated leave
(*Array or Table).

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] error creating table with many columns

[Stefan K. <pon...@ya...>]

I am getting an error creating a table with 1000+ columns, the error is


  #005: H5O.c line 2157 in H5O_new_mesg(): object header message is too
large (16k max)

the full error message is below, is there a limit on the number of
columns  I can put in a table?  There error also implies there is a
limit to the size of the attributes on a table?

thanks,
Stefan


HDF5-DIAG: Error detected in HDF5 library version: 1.6.4 thread 0. 
Back trace follows.
  #000: H5D.c line 1087 in H5Dcreate(): unable to create dataset
    major(15): Dataset interface
    minor(29): Unable to initialize object
  #001: H5D.c line 2258 in H5D_create(): can't update the metadata
cache
    major(15): Dataset interface
    minor(29): Unable to initialize object
  #002: H5D.c line 1843 in H5D_update_entry_info(): unable to update
type or space header \
messages
    major(15): Dataset interface
    minor(29): Unable to initialize object
  #003: H5O.c line 2037 in H5O_append(): unable to append to object
header
    major(12): Object header layer
    minor(25): Write failed
  #004: H5O.c line 2082 in H5O_append_real(): unable to create new
message
    major(12): Object header layer
    minor(29): Unable to initialize object
  #005: H5O.c line 2157 in H5O_new_mesg(): object header message is too
large (16k max)
    major(12): Object header layer
    minor(29): Unable to initialize object
Traceback (most recent call last):
  File "join.py", line 85, in ?
    join( [( table_a, 'obs' ), ( table_b, 'perf' )],
'crpAccountMatchKey' )
  File "join.py", line 59, in join
    expectedrows=len(master_table))
  File "/apps/lib/python2.4/site-packages/tables/File.py", line 571, in
createTable
    return self._createNode(where, name, Table, doCreateNode, _log)
  File "/apps/lib/python2.4/site-packages/tables/File.py", line 499, in
_createNode
    setattr(parent, name, node)
  File "/apps/lib/python2.4/site-packages/tables/Group.py", line 635,
in __setattr__
    value._g_putUnder(self, name)
  File "/apps/lib/python2.4/site-packages/tables/Leaf.py", line 232, in
_g_putUnder
    self._create()
  File "/apps/lib/python2.4/site-packages/tables/Table.py", line 396,
in _create
    self._createTable(self._v_new_title, self.filters.complib,
obversion)
  File "TableExtension.pyx", line 265, in
TableExtension.Table._createTable
tables.exceptions.HDF5ExtError: Problems creating the table



		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: [Pytables-users] Reading vlen data created by libhdf5 with pytables

[Francesc A. <fa...@ca...>]

A Dimarts 04 Octubre 2005 02:30, Cournapeau David va escriure:
> On Mon, 2005-10-03 at 17:11 +0200, Francesc Altet wrote:
> > Great, this looks like a good workaround.
>
> Ok. Would you like a diff against the last version ( 1.2-b6) ?

No thanks. In fact, a patch is already applied in the last snapshot:

http://www.carabos.com/downloads/pytables/snapshots/pytables-latest.tar.gz

> BTW, Ivan has recently implemented a new module to test compatibility
> Should I send the the files to your carabos.com address ? (they are
> really small anyway, a few kb only).

Yes, that's fine. Have in mind that we will try to add these files to
the HDF5 compatibility test unit, so try to reduce the size to a bare
minimum.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] Reading vlen data created by libhdf5 with pytables

[Cournapeau D. <cou...@at...>]

On Mon, 2005-10-03 at 17:11 +0200, Francesc Altet wrote:

> Great, this looks like a good workaround.
Ok. Would you like a diff against the last version ( 1.2-b6) ?
> 
> BTW, Ivan has recently implemented a new module to test compatibility
> with pure HDF5 files. Can you send us your file for inclusion in
> tests, please? That way, new modifications will always check your
> file.
[snip]
> Please, send me a sample of this too. I'll see what I can do.
Should I send the the files to your carabos.com address ? (they are
really small anyway, a few kb only).

David

Re: [Pytables-users] Reading vlen data created by libhdf5 with pytables

[Francesc A. <fa...@ca...>]

A Dilluns 03 Octubre 2005 08:31, Cournapeau David va escriure:
> Anyway, as the Flavor attribute does not exist, it is set to Unknown,
> and the  the Atom function call fails in VLArray.py:VLArray._g_open.
> (specifically the checkflavor call in the Atom.__init__ fails).

Yes, you discovered a flaw in implementation.

> But this means the first call, even if unsuccessfull, manages to read
> the vlen array in the file. (the data given back by the second call are
> correct). So I tried the following hack (in the file VLArray.py,
> function _g_open of the class VLArray):
[snipped]
> Then, it works in my case. As I don't know much yet about pytable,
> I don't know what is the correct way achieve the same result.

Great, this looks like a good workaround.

BTW, Ivan has recently implemented a new module to test compatibility
with pure HDF5 files. Can you send us your file for inclusion in
tests, please? That way, new modifications will always check your
file.

> Concerning the case where the dimensions are fixed, I Unfortunately
> don't have the time right now to investigate it.

Please, send me a sample of this too. I'll see what I can do.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] delete a column

[Francesc A. <fa...@ca...>]

A Divendres 30 Setembre 2005 18:47, dragan savic va escriure:
> Hi!
>
> Does anybody know a way to delete a column from
> already created table?

Nope. Although providing such a facility should be very easy (however,
it would imply a copy of all the columns, except the deleted one), it
is not done yet. I think it would be a nice addition, so perhaps that
would be implemented sooner than later.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] Experience with PyTables & Scipy Core 0.4 (Beta)?

[Francesc A. <fa...@ca...>]

A Dilluns 03 Octubre 2005 15:27, Norbert Nemec va escriure:
> Travis Oliphant just announced the release of SciPy Core 0.4 (Beta).
> This code is designed to be the successor of both Numeric and numarray.
>
> Has anybody yet gathered some experience in cooperation with PyTables?

Not yet, at least here in C=E1rabos. Although I think that the migration
from numarray to scipy_core should be relatively easy (if we believe
all the promises annonced by Travis), the change would imply the use
of a completely new package instead of numarray as the core data
container in PyTables.

Because of that, I think that it would safer for the PyTables
community to wait a bit until scipy_core gets ready for production.
Given that numarray took at least a couple of years to reach this
stage, we should not expect doing this migration before a year (unless
scipy_core would be a great success not only in stability, but also
that its use would eventually spreads very rapidly).

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] Experience with PyTables & Scipy Core 0.4 (Beta)?

[Norbert N. <Nor...@gm...>]

Hi there,

Travis Oliphant just announced the release of SciPy Core 0.4 (Beta).
This code is designed to be the successor of both Numeric and numarray.

Has anybody yet gathered some experience in cooperation with PyTables?

Greetings,
Norbert Nemec

Re: [Pytables-users] Reading vlen data created by libhdf5 with pytables

[Cournapeau D. <cou...@at...>]

> 
> Strange. Anyway, I recommend you to stick with pytables 1.2b4, which
> is should be quite stable right now.

Ok, this is the version I am using right now, hence all problems
reported here are against this version.

> If you are going to modify the Pyrex sources, the best to rebuild the
> extensions is to use the distutils:
> 
[snip]

Thanks for the information. 

Here is what I get. First, when I try to access my vlen array 'cell'
through this small script (boilerplate code omitted):

    h5file  = openFile(filename, "r")
    root    = h5file.root
    print "=================== First call ===================="
    try:
        cell    = root.cell
    except:
        print 'exception during first call'

    print "=================== Second call ===================="
    cell    = root.cell
    for i in cell:
        print i

the h5f file 'filename' I tried to read as reported by h5dump (only the
interesting group is mentioned):

DATASET "cell" {
      DATATYPE  H5T_VLEN { H5T_IEEE_F64LE}
      DATASPACE  SIMPLE { ( 2 ) / ( H5S_UNLIMITED ) }
      DATA {
      (0): (-0.432565, -1.66558, 0.125332), (0.287676, -1.14647)
      }
   }

I then have the following error:

HDF5-DIAG: Error detected in HDF5 library version: 1.6.4 thread
3084815008.  Back trace follows.
  #000: ../../../src/H5A.c line 457 in H5Aopen_name(): attribute not
found
    major(18): Attribute layer
    minor(05): Bad value
  #001: ../../../src/H5A.c line 404 in H5A_get_index(): attribute not
found
    major(18): Attribute layer
    minor(48): Object not found
/usr/src/misc/pytables/pytables-1.2b4/tables/Group.py:397: UserWarning:
problems loading leaf ``/cell``::

  flavor of type 'Float64' must be one of the "NumArray", "Numeric",
"Tuple" or "List" values, and you tried to set it to "unknown".


The leaf will become an ``UnImplemented`` node.


The hf5 warning is coming from the call of H5LTget_attribute_string,
in the function _openArray in the pyrex file /src/hdf5Extension.pyx.
This of course happens because my file is not created by pytables, hence
does not have any FLAVOR attribute. Is it intentional to keep the Hf5
warning on? 

Anyway, as the Flavor attribute does not exist, it is set to Unknown,
and the  the Atom function call fails in VLArray.py:VLArray._g_open. 
(specifically the checkflavor call in the Atom.__init__ fails).

As I said in my previous email, the second attempt to read this vlen
array succeeds, because pytables has it in memory, hence the
File.py:File._getNode does not load it from the file, but from memory.

But this means the first call, even if unsuccessfull, manages to read
the vlen array in the file. (the data given back by the second call are
correct). So I tried the following hack (in the file VLArray.py,
function _g_open of the class VLArray):


	# First, check the special cases VLString and Object types
	if flavor == "VLString":
		self.atom = VLStringAtom()
	elif flavor == "Object":
        	self.atom = ObjectAtom()
	elif stype == 'CharType':
        	self.atom = StringAtom(self._atomicshape,
self._basesize,flavor)
	elif stype == 'Enum':
        	(enum, type_) = self._loadEnum()
        	self.atom = EnumAtom(enum, type_, self._atomicshape, flavor)
        	self._atomictype = type_
	else:
     	if flavor == 'unknown':
        	self.atom = Atom(stype, self._atomicshape)
     	else:
        	self.atom = Atom(stype, self._atomicshape, flavor)

return objectId

Then, it works in my case. As I don't know much yet about pytable,
I don't know what is the correct way achieve the same result. 

Concerning the case where the dimensions are fixed, I Unfortunately
don't have the time right now to investigate it.

David

[Pytables-users] delete a column

[dragan s. <sav...@ya...>]

Hi!

Does anybody know a way to delete a column from
already created table?

Thanks in advance.
Best regards, Dragan.


		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: [Pytables-users] performance question ( with source )

[Francesc A. <fa...@ca...>]

A Dijous 29 Setembre 2005 16:44, Stefan Kuzminski va escriure:
> Is there a way to specify the order of the column?  Or do I have to
> just ensure that any data I append to the table is in that order?

There is. Use the qualifier 'pos' in Col declarations. Look at:

http://pytables.sourceforge.net/html-doc/usersguide4.html#subsection4.16.1

and, for set the order in general nested types, look:

http://pytables.sourceforge.net/html-doc/usersguide3.html#section3.6

> The selections are great, we are building tools like merges and sorts,
> frequency counting and statistics.  The merge and sort things tend to
> be rowwise operations, ( pattering on SAS to deal with very large data
> sets )  but I am beginning to realize that column operations are
> faster.  I guess I had naivly assumed that rows were contiguous in
> memory ( and on disk ).

Well, in fact they are (both in-memory and on-disk). Other thing is
how numarray (and in particular) RecArray objects manage more
efficiently this information. RecArray is, roughly speaking, a
container for heterogeneous data types made of a set of NumArray
objects that contains the (homogenous) data in columns. These NumArray
objects has gaps in-between the elements of columns, being the size of
the gap the size of the record. So, operations column-wise tend to be
very fast as well.

If you are getting the impression that row-wise operation are not
efficient is probably because your code is making conversions to
Python objects (most probably lists or tuples) behind the scenes, so
loosing much of its potential speed.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] Re: Running PyTables on Red Hat 9.0

[Francesc A. <fa...@ca...>]

A Dimarts 27 Setembre 2005 23:39, dragan savic va escriure:
> Hi Francesc!
>
> I have installed pytables-1.1.1 on my Red Hat 9.0
> machine. I've tested the program and got:
> Warning: Encountered invalid numeric result(s)  in
> less
> Should I worry about that?

I don't think so. I'm not certain of where these warnings come from,
but, as the tests actually pass, I don't think this would too grave
anyway.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] performance question ( with source )

[Francesc A. <fa...@ca...>]

A Dimarts 27 Setembre 2005 20:21, Stefan Kuzminski va escriure:
> Ahh, that's the ticket.  I can get the throughput I need now ( >4000tps
> )!!  Another quick questions or so..

Excellent!

> 1.  If I use a dictionary as a table description ( rather than a
> IsDescrition subclass ), how do I tell it what order the columns are?
> Dictionaries are non-ordered?

You can use the ._v_names attribute of table description. For example:

In [8]:f.root.newgroup.table.description._v_names
Out[8]:['name', 'lati', 'longi', 'pressure', 'temperature']


> 2.  Is there some type of SQL interface built on top of pyttables?

Not as such, but there are several ways to do data selections in
tables, namely, general selections, in-kernel selections and indexed
selections. Look at:

http://pytables.sourceforge.net/doc/SciPy04.pdf

for some directions, experiments, recomendations and limitations about
data selections.

Or do you need something different from selections?

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] performance question ( with source )

[Stefan K. <pon...@ya...>]

Ahh, that's the ticket.  I can get the throughput I need now ( >4000tps
)!!  Another quick questions or so..

1.  If I use a dictionary as a table description ( rather than a
IsDescrition subclass ), how do I tell it what order the columns are? 
Dictionaries are non-ordered?

2.  Is there some type of SQL interface built on top of pyttables?

thanks,
Stefan

--- Francesc Altet <fa...@ca...> wrote:

> A Dilluns 26 Setembre 2005 20:50, Stefan Kuzminski va escriure:
> > Once I calculate an index ( not a specific pytables index, just a
> > vector of row numbers ) that is the new order that the table has to
> be
> > in, I tried to just random access the source table, but that was
> slow..
> >
> > for idx in new_order:
> >    new_table.append( src_table[idx] )
> >
> > I wonder if I should try this approach but in a column oriented
> way?
> > Is there a way to pass to pytables the order that you want the data
> > returned?
> 
> Yes, there is. Try:
> 
>    new_table.append( src_table.readCoordinates(new_order) )
> 
> In the future, one should be able to do:
> 
>    new_table.append( src_table[new_order] )
> 
> as well :-)
> 
> I'm not sure, but you may need to try with 1.2bx accessible in:
> 
> http://www.carabos.com/downloads/pytables/preliminary/
> 
> in order to make readCoordinates to work well (although 1.1.1 should
> work fine, I believe)
> 
> Cheers,
> 
> -- 
> >0,0<   Francesc Altet     http://www.carabos.com/
> V   V   Cárabos Coop. V.   Enjoy Data
>  "-"
> 
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by:
> Tame your development challenges with Apache's Geronimo App Server.
> Download it for free - -and be entered to win a 42" plasma tv or your
> very
> own Sony(tm)PSP.  Click here to play:
> http://sourceforge.net/geronimo.php
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Pytables-users] performance question ( with source )

[Francesc A. <fa...@ca...>]

A Dilluns 26 Setembre 2005 20:50, Stefan Kuzminski va escriure:
> Once I calculate an index ( not a specific pytables index, just a
> vector of row numbers ) that is the new order that the table has to be
> in, I tried to just random access the source table, but that was slow..
>
> for idx in new_order:
>    new_table.append( src_table[idx] )
>
> I wonder if I should try this approach but in a column oriented way?
> Is there a way to pass to pytables the order that you want the data
> returned?

Yes, there is. Try:

   new_table.append( src_table.readCoordinates(new_order) )

In the future, one should be able to do:

   new_table.append( src_table[new_order] )

as well :-)

I'm not sure, but you may need to try with 1.2bx accessible in:

http://www.carabos.com/downloads/pytables/preliminary/

in order to make readCoordinates to work well (although 1.1.1 should
work fine, I believe)

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"