pytables-users — PyTables users discussion list

[Pytables-users] nctoh5

[Jeff W. <jef...@no...>]

Hi:

I just wrote a script to convert a netcdf datasets to hdf5.  It's posted 
below, in case others might find it useful (you can try it on the files 
at ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis).   I have a question 
regarding EArrays - I see how to append values and to write the entire 
array at once, but is there a way to update an array record without 
re-writing the whole thing?  

Here's the nctoh5 script (requires Scientific from 
http://starship.python.net/~hinsen/ScientificPython).

"""
convert netCDF file to HDF5 using Scientific.IO.NetCDF and PyTables.
Jeff Whitaker <jef...@no...>
"""
import Scientific.IO.NetCDF as NetCDF
import tables, sys
# open netCDF file
ncfile = NetCDF.NetCDFFile(sys.argv[1],mode = "r")
# open h5 file.
h5file = tables.openFile(sys.argv[2], mode = "w")
# loop over variables in netCDF file.
for varname in ncfile.variables.keys():
    var = ncfile.variables[varname]
    vardims = list(var.dimensions)
    vardimsizes = [ncfile.dimensions[vardim] for vardim in vardims]
    # use long_name for title.
    if hasattr(var,'long_name'):
       title = var.long_name
    else: # or, just use some bogus title.
       title = varname + ' array'
    # if variable has unlimited dimension or has rank>1,
    # make it enlargeable (with zlib compression).
    if vardimsizes[0] == None or len(vardimsizes) > 1:
        vardimsizes[0] = 0
        vardata = h5file.createEArray(h5file.root, varname,
        tables.Atom(shape=tuple(vardimsizes),dtype=var.typecode(),),
        title,filters=tables.Filters(complevel=6,complib='zlib'))
    # write data to enlargeable array on record at a time.
    # (so the whole array doesn't have to be kept in memory).
        for n in range(var.shape[0]):
            vardata.append(var[n:n+1])
    # or else, create regular array write data to it all at once.
    else:
        vardata=h5file.createArray(h5file.root,varname,var[:],title)
    # set variable attributes.
    for key,val in var.__dict__.iteritems():
        setattr(vardata.attrs,key,val)
    setattr(vardata.attrs,'dimensions',tuple(vardims))
# set global (file) attributes.
for key,val in ncfile.__dict__.iteritems():
    setattr(h5file.root._v_attrs,key,val)
# Close the file
h5file.close()

-Jeff



-- 
Jeffrey S. Whitaker         Phone : (303)497-6313
NOAA/OAR/CDC  R/CDC1        FAX   : (303)497-6449
325 Broadway                Web   : http://www.cdc.noaa.gov/~jsw
Boulder, CO, USA 80305-3328 Office: Skaggs Research Cntr 1D-124

Re: [Pytables-users] Writing to arrays?

[Francesc A. <fa...@py...>]

A Dilluns 18 Octubre 2004 15:19, Norbert Nemec va escriure:
> maybe I'm just blind trying to read the docs: Is there any way to write to an 
> existing array within a h5file? I would have expected the following to work, 
> but it does not:
> ---------------------------
>  from tables import *
>  from scipy import *
>  h5 = openFile("tryout.h5",'a')
>  h5.createArray(h5.root,"tryarray",zeros((3,4)))
>  h5.root.tryarray[2,2] = 17.
> ---------------------------
> but the last line gives "TypeError: object does not support item assignment"
> 
> What would be the correct approach?

That would be the correct approach if modification of values in Arrays would
be supported, but it is not.

However, I've implemented value modification in Table objects for
forthcoming PyTables 0.9 (you can fetch the RC1 in
http://pytables.sourceforge.net/pytables-0.9-rc1.tar.gz). Maybe you can use
use it until I (or somebody else) find time to implement value modification
in *Arrays as well.

Regards,

-- 
Francesc Alted

[Pytables-users] Writing to arrays?

[Norbert N. <Nor...@gm...>]

Hi there,

maybe I'm just blind trying to read the docs: Is there any way to write to an 
existing array within a h5file? I would have expected the following to work, 
but it does not:
---------------------------
	from tables import *
	from scipy import *
	h5 = openFile("tryout.h5",'a')
	h5.createArray(h5.root,"tryarray",zeros((3,4)))
	h5.root.tryarray[2,2] = 17.
---------------------------
but the last line gives "TypeError: object does not support item assignment"

What would be the correct approach?

Ciao,
Nobbi

-- 
_________________________________________Norbert Nemec
         Bernhardstr. 2 ... D-93053 Regensburg
    Tel: 0941 / 2009638 ... Mobile: 0179 / 7475199
           eMail: <No...@Ne...>

Re: [Pytables-users] ValueError when a list of numarray.arrays used.

[Francesc A. <fa...@py...>]

A Diumenge 10 Octubre 2004 05:59, Shin, Daehyok va escriure:
> Assuming at every simulation step I got a numeric array of 100 rows and 10
> cols,
> what is the most efficient way to append it to Table?

You can do something like:

from numarray import *
from tables import *

class Cols(IsDescription):
    V1 = FloatCol(shape=10)

f = openFile("test.hd5", "w")
tab = f.createTable("/","Data",Cols,"Data")

for i in range(10):
    # d simulates the output of your simulation step
    d = arange(i, 1000+i, shape=(100,10), type=Float64)
    tab.append([d])
    
f.close()

i.e. declaring a column with dimension 10.

> I want  to use Table rather than EArray because of explicit column names.

In that case, you can't use the recipe above. I would suggest you getting a
numarray.records.RecArray object from every step of your simulation code.
Then, you can feed the RecArray to Table.append() without any problem.

By the way, after thinking a bit about passing lists of numarray/Numeric to
the Table.append() method, I came to the conclusion that this is possible,
although this use is meant for multidimensional columns. For example:

from numarray import *
from tables import *

class Cols(IsDescription):
    V1 = FloatCol(shape=10)
    V2 = IntCol(shape=5)

f = openFile("test2.hd5", "w")
tab = f.createTable("/","Data",Cols,"Data")

for i in range(10):
    # d simulates the output of your simulation step
    d = arange(i, 1000+i, shape=(100,10), type=Float64)
    e = arange(i, 500+i, shape=(100,5), type=Int32)
    tab.append([d, e])
    
f.close()

works without problems. Maybe I should specify in the docs that you can use
lists of Numeric/numarray objects *only* in case of multimensional columns.

Cheers,

-- 
Francesc Alted

RE: [Pytables-users] ValueError when a list of numarray.arrays used.

[Shin, D. <sd...@ca...>]

I got it.
Then, let me ask one question about it.
Assuming at every simulation step I got a numeric array of 100 rows and 10
cols,
what is the most efficient way to append it to Table?
I want  to use Table rather than EArray because of explicit column names.
Thank for your kind reply.

Daehyok Shin (Peter)

> -----Original Message-----
> From: pyt...@li...
> [mailto:pyt...@li...]On Behalf Of
> Francesc Alted
> Sent: Saturday, October 09, 2004 AM 5:55
> To: pyt...@li...
> Subject: Re: [Pytables-users] ValueError when a list of
> numarray.arrays used.
>
>
> Hi Daehyok,
>
> The ultimate reason for this is that numarray.records.array
> cannot convert a
> list of arrays to a RecArray. See:
>
> In [27]: records.array([[1.,2.]]*3, formats='f8,f8')
> Out[27]:
> array(
> [(1.0, 2.0),
> (1.0, 2.0),
> (1.0, 2.0)],
> formats=['1Float64', '1Float64'],
> shape=3,
> names=['c1', 'c2'])
>
> That works well. But...
>
> In [28]: records.array([array([1.,2.])]*3, formats='f8,f8')
> ------------------------------------------------------------------
> ---------
> IndexError                                Traceback (most recent
> call last)
>
> /home/falted/python.nobackup/IM-2004.01.05/<console>
>
> /usr/local/lib/python2.3/site-packages/numarray/records.py in
> array(buffer, formats, shape, names, byteorder, aligned)
>     382         if isinstance(buffer[0], num.NumArray) or
> isinstance(buffer[0], chararray.CharArray):
>     383             return fromarrays(buffer, formats=formats,
> shape=shape, names=names,
> --> 384                         byteorder=byteorder, aligned=aligned)
>     385         else:
>     386             return fromrecords(buffer, formats=formats,
> shape=shape, names=names,
>
> /usr/local/lib/python2.3/site-packages/numarray/records.py in
> fromarrays(arrayList, formats, names, shape, byteorder, aligned)
>     183     for i in range(len(arrayList)):
>     184         try:
> --> 185             _array.field(_array._names[i])[:] = arrayList[i]
>     186         except TypeError: # trap only TypeError to get
> better messages
>     187             raise TypeError, "Incorrect format %s, copy
> unsuccessful." % _array._formats[i]
>
> IndexError: list index out of range
>
> does not work.
>
> Perhaps, this worked with previous versions of numarray, I can't remember.
> Anyway, I'll remove the part of manual where I say that Table.append() do
> accept lists of numarrays/Numeric/chararray.
>
> Cheers,
>
> Francesc
>
> A Divendres 08 Octubre 2004 23:17, Shin va escriure:
> > While filling up a table, I met a strange error message.
> > When the appended program is tried with version 0.8.1,
> > only the second trial to use a list of arrays raises an ValueError.
> > But, the manual says I can use a list of numarrays to append multi-rows.
> > What did I miss? Or, a bug? Thanks.
> >
> > >>>>>>>>>>>>>>> test3.py <<<<<<<<<<<<<<<<<<
> >
> > #!/usr/bin/env python
> >
> > from numarray import *
> > from tables import *
> >
> > # By using a list of lists.
> > class Cols(IsDescription):
> >     V1 = FloatCol()
> >     V2 = FloatCol()
> >
> > f = openFile("test.hd5", "w")
> > tab = f.createTable("/","Data",Cols,"Data")
> >
> > d = []
> > for i in range(10):
> >     d = d + [[1.0,1.0]]
> >
> > tab.append(d)
> > f.close()
> >
> > # By using a list of arrays.
> >
> > f = openFile("test.hd5", "w")
> > tab = f.createTable("/","Data",Cols,"Data")
> >
> > d = []
> > for i in range(10):
> >     d = d + [array([1.0,1.0])]
> >
> > tab.append(d)
> > f.close()
> >
> > >>>>>>>>> Here is the error message caught in ipython <<<<<<
> >
> > In [1]: execfile("test3.py")
> >
> ------------------------------------------------------------------
> ---------
> > ValueError                                Traceback (most recent call
> > last)
> >
> > /home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/<console>
> >
> > /home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/test3.py
> >      24 for i in range(10):
> >      25     d = d + [array([1.0,1.0])]
> >      26
> > ---> 27 tab.append(d)
> >         tab = /Data (Table(0,)) 'Data'
> >   description := {
> >     ...pe=1, dflt=0.0, pos=None) }
> >   byteorder := little, global append = undefined, d = [array([ 1.,
> > 1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
> > 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]),
> > array([ 1.,  1.]), array([ 1.,  1.])]
> >      28 f.close()
> >
> > /usr/lib/python2.3/site-packages/tables/Table.py in append(self=/Data
> > (Table(0,)) 'Data'
> >   description := {
> >     ...pe=1, dflt=0.0, pos=None) }
> >   byteorder := little, rows=[array([ 1.,  1.]), array([ 1.,  1.]),
> > array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,
> > 1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
> > 1.,  1.])])
> >     588         except:
> >     589             (type, value, traceback) = sys.exc_info()
> > --> 590             raise ValueError, \
> >         global ValueError = undefined, global str = undefined, self =
> > /Data (Table(0,)) 'Data'
> >   description := {
> >     ...pe=1, dflt=0.0, pos=None) }
> >   byteorder := little, value = <exceptions.IndexError instance>
> >     591 "rows parameter cannot be converted into a recarray object
> > compliant with table '%s'. The error was: <%s>" % (str(self), value)
> >     592         lenrows = recarray.shape[0]
> >
> > ValueError: rows parameter cannot be converted into a recarray object
> > compliant with table '/Data (Table(0,)) 'Data''. The error was: <list
> > index out of range>
> >
>
> --
> Francesc Alted
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to
> find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] How to subclass tables.File?

[Francesc A. <fa...@py...>]

A Divendres 08 Octubre 2004 20:30, Shin va escriure:
> I want to subclass tables.File to add some customized behaviors.
> But, it turned out not as simple as I had expected.
> File class has __init__ method, but HDF5Extension.File, one of
> tables.File's superclasses, has only __new__ method.
> I am not familiar with new class system in Python.

hdf5Extension.File is not a class, is an extension type.

> To subclass tables.File, how should I implement __init__ or __new__
> method of the new class? Thanks in advance.

I've never tried that. I'll have a look into that and I'll inform you about
my findings. Anybody in the list has tried this before?

Cheers,

-- 
Francesc Alted

Re: [Pytables-users] ValueError when a list of numarray.arrays used.

[Francesc A. <fa...@py...>]

Hi Daehyok,

The ultimate reason for this is that numarray.records.array cannot convert a
list of arrays to a RecArray. See:

In [27]: records.array([[1.,2.]]*3, formats='f8,f8')
Out[27]:
array(
[(1.0, 2.0),
(1.0, 2.0),
(1.0, 2.0)],
formats=['1Float64', '1Float64'],
shape=3,
names=['c1', 'c2'])

That works well. But...

In [28]: records.array([array([1.,2.])]*3, formats='f8,f8')
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)

/home/falted/python.nobackup/IM-2004.01.05/<console>

/usr/local/lib/python2.3/site-packages/numarray/records.py in array(buffer, formats, shape, names, byteorder, aligned)
    382         if isinstance(buffer[0], num.NumArray) or isinstance(buffer[0], chararray.CharArray):
    383             return fromarrays(buffer, formats=formats, shape=shape, names=names,
--> 384                         byteorder=byteorder, aligned=aligned)
    385         else:
    386             return fromrecords(buffer, formats=formats, shape=shape, names=names,

/usr/local/lib/python2.3/site-packages/numarray/records.py in fromarrays(arrayList, formats, names, shape, byteorder, aligned)
    183     for i in range(len(arrayList)):
    184         try:
--> 185             _array.field(_array._names[i])[:] = arrayList[i]
    186         except TypeError: # trap only TypeError to get better messages
    187             raise TypeError, "Incorrect format %s, copy unsuccessful." % _array._formats[i]

IndexError: list index out of range

does not work.

Perhaps, this worked with previous versions of numarray, I can't remember.
Anyway, I'll remove the part of manual where I say that Table.append() do
accept lists of numarrays/Numeric/chararray.

Cheers,

Francesc

A Divendres 08 Octubre 2004 23:17, Shin va escriure:
> While filling up a table, I met a strange error message.
> When the appended program is tried with version 0.8.1,
> only the second trial to use a list of arrays raises an ValueError.
> But, the manual says I can use a list of numarrays to append multi-rows.
> What did I miss? Or, a bug? Thanks.
> 
> >>>>>>>>>>>>>>> test3.py <<<<<<<<<<<<<<<<<<
> 
> #!/usr/bin/env python
> 
> from numarray import *
> from tables import *
> 
> # By using a list of lists.
> class Cols(IsDescription):
>     V1 = FloatCol()
>     V2 = FloatCol()    
> 
> f = openFile("test.hd5", "w")
> tab = f.createTable("/","Data",Cols,"Data")
> 
> d = []
> for i in range(10):
>     d = d + [[1.0,1.0]]
> 
> tab.append(d)
> f.close()
> 
> # By using a list of arrays.
> 
> f = openFile("test.hd5", "w")
> tab = f.createTable("/","Data",Cols,"Data")
> 
> d = []
> for i in range(10):
>     d = d + [array([1.0,1.0])]
> 
> tab.append(d)
> f.close()
> 
> >>>>>>>>> Here is the error message caught in ipython <<<<<<
> 
> In [1]: execfile("test3.py")
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call
> last)
> 
> /home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/<console> 
> 
> /home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/test3.py 
>      24 for i in range(10):
>      25     d = d + [array([1.0,1.0])]
>      26 
> ---> 27 tab.append(d)
>         tab = /Data (Table(0,)) 'Data'
>   description := {
>     ...pe=1, dflt=0.0, pos=None) }
>   byteorder := little, global append = undefined, d = [array([ 1., 
> 1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
> 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]),
> array([ 1.,  1.]), array([ 1.,  1.])]
>      28 f.close()
> 
> /usr/lib/python2.3/site-packages/tables/Table.py in append(self=/Data
> (Table(0,)) 'Data'
>   description := {
>     ...pe=1, dflt=0.0, pos=None) }
>   byteorder := little, rows=[array([ 1.,  1.]), array([ 1.,  1.]),
> array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1., 
> 1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
> 1.,  1.])])
>     588         except:
>     589             (type, value, traceback) = sys.exc_info()
> --> 590             raise ValueError, \
>         global ValueError = undefined, global str = undefined, self =
> /Data (Table(0,)) 'Data'
>   description := {
>     ...pe=1, dflt=0.0, pos=None) }
>   byteorder := little, value = <exceptions.IndexError instance>
>     591 "rows parameter cannot be converted into a recarray object
> compliant with table '%s'. The error was: <%s>" % (str(self), value)
>     592         lenrows = recarray.shape[0]
> 
> ValueError: rows parameter cannot be converted into a recarray object
> compliant with table '/Data (Table(0,)) 'Data''. The error was: <list
> index out of range>
> 

-- 
Francesc Alted

[Pytables-users] ValueError when a list of numarray.arrays used.

[Shin <sd...@em...>]

While filling up a table, I met a strange error message.
When the appended program is tried with version 0.8.1,
only the second trial to use a list of arrays raises an ValueError.
But, the manual says I can use a list of numarrays to append multi-rows.
What did I miss? Or, a bug? Thanks.

>>>>>>>>>>>>>>> test3.py <<<<<<<<<<<<<<<<<<

#!/usr/bin/env python

from numarray import *
from tables import *

# By using a list of lists.
class Cols(IsDescription):
    V1 = FloatCol()
    V2 = FloatCol()    

f = openFile("test.hd5", "w")
tab = f.createTable("/","Data",Cols,"Data")

d = []
for i in range(10):
    d = d + [[1.0,1.0]]

tab.append(d)
f.close()

# By using a list of arrays.

f = openFile("test.hd5", "w")
tab = f.createTable("/","Data",Cols,"Data")

d = []
for i in range(10):
    d = d + [array([1.0,1.0])]

tab.append(d)
f.close()

>>>>>>>>> Here is the error message caught in ipython <<<<<<

In [1]: execfile("test3.py")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call
last)

/home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/<console> 

/home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/test3.py 
     24 for i in range(10):
     25     d = d + [array([1.0,1.0])]
     26 
---> 27 tab.append(d)
        tab = /Data (Table(0,)) 'Data'
  description := {
    ...pe=1, dflt=0.0, pos=None) }
  byteorder := little, global append = undefined, d = [array([ 1., 
1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]),
array([ 1.,  1.]), array([ 1.,  1.])]
     28 f.close()

/usr/lib/python2.3/site-packages/tables/Table.py in append(self=/Data
(Table(0,)) 'Data'
  description := {
    ...pe=1, dflt=0.0, pos=None) }
  byteorder := little, rows=[array([ 1.,  1.]), array([ 1.,  1.]),
array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1., 
1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
1.,  1.])])
    588         except:
    589             (type, value, traceback) = sys.exc_info()
--> 590             raise ValueError, \
        global ValueError = undefined, global str = undefined, self =
/Data (Table(0,)) 'Data'
  description := {
    ...pe=1, dflt=0.0, pos=None) }
  byteorder := little, value = <exceptions.IndexError instance>
    591 "rows parameter cannot be converted into a recarray object
compliant with table '%s'. The error was: <%s>" % (str(self), value)
    592         lenrows = recarray.shape[0]

ValueError: rows parameter cannot be converted into a recarray object
compliant with table '/Data (Table(0,)) 'Data''. The error was: <list
index out of range>

-- 
Daehyok Shin (Peter)
Geography Department
Univ. of North Carolina-Chapel Hill

[Pytables-users] How to subclass tables.File?

[Shin <sd...@em...>]

I want to subclass tables.File to add some customized behaviors.
But, it turned out not as simple as I had expected.
File class has __init__ method, but HDF5Extension.File, one of
tables.File's superclasses, has only __new__ method.
I am not familiar with new class system in Python.

To subclass tables.File, how should I implement __init__ or __new__
method of the new class? Thanks in advance.

-- 
Daehyok Shin (Peter)
Geography Department
Univ. of North Carolina-Chapel Hill

[Pytables-users] Help on testing PyTables 0.9

[Francesc A. <fa...@py...>]

Hi,

I'm in the final phase of testing PyTables 0.9 before releasing it. As the
number of supported platforms is increasing, I'm asking for your help here.

As of now, I've successfully tested PyTables mainly with Linux machines,
both 32-bits (Pentium IV) and 64-bits (Intel Itanium, AMD Opteron). However,
I would like to test, at very least, the Windows and MacOSX implementation.
If someone is able to test a BSD, Solaris or Irix platforms, much better :)

The procedure of compilation/installation has not been updated in
documentation yet, but it should be fairly similar to 0.8.1. You should be
able to run PyTables with both HDF5 1.6.2 and HDF5 1.6.3-post1, and either
numarray 1.0 or numarray 1.1 (although I've recently only tested numarray
1.1). 

You can find the 0.9 pre-release in:

http://pytables.sourceforge.net/html/GetIt.html

and clicking in "Preview of the forthcoming 0.9 release"

Please, have in mind that the new "official" version of HDF5, namely 1.6.3,
version came out with a nasty bug that prevents to delete an Index object
(which has been introduced in PyTables 0.9) and besides, it causes a
segmentation fault if you try to do that. I've asked to the NCSA to provide
binary libraries with that bug corrected (the -post1 version), and they are
willing to do that, at least for Windows and MacOSX platforms. So, any user
on Win or MacOSX can test HDF5 1.6.3-post1 binaries as well? I'm attaching
below the message from Elena Pourmal (the Maintenance and Support Team
Leader) with the location for these libraries and asking for confirmation
that the libraries works well.

One thing more, a Windows user has already told me that he is having
problems with the newly introduced calls to nextafter and nextafterf
mathematical functions that appear not to be on "math.h", which is the
official place in C99 standard. Someone can provide a patch for this (using
the Microsoft C Library)?.

Many thanks for your time!

----------  Missatge transmes  ----------

Subject: binaries with applied patch Re: Segmentation fault with 1.6.3
Date: Divendres 01 Octubre 2004 01:22
From: Elena Pourmal <epo...@nc...>
To: Francesc Alted <fa...@py...>
Cc: hd...@nc...

Francesc,

I've built binaries with applied path for Windows, Linux 2.4 and Mac OSX. 
Binaries are not configured with szip support but gzip support is included.

Files are on our ftp server in my outgoing directory:

ftp://hdf.ncsa.uiuc.edu/pub/outgoing/epourmal/patch

Could you please test at least one binary distribution and make sure it 
works for you?
If it works, we will move the files to our official distribution site and 
we will let you know the exact location.

If you want to build from scratch yourself, just drop H5Distore.c file into 
the src directory of the hdf5-1.6.3 source.

Thank you!

Elena

At 01:33 PM 9/28/2004, you wrote:
>A Dimarts 28 Setembre 2004 20:16, vareu escriure:
> > Francesc,
> >
> > We do not keep old binaries on the ftp server. Do you know for which
> > platforms your users need HDF5 binaries?
> >
>
>I was thinking mostly in Windows platforms, but I know that there are some
>other PyTables users using MacOSX.
>
>Another unfortunate consequence of the bug in the official release is that,
>normally, this is the one choosed by package maintainers for including in
>linux distributions (at least, this is the case for Debian), so most
>probably, most Linux distributions will distribute the buggy release in form
>of binary packages.
>
>Anyway, if it is difficult to you to integrate the patch for 1.6.3, I'm
>afraid that I'll have to disable indexing features for the forthcoming
>pytables 0.9. That is not necessarily too bad, only that users will have to
>wait for this functionality until HDF5 1.6.4 or whatever.
>
>Regards,
>
>--
>Francesc Alted

-----------------------------------------------------------------------------------------------
Elena Pourmal, HDF QA, Maintenance and Support Team Leader
NCSA University of Illinois at Urbana-Champaign
605 E. Springfield Ave.
Champaign, IL 61820

epo...@nc...
(217)333-0238 (office)
(217)244-1987 (fax)
----------------------------------------------------------------------------------------------



-------------------------------------------------------

-- 
Francesc Alted

Re: Fwd: Re: [Pytables-users] Danger of file corruption?

[Francesc A. <fa...@py...>]

A Dijous 30 Setembre 2004 18:44, Norbert Nemec va escriure:
> Sounds like a fundamental flaw in the design of HDF5 - a high-end data-storage 
> system like HDF5 without protection against file corruption sounds 
> unbelievable to me.
> 
> The argument that this is the same for any kind of file-storage does not hold 
> here: I you use a journalled filesystem, you know that basic write operations 
> are atomic. Based on that, it is possible with some care to design a 
> file-format and protocol of writing that leaves the file in a well-defined 
> state at any moment.

I agree. The problem is that HDF5 is not journaled, so you cannot be certain
that an on-going operation would be atomic.

Making HDF5 to support journaled files would be a nice idea, but the problem
is who is willing to do that.

Anyway, I think it might be better to debate this on HDF5 list, rather than
here.

Cheers,

-- 
Francesc Alted

Fwd: Re: [Pytables-users] Danger of file corruption?

[Francesc A. <fa...@py...>]

Another contribution of Quincey Koziol, one of the HDF5 developers.

In next PyTables 0.9 H5Fflush() will be called automatically when
File.flush() method is invoked. For 0.8.1 there is a patch for that (see
patch #947006 at the pytables project:
http://sourceforge.net/tracker/?group_id=63486&atid=504146)

Cheers,

----------  Missatge transmes  ----------

Subject: Re: Re: [Pytables-users] Danger of file corruption?
Date: Dijous 30 Setembre 2004 18:30
From: Quincey Koziol <ko...@nc...>
To: Matthew Street <Mat...@aw...>
Cc: fa...@py..., hd...@nc...

Hi Matt, Francesc, et al.
    Yes, you are correct - due to the caching that the HDF5 library performs,
it is highly likely that a file opened for write access (and actually modified)
will be corrupt if the application terminates abruptly.

    The preferred solution in this case is not to close and re-open the
file repeatedly, which will produce poor performance, but to call H5Fflush(),
which will syncronize the information on disk with the information cached in
memory.  It is also possible to turn off the various caches in the library with
API calls, but I don't recommend doing this due to the loss in performance.

    Quincey

> My experience has been that the file does indeed become corrupted, and the
> data in it unaccessable, if the code is terminated for some reason.  We have
> found this to happen if the code crashes, is removed by a scheduler, and
> also if the filesystem it is writing to becomes temporarily unavailable.
> These kinds of issues would affect most file operations though - not just
> HDF5 ones.
> 
> A lot of the users I support write timestep dumps, and I always advise
> writing a number of dumps containing N timesteps, rather than putting them
> all in one, just so that all data is not lost if the worst happens.  I
> always get them to close and re-open the file between dumps too as leaving
> it open can be dangerous - I think the H5Fclose routine needs to complete
> properly for the file to be non-corrupt, if this is the case 'atomic'-type
> operations probably wont help.
> 
> Hope this helps,
> 
> Matt
> 
> 
> --
> _______________________________________________________________________
> Matt Street
> Parallel Technology Support	Tel:	+44 (0) 118 982 4528
> Applied Computer Science 	AWE, Aldermaston, Reading, RG7 4PR. UK.
> 
> 
> > -----Original Message-----
> > From: Francesc Alted
> > Sent: 30 September 2004 12:42
> > 
> > 
> > Hi,
> > 
> > A PyTables user has sent me couple of questions about HDF5 
> > behaviour in case of a sudden sutdown (or crash) of the 
> > program that is writing a file.
> > 
> > I'm forwarding the questions more related with HDF5, 
> > toghether with my answers. Can anybody give us a bit of light 
> > on this issues?
> > 
> > Thanks,
> > 
> > ----------  Missatge transmes  ----------
> > 
> > Subject: Re: [Pytables-users] Danger of file corruption? 
> > (was: Bug? flush() ignores new tables)
> > Date: Dijous 30 Setembre 2004 11:00
> > From: Francesc Alted <fa...@py...>
> > To: pyt...@li...
> > 
> > A Dijous 30 Setembre 2004 10:27, Norbert Nemec va escriure:
> > > * Are the elementary operations of the underlying hdf5 
> > library atomic? 
> > > I.e.:
> > > If I shut down the program in the middle of a pytables 
> > write action, might 
> > > the resulting file be fundamentally corrupted at the hdf5 
> > level, or would the 
> > > corruption be well localized and reparable?
> > 
> > I don't really know that.  I do know that HDF5 has a quite 
> > sophisticated cache system (both for data and metadata), but 
> > I know nothing about possible corruption scenarios and 
> > neither if they would be reparable or not. That might be a 
> > good question for the HDF5 list. Do you want me to ask this? 
> > Anyway, if you want to ask yourself, the HDF5 list is 
> > hd...@nc....
> > 
> > > All leading to the third question:
> > > 
> > > * Might there be some simple way to make a set of changes 
> > atomic? Even
> > > if that means restricting the use of pytables?
> > 
> > By atomic you mean in a way that always let the file 
> > uncorrupted?. I just don't know, it depends on how HDF5 deals 
> > with this situations, but a priori my feeling is negative.
> > 
> > Cheers,
> > 
> > -- 
> > Francesc Alted
> --
> _______________________________________________________________________________
> 
> The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited.  Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer.  While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected.
> 
> AWE Plc
> Registered in England and Wales
> Registration No 02763902
> AWE, Aldermaston, Reading, RG7 4PR
> 



-------------------------------------------------------

-- 
Francesc Alted

Re: Fwd: Re: [Pytables-users] Danger of file corruption?

[Norbert N. <Nor...@gm...>]

Sounds like a fundamental flaw in the design of HDF5 - a high-end data-storage 
system like HDF5 without protection against file corruption sounds 
unbelievable to me.

The argument that this is the same for any kind of file-storage does not hold 
here: I you use a journalled filesystem, you know that basic write operations 
are atomic. Based on that, it is possible with some care to design a 
file-format and protocol of writing that leaves the file in a well-defined 
state at any moment.

If, in HDF5, the basic operations are not atomic in the same sense, it will 
never be possible to build a secure system on top of it.

Writing additional copies of the file every now and then is, of course, a 
solution, but it really destroys all the benefits from the high-performance 
file format HDF5...

On Thursday 30 September 2004 17:02, Francesc Alted wrote:
> Someone at the list of HDF5 has answered. It looks like the danger of
> corruption is real. Matt suggest writing a number of dumps to avoid the
> worst to happen.
>
> Cheers,

-- 
_________________________________________Norbert Nemec
        Altdorfer Str. 9a ... D-93049 Regensburg
    Tel: 0941 / 2009638 ... Mobile: 0179 / 7475199
           eMail: <No...@Ne...>

Fwd: Re: [Pytables-users] Danger of file corruption?

[Francesc A. <fa...@py...>]

Someone at the list of HDF5 has answered. It looks like the danger of
corruption is real. Matt suggest writing a number of dumps to avoid the
worst to happen.

Cheers,

-- 
Francesc Alted

----------  Missatge transmes  ----------

Subject: RE: Re: [Pytables-users] Danger of file corruption?
Date: Dijous 30 Setembre 2004 15:57
From: "Matthew Street" <Mat...@aw...>
To: fa...@py..., hd...@nc...


Hi,

My experience has been that the file does indeed become corrupted, and the
data in it unaccessable, if the code is terminated for some reason.  We have
found this to happen if the code crashes, is removed by a scheduler, and
also if the filesystem it is writing to becomes temporarily unavailable.
These kinds of issues would affect most file operations though - not just
HDF5 ones.

A lot of the users I support write timestep dumps, and I always advise
writing a number of dumps containing N timesteps, rather than putting them
all in one, just so that all data is not lost if the worst happens.  I
always get them to close and re-open the file between dumps too as leaving
it open can be dangerous - I think the H5Fclose routine needs to complete
properly for the file to be non-corrupt, if this is the case 'atomic'-type
operations probably wont help.

Hope this helps,

Matt


--
_______________________________________________________________________
Matt Street
Parallel Technology Support	Tel:	+44 (0) 118 982 4528
Applied Computer Science 	AWE, Aldermaston, Reading, RG7 4PR. UK.


> -----Original Message-----
> From: Francesc Alted
> Sent: 30 September 2004 12:42
> 
> 
> Hi,
> 
> A PyTables user has sent me couple of questions about HDF5 
> behaviour in case of a sudden sutdown (or crash) of the 
> program that is writing a file.
> 
> I'm forwarding the questions more related with HDF5, 
> toghether with my answers. Can anybody give us a bit of light 
> on this issues?
> 
> Thanks,
> 
> ----------  Missatge transmes  ----------
> 
> Subject: Re: [Pytables-users] Danger of file corruption? 
> (was: Bug? flush() ignores new tables)
> Date: Dijous 30 Setembre 2004 11:00
> From: Francesc Alted <fa...@py...>
> To: pyt...@li...
> 
> A Dijous 30 Setembre 2004 10:27, Norbert Nemec va escriure:
> > * Are the elementary operations of the underlying hdf5 
> library atomic? 
> > I.e.:
> > If I shut down the program in the middle of a pytables 
> write action, might 
> > the resulting file be fundamentally corrupted at the hdf5 
> level, or would the 
> > corruption be well localized and reparable?
> 
> I don't really know that.  I do know that HDF5 has a quite 
> sophisticated cache system (both for data and metadata), but 
> I know nothing about possible corruption scenarios and 
> neither if they would be reparable or not. That might be a 
> good question for the HDF5 list. Do you want me to ask this? 
> Anyway, if you want to ask yourself, the HDF5 list is 
> hd...@nc....
> 
> > All leading to the third question:
> > 
> > * Might there be some simple way to make a set of changes 
> atomic? Even
> > if that means restricting the use of pytables?
> 
> By atomic you mean in a way that always let the file 
> uncorrupted?. I just don't know, it depends on how HDF5 deals 
> with this situations, but a priori my feeling is negative.
> 
> Cheers,
> 
> -- 
> Francesc Alted
--
_______________________________________________________________________________

The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited.  Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer.  While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected.

AWE Plc
Registered in England and Wales
Registration No 02763902
AWE, Aldermaston, Reading, RG7 4PR



-------------------------------------------------------

Re: [Pytables-users] Danger of file corruption? (was: Bug? flush() ignores new tables)

[Francesc A. <fa...@py...>]

A Dijous 30 Setembre 2004 10:27, Norbert Nemec va escriure:
> * Does pytables ever initiate writes except in explicit flush() and close() 
> commands?

Well, there are two levels of cache buffers: table buffers in PyTables and
HDF5 buffers. Now, PyTables triggers a write operation by calling an HDF5
function in two situations: 

1) When table buffers are full

2) In any other write operation (writing attributes or *Array elements)

then, is responsability of the underlying HDF5 library to write the data on
disk.

> * Are the elementary operations of the underlying hdf5 library atomic? I.e.: 
> If I shut down the program in the middle of a pytables write action, might 
> the resulting file be fundamentally corrupted at the hdf5 level, or would the 
> corruption be well localized and reparable?

I don't really know that.  I do know that HDF5 has a quite sophisticated
cache system (both for data and metadata), but I know nothing about
possible corruption scenarios and neither if they would be reparable or not.
That might be a good question for the HDF5 list. Do you want me to ask this?
Anyway, if you want to ask yourself, the HDF5 list is hd...@nc....

> All leading to the third question:
> 
> * Might there be some simple way to make a set of changes atomic? Even 
> if that means restricting the use of pytables?

By atomic you mean in a way that always let the file uncorrupted?. I just
don't know, it depends on how HDF5 deals with this situations, but a priori
my feeling is negative.

Cheers,

-- 
Francesc Alted

Re: [Pytables-users] Danger of file corruption? (was: Bug? flush() ignores new tables)

[Norbert N. <Nor...@gm...>]

On Thursday 30 September 2004 09:56, Francesc Alted wrote:
> A Dijous 30 Setembre 2004 09:32, Norbert Nemec va escriure:
> > Even though: I would strongly hope that it is not necessary. Does
> > h5file.close() do anything on the file but flush() it and then close the
> > physical file? In other words: is there any danger of file corruption if
> > the python program is just shut down cold without a chance to save
> > anything? Like - when the file is written via NFS and the computer
> > running the program is switched off hard?
>
> I'm not completely sure, but muy guts say that this would not be a problem.
> If your program crashes (or is shut down) and there are buffers that are
> not written yet, then the file will miss this information, but will not get
> corrupted.
>
> The only problem may appear if you shut down the program *during* a write
> process. In that case I guess you will end with a corrupted file, as there
> are no provision of I/O transactions yet.

OK, that leaves me two specific questions:

* Does pytables ever initiate writes except in explicit flush() and close() 
commands?

* Are the elementary operations of the underlying hdf5 library atomic? I.e.: 
If I shut down the program in the middle of a pytables write action, might 
the resulting file be fundamentally corrupted at the hdf5 level, or would the 
corruption be well localized and reparable?

All leading to the third question:

* Might there be some simple way to make a set of changes atomic? Even if that 
means restricting the use of pytables?

Ciao,
Nobbi

-- 
_________________________________________Norbert Nemec
        Altdorfer Str. 9a ... D-93049 Regensburg
    Tel: 0941 / 2009638 ... Mobile: 0179 / 7475199
           eMail: <No...@Ne...>

Re: [Pytables-users] Danger of file corruption? (was: Bug? flush() ignores new tables)

[Francesc A. <fa...@py...>]

A Dijous 30 Setembre 2004 09:32, Norbert Nemec va escriure:
> Even though: I would strongly hope that it is not necessary. Does 
> h5file.close() do anything on the file but flush() it and then close the 
> physical file? In other words: is there any danger of file corruption if the 
> python program is just shut down cold without a chance to save anything? Like 
> - when the file is written via NFS and the computer running the program is 
> switched off hard?

I'm not completely sure, but muy guts say that this would not be a problem.
If your program crashes (or is shut down) and there are buffers that are not
written yet, then the file will miss this information, but will not get
corrupted.

The only problem may appear if you shut down the program *during* a write
process. In that case I guess you will end with a corrupted file, as there
are no provision of I/O transactions yet.

Cheers,

-- 
Francesc Alted

[Pytables-users] Danger of file corruption? (was: Bug? flush() ignores new tables)

[Norbert N. <Nor...@gm...>]

On Wednesday 29 September 2004 20:10, Ivan Vilata i Balaguer wrote:
> On Wed, Sep 29, 2004 at 01:51:32PM +0200, Francesc Alted wrote:
> > A Dimecres 29 Setembre 2004 11:57, vareu escriure:
> > [...]
> >
> > > Maybe there is another way to go? My intention is to have one script
> > > that opens an existing hdf5 datafile or creates it from scratch if it
> > > does not exist and then starts a lengthy calculation saving the curre=
nt
> > > state every few minutes. Usually, the script would never finish but r=
un
> > > indefinitely until interrupted by Ctrl-C (therefore close() is never
> > > called)
> > >
> > > Another script would run interactively in parallel (maybe even on
> > > another machine) reading the current state from disk and evaluating t=
he
> > > data.
> >
> > I do think so. At least there should be someone in the pytables users
> > list that are using pytables for doing similar things.
> > [...]
>
>       Anyway, I would suggest enclosing the loop in a =E2=80=9Ctry=E2=80=
=9D clause to
>   try and catch the KeyboardInterrupt exception, or maybe use a
>   =E2=80=9Cfinally=E2=80=9C clause to close the file in any situation.  B=
ye!

Good idea, thanks!

Even though: I would strongly hope that it is not necessary. Does=20
h5file.close() do anything on the file but flush() it and then close the=20
physical file? In other words: is there any danger of file corruption if th=
e=20
python program is just shut down cold without a chance to save anything? Li=
ke=20
=2D when the file is written via NFS and the computer running the program i=
s=20
switched off hard?

=2D-=20
_________________________________________Norbert Nemec
        Altdorfer Str. 9a ... D-93049 Regensburg
    Tel: 0941 / 2009638 ... Mobile: 0179 / 7475199
           eMail: <No...@Ne...>

Re: [Pytables-users] Re: Bug? flush() ignores new tables

[Ivan V. i B. <iv...@ca...>]

On Wed, Sep 29, 2004 at 01:51:32PM +0200, Francesc Alted wrote:
> A Dimecres 29 Setembre 2004 11:57, vareu escriure:
> [...]
> > Maybe there is another way to go? My intention is to have one script th=
at=20
> > opens an existing hdf5 datafile or creates it from scratch if it does n=
ot=20
> > exist and then starts a lengthy calculation saving the current state ev=
ery=20
> > few minutes. Usually, the script would never finish but run indefinitel=
y=20
> > until interrupted by Ctrl-C (therefore close() is never called)
> >=20
> > Another script would run interactively in parallel (maybe even on anoth=
er=20
> > machine) reading the current state from disk and evaluating the data.
>=20
> I do think so. At least there should be someone in the pytables users list
> that are using pytables for doing similar things.
> [...]

      Anyway, I would suggest enclosing the loop in a =E2=80=9Ctry=E2=80=9D=
 clause to
  try and catch the KeyboardInterrupt exception, or maybe use a
  =E2=80=9Cfinally=E2=80=9C clause to close the file in any situation.  Bye!

--=20
Ivan Vilata i Balaguer  	@ =C2=A1Banda ancha asequible para Espa=C3=B1a ya!=
 @
http://www.selidor.net/ 	@  http://www.petitiononline.com/prot1/  @

[Pytables-users] Re: Bug? flush() ignores new tables

[Francesc A. <fa...@py...>]

Hi Nobbi,

A Dimecres 29 Setembre 2004 11:57, vareu escriure:
> Hi there,
> 
> just starting to use pytables. I was rather surprised by the fact that flush() 
> does not write new tables to disk. Instead, I have to close() the file and 
> then reopen it again.

Yes, this is is a known bug. This is solved in CVS version, and some people
has made a patch for 0.8, and I guess it will apply well to 0.8.1 as well
(see patch #947006 at the pytables project:
http://sourceforge.net/tracker/?group_id=63486&atid=504146)

> Maybe there is another way to go? My intention is to have one script that 
> opens an existing hdf5 datafile or creates it from scratch if it does not 
> exist and then starts a lengthy calculation saving the current state every 
> few minutes. Usually, the script would never finish but run indefinitely 
> until interrupted by Ctrl-C (therefore close() is never called)
> 
> Another script would run interactively in parallel (maybe even on another 
> machine) reading the current state from disk and evaluating the data.

I do think so. At least there should be someone in the pytables users list
that are using pytables for doing similar things.

> Is this setup possible with pytables at all? I read somewhere that HDF5 
> supports parallel file access in some way. Does PyTables allow this as well?

No, pytables does not support parallel file access as of now.

Cheers,

-- 
Francesc Alted

[Pytables-users] Heaven and earth interface [Was: Special __call__ method...]

[Ivan V. i B. <iv...@ca...>]

On Thu, Sep 16, 2004 at 01:14:12PM -0400, st...@st... wrote:
> Removing the __call__() methods would make me happier.
> In code I (should) always write the long version anyway,
> because it is easier to read. And interactively, the
> call methods save steps only at a normal python prompt,
> they are an inconvenience in ipython.=20

      I absolutely agree with removing those methods, since they quite
  obscure the source.

      In fact I am writing this mail to discuss a new interface I have
  devised for navigation in PyTables.  Of course it is not my intention
  for it to go into the next version, but I'd appreciate your opinion.

      I call this the =E2=80=9Cheaven and earth=E2=80=9D interface, but I r=
ecognise the
  name is quite arbitrary.  Given the following hierarcy:

	          (root)
	 at1-.   /    \   .-at1
	 at2--(g1)    [t2]--at2
	       /  \
	    [t1]  [a1]

      This hierarchy has two parallel =E2=80=9Cnatures=E2=80=9D, one that c=
an be
  accessed by string paths (such as '/g1/a1') using the getNode() method,
  and one that can be accessed by natural naming.  Given a node in the
  hierarchy, one can =E2=80=9Cwarp=E2=80=9D from one nature to the other.  =
We will call
  the first nature the =E2=80=9Cheaven=E2=80=9C nature, and the second one =
the =E2=80=9Cearth=E2=80=9D
  nature.

      In the heaven nature, node and attribute identifiers are never used
  as Python identifiers, but strings, which would allow them to have any
  form (as long as they have no slashes).  As said, nodes are accessed via
  the getNode() method, so any other method name can be used with no risk
  of clashing.  Attribute sets are accessed via a node's getAttributes()
  method (alternatives: an attributes() method or an =E2=80=9Cattrs=E2=80=
=9D attribute).
  Attributes in an attribute set may be accessed using some method, which
  could easily be __getitem__() (i.e. attrs['attrname']).

      In the earth nature, node and attribute identifiers are used as
  Python identifiers, so their form is restricted.  Method names and
  special node attributes *always* start with =E2=80=9C_=E2=80=9D.

      Since the earth nature is aimed to interactive usage, the set of
  methods supported by a node or attribute set might be quite reduced,
  maybe only covering read-only access methods.  The reason for this is
  avoiding redundance between heaven and earth classes.  However, if the
  whole functionality is desired in both class types, coding the heaven
  methods in terms of the earth ones would simply be a matter of warping,
  calling the earth method and possibly warping back the result.

	root =3D h5file.getNode('/')

	g1 =3D h5file.getNode('/g1')
	g1 =3D root.getNode('g1')     # Non-leaf nodes have a getNode() method ...
	t1 =3D g1.getNode('t1')       # ... but only relative paths are allowed.
	a1 =3D root.getNode('g1/a1')
	t2 =3D h5file.getNode('/t2')  # Absolute paths are only allowed on File ob=
jects.


	# What if '.', '..', and absolute paths were allowed (using weakrefs)
	# on any (not only non-leaf) object?
	# t1 =3D g1.getNode('/g1/t1')
	# t1 =3D g1.getNode('./t1')
	# a1 =3D t1.getNode('../a1')

	# Getting a node's attribute in heaven.
	g1_at1 =3D root.getNode('g1').getAttributes()['at1']
	# g1_at1 =3D root.getNode('g1').attributes()['at1']
	# g1_at1 =3D root.getNode('g1').attrs['at1']

	# Warping to earth.
	e_root =3D root.warp()
	e_g1 =3D e_root.g1
	e_a1 =3D e_g1.a1

	# Getting a node's attribute in earth.
	e_g1_at1 =3D e_g1._attrs.at1
	e_g1_at2 =3D e_g1._attrs._warp()['at2']
	e_t2_at2 =3D e_root.t2._attrs.t2

	# Warping back to heaven.
	t1 =3D e_g1._warp().getNode('t1')

	# Example of earth node methods.
	def _someMethodReturningANode(self, args):
		return self._warp().someMethodReturningANode(args).warp()

	def _someMethodNotReturningANode(self, args):
		return self._warp().someMethodNotReturningANode(args)

      What do you think of this proposal?  Does it look more or less
  intuitive?  Do you guess more or less immediate problems (apart of
  interface incompatibility, of course)?

  Regards,
      Ivan

--=20
Ivan Vilata i Balaguer  	@      FIGHT AGAINST SOFTWARE PATENTS!     @
http://www.selidor.net/ 	@ http://EuropeSwPatentFree.hispalinux.es/ @

Re: [Pytables-users] Complex numbers

[Francesc A. <fa...@py...>]

Hi List,

After a couple of days of hacking, the patch for supporting complex numbers
has been applied. In addition I've managed to add support of complex numbers
to Tables as well. Documentation has been updated as well.

So, everything is ready for complex datatypes support for next public
release of PyTables. 

Rejoice!

-- 
Francesc Alted

Re: [Pytables-users] Special __call__ method in PyTables objects

[Norm P. <nj...@nj...>]

+1 for "explicit is better" (the suggested change is fine, and as a complete
pytables newbie, I don't have any affected production code)

Thanks for making just the tool I need right now to manage a lot of
industrial real-time data. Nice work!

----- Original Message ----- 
From: "Francesc Alted" <fa...@py...>
To: <pyt...@li...>
Sent: Thursday, September 16, 2004 12:47 PM
Subject: [Pytables-users] Special __call__ method in PyTables objects


> Dear PyTables users,
>
> I'm pondering to disable most of __call__() methods is PyTables objects
> because of two things:
>
> 1.- ipython (http://ipython.scipy.org/), which is a very good shell for
> Python adds by default the parenthesis '()' if you type just a name at the
> prompt and nothing more. Example:
>
> In [7]: float
> ------> float()
> Out[7]: 0.0
>
> This is considered a good thing for most uses. Unfortunately, PyTables is
> designed to make use of bare object names to print a description of the
> object. That is, in the standard python console:
>
> >>> f.root.small
> /small (Group) 'small Group'
>   children := ['create_worst' (Table), 'search_best' (Group),
'search_worst' (Group), 'create_best' (Table)]
>
> but the same in ipython gives:
>
> In [9]: f.root.small
> ------> f.root.small()
> Out[9]: <generator object at 0x415d490c>
>
> which is not was the user intended usually.
>
> 2.- More importantly, in my struggle to make interactive work with
pytables
> easy, I'm afraid that I've abused a bit of the __call__ method, in most
> cases being introduced as a shortcut basically. For example, this is a
> typical use:
>
> >>> names=[ x['name'] for x in table(start=2, stop=1000) if
x['TDCcount']>3]
>
> which is the same as:
>
> >>> names=[ x['name'] for x in table.iterrows(start=2, stop=1000) if
x['TDCcount']>3]
>
> As I see it now, I find the second version much more clear, and although
it
> is a bit more expensive to type, if we follow the mantra "explicit is
better
> than implicit" I definitely think that the second is the way to go.
>
> So, these are the reasons why I'm planning to remove most of the
__call__()
> methods and, if necessary, replace them by acurate method names. Although
I
> think this is a good thing (TM), this may pose problems of backward
> compatibility with existing programs, and this is why I'm asking here.
>
> What do you think? You would like the change? That might cause problems
with
> your programs? Do you have a better alternative?
>
> Cheers,
>
> -- 
> Francesc Alted
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>

Re: [Pytables-users] Special __call__ method in PyTables objects

[<st...@st...>]

Removing the __call__() methods would make me happier.
In code I (should) always write the long version anyway,
because it is easier to read. And interactively, the
call methods save steps only at a normal python prompt,
they are an inconvenience in ipython. 

-- 
Stan

[Pytables-users] Special __call__ method in PyTables objects

[Francesc A. <fa...@py...>]

Dear PyTables users,

I'm pondering to disable most of __call__() methods is PyTables objects
because of two things:

1.- ipython (http://ipython.scipy.org/), which is a very good shell for
Python adds by default the parenthesis '()' if you type just a name at the
prompt and nothing more. Example:

In [7]: float
------> float()
Out[7]: 0.0

This is considered a good thing for most uses. Unfortunately, PyTables is
designed to make use of bare object names to print a description of the
object. That is, in the standard python console:

>>> f.root.small
/small (Group) 'small Group'
  children := ['create_worst' (Table), 'search_best' (Group), 'search_worst' (Group), 'create_best' (Table)]

but the same in ipython gives:

In [9]: f.root.small
------> f.root.small()
Out[9]: <generator object at 0x415d490c>

which is not was the user intended usually.

2.- More importantly, in my struggle to make interactive work with pytables
easy, I'm afraid that I've abused a bit of the __call__ method, in most
cases being introduced as a shortcut basically. For example, this is a
typical use:

>>> names=[ x['name'] for x in table(start=2, stop=1000) if x['TDCcount']>3]

which is the same as:

>>> names=[ x['name'] for x in table.iterrows(start=2, stop=1000) if x['TDCcount']>3]

As I see it now, I find the second version much more clear, and although it
is a bit more expensive to type, if we follow the mantra "explicit is better
than implicit" I definitely think that the second is the way to go.

So, these are the reasons why I'm planning to remove most of the __call__()
methods and, if necessary, replace them by acurate method names. Although I
think this is a good thing (TM), this may pose problems of backward
compatibility with existing programs, and this is why I'm asking here.

What do you think? You would like the change? That might cause problems with
your programs? Do you have a better alternative?

Cheers,

-- 
Francesc Alted