pytables-users — PyTables users discussion list

Re: [Pytables-users] creating numarray.array from python comprehension list

[Francesc A. <fa...@ca...>]

A Thursday 16 June 2005 14:17, phi...@ho... va escriure:
> for retrieving all the data from an Array (or a Table) object, i read
> the data and save it into a numarray.array using the read() function:
> my_numarray_array =3D my_object.read().
>
> for example:
> readout =3D h.root.detector.readout
> b=3Dreadout.read()
> print b
>
> That's really nice to use and i can easilly treat data after that.

With *Array objects you can just select multimensional slices:

readout[2,3:20,...,:]

But filtering by value is not supported for *Array objects, at least
in a efficient way. You should use Table objects instead.

> My question is about selecting only a part of data:
> Can i create directly a numarray.array object using a python list
> comprehension on an Array or a Table?
>
> For the moment using list comprehension, i only get a list.
>
> for example:
>  >>> pressure =3D h.root.columns.pressure
>  >>> a=3D[p for p in pressure]
>  >>> print a
>
> [25.0, 36.0, 49.0]
>
>  >>> a=3D[p for p in pressure if p <36]
>  >>> print a
>
> [25.0]
>
>  >>> type(a)
>
> <type 'list'>
>
> I want to get a numarray.array.
>
> 1. Can i cast the list into a numarray.array?

Yes: array(a). Read the numarray manual.

> 2. Can i copy all the data from the list to a numarray.array once?

Yes, as above.

> 3. Must i instrospect the list using the for elt in list: and add each
> data in a numarray?

You can, but this is quite inefficient.

Anyhow, if you want to do efficient searches and get numarray object
as a result take a look at this example:

In [25]:rout =3D f.root.detector.readout

In [26]:rout
Out[26]:
/detector/readout (Table(10L,)) 'Readout example'
  description :=3D {
  "ADCcount": Col(dtype=3D'UInt16', shape=3D1, dflt=3DNone, pos=3D0, indexe=
d=3DFalse),
  "TDCcount": Col(dtype=3D'UInt8', shape=3D1, dflt=3DNone, pos=3D1, indexed=
=3DFalse),
  "energy": Col(dtype=3D'Float64', shape=3D1, dflt=3DNone, pos=3D2, indexed=
=3DFalse),
  "grid_i": Col(dtype=3D'Int32', shape=3D1, dflt=3DNone, pos=3D3, indexed=
=3DFalse),
  "grid_j": Col(dtype=3D'Int32', shape=3D1, dflt=3DNone, pos=3D4, indexed=
=3DFalse),
  "idnumber": Col(dtype=3D'Int64', shape=3D1, dflt=3DNone, pos=3D5, indexed=
=3DFalse),
  "name": StringCol(length=3D16, dflt=3DNone, shape=3D1, pos=3D6, indexed=
=3DFalse),
  "pressure": Col(dtype=3D'Float32', shape=3D1, dflt=3DNone, pos=3D7, index=
ed=3DFalse)}
  byteorder :=3D little

In [28]:rout.cols.pressure[:]
Out[28]:array([  0.,   1.,   4.,   9.,  16.,  25.,  36.,  49.,  64.,  81.],=
=20
type=3DFloat32)

In [31]:idx =3D rout.getWhereList(rout.cols.pressure < 36, flavor=3D"NumArr=
ay")

In [32]:idx
Out[32]:array([0, 1, 2, 3, 4, 5], type=3DInt64)

In [33]:f.root.detector.readout.readCoordinates(idx, field=3D"pressure")
Out[33]:array([  0.,   1.,   4.,   9.,  16.,  25.], type=3DFloat32)

This is really fast as most of the operations are made in C space;
besides, you don't need to load all your table data in memory. If you
want further speed, you might want to index any of the columns you are
using to filter the data. See the PyTables manual for more info on
indexing capabilities.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] creating numarray.array from python comprehension list

[<phi...@ho...>]

Hi list,

I want to select data from an hdf5 file using pytables.

for retrieving all the data from an Array (or a Table) object, i read 
the data and save it into a numarray.array using the read() function:
my_numarray_array = my_object.read().

for example:
readout = h.root.detector.readout
b=readout.read()
print b

That's really nice to use and i can easilly treat data after that.

My question is about selecting only a part of data:
Can i create directly a numarray.array object using a python list 
comprehension on an Array or a Table?

For the moment using list comprehension, i only get a list.

for example:
 >>> pressure = h.root.columns.pressure
 >>> a=[p for p in pressure]
 >>> print a
[25.0, 36.0, 49.0]
 >>> a=[p for p in pressure if p <36]
 >>> print a
[25.0]
 >>> type(a)
<type 'list'>

I want to get a numarray.array.

1. Can i cast the list into a numarray.array?
2. Can i copy all the data from the list to a numarray.array once?
3. Must i instrospect the list using the for elt in list: and add each 
data in a numarray?

Thanks a lot for your help,
Philippe

[Pytables-users] Re: Tables with arbitrary number of columns

[Francesc A. <fa...@ca...>]

[I'm CC-ing this to pytables-users. Hope you don't mind]

A Wednesday 15 June 2005 11:09, v=E0reu escriure:
> Hi Francesc!
>
> I have contacted you before about using PyTables for storing simulation
> data. We have decided to make use of your program since it has so many
> powerful features. Nevertheless I have a question regarding the creation =
of
> tables. Suppose we have a class like this (taken from your example):
>
> class Particle(IsDescription):
>     name      =3D StringCol(16)   # 16-character String
>     idnumber  =3D Int64Col()      # Signed 64-bit integer
>     ADCcount  =3D UInt16Col()     # Unsigned short integer
>     TDCcount  =3D UInt8Col()      # unsigned byte
>     grid_i    =3D Int32Col()      # integer
>     grid_j    =3D IntCol()        # integer (equivalent to Int32Col)
>     pressure  =3D Float32Col()    # float  (single-precision)
>     energy    =3D FloatCol()      # double (double-precision)
>
> and later on we create a table with:
>
> table =3D h5file.createTable(group, 'readout', Particle, "Readout example=
")
>
> The third parameter of the createTable function is the name of the class,
> in this example Particle - a user record for characterizing 8 attributes
> (name, idnumber, ADCcount …), which correspond to 8 columns of a
> table. Now, suppose I want to write a new function for dealing with tables
> where a user can specify an arbitrary number of columns with different da=
ta
> types. How can I make a custom class (Particle in this example) to change
> its number of attributes corresponding to user request/input of the number
> of columns that it wishes to have?
> I hope I was clear enough.

I am not sure to completely understand you, but perhaps creating a
table using a dictionary as a description would be enough for you?

Look at:

http://pytables.sourceforge.net/html-doc/x912.html#secondExample

for an example of using dictionaries as table descriptors.

HTH,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] ptrepack and Windows

[Bernard K. <ber...@be...>]

Dear Francesc,

Thank you very much for your quick answer.

I just realized that "ptrepack" was simply a python script but without a 
".py" extension. I thought it was some kind of unix executable hence my 
email and incapability to run it.

Sincerely,

Bernard KAPLAN

Re: [Pytables-users] ptrepack and Windows

[Francesc A. <fa...@ca...>]

Hello Bernard,

A Dimarts 07 Juny 2005 17:19, Bernard KAPLAN va escriure:
> My questions are as follows :
> - How can I run "ptrepack" under Windows XP ?

Well, I personally never try it, but you should be able to do it. Look
were ptrepack was installed, put this directory in your path and you
should be done.

> - Same question with "ptdump" ? (for my information)

Same way.

> - Is it possible to find a way to convert the files without requiring a
> software like Cygwin ? I am asking this because if I decide to upgrade my
> software, I will have to convert the files that were created by the users
> and there is no way I can install a program like Cygwin.

To my understanding, you don't need any Cygwin software installed.

As an aside, please, note that you only need to repack your files if
they contains indexes and you want to use them.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] ptrepack and Windows

[Bernard K. <ber...@be...>]

Hello,

Recently I released a software that uses pytables 0.9 (python 2.3) and I would like to move to the new 1.0 version of pytables.

In the release notes I read :
"The format of indexes has been changed and indexes in files created with PyTables pre-1.0 versions are ignored now.  However,``ptrepack`` can still save your life because it is able to convert your old files into the new indexing format..."

My questions are as follows :
- How can I run "ptrepack" under Windows XP ?
- Same question with "ptdump" ? (for my information)
- Is it possible to find a way to convert the files without requiring a software like Cygwin ?
  I am asking this because if I decide to upgrade my software, I will have to convert the files that were created by the users and there is no way I can install a program like Cygwin.

Sincerely,

Bernard KAPLAN

[Pytables-users] Problem walking through structure of an hdf5 file and pygtk treeview

[<phi...@ho...>]

Hi again list,
Sorry using thunderbird, i send my mail under my default mail account so 
it's not the adress mail registered to the list.
Sorry for flooding :) (time to go into week end)

I want to display all the node of an hdf5 in a treeview but reflecting 
exactly the hdf5 structure of the file.
For example i try to display this structure in a treeview:

/ (Group)
-> columns (Group)
   -> name (Array)
   -> pressure (Array)
   -> test (Group)
-> detector (Group)
   -> readout (Table)

Here you probably recognize the structure of the tutorial hdf5 file  
except that i add an empty group test in columns group. ;)
My problem is i didn't succeed to display the real structure of the hdf5 
file in a treeview.

When i use a for loop listing nodes, it's not exactly the structure of 
the hdf5 file:
for node in h5file:
   components_list.append({'name':node._v_name, 
'type':node._v_attrs.CLASS})

the result is:
/
columns
detector
name
pressure
readin
test
readout

The problem is the group detector is displayed after the group columns 
and before the childs of the group columns.
So when i display it in a treeview, detector is part of columns group.

I use for the moment this loop:
for group in h5file.walkGroups("/"):
           if  group._v_name == "/":
               iter0 = self.__model.append(None, [group._v_name,"group"])
           else:
               iter1 = self.__model.append(iter0, [group._v_name,"group"])
           for array in h5file.listNodes(group, classname = 'Array'):
               if iter1 == None:
                   iter2 = self.__model.append(iter0, 
[array.hdf5name,"array"])
               else:
                   iter2 = self.__model.append(iter1, 
[array.hdf5name,"array"])
           for Table in h5file.listNodes(group, "Table"):#"Table"):
               if iter1 == None:
                   iter3 = self.__model.append(iter0, 
[Table.hdf5name,"Table"])
               else:
                   iter3 = self.__model.append(iter1, 
[Table.hdf5name,"Table"])

and add a different but not better result:

/ (Group)
-> columns (Group)
   -> name (Array)
   -> pressure (Array)
-> detector (Group)
   -> readout (Table)
-> test (Group)

How can i display the real structure of an hdf5 in a treeview?
Thanks in advance,
I'm sad to go into week end without solving this problem :/.
Have a good week end,
Philippe Collet

[Pytables-users] Problem walking through structure of an hdf5 file and pygtk treeview

[Philou <Kal...@wa...>]

Hi list,

I want to display all the node of an hdf5 in a treeview but reflecting 
exactly the hdf5 structure of the file.
For example i try to display this structure in a treeview:

 / (Group)
-> columns (Group)
    -> name (Array)
    -> pressure (Array)
    -> test (Group)
-> detector (Group)
    -> readout (Table)

Here you probably recognize the structure of the tutorial hdf5 file  
except that i add an empty group test in columns group. ;)
My problem is i didn't succeed to display the real structure of the hdf5 
file in a treeview.

When i use a for loop listing nodes, it's not exactly the structure of 
the hdf5 file:
for node in h5file:
    components_list.append({'name':node._v_name, 
'type':node._v_attrs.CLASS})

the result is:
/
columns
detector
name
pressure
readin
test
readout

The problem is the group detector is displayed after the group columns 
and before the childs of the group columns.
So when i display it in a treeview, detector is part of columns group.

I use for the moment this loop:
for group in h5file.walkGroups("/"):
            if  group._v_name == "/":
                iter0 = self.__model.append(None, [group._v_name,"group"])
            else:
                iter1 = self.__model.append(iter0, [group._v_name,"group"])
            for array in h5file.listNodes(group, classname = 'Array'):
                if iter1 == None:
                    iter2 = self.__model.append(iter0, 
[array.hdf5name,"array"])
                else:
                    iter2 = self.__model.append(iter1, 
[array.hdf5name,"array"])
            for Table in h5file.listNodes(group, "Table"):#"Table"):
                if iter1 == None:
                    iter3 = self.__model.append(iter0, 
[Table.hdf5name,"Table"])
                else:
                    iter3 = self.__model.append(iter1, 
[Table.hdf5name,"Table"])

and add a different but not better result:

 / (Group)
-> columns (Group)
    -> name (Array)
    -> pressure (Array)
-> detector (Group)
    -> readout (Table)
-> test (Group)

How can i display the real structure of an hdf5 in a treeview?
Thanks in advance,
I'm sad to go into week end without solving this problem :/.
Have a good week end,
Philippe Collet

Re: [Pytables-users] PyTables Error: from tables import *

[Francesc A. <fa...@ca...>]

On Thursday 02 June 2005 20:36, Benjamin Scott wrote:
> Note that the installation seems to go fine.  Although, when I run the
> test_all.py file it seems to fall over due to references to numarray;
> I am using Numeric.

Oops, you didn't mentioned this before: you absolutely need numarray,
even though you use only Numeric as data containers afterwards.

HTH,

-- 
Francesc

[Pytables-users] PyTables Error: from tables import *

[Benjamin S. <ben...@gm...>]

hello,

yesterday i posted my PyTables error to comp.lang.python (MsgID:
1117647060).  i haven't had a reply. Francesc Altet recommended that i
post my error here.  thanks in advance for your comments:

________________________________________

Hello,

I am using an XP box and Python 2.3 (Enthought Edition).  I am getting
the same error with both of the .exe's listed on sourceforge:

tables-1.0.win32-py2.3.exe
tables-1.0.LB.win32-py2.3.exe

Note that the installation seems to go fine.  Although, when I run the
test_all.py file it seems to fall over due to references to numarray;
I am using Numeric.

I am trying to follow the PyTables User Guide:

http://pytables.sourceforge.net/html-doc/

Per installation instructions I copied the 3 prereq files as noted
here:

http://pytables.sourceforge.net/html-doc/x420.html#subsection2.2.1

I am getting an error in the following section:

http://pytables.sourceforge.net/html-doc/c474.html#subsection3.1.1

In particular, "from tables import *" gives the following error:

Traceback (most recent call last):
 File "<pyshell#0>", line 1, in -toplevel-
   from tables import *
 File "...\site-packages\tables\__init__.py", line 33, in
-toplevel-
   from tables.hdf5Extension import\
ImportError: No module named hdf5Extension

Note that the only file with the name "hdf5Extension" in the
"...\site-packages\tables\" directory has the extension ".pyd" (a
.dll), not ".py" or ".pyc".

Suggestions?

Thanks in advance,

-Ben

Re: [Pytables-users] How persistent are the object ids?

[Francesc A. <fa...@ca...>]

A Dimarts 31 Maig 2005 17:18, Lars Heuer va escriure:
> Hi all,
>
> Each node in HDF gets an object id (_v_objectID).
> Is this object id stable and unique in each file or may the object id
> change under some circumstances (i.e. packing of the file etc.)?

Well, I'm not completely sure (as this depends on the underlying HDF5
library), but I'd bet that _v_objectID is immutable and unique for
each file.

However, I don't know if the same is true for files that have been
copied (for example, using tables.copyFile) or repacked (using
h5repack or ptrepack). I've made a couple of repackings and the ID's
for objects seem unchanged in new files, but I guess it may be a bit
risky to conclude that this is going to be the case for all cases.
Maybe it is safer to ask to the HDF5 list.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] Problem browsing the object tree of an hdf5 file

[Francesc A. <fa...@ca...>]

A Dimarts 31 Maig 2005 14:30, phi...@ho... va escriure:
> Hi list,
>
> I'm browsing an hdf5 file using pytables 1.0 with python 2.3 and fill a
> gtk treeview.
> I use a list composed with dictionaries to store the name and the type
> of node in the hdf5 file.
> Here is the method i use:
<stripped...>
> I didn't find a better way to do it. The problems are:
>     - i use lots of for loop -that's a memory waste-

Not necessarily, but I accept that is more CPU intensive.

>    -  i use three differents functions to get the name of node
> (group._v_name, array.hdf5name, Table.hdf5name). Is it appropriate.

Well, this can be somewhat improved. hdf5name is the real name of the
dataset on disk while name (or _v_name) are the names in PyTables
space (memory). You can use _v_name for all nodes for simplicity.

>    - When i see the result, a table which is in a group is displayed
> also in the root node '/'. I don't know why.

Can you provide a simple example that shows this behaviour?

> Is there a better way to do this?

What about:

for node in h5file:
    components_list.append({'name':node._v_name, 'type':node._v_attrs.CLASS=
})


Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] How persistent are the object ids?

[Lars H. <he...@se...>]

Hi all,

Each node in HDF gets an object id (_v_objectID).
Is this object id stable and unique in each file or may the object id
change under some circumstances (i.e. packing of the file etc.)?

Thanks in advance and best regards,
Lars
-- 
http://semagia.com

[Pytables-users] Problem browsing the object tree of an hdf5 file

[<phi...@ho...>]

Hi list,

I'm browsing an hdf5 file using pytables 1.0 with python 2.3 and fill a 
gtk treeview.
I use a list composed with dictionaries to store the name and the type 
of node in the hdf5 file.
Here is the method i use:

   def fill_treeview(self,filename=""):
       components_list=[]
       h5file = tables.openFile(filename, "a")
       for group in h5file.walkGroups("/"):
           components_list.append({'name':group._v_name,'type':"Group"})
           for array in h5file.listNodes(group, classname = 'Array'):
               
components_list.append({'name':array.hdf5name,'type':"Array"})
           for Table in h5file.walkNodes(group, "Table"):
               
components_list.append({'name':Table.hdf5name,'type':"Table"})

       for i in range(len(components_list)):
           iter = self.__model.insert_before(None, None)            
self.__model.set_value(iter, 0, components_list[i]['name'])
           self.__model.set_value(iter, 1, components_list[i]['type'])

I browsed the pytables several times.
I didn't find a better way to do it. The problems are:
    - i use lots of for loop -that's a memory waste-
   -  i use three differents functions to get the name of node 
(group._v_name, array.hdf5name, Table.hdf5name). Is it appropriate.
   - When i see the result, a table which is in a group is displayed 
also in the root node '/'. I don't know why.

Is there a better way to do this?

Perhaps it could be using only one loop with the node element?
for node in h5file:
   ...
In this case i don't know how to get the type of the node 
(i.e:Table,Array or Group).

Thans a lot for your answers,
Regards,
Philippe Collet

[Pytables-users] Problem browsing the object tree

[Philou <Kal...@wa...>]

Hi list,

I'm browsing an hdf5 file using pytables 1.0 with python 2.3 and fill a 
gtk treeview.
I use a list composed with dictionaries to store the name and the type 
of node in the hdf5 file.
Here is the method i use:

    def fill_treeview(self,filename=""):
        components_list=[]
        h5file = tables.openFile(filename, "a")
        for group in h5file.walkGroups("/"):
            components_list.append({'name':group._v_name,'type':"Group"})
            for array in h5file.listNodes(group, classname = 'Array'):
                
components_list.append({'name':array.hdf5name,'type':"Array"})
            for Table in h5file.walkNodes(group, "Table"):
                
components_list.append({'name':Table.hdf5name,'type':"Table"})

        for i in range(len(components_list)):
            iter = self.__model.insert_before(None, None) 
            self.__model.set_value(iter, 0, components_list[i]['name'])
            self.__model.set_value(iter, 1, components_list[i]['type'])

I browsed the pytables several times.
I didn't find a better way to do it. The problems are:
     - i use lots of for loop -that's a memory waste-
    -  i use three differents functions to get the name of node 
(group._v_name, array.hdf5name, Table.hdf5name). Is it appropriate.
    - When i see the result, a table which is in a group is displayed 
also in the root node '/'. I don't know why.

Is there a better way to do this?

Perhaps it could be using only one loop with the node element?
for node in h5file:
    ...
In this case i don't know how to get the type of the node 
(i.e:Table,Array or Group).

Thans a lot for your answers,
Regards,
Philippe Collet

Re: [Pytables-users] pytables matlab compatibility

[Francesc A. <fa...@ca...>]

Hi Joost,

I'm glad that you like PyTables.

Regarding matlab importing problems I realize the best thing should be
contact them and report the problem. You can also try to export a
sample HDF5 file in matlab and send it to me so that I can look at its
internal structure and recommend you the best way to create the file
in PyTables in order to be compatible with the matlab format (if this
is possible at all!).

Cheers,

A Dijous 26 Maig 2005 19:31, joost va escriure:
> Dear pytables people,
>
> thank you for creating pytables, I like it very much. It seems to me an
> ideal way of organizing my numarray files. And while organizing it makes
> my data easy accessable to other programs like matlab. But matlab
> complains that the hdf5 file I generated using pytables is not valid.
>
> I use pytables 1.0 and hdf5 1.6.4.
> The array i am storing is an Int32 array and the hdf5 file is readable
> by NCSA's hdfview application. It doesn't use any compression filters.
>
> Does anyone know why matlabs hdfinfo complains to me.
>
> Joost

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] pytables matlab compatibility

[joost <j.g...@ew...>]

Dear pytables people,

thank you for creating pytables, I like it very much. It seems to me an
ideal way of organizing my numarray files. And while organizing it makes
my data easy accessable to other programs like matlab. But matlab
complains that the hdf5 file I generated using pytables is not valid. 

I use pytables 1.0 and hdf5 1.6.4.
The array i am storing is an Int32 array and the hdf5 file is readable
by NCSA's hdfview application. It doesn't use any compression filters.

Does anyone know why matlabs hdfinfo complains to me.

Joost

-- 
joost <j.g...@ew...>

[Pytables-users] ANNOUNCE: CSTables 1.0b

[Andreu A. A. <aa...@ar...>]

Announcing CSTables 1.0b
-------------------------
 
This is a beta release.
 
What it is
----------
 
CSTables is a Python client-server database layer library that provides
concurrency, client metadata cache, lock system, authentication and other
features.
 
CSTables is built on top of the HDF5 and PyTables libraries (that provide 
the data API and data disk management). The union of the forementioned 
libraries plus the Twisted Framework (in charge of the communication stuff) 
and NumArray (as a provider of containers for presenting and transporting 
data) outcomes in a complete client-server database with very interesting 
features.
 
Features
--------
 
Client-Server
CSTables allows remote HDF5 files to be accessed from clients using the 
PyTables API. Some classes and new methods are added, for dealing with 
client-server execution and other features. 
The Twisted Framework has been used to implement the client-server 
communication.
 
Client metadata cache
CSTables uses internally a client cache for storing metadata of the remote 
file objects. 
What is metadata? 
*The attributes of the remote file objects (File, Group, Leaf, AttributeSet,
 etc).
*The remote file tree hierarchy. Relations of ascendats to descendants are 
cached when they are used for the first time, allowing a smooth navigation 
in subsequent accesses.
 
Client metadata cache refresh
Attributes that are cached on the clients will be refreshed when they are 
changed in the server. A timestamp mechanism is provided to discard out of 
phase modifications, and guarantee that older modifications that arrive to 
the client after a newest modification will be discarded. The cache refresh 
is very useful for inter-client communication, because clients no need to 
poll the database for messages of other clients. 
 
Concurrency
Multiple clients run against the same CSTables server. No transactional 
support is added. However the Lock System can be used to implement 
applications with transactional features. All remote operations are 
guaranteed to be atomic, including those, such as iterrows(), that can 
require several communications with the server. 
 
Authentication
CSTables allows client authentication. The server can be configured to
accept 
anonymous logins, or to allow only connections that are authenticated
against 
its user database. Authentication is made through a challenge/authentication

protocol with double hashed MD5 passwords.
 
Remote file system management
Methods to allow creating, opening, removing and listing files in the server

files are supported. 
 
Lock System
In multi-client applications with clients accessing to the same database, a 
mechanism should be provided to allow to execute portions of code preventing

other clients from making changes to the desired objects while the 
modifications are not over. CSTables provides a lock mechanism, allowing 
applications to specifically put a lock in a node or in a full subtree. 
When a lock is set in a node and other client attempts to modify the node 
in an incompatible way, that client depending on a switch variable that can 
be set programatically either becomes blocked and remains blocked until 
the client that has put the lock releases it or raises a BlockedException 
exception. 
 
Full compatibility with PyTables API
There are several ways to program with CSTables. One of these allows to 
wrap PyTables monolitic scripts without modifying a single line and execute 
them against a remote server where reside the files the script will expect 
to find.
 

What is HDF5?
-------------
 
For those people who know nothing about HDF5, it is a general purpose
library and file format for storing scientific data made at NCSA. HDF5
can store two primary objects: datasets and groups. A dataset is
essentially a multidimensional array of data elements, and a group is a
structure for organizing objects in an HDF5 file. Using these two basic
constructs, one can create and store almost any kind of scientific data
structure, such as images, arrays of vectors, and structured and
unstructured grids.
 
What is PyTables?
-----------------
 
PyTables is a hierarchical database package designed to efficiently manage
very large amounts of data. 
 
PyTables is built on top of the HDF5 library and the numarray package. 
It features an object-oriented interface that, combined with C extensions 
for the peformance-critical parts of the code (generated using Pyrex), makes

it a fast, yet extremely easy to use tool for interactively save and
retrieve 
very large amounts of data. One important feature of PyTables is that it 
optimizes memory and disk resources so that data take much less space 
(between a factor 3 to 5, and more if the data is compressible) than other 
solutions, like for example, relational or object oriented databases. 
 

Platforms
---------
 
Currently tested and cross-tested in Windows XP and Red Hat Linux 9.0.
 
Web site
--------
 
Go to the CSTables page for more details:
 
http://www.cstables.org <http://www.cstables.org/> 
 
-- Andreu Alted

Re: R: R: [Pytables-users] Re: [ pytables-Patches-1192700 ] Multi-dimensional HDF5 attributes

[Francesc A. <fa...@ca...>]

A Dilluns 16 Maig 2005 16:41, Antonio Valentino va escriure:
> > I have run the the test in heavy mode today on the PC at work (not my
> > laptop) and got 3 errors for the snapshot-20060516
>
> [...]
>
> ops. This also happens in *non* heavy mode.

Well, I've commited a patch to solve this. I'm attaching it in case
you are interested in looking at it.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: R: R: [Pytables-users] Re: [ pytables-Patches-1192700 ] Multi-dimensional HDF5 attributes

[Antonio V. <val...@co...>]

Alle 16:28, luned=EC 16 maggio 2005, Antonio Valentino ha scritto:
> Alle 13:33, domenica 15 maggio 2005, Francesc Altet ha scritto:
> > Hola Antonio,
> >

[...]

> > That's good. You had a small glitch converting NROWS to Python ints
> > instead of longs (remember that PyTables supports objects with more
>
> I'm sorry.
>
> > than 2**31 rows). This has been corrected. If you would have run the
> > tests in heavy mode you should have been warned about that.
>
> I have run the the test in heavy mode today on the PC at work (not my
> laptop) and got 3 errors for the snapshot-20060516

[...]

ops. This also happens in *non* heavy mode.

[...]
Reading an old version file node. ...=20
/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py:248:=20
UserWarning: problems loading leaf ``/test``: unsubscriptable object; it =
will=20
become an ``UnImplemented`` node
  warnings.warn("""\
ERROR
Writing an old version file node. ... ERROR
Accessing attributes in an old version file node. ... ERROR

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
ERROR: Reading an old version file node.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/valentino/pyvm/src/pytables-20050516/test/test_filenode.py"=
,=20
line 909, in setUp
    oldh5f =3D tables.openFile(self.oldh5fname)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
229, in openFile
    return File(path, mode, title, new, trMap, rootUEP, isPTFile, filters=
)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
409, in __init__
    self.root =3D self.__getRootGroup(rootUEP)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
460, in __getRootGroup
    rootGroup =3D RootGroup(self, '/', rootUEP, new=3Dself._v_new)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 831, in __init__
    self._g_openFile()
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 253, in _g_openFile
    objleaf._g_putUnder(objgroup, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Leaf.py",=
 line=20
222, in _g_putUnder
    super(Leaf, self)._g_putUnder(parent, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Node.py",=
 line=20
693, in _g_putUnder
    parent._g_refNode(self, ptname, validate)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 323, in _g_refNode
    raise NodeError(
NodeError: group ``/`` already has a child node named ``test``

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
ERROR: Writing an old version file node.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/valentino/pyvm/src/pytables-20050516/test/test_filenode.py"=
,=20
line 909, in setUp
    oldh5f =3D tables.openFile(self.oldh5fname)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
229, in openFile
    return File(path, mode, title, new, trMap, rootUEP, isPTFile, filters=
)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
409, in __init__
    self.root =3D self.__getRootGroup(rootUEP)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
460, in __getRootGroup
    rootGroup =3D RootGroup(self, '/', rootUEP, new=3Dself._v_new)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 831, in __init__
    self._g_openFile()
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 253, in _g_openFile
    objleaf._g_putUnder(objgroup, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Leaf.py",=
 line=20
222, in _g_putUnder
    super(Leaf, self)._g_putUnder(parent, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Node.py",=
 line=20
693, in _g_putUnder
    parent._g_refNode(self, ptname, validate)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 323, in _g_refNode
    raise NodeError(
NodeError: group ``/`` already has a child node named ``test``

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
ERROR: Accessing attributes in an old version file node.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/valentino/pyvm/src/pytables-20050516/test/test_filenode.py"=
,=20
line 909, in setUp
    oldh5f =3D tables.openFile(self.oldh5fname)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
229, in openFile
    return File(path, mode, title, new, trMap, rootUEP, isPTFile, filters=
)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
409, in __init__
    self.root =3D self.__getRootGroup(rootUEP)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
460, in __getRootGroup
    rootGroup =3D RootGroup(self, '/', rootUEP, new=3Dself._v_new)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 831, in __init__
    self._g_openFile()
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 253, in _g_openFile
    objleaf._g_putUnder(objgroup, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Leaf.py",=
 line=20
222, in _g_putUnder
    super(Leaf, self)._g_putUnder(parent, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Node.py",=
 line=20
693, in _g_putUnder
    parent._g_refNode(self, ptname, validate)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 323, in _g_refNode
    raise NodeError(
NodeError: group ``/`` already has a child node named ``test``

----------------------------------------------------------------------
Ran 1798 tests in 206.001s

FAILED (errors=3D3)
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
-=3D
PyTables version:  1.1-alpha
Extension version: $Id: hdf5Extension.pyx 852 2005-05-15 11:31:50Z faltet=
 $
HDF5 version:      1.6.4
numarray version:  1.3.1
Zlib version:      1.1.4
BZIP2 version:     1.0.2 (30-Dec-2001)
Python version:    2.4.1 (#1, May  4 2005, 11:47:31)
[GCC 3.2.2]
Platform:          linux2-i686
Byte-ordering:     little
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
-=3D
Performing only a light (yet comprehensive) subset of the test
suite.  If you have a big system and lots of CPU to waste and want to
do a more complete test, try passing the --heavy flag to this script.
The whole suite will take more than 10 minutes to complete on a
relatively modern CPU and around 100 MB of main memory.
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
-=3D
Skipping Numeric test suite
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
-=3D


I will try on my laptop as soon as possible.

[...]

 ciao

--=20
Antonio Valentino
INNOVA - Consorzio per l'Informatica e la Telematica
via della Scienza - Zona Paip I
75100 Matera (MT) Italy
Tel.:      +39 0835 309180
Fax:       +39 0835 264705
Home Page: www.consorzio-innova.it
Email:     val...@co...

Re: R: R: [Pytables-users] Re: [ pytables-Patches-1192700 ] Multi-dimensional HDF5 attributes

[Antonio V. <val...@co...>]

Alle 13:33, domenica 15 maggio 2005, Francesc Altet ha scritto:
> Hola Antonio,
>
> On Sunday 15 May 2005 11:37, you wrote:
> > Hi Francesc,
> > I have uploaded the patch that implements what we discussed in the
> > previous posts on the patch section of the pytables project on
> > SourceForge (pytables-20050514-mdattr.patch - file format_version >=3D
> > 1.4).
> >
> > Some notes:
> >
> > - now all python objects are saved as cpickle strings and all numarra=
y
> >   objects are saved as HDF5 numeric attributes. The only exceptions
> >   are the numeric sys attributes (NROWS, EXTDIM, AUTOMATIC_INDEX,
> >   REINDEX, DIRTY, NODE_TYPE_VERSION) that are still stored as scalar
> >   HDF5 attributes and retrieved as python int.
> >   This because I found that in some cases they are set from the C
> >   extension code:
> >     H5ARRAY.c       H5ARRAYmake            EXTDIM
> >     H5TB.c          H5TBdelete_record      NROWS
> >     H5TB.c          H5TBmake_table         NROWS
> >     H5TB-opt.c      H5TBOclose_append      NROWS
> >   In my opinion this is not so bad in that they are treated as specia=
l
> >   attributes in any case.
>
> That's good. You had a small glitch converting NROWS to Python ints
> instead of longs (remember that PyTables supports objects with more

I'm sorry.

> than 2**31 rows). This has been corrected. If you would have run the
> tests in heavy mode you should have been warned about that.

I have run the the test in heavy mode today on the PC at work (not my lap=
top)
and got 3 errors for the snapshot-20060516

[...]

Reading an old version file node. ...=20
/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py:248:=20
UserWarning: problems loading leaf ``/test``: unsubscriptable object; it =
will=20
become an ``UnImplemented`` node
  warnings.warn("""\
ERROR
Writing an old version file node. ... ERROR
Accessing attributes in an old version file node. ... ERROR

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
ERROR: Reading an old version file node.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/valentino/pyvm/src/pytables-20050516/test/test_filenode.py"=
,=20
line 909, in setUp
    oldh5f =3D tables.openFile(self.oldh5fname)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
229, in openFile
    return File(path, mode, title, new, trMap, rootUEP, isPTFile, filters=
)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
409, in __init__
    self.root =3D self.__getRootGroup(rootUEP)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
460, in __getRootGroup
    rootGroup =3D RootGroup(self, '/', rootUEP, new=3Dself._v_new)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 831, in __init__
    self._g_openFile()
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 253, in _g_openFile
    objleaf._g_putUnder(objgroup, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Leaf.py",=
 line=20
222, in _g_putUnder
    super(Leaf, self)._g_putUnder(parent, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Node.py",=
 line=20
693, in _g_putUnder
    parent._g_refNode(self, ptname, validate)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 323, in _g_refNode
    raise NodeError(
NodeError: group ``/`` already has a child node named ``test``

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
ERROR: Writing an old version file node.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/valentino/pyvm/src/pytables-20050516/test/test_filenode.py"=
,=20
line 909, in setUp
    oldh5f =3D tables.openFile(self.oldh5fname)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
229, in openFile
    return File(path, mode, title, new, trMap, rootUEP, isPTFile, filters=
)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
409, in __init__
    self.root =3D self.__getRootGroup(rootUEP)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
460, in __getRootGroup
    rootGroup =3D RootGroup(self, '/', rootUEP, new=3Dself._v_new)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 831, in __init__
    self._g_openFile()
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 253, in _g_openFile
    objleaf._g_putUnder(objgroup, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Leaf.py",=
 line=20
222, in _g_putUnder
    super(Leaf, self)._g_putUnder(parent, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Node.py",=
 line=20
693, in _g_putUnder
    parent._g_refNode(self, ptname, validate)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 323, in _g_refNode
    raise NodeError(
NodeError: group ``/`` already has a child node named ``test``

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
ERROR: Accessing attributes in an old version file node.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/valentino/pyvm/src/pytables-20050516/test/test_filenode.py"=
,=20
line 909, in setUp
    oldh5f =3D tables.openFile(self.oldh5fname)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
229, in openFile
    return File(path, mode, title, new, trMap, rootUEP, isPTFile, filters=
)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
409, in __init__
    self.root =3D self.__getRootGroup(rootUEP)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/File.py",=
 line=20
460, in __getRootGroup
    rootGroup =3D RootGroup(self, '/', rootUEP, new=3Dself._v_new)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 831, in __init__
    self._g_openFile()
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 253, in _g_openFile
    objleaf._g_putUnder(objgroup, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Leaf.py",=
 line=20
222, in _g_putUnder
    super(Leaf, self)._g_putUnder(parent, name)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Node.py",=
 line=20
693, in _g_putUnder
    parent._g_refNode(self, ptname, validate)
  File "/home/valentino/pyvm/lib/python2.4/site-packages/tables/Group.py"=
,=20
line 323, in _g_refNode
    raise NodeError(
NodeError: group ``/`` already has a child node named ``test``

----------------------------------------------------------------------
Ran 2456 tests in 781.418s

FAILED (errors=3D3)
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
-=3D
PyTables version:  1.1-alpha
Extension version: $Id: hdf5Extension.pyx 852 2005-05-15 11:31:50Z faltet=
 $
HDF5 version:      1.6.4
numarray version:  1.3.1
Zlib version:      1.1.4
BZIP2 version:     1.0.2 (30-Dec-2001)
Python version:    2.4.1 (#1, May  4 2005, 11:47:31)
[GCC 3.2.2]
Platform:          linux2-i686
Byte-ordering:     little
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
-=3D
Performing the complete test suite!
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
-=3D
Skipping Numeric test suite
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
-=3D



> > - I run the complete test suite an all tests are passed (see the
> >   attached testlog.txt). Unfortunately now come out some warning
> >   that I am unable to fix, I'm sorry :((
> >     Warning: Encountered invalid numeric result(s)  in greater_equal
> >     Warning: Encountered invalid numeric result(s)  in less_equal
> >     Warning: Encountered invalid numeric result(s)  in greater_equal
> >     Warning: Encountered invalid numeric result(s)  in less
> >     Warning: Encountered invalid numeric result(s)  in greater
> >     Warning: Encountered invalid numeric result(s)  in less_equal
> >     Warning: Encountered invalid numeric result(s)  in greater
> >     Warning: Encountered invalid numeric result(s)  in less
> >   I only know that they come from the test_Numeric.py test suite.
>
> I'm not so sure that these warnings come from test_Numeric.py. I know
> that these warnings appears from time to time when running the tests,
> even before applying your patches. They seems unproblematic, though.

OK

[...]

> > Comments and suggestions are welcome.
>
> Very good work indeed. Many thanks in name of the PyTables community!

Thank you for pytables

> The changes has been uploaded into the SVN repository.
>
> Ciao,
>
> Francesc

ciao

--=20
Antonio Valentino
INNOVA - Consorzio per l'Informatica e la Telematica
via della Scienza - Zona Paip I
75100 Matera (MT) Italy
Tel.:      +39 0835 309180
Fax:       +39 0835 264705
Home Page: www.consorzio-innova.it
Email:     val...@co...

Re: [Pytables-users] fat32 and pytables on Linux

[Ivan V. i B. <iv...@ca...>]

Francesc Altet (el 2005-05-14 a les 09:24:03 +0200) va dir::

> This seems an incompatibility between HDF5 library and the driver for
> the fat32 in Linux. I don't know who might be the ultimate responsible
> of the issue. You may want to issue a bug to the HDF5 (see
> http://hdf.ncsa.uiuc.edu/HDF5-FAQ.html#21) and see what they say
> about that.
>=20
> My opinion is, though, that using HDF5 on top of Linux and VFAT may
> kill the performance. It would be much better to create the files on a
> native Linux partition and then move the resulting files to the VFAT
> one.

Using an Ext2/Ext3 driver for Windows could also do the trick.  Ext2fsd_
seems to be quite complete, though bugs still exist and most probably
performance will be sub-standard.

=2E. _Ext2fsd: http://ext2fsd.sourceforge.net/

Good luck,

::

	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
	       C=E1rabos Coop. V.  V  V   Enjoy Data
	                          ""

Re: R: R: [Pytables-users] Re: [ pytables-Patches-1192700 ] Multi-dimensional HDF5 attributes

[Francesc A. <fa...@ca...>]

Hola Antonio,

On Sunday 15 May 2005 11:37, you wrote:
> Hi Francesc,
> I have uploaded the patch that implements what we discussed in the
> previous posts on the patch section of the pytables project on
> SourceForge (pytables-20050514-mdattr.patch - file format_version >=
> 1.4).
>
> Some notes:
>
> - now all python objects are saved as cpickle strings and all numarray
>   objects are saved as HDF5 numeric attributes. The only exceptions
>   are the numeric sys attributes (NROWS, EXTDIM, AUTOMATIC_INDEX,
>   REINDEX, DIRTY, NODE_TYPE_VERSION) that are still stored as scalar
>   HDF5 attributes and retrieved as python int.
>   This because I found that in some cases they are set from the C
>   extension code:
>     H5ARRAY.c       H5ARRAYmake            EXTDIM
>     H5TB.c          H5TBdelete_record      NROWS
>     H5TB.c          H5TBmake_table         NROWS
>     H5TB-opt.c      H5TBOclose_append      NROWS
>   In my opinion this is not so bad in that they are treated as special
>   attributes in any case.


That's good. You had a small glitch converting NROWS to Python ints
instead of longs (remember that PyTables supports objects with more
than 2**31 rows). This has been corrected. If you would have run the
tests in heavy mode you should have been warned about that.

>
> - I run the complete test suite an all tests are passed (see the
>   attached testlog.txt). Unfortunately now come out some warning
>   that I am unable to fix, I'm sorry :((
>     Warning: Encountered invalid numeric result(s)  in greater_equal
>     Warning: Encountered invalid numeric result(s)  in less_equal
>     Warning: Encountered invalid numeric result(s)  in greater_equal
>     Warning: Encountered invalid numeric result(s)  in less
>     Warning: Encountered invalid numeric result(s)  in greater
>     Warning: Encountered invalid numeric result(s)  in less_equal
>     Warning: Encountered invalid numeric result(s)  in greater
>     Warning: Encountered invalid numeric result(s)  in less
>   I only know that they come from the test_Numeric.py test suite.
>

I'm not so sure that these warnings come from test_Numeric.py. I know
that these warnings appears from time to time when running the tests,
even before applying your patches. They seems unproblematic, though.

> - I noticed that, due to the rework of the attributes related code,
>   now there are many functions that are no more used:
>     # hdfExtension.pyx
>     AttributeSet._g_setAttrChar
>     AttributeSet._g_setAttrShort
>     AttributeSet._g_setAttrInt
>     AttributeSet._g_setAttrFloat
>     AttributeSet._g_setAttrDouble
>
>     #H5LT.h/H5LT.c
>     H5LTset_attribute_char
>     H5LTset_attribute_short
>     H5LTset_attribute_int
>     H5LTset_attribute_long
>
>     H5LTset_attribute_uchar
>     H5LTset_attribute_ushort
>     H5LTset_attribute_uint
>     H5LTset_attribute_ulong
>
>     H5LTset_attribute_float
>     H5LTset_attribute_double
>
>     H5LTget_attribute_char
>     H5LTget_attribute_short
>     H5LTget_attribute_long
>
>     H5LTget_attribute_uchar
>     H5LTget_attribute_ushort
>     H5LTget_attribute_uint
>     H5LTget_attribute_ulong
>
>     H5LTget_attribute_float
>     H5LTget_attribute_double
>
>   The H5LTget_attribute_int is still used in hdf5Extension.pyx from
>   the Array._openArray method.

Good. I should do some timmings in order to see how the changes affect
to opening times of complex PyTables files.

> Comments and suggestions are welcome.

Very good work indeed. Many thanks in name of the PyTables community!

The changes has been uploaded into the SVN repository.

Ciao,

Francesc

Re: [Pytables-users] fat32 and pytables on Linux

[Francesc A. <fa...@ca...>]

Hi Ryan,

This seems an incompatibility between HDF5 library and the driver for
the fat32 in Linux. I don't know who might be the ultimate responsible
of the issue. You may want to issue a bug to the HDF5 (see
http://hdf.ncsa.uiuc.edu/HDF5-FAQ.html#21) and see what they say
about that.

My opinion is, though, that using HDF5 on top of Linux and VFAT may
kill the performance. It would be much better to create the files on a
native Linux partition and then move the resulting files to the VFAT
one.

Cheers,

Francesc

On Friday 13 May 2005 22:19, Ryan Krauss wrote:
> I have a problem running pytables on a Fat32 partition on Linux.  If I
> run the table2.py example from anywhere on my Linux partition, it works
> fine.  But if I copy it to somewhere on my fat32 partion, file.close()
> generates the following error:
>
> HDF5-DIAG: Error detected in HDF5 library version: 1.6.4 thread 0.  Back
> trace follows.
>   #000: H5F.c line 2175 in H5Fflush(): flush failed
>     major(04): File interface
>     minor(29): Unable to initialize object
>   #001: H5F.c line 2849 in H5F_flush(): low level flush failed
>     major(05): Low-level I/O layer
>     minor(25): Write failed
>   #002: H5FD.c line 3578 in H5FD_flush(): driver flush request failed
>     major(22): Virtual File Layer
>     minor(29): Unable to initialize object
>   #003: H5FDsec2.c line 876 in H5FD_sec2_flush(): unable to extend file
> properly
>     major(05): Low-level I/O layer
>     minor(23): Seek failed
>
> The same error occurs for any flush() or close() comand if the *.h5 file
> is on the fat partition.  My system is dual boot and all my research
> files need to be on the fat partition to be accessible to both windows
> and linux.
>
> Can someone help me with this?
>
> Thanks,
>
> Ryan
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by Oracle Space Sweepstakes
> Want to be the first software developer in space?
> Enter now for the Oracle Space Sweepstakes!
> http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

[Pytables-users] fat32 and pytables on Linux

[Ryan K. <rya...@co...>]

I have a problem running pytables on a Fat32 partition on Linux.  If I 
run the table2.py example from anywhere on my Linux partition, it works 
fine.  But if I copy it to somewhere on my fat32 partion, file.close() 
generates the following error:

HDF5-DIAG: Error detected in HDF5 library version: 1.6.4 thread 0.  Back 
trace follows.
  #000: H5F.c line 2175 in H5Fflush(): flush failed
    major(04): File interface
    minor(29): Unable to initialize object
  #001: H5F.c line 2849 in H5F_flush(): low level flush failed
    major(05): Low-level I/O layer
    minor(25): Write failed
  #002: H5FD.c line 3578 in H5FD_flush(): driver flush request failed
    major(22): Virtual File Layer
    minor(29): Unable to initialize object
  #003: H5FDsec2.c line 876 in H5FD_sec2_flush(): unable to extend file 
properly
    major(05): Low-level I/O layer
    minor(23): Seek failed

The same error occurs for any flush() or close() comand if the *.h5 file 
is on the fat partition.  My system is dual boot and all my research 
files need to be on the fat partition to be accessible to both windows 
and linux.

Can someone help me with this?

Thanks,

Ryan