Re: [Pytables-users] Reading vlen data created by libhdf5 with pytables

[Cournapeau D. <cou...@at...>]

> 
> Strange. Anyway, I recommend you to stick with pytables 1.2b4, which
> is should be quite stable right now.

Ok, this is the version I am using right now, hence all problems
reported here are against this version.

> If you are going to modify the Pyrex sources, the best to rebuild the
> extensions is to use the distutils:
> 
[snip]

Thanks for the information. 

Here is what I get. First, when I try to access my vlen array 'cell'
through this small script (boilerplate code omitted):

    h5file  = openFile(filename, "r")
    root    = h5file.root
    print "=================== First call ===================="
    try:
        cell    = root.cell
    except:
        print 'exception during first call'

    print "=================== Second call ===================="
    cell    = root.cell
    for i in cell:
        print i

the h5f file 'filename' I tried to read as reported by h5dump (only the
interesting group is mentioned):

DATASET "cell" {
      DATATYPE  H5T_VLEN { H5T_IEEE_F64LE}
      DATASPACE  SIMPLE { ( 2 ) / ( H5S_UNLIMITED ) }
      DATA {
      (0): (-0.432565, -1.66558, 0.125332), (0.287676, -1.14647)
      }
   }

I then have the following error:

HDF5-DIAG: Error detected in HDF5 library version: 1.6.4 thread
3084815008.  Back trace follows.
  #000: ../../../src/H5A.c line 457 in H5Aopen_name(): attribute not
found
    major(18): Attribute layer
    minor(05): Bad value
  #001: ../../../src/H5A.c line 404 in H5A_get_index(): attribute not
found
    major(18): Attribute layer
    minor(48): Object not found
/usr/src/misc/pytables/pytables-1.2b4/tables/Group.py:397: UserWarning:
problems loading leaf ``/cell``::

  flavor of type 'Float64' must be one of the "NumArray", "Numeric",
"Tuple" or "List" values, and you tried to set it to "unknown".


The leaf will become an ``UnImplemented`` node.


The hf5 warning is coming from the call of H5LTget_attribute_string,
in the function _openArray in the pyrex file /src/hdf5Extension.pyx.
This of course happens because my file is not created by pytables, hence
does not have any FLAVOR attribute. Is it intentional to keep the Hf5
warning on? 

Anyway, as the Flavor attribute does not exist, it is set to Unknown,
and the  the Atom function call fails in VLArray.py:VLArray._g_open. 
(specifically the checkflavor call in the Atom.__init__ fails).

As I said in my previous email, the second attempt to read this vlen
array succeeds, because pytables has it in memory, hence the
File.py:File._getNode does not load it from the file, but from memory.

But this means the first call, even if unsuccessfull, manages to read
the vlen array in the file. (the data given back by the second call are
correct). So I tried the following hack (in the file VLArray.py,
function _g_open of the class VLArray):


	# First, check the special cases VLString and Object types
	if flavor == "VLString":
		self.atom = VLStringAtom()
	elif flavor == "Object":
        	self.atom = ObjectAtom()
	elif stype == 'CharType':
        	self.atom = StringAtom(self._atomicshape,
self._basesize,flavor)
	elif stype == 'Enum':
        	(enum, type_) = self._loadEnum()
        	self.atom = EnumAtom(enum, type_, self._atomicshape, flavor)
        	self._atomictype = type_
	else:
     	if flavor == 'unknown':
        	self.atom = Atom(stype, self._atomicshape)
     	else:
        	self.atom = Atom(stype, self._atomicshape, flavor)

return objectId

Then, it works in my case. As I don't know much yet about pytable,
I don't know what is the correct way achieve the same result. 

Concerning the case where the dimensions are fixed, I Unfortunately
don't have the time right now to investigate it.

David