From: Milad F. <ic...@gm...> - 2010-05-02 21:33:08
|
Hello: I have an HDF5 file that was generated outside of pytables (in a C++ program). It wrote a 3D array where the atom is: Float64Atom(shape=(3,), dflt=array([ 0., 0., 0.])) In other words, it is a 3D array of 1D arrays of doubles. When I open the file in python using pytables, it correctly identifies the array shape and atom type. However, if i copy the dataset using the _f_copy method to another location the atom type changes to: Float64Atom(shape=(), dflt=0.0) In this case the copy is a 4D array of doubles, instead of a 3D array whose elements are 1D arrays. If I copy arrays that were made by pytables, no problem exists - it only occurs when I make the array outside of pytables and copy it using pytables. What is interesting is that pytables does identiy the atom correctly but changes it when copying... I've managed to circumvent the problem temporarily by making a new chunked array in the file then manually copying the data. Is this a bug, or is there some limitation on copying arrays made outisde of pytables? Thanks for your help. I hope this was clear. Milad |
From: Francesc A. <fa...@py...> - 2010-05-03 06:41:36
|
Hello Milad, A Sunday 02 May 2010 23:32:41 Milad Fatenejad escrigué: > Hello: > > I have an HDF5 file that was generated outside of pytables (in a C++ > program). It wrote a 3D array where the atom is: > > Float64Atom(shape=(3,), dflt=array([ 0., 0., 0.])) > > In other words, it is a 3D array of 1D arrays of doubles. > > When I open the file in python using pytables, it correctly identifies the > array shape and atom type. However, if i copy the dataset using the _f_copy > method to another location the atom type changes to: > > Float64Atom(shape=(), dflt=0.0) > > In this case the copy is a 4D array of doubles, instead of a 3D array whose > elements are 1D arrays. If I copy arrays that were made by pytables, no > problem exists - it only occurs when I make the array outside of pytables > and copy it using pytables. What is interesting is that pytables does > identiy the atom correctly but changes it when copying... > > I've managed to circumvent the problem temporarily by making a new chunked > array in the file then manually copying the data. Is this a bug, or is > there some limitation on copying arrays made outisde of pytables? That's a bit weird, yes, and smells like a bug. Could you please provide a small datafile so that I can use it to reproduce the behaviour? I'd like to fix this as soon as possible. Cheers, -- Francesc Alted |
From: Francesc A. <fa...@py...> - 2010-05-03 16:49:35
|
A Monday 03 May 2010 16:00:00 escriguéreu: > Hi Francesc: > > I've attached a couple of files: > > 1) main.cpp.gz - the program I used to generate the hdf5 file. I should > mention that I am using HDF5 1.8.4 with enable-parallel > > 2) test.h5 - the hdf5 file that was generated. The file contains only 1 > data set at /arr that is 5x5x5 with length 3 array atoms. > > I opened python and entered: > > import tables > f = tables.openFile("test.h5", "a") > f.root.arr.copy(newparent=f.root, newname="arr2") > > I am using pytables 2.1.2 > > I hope this helps. Okay. This did not work because NumPy does not support having multidimensional types with homogeneous arrays. But as HDF5 does have support for this, I went ahead and managed to keep the original shape of the atom during the copy. More details in: http://www.pytables.org/trac/ticket/275 Thanks for reporting! -- Francesc Alted |