pytables-users

[Pytables-users] ValueError when a list of numarray.arrays used.

[Shin <sd...@em...>]

While filling up a table, I met a strange error message.
When the appended program is tried with version 0.8.1,
only the second trial to use a list of arrays raises an ValueError.
But, the manual says I can use a list of numarrays to append multi-rows.
What did I miss? Or, a bug? Thanks.

>>>>>>>>>>>>>>> test3.py <<<<<<<<<<<<<<<<<<

#!/usr/bin/env python

from numarray import *
from tables import *

# By using a list of lists.
class Cols(IsDescription):
    V1 = FloatCol()
    V2 = FloatCol()    

f = openFile("test.hd5", "w")
tab = f.createTable("/","Data",Cols,"Data")

d = []
for i in range(10):
    d = d + [[1.0,1.0]]

tab.append(d)
f.close()

# By using a list of arrays.

f = openFile("test.hd5", "w")
tab = f.createTable("/","Data",Cols,"Data")

d = []
for i in range(10):
    d = d + [array([1.0,1.0])]

tab.append(d)
f.close()

>>>>>>>>> Here is the error message caught in ipython <<<<<<

In [1]: execfile("test3.py")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call
last)

/home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/<console> 

/home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/test3.py 
     24 for i in range(10):
     25     d = d + [array([1.0,1.0])]
     26 
---> 27 tab.append(d)
        tab = /Data (Table(0,)) 'Data'
  description := {
    ...pe=1, dflt=0.0, pos=None) }
  byteorder := little, global append = undefined, d = [array([ 1., 
1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]),
array([ 1.,  1.]), array([ 1.,  1.])]
     28 f.close()

/usr/lib/python2.3/site-packages/tables/Table.py in append(self=/Data
(Table(0,)) 'Data'
  description := {
    ...pe=1, dflt=0.0, pos=None) }
  byteorder := little, rows=[array([ 1.,  1.]), array([ 1.,  1.]),
array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1., 
1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
1.,  1.])])
    588         except:
    589             (type, value, traceback) = sys.exc_info()
--> 590             raise ValueError, \
        global ValueError = undefined, global str = undefined, self =
/Data (Table(0,)) 'Data'
  description := {
    ...pe=1, dflt=0.0, pos=None) }
  byteorder := little, value = <exceptions.IndexError instance>
    591 "rows parameter cannot be converted into a recarray object
compliant with table '%s'. The error was: <%s>" % (str(self), value)
    592         lenrows = recarray.shape[0]

ValueError: rows parameter cannot be converted into a recarray object
compliant with table '/Data (Table(0,)) 'Data''. The error was: <list
index out of range>

-- 
Daehyok Shin (Peter)
Geography Department
Univ. of North Carolina-Chapel Hill

Re: [Pytables-users] ValueError when a list of numarray.arrays used.

[Francesc A. <fa...@py...>]

Hi Daehyok,

The ultimate reason for this is that numarray.records.array cannot convert a
list of arrays to a RecArray. See:

In [27]: records.array([[1.,2.]]*3, formats='f8,f8')
Out[27]:
array(
[(1.0, 2.0),
(1.0, 2.0),
(1.0, 2.0)],
formats=['1Float64', '1Float64'],
shape=3,
names=['c1', 'c2'])

That works well. But...

In [28]: records.array([array([1.,2.])]*3, formats='f8,f8')
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)

/home/falted/python.nobackup/IM-2004.01.05/<console>

/usr/local/lib/python2.3/site-packages/numarray/records.py in array(buffer, formats, shape, names, byteorder, aligned)
    382         if isinstance(buffer[0], num.NumArray) or isinstance(buffer[0], chararray.CharArray):
    383             return fromarrays(buffer, formats=formats, shape=shape, names=names,
--> 384                         byteorder=byteorder, aligned=aligned)
    385         else:
    386             return fromrecords(buffer, formats=formats, shape=shape, names=names,

/usr/local/lib/python2.3/site-packages/numarray/records.py in fromarrays(arrayList, formats, names, shape, byteorder, aligned)
    183     for i in range(len(arrayList)):
    184         try:
--> 185             _array.field(_array._names[i])[:] = arrayList[i]
    186         except TypeError: # trap only TypeError to get better messages
    187             raise TypeError, "Incorrect format %s, copy unsuccessful." % _array._formats[i]

IndexError: list index out of range

does not work.

Perhaps, this worked with previous versions of numarray, I can't remember.
Anyway, I'll remove the part of manual where I say that Table.append() do
accept lists of numarrays/Numeric/chararray.

Cheers,

Francesc

A Divendres 08 Octubre 2004 23:17, Shin va escriure:
> While filling up a table, I met a strange error message.
> When the appended program is tried with version 0.8.1,
> only the second trial to use a list of arrays raises an ValueError.
> But, the manual says I can use a list of numarrays to append multi-rows.
> What did I miss? Or, a bug? Thanks.
> 
> >>>>>>>>>>>>>>> test3.py <<<<<<<<<<<<<<<<<<
> 
> #!/usr/bin/env python
> 
> from numarray import *
> from tables import *
> 
> # By using a list of lists.
> class Cols(IsDescription):
>     V1 = FloatCol()
>     V2 = FloatCol()    
> 
> f = openFile("test.hd5", "w")
> tab = f.createTable("/","Data",Cols,"Data")
> 
> d = []
> for i in range(10):
>     d = d + [[1.0,1.0]]
> 
> tab.append(d)
> f.close()
> 
> # By using a list of arrays.
> 
> f = openFile("test.hd5", "w")
> tab = f.createTable("/","Data",Cols,"Data")
> 
> d = []
> for i in range(10):
>     d = d + [array([1.0,1.0])]
> 
> tab.append(d)
> f.close()
> 
> >>>>>>>>> Here is the error message caught in ipython <<<<<<
> 
> In [1]: execfile("test3.py")
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call
> last)
> 
> /home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/<console> 
> 
> /home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/test3.py 
>      24 for i in range(10):
>      25     d = d + [array([1.0,1.0])]
>      26 
> ---> 27 tab.append(d)
>         tab = /Data (Table(0,)) 'Data'
>   description := {
>     ...pe=1, dflt=0.0, pos=None) }
>   byteorder := little, global append = undefined, d = [array([ 1., 
> 1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
> 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]),
> array([ 1.,  1.]), array([ 1.,  1.])]
>      28 f.close()
> 
> /usr/lib/python2.3/site-packages/tables/Table.py in append(self=/Data
> (Table(0,)) 'Data'
>   description := {
>     ...pe=1, dflt=0.0, pos=None) }
>   byteorder := little, rows=[array([ 1.,  1.]), array([ 1.,  1.]),
> array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1., 
> 1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
> 1.,  1.])])
>     588         except:
>     589             (type, value, traceback) = sys.exc_info()
> --> 590             raise ValueError, \
>         global ValueError = undefined, global str = undefined, self =
> /Data (Table(0,)) 'Data'
>   description := {
>     ...pe=1, dflt=0.0, pos=None) }
>   byteorder := little, value = <exceptions.IndexError instance>
>     591 "rows parameter cannot be converted into a recarray object
> compliant with table '%s'. The error was: <%s>" % (str(self), value)
>     592         lenrows = recarray.shape[0]
> 
> ValueError: rows parameter cannot be converted into a recarray object
> compliant with table '/Data (Table(0,)) 'Data''. The error was: <list
> index out of range>
> 

-- 
Francesc Alted

RE: [Pytables-users] ValueError when a list of numarray.arrays used.

[Shin, D. <sd...@ca...>]

I got it.
Then, let me ask one question about it.
Assuming at every simulation step I got a numeric array of 100 rows and 10
cols,
what is the most efficient way to append it to Table?
I want  to use Table rather than EArray because of explicit column names.
Thank for your kind reply.

Daehyok Shin (Peter)

> -----Original Message-----
> From: pyt...@li...
> [mailto:pyt...@li...]On Behalf Of
> Francesc Alted
> Sent: Saturday, October 09, 2004 AM 5:55
> To: pyt...@li...
> Subject: Re: [Pytables-users] ValueError when a list of
> numarray.arrays used.
>
>
> Hi Daehyok,
>
> The ultimate reason for this is that numarray.records.array
> cannot convert a
> list of arrays to a RecArray. See:
>
> In [27]: records.array([[1.,2.]]*3, formats='f8,f8')
> Out[27]:
> array(
> [(1.0, 2.0),
> (1.0, 2.0),
> (1.0, 2.0)],
> formats=['1Float64', '1Float64'],
> shape=3,
> names=['c1', 'c2'])
>
> That works well. But...
>
> In [28]: records.array([array([1.,2.])]*3, formats='f8,f8')
> ------------------------------------------------------------------
> ---------
> IndexError                                Traceback (most recent
> call last)
>
> /home/falted/python.nobackup/IM-2004.01.05/<console>
>
> /usr/local/lib/python2.3/site-packages/numarray/records.py in
> array(buffer, formats, shape, names, byteorder, aligned)
>     382         if isinstance(buffer[0], num.NumArray) or
> isinstance(buffer[0], chararray.CharArray):
>     383             return fromarrays(buffer, formats=formats,
> shape=shape, names=names,
> --> 384                         byteorder=byteorder, aligned=aligned)
>     385         else:
>     386             return fromrecords(buffer, formats=formats,
> shape=shape, names=names,
>
> /usr/local/lib/python2.3/site-packages/numarray/records.py in
> fromarrays(arrayList, formats, names, shape, byteorder, aligned)
>     183     for i in range(len(arrayList)):
>     184         try:
> --> 185             _array.field(_array._names[i])[:] = arrayList[i]
>     186         except TypeError: # trap only TypeError to get
> better messages
>     187             raise TypeError, "Incorrect format %s, copy
> unsuccessful." % _array._formats[i]
>
> IndexError: list index out of range
>
> does not work.
>
> Perhaps, this worked with previous versions of numarray, I can't remember.
> Anyway, I'll remove the part of manual where I say that Table.append() do
> accept lists of numarrays/Numeric/chararray.
>
> Cheers,
>
> Francesc
>
> A Divendres 08 Octubre 2004 23:17, Shin va escriure:
> > While filling up a table, I met a strange error message.
> > When the appended program is tried with version 0.8.1,
> > only the second trial to use a list of arrays raises an ValueError.
> > But, the manual says I can use a list of numarrays to append multi-rows.
> > What did I miss? Or, a bug? Thanks.
> >
> > >>>>>>>>>>>>>>> test3.py <<<<<<<<<<<<<<<<<<
> >
> > #!/usr/bin/env python
> >
> > from numarray import *
> > from tables import *
> >
> > # By using a list of lists.
> > class Cols(IsDescription):
> >     V1 = FloatCol()
> >     V2 = FloatCol()
> >
> > f = openFile("test.hd5", "w")
> > tab = f.createTable("/","Data",Cols,"Data")
> >
> > d = []
> > for i in range(10):
> >     d = d + [[1.0,1.0]]
> >
> > tab.append(d)
> > f.close()
> >
> > # By using a list of arrays.
> >
> > f = openFile("test.hd5", "w")
> > tab = f.createTable("/","Data",Cols,"Data")
> >
> > d = []
> > for i in range(10):
> >     d = d + [array([1.0,1.0])]
> >
> > tab.append(d)
> > f.close()
> >
> > >>>>>>>>> Here is the error message caught in ipython <<<<<<
> >
> > In [1]: execfile("test3.py")
> >
> ------------------------------------------------------------------
> ---------
> > ValueError                                Traceback (most recent call
> > last)
> >
> > /home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/<console>
> >
> > /home/sdhyok/catchlab/catchlab/catchlab/pkg/rhessys/test/test3.py
> >      24 for i in range(10):
> >      25     d = d + [array([1.0,1.0])]
> >      26
> > ---> 27 tab.append(d)
> >         tab = /Data (Table(0,)) 'Data'
> >   description := {
> >     ...pe=1, dflt=0.0, pos=None) }
> >   byteorder := little, global append = undefined, d = [array([ 1.,
> > 1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
> > 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]),
> > array([ 1.,  1.]), array([ 1.,  1.])]
> >      28 f.close()
> >
> > /usr/lib/python2.3/site-packages/tables/Table.py in append(self=/Data
> > (Table(0,)) 'Data'
> >   description := {
> >     ...pe=1, dflt=0.0, pos=None) }
> >   byteorder := little, rows=[array([ 1.,  1.]), array([ 1.,  1.]),
> > array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,
> > 1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([ 1.,  1.]), array([
> > 1.,  1.])])
> >     588         except:
> >     589             (type, value, traceback) = sys.exc_info()
> > --> 590             raise ValueError, \
> >         global ValueError = undefined, global str = undefined, self =
> > /Data (Table(0,)) 'Data'
> >   description := {
> >     ...pe=1, dflt=0.0, pos=None) }
> >   byteorder := little, value = <exceptions.IndexError instance>
> >     591 "rows parameter cannot be converted into a recarray object
> > compliant with table '%s'. The error was: <%s>" % (str(self), value)
> >     592         lenrows = recarray.shape[0]
> >
> > ValueError: rows parameter cannot be converted into a recarray object
> > compliant with table '/Data (Table(0,)) 'Data''. The error was: <list
> > index out of range>
> >
>
> --
> Francesc Alted
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to
> find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] ValueError when a list of numarray.arrays used.

[Francesc A. <fa...@py...>]

A Diumenge 10 Octubre 2004 05:59, Shin, Daehyok va escriure:
> Assuming at every simulation step I got a numeric array of 100 rows and 10
> cols,
> what is the most efficient way to append it to Table?

You can do something like:

from numarray import *
from tables import *

class Cols(IsDescription):
    V1 = FloatCol(shape=10)

f = openFile("test.hd5", "w")
tab = f.createTable("/","Data",Cols,"Data")

for i in range(10):
    # d simulates the output of your simulation step
    d = arange(i, 1000+i, shape=(100,10), type=Float64)
    tab.append([d])
    
f.close()

i.e. declaring a column with dimension 10.

> I want  to use Table rather than EArray because of explicit column names.

In that case, you can't use the recipe above. I would suggest you getting a
numarray.records.RecArray object from every step of your simulation code.
Then, you can feed the RecArray to Table.append() without any problem.

By the way, after thinking a bit about passing lists of numarray/Numeric to
the Table.append() method, I came to the conclusion that this is possible,
although this use is meant for multidimensional columns. For example:

from numarray import *
from tables import *

class Cols(IsDescription):
    V1 = FloatCol(shape=10)
    V2 = IntCol(shape=5)

f = openFile("test2.hd5", "w")
tab = f.createTable("/","Data",Cols,"Data")

for i in range(10):
    # d simulates the output of your simulation step
    d = arange(i, 1000+i, shape=(100,10), type=Float64)
    e = arange(i, 500+i, shape=(100,5), type=Int32)
    tab.append([d, e])
    
f.close()

works without problems. Maybe I should specify in the docs that you can use
lists of Numeric/numarray objects *only* in case of multimensional columns.

Cheers,

-- 
Francesc Alted