[Pytables-users] Re: PyTables help

[Francesc A. <fa...@op...>]

Hi Joseph,

A Dijous 04 Desembre 2003 06:35, vareu escriure:
> Francesc,
>
> I have a PyTables question I was hoping you could help me out with.
> I want to program a table of tables.
>
> Specifically, each row in the top level table will contain another table,
> of variable length.
>
> For example, one table row might contain [9, 2, 60, 30, 29, 54], and the
> second table row might contain [2, 6, 3], and the third row might contain
> [1].
>
> In other words, I want to have arrays (or tables) within my top-level
> tables.
>
> The documentation for pytables does not suggest how to do this.
> Nonetheless, is it possible for me to achieve this data structure using
> pytables?
>

Well, I've both bad and good news to you. Bad news is that pytables, up to
0.7.2 (and most probably 0.8 as well), only allows scalars or *fixed* length
numarray objects to be elements of columns, so you can't define a column to
be as *variable* length.

Good news is that VLArray (i.e. variable length arrays) objects will be
introduced in the forthcoming 0.8 release (among other niceties, like
enlargeable Array objects, support for boolean types and more). In fact, the
VLArray class is already in CVS, with proper unittests in place, and fairly
bug-free, as fas as I can tell. So, you can start using it today as I don't
think I'm going to change the API too much for the final version.

However, documentation has not been updated to include this new feature yet.
Meanwhile, you can look into the examples/vlarray1.py small script to get a
feeling of what you can do with this new creature. Just to wet your
appetite, some tiny example follows:

import tables
from numarray import *

# Create a VLArray:
fileh = tables.openFile("vlarray2.h5", mode = "w")
root = fileh.root
vlarray = fileh.createVLArray(root, 'vlarray1', tables.Int32Atom(),
                              "ragged array of ints")
vlarray.append(array([5, 6]))
vlarray.append(array([5, 6, 7]))
vlarray.append([5, 6, 9, 8])
vlarray.append(5, 6, 9, 10, 12)
print "Created VLArray:", repr(vlarray)
# Now, read it through the use of the iterator:
for x in vlarray:
    print vlarray.name, "[", vlarray.nrow, "]-->", x
# Close the file
fileh.close()

and the output is:

Created VLArray: /vlarray1 (VLArray(4,)) 'ragged array of ints'
  atom = Atom(type=Int32, shape=1, flavor='Numeric')
  nrows = 4
  flavor = 'Numeric'
  byteorder = 'little'
vlarray1 [ 0 ]--> [5 6]
vlarray1 [ 1 ]--> [5 6 7]
vlarray1 [ 2 ]--> [5 6 9 8]
vlarray1 [ 3 ]--> [ 5  6  9 10 12]

So, by joining VLArray and Table entities, you can simulate the
variable-length records you want to, just by defining a VLArray where you
will put your variable-length objects, and a new column in your table with
the row number of the VLArray where your desired data is. You can regard
this as a temporary workaround until I find time to implement fully
variable-length columns in Tables.

Cheers,

-- 
Francesc Alted