Thanks for all the tips on this. It is finally clear what is going on. I think they fact that you can specify a shape for table columns leads beginners down a the road I'm on. When I chose to use pytables after reading some of the documentation and playing with the code, it never occurred to me that I'd have this issue.

Turns out that I'm only storing arrays of shape (1,) in the columns, so it is very easy to pad and un-pad them and this doesn't really add much size to my database. I'll stick with that for now, but in version two of my software, I'll certainly look into the ability to add a VLArrayCol. It would be a nice feature.

Jason

On Wed, Jun 22, 2011 at 6:10 PM, Anthony Scopatz <scopatz@gmail.com> wrote:
Hi Jason, 

Upon further inspection, I think that this comes down to a spelling issue.  

Really, what you want is a VL structure inside of a table.  Tables are themselves
VL structures and have been special cased to nest themselves.  This could be 
changed to allow more general VL structures to be nested, but that would require
a good amount of effort (that I sadly don't have time for now).

But really the only thing that you are doing when you nest a table within a table
rather than nesting a vlarray in a table is that you are providing *extra* information.
This isn't really a bad thing, and gets to what I mean by spelling.

Nesting a table in a table you have,

desc = {'A': Float64Col, 'B': {'C': Int32Col}}

Nesting a vlarry inside of a column would presumably be

desc = {'A': Float64Col, 'B': VLArrayCol(atom=Int32Col)}

Unfortunately, the VLArrayCol object doesn't exist.  The reason it doesn't 
I suspect is because of how you would spell the path to the nested data.
In nested tables you use the column names:

t['B/C']

In nested vlarrays, you'd have to use the index, which is weird.

t['B/0'], t['B/1'], t['B/2'], ...

Nested tables are everything you want, plus a tiny bit more in terms of having to 
name the dimension you want to be variable.

It should be noted that you end up with the same weird spelling issues in numpy
structured arrays, which similarly disallow such funny business.

Of course, as Dav mentions, there are other options.  Store your VLArrays outside of
the table, then link to them in the table either through path or hard links, etc.

At the end of the day I would still suggest the table in a table idea.  It isn't that much
extra work, and on the HDF5 level it is exactly what you want.

Be Well
Anthony

On Wed, Jun 22, 2011 at 6:26 PM, Jason Moore <moorepants@gmail.com> wrote:
Anthony,

I read of the documentation and tried creating a table with a VLArray nested inside, but can't seem to figure out how to nest anything inside a table other than another table description.

I found this question that is trying to do exactly what I want to do:

http://stackoverflow.com/questions/5366099/in-pytables-how-to-create-nested-array-of-variable-length

It is well written but has no answers.

Any specific ideas on how to make this example work?

Jason


On Wed, Jun 22, 2011 at 2:57 PM, Anthony Scopatz <scopatz@gmail.com> wrote:
Also note that whenever you are dealing with variable length stuctures, chuncksize is very important.


On Wed, Jun 22, 2011 at 4:55 PM, Anthony Scopatz <scopatz@gmail.com> wrote:
Hi Jason, 

You can try nesting variable length arrays inside of your table (http://www.pytables.org/docs/manual/ch04.html#VLArrayClassDescr).  You might need to refer to nested tables to get an idea of how to do this (http://www.pytables.org/docs/manual/ch03.html#id332794).  On the HDF5 level, tables are just a convenient wrapper around variable length arrays of structs.  Since you can nest tables, you should be able to nest VLArrays inside of tables.

Let us know if you run into specific problems.

Be Well
Anthony   

On Wed, Jun 22, 2011 at 4:46 PM, Jason Moore <moorepants@gmail.com> wrote:
Hi,

I've been attempting to use pyTables to store data from a data acquisition system that I'm using for my experimental work. I currently make use of several pytables tables to store the data and really like the querying ability that the table object provides. But I have an issue with storing arrays in a table column. For each experiment that I do, I record data for a different amount of time and potentially a different sampling rate. I'd like the rows of the table to correspond with each experiment and the columns with the various data signals that I collect. But it seems that a pytables table object requires every array in a column to be of the same shape. Right now I choose an arbitrarily large shape in my table definition and I'm awkwardly padding the data signals before I store them in the table. Are there other ways to do this?

Jason

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger.
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today.
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users




------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger.
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today.
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users




--
http://mae.ucdavis.edu/~biosport/jkm/
Sports Biomechanics Lab, UC Davis
Davis Bike Collective Minister, Davis, CA
BikeDavis.info
Office: +01 530-752-2163

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger.
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today.
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users



------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger.
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today.
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users




--
http://mae.ucdavis.edu/~biosport/jkm/
Sports Biomechanics Lab, UC Davis
Davis Bike Collective Minister, Davis, CA
BikeDavis.info
Office: +01 530-752-2163
Lab: +01 530-752-2235
Home: +01 530-753-0794