[Pytables-users] Storing large images in PyTable

[Mathieu D. <dub...@ya...>]

Hello,

I'm a beginner with Pyable.

I wanted to store a database in a HDF5 file using PyTable. The DB is 
made by a CSV file (which contains the subject information) and a lot of 
images (I work on MRI so the images are 3 dimensional float32 arrays of 
shape (121, 145, 121)). The relation is very simple: there are a 3 
images per subject.

My first idea was to create a class  Subject like this:
class Subject(tables.IsDescription):
     # Subject information
     Id           = tables.UInt16Col()
     ...
     Image    = tables.Float32Col(shape=IMAGE_SIZE)

And the proceed like in the tutorial (open a file, create a group and a 
table associated to the Subject class and then append data to this table).

Unfortunately I got an error when creating the table (even before 
inserting data):
HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 140612945950464:
   #000: ../../../src/H5Ddeprec.c line 170 in H5Dcreate1(): unable to 
create dataset
     major: Dataset
     minor: Unable to initialize object
   #001: ../../../src/H5Dint.c line 428 in H5D_create_named(): unable to 
create and link to dataset
     major: Dataset
     minor: Unable to initialize object
   #002: ../../../src/H5L.c line 1639 in H5L_link_object(): unable to 
create new link to object
     major: Links
     minor: Unable to initialize object
   #003: ../../../src/H5L.c line 1862 in H5L_create_real(): can't insert 
link
     major: Symbol table
     minor: Unable to insert object
   #004: ../../../src/H5Gtraverse.c line 877 in H5G_traverse(): internal 
path traversal failed
     major: Symbol table
     minor: Object not found
   #005: ../../../src/H5Gtraverse.c line 703 in H5G_traverse_real(): 
traversal operator failed
     major: Symbol table
     minor: Callback failed
   #006: ../../../src/H5L.c line 1685 in H5L_link_cb(): unable to create 
object
     major: Object header
     minor: Unable to initialize object
   #007: ../../../src/H5O.c line 2677 in H5O_obj_create(): unable to 
open object
     major: Object header
     minor: Can't open object
   #008: ../../../src/H5Doh.c line 296 in H5O_dset_create(): unable to 
create dataset
     major: Dataset
     minor: Unable to initialize object
   #009: ../../../src/H5Dint.c line 1034 in H5D_create(): can't update 
the metadata cache
     major: Dataset
     minor: Unable to initialize object
   #010: ../../../src/H5Dint.c line 799 in H5D_update_oh_info(): unable 
to update new fill value header message
     major: Dataset
     minor: Unable to initialize object
   #011: ../../../src/H5Omessage.c line 188 in H5O_msg_append_oh(): 
unable to create new message in header
     major: Attribute
     minor: Unable to insert object
   #012: ../../../src/H5Omessage.c line 228 in H5O_msg_append_real(): 
unable to create new message
     major: Object header
     minor: No space available for allocation
   #013: ../../../src/H5Omessage.c line 1940 in H5O_msg_alloc(): unable 
to allocate space for message
     major: Object header
     minor: Unable to initialize object
   #014: ../../../src/H5Oalloc.c line 1032 in H5O_alloc(): object header 
message is too large
     major: Object header
     minor: Unable to initialize object
Traceback (most recent call last):
   File "00_build_dataset.tmp.py", line 52, in <module>
     dump_in_hdf5(**vars(args))
   File "00_build_dataset.tmp.py", line 32, in dump_in_hdf5
     data_api.Subject)
   File "/usr/lib/python2.7/dist-packages/tables/file.py", line 770, in 
createTable
     chunkshape=chunkshape, byteorder=byteorder)
   File "/usr/lib/python2.7/dist-packages/tables/table.py", line 832, in 
__init__
     byteorder, _log)
   File "/usr/lib/python2.7/dist-packages/tables/leaf.py", line 291, in 
__init__
     super(Leaf, self).__init__(parentNode, name, _log)
   File "/usr/lib/python2.7/dist-packages/tables/node.py", line 296, in 
__init__
     self._v_objectID = self._g_create()
   File "/usr/lib/python2.7/dist-packages/tables/table.py", line 983, in 
_g_create
     self._v_new_title, self.filters.complib or '', obversion )
   File "tableExtension.pyx", line 195, in 
tables.tableExtension.Table._createTable (tables/tableExtension.c:2181)
tables.exceptions.HDF5ExtError: Problems creating the table

I think that the size of the column is too large (if I remove the Image 
field, everything works perfectly).

Therefore what is the best way to store the images (while keeping the 
relation)? I have read various post about this subject on the web but 
could not find a definitive answer (the more helpful was 
http://stackoverflow.com/questions/8843062/python-how-to-store-a-numpy-multidimensional-array-in-pytables).

I was thinking to create an extensible array and store each image in the 
same order than the subject. However, I would feel more comfortable if 
the subject Id could be inserted too (to join the tables).

Any help?

Mathieu