Re: [Pytables-users] Storing large images in PyTable

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Le 05/07/2013 00:31, Anthony Scopatz a écrit :
>
>
>
> On Thu, Jul 4, 2013 at 4:13 PM, Mathieu Dubois 
> <dub...@ya... <mailto:dub...@ya...>> wrote:
>
>     Hello,
>
>     I'm a beginner with Pyable.
>
>     I wanted to store a database in a HDF5 file using PyTable. The DB is
>     made by a CSV file (which contains the subject information) and a
>     lot of
>     images (I work on MRI so the images are 3 dimensional float32
>     arrays of
>     shape (121, 145, 121)). The relation is very simple: there are a 3
>     images per subject.
>
>     My first idea was to create a class  Subject like this:
>     class Subject(tables.IsDescription):
>          # Subject information
>          Id           = tables.UInt16Col()
>          ...
>          Image    = tables.Float32Col(shape=IMAGE_SIZE)
>
>     And the proceed like in the tutorial (open a file, create a group
>     and a
>     table associated to the Subject class and then append data to this
>     table).
>
>     Unfortunately I got an error when creating the table (even before
>     inserting data):
>     HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread
>     140612945950464:
>        #000: ../../../src/H5Ddeprec.c line 170 in H5Dcreate1(): unable to
>     create dataset
>          major: Dataset
>          minor: Unable to initialize object
>        #001: ../../../src/H5Dint.c line 428 in H5D_create_named():
>     unable to
>     create and link to dataset
>          major: Dataset
>          minor: Unable to initialize object
>        #002: ../../../src/H5L.c line 1639 in H5L_link_object(): unable to
>     create new link to object
>          major: Links
>          minor: Unable to initialize object
>        #003: ../../../src/H5L.c line 1862 in H5L_create_real(): can't
>     insert
>     link
>          major: Symbol table
>          minor: Unable to insert object
>        #004: ../../../src/H5Gtraverse.c line 877 in H5G_traverse():
>     internal
>     path traversal failed
>          major: Symbol table
>          minor: Object not found
>        #005: ../../../src/H5Gtraverse.c line 703 in H5G_traverse_real():
>     traversal operator failed
>          major: Symbol table
>          minor: Callback failed
>        #006: ../../../src/H5L.c line 1685 in H5L_link_cb(): unable to
>     create
>     object
>          major: Object header
>          minor: Unable to initialize object
>        #007: ../../../src/H5O.c line 2677 in H5O_obj_create(): unable to
>     open object
>          major: Object header
>          minor: Can't open object
>        #008: ../../../src/H5Doh.c line 296 in H5O_dset_create(): unable to
>     create dataset
>          major: Dataset
>          minor: Unable to initialize object
>        #009: ../../../src/H5Dint.c line 1034 in H5D_create(): can't update
>     the metadata cache
>          major: Dataset
>          minor: Unable to initialize object
>        #010: ../../../src/H5Dint.c line 799 in H5D_update_oh_info():
>     unable
>     to update new fill value header message
>          major: Dataset
>          minor: Unable to initialize object
>        #011: ../../../src/H5Omessage.c line 188 in H5O_msg_append_oh():
>     unable to create new message in header
>          major: Attribute
>          minor: Unable to insert object
>        #012: ../../../src/H5Omessage.c line 228 in H5O_msg_append_real():
>     unable to create new message
>          major: Object header
>          minor: No space available for allocation
>        #013: ../../../src/H5Omessage.c line 1940 in H5O_msg_alloc():
>     unable
>     to allocate space for message
>          major: Object header
>          minor: Unable to initialize object
>        #014: ../../../src/H5Oalloc.c line 1032 in H5O_alloc(): object
>     header
>     message is too large
>          major: Object header
>          minor: Unable to initialize object
>     Traceback (most recent call last):
>        File "00_build_dataset.tmp.py
>     <http://00_build_dataset.tmp.py>", line 52, in <module>
>          dump_in_hdf5(**vars(args))
>        File "00_build_dataset.tmp.py
>     <http://00_build_dataset.tmp.py>", line 32, in dump_in_hdf5
>          data_api.Subject)
>        File "/usr/lib/python2.7/dist-packages/tables/file.py", line
>     770, in
>     createTable
>          chunkshape=chunkshape, byteorder=byteorder)
>        File "/usr/lib/python2.7/dist-packages/tables/table.py", line
>     832, in
>     __init__
>          byteorder, _log)
>        File "/usr/lib/python2.7/dist-packages/tables/leaf.py", line
>     291, in
>     __init__
>          super(Leaf, self).__init__(parentNode, name, _log)
>        File "/usr/lib/python2.7/dist-packages/tables/node.py", line
>     296, in
>     __init__
>          self._v_objectID = self._g_create()
>        File "/usr/lib/python2.7/dist-packages/tables/table.py", line
>     983, in
>     _g_create
>          self._v_new_title, self.filters.complib or '', obversion )
>        File "tableExtension.pyx", line 195, in
>     tables.tableExtension.Table._createTable
>     (tables/tableExtension.c:2181)
>     tables.exceptions.HDF5ExtError: Problems creating the table
>
>     I think that the size of the column is too large (if I remove the
>     Image
>     field, everything works perfectly).
>
>
> Hi Mathieu,
>
> This shouldn't be the case.  What is the value of IMAGE_SIZE?

IMAGE_SIZE is a tuple containing (121, 145, 121).

>
> Be Well
> Anthony
>
>
>     Therefore what is the best way to store the images (while keeping the
>     relation)? I have read various post about this subject on the web but
>     could not find a definitive answer (the more helpful was
>     http://stackoverflow.com/questions/8843062/python-how-to-store-a-numpy-multidimensional-array-in-pytables).
>
>     I was thinking to create an extensible array and store each image
>     in the
>     same order than the subject. However, I would feel more comfortable if
>     the subject Id could be inserted too (to join the tables).
>
>     Any help?
>
>     Mathieu
>
>     ------------------------------------------------------------------------------
>     This SF.net email is sponsored by Windows:
>
>     Build for Windows Store.
>
>     http://p.sf.net/sfu/windows-dev2dev
>     _______________________________________________
>     Pytables-users mailing list
>     Pyt...@li...
>     <mailto:Pyt...@li...>
>     https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
>
>
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users