Re: [Pytables-users] Storing large images in PyTable

[Anthony S. <sc...@gm...>]

Thanks Mathieu!

I am glad this is working for you now.  File this one under "Mysterious
Errors of the Universe" :).

Be Well
Anthony


On Fri, Jul 5, 2013 at 6:51 PM, Mathieu Dubois
<dub...@ya...>wrote:

>  Hi,
>
> Sorry for the late response.
>
> First of all, I have managed to achieve what I wanted to do differently.
>
> Then the code Francesc send works well (I had to adapt it because I use
> version 2.3.1 under Ubuntu 12.04).
>
> I was able to reproduce something similar with a class like this (copied &
> pasted from the tutorial):
>
> import tables as tb
>
> import numpy as np
>
> class Subject(tb.IsDescription):
>
>      # Subject information
>
>      Id       = tb.UInt16Col()
>
>      Image    = tb.Float32Col(shape=(121, 145, 121))
>
> h5file = tb.openFile("tutorial1.h5", mode = "w", title = "Test file")
>
> group = h5file.createGroup("/", 'subject', 'Suject information')
>
> table = h5file.createTable(group, 'readout', Subject, "Readout example")
>
> subject = table.row
>
> for i in xrange(10):
>
>      subject['Id'] = i
>
>      subject['Image'] = np.ones((121, 145, 121))
>
>      subject.append()
>
> This code works well  too.
>
> So I don't really know why nothing was working yesterday: this was the
> same class and a very close program. I will try to investigate later on
> this.
>
> Thanks for everything,
> Mahtieu
>
> Le 05/07/2013 16:54, Anthony Scopatz a écrit :
>
>
>
>
> On Fri, Jul 5, 2013 at 8:40 AM, Francesc Alted <fa...@gm...> wrote:
>
>> On 7/5/13 1:33 AM, Mathieu Dubois wrote:
>> > tables.tableExtension.Table._createTable (tables/tableExtension.c:2181)
>> >>
>> >>     tables.exceptions.HDF5ExtError: Problems creating the table
>> >>
>> >>     I think that the size of the column is too large (if I remove the
>> >>     Image
>> >>     field, everything works perfectly).
>> >>
>> >>
>> >> Hi Mathieu,
>> >>
>> >> This shouldn't be the case.  What is the value of IMAGE_SIZE?
>> >
>> > IMAGE_SIZE is a tuple containing (121, 145, 121).
>>
>>  This is a bit large for a row in the Table object.  My recommendation
>> for these cases is to use an associated EArray with shape (0, 121, 145,
>> 121) and then append the images there.  You can always refer to the
>> image by issuing a __getitem__() operation on the EArray object with the
>> index of the row in the table.  Easy as a pie and you will allow the
>> compression library (in case you are using compression) to work much
>> more efficiently for the table.
>>
>
>
>  Hi Francesc,
>
>  I disagree that this shape is too large for a table.  Here is a minimal
> example that works for me:
>
>  import tables as tb
> import numpy as np
>
>  images = np.ones(100, dtype=[('id', np.uint16),
>                              ('image', np.float32, (121, 145, 121))
>                              ])
>
>  with tb.open_file('temp.h5', 'w') as f:
>     f.create_table('/', 'images', images)
>
>  I think that there is something else going on with the initialization
> but Mathieu hasn't given us enough information to figure it out =/.  A
> minimal failing script would be super helpful here!
>
>  (BTW Mathieu, Tables can also take advantage of compression.  Though
> Francesc's solution is nicer for a lot of reason too.)
>
>  Be Well
>  Anthony
>
>
>>
>> HTH,
>>
>> -- Francesc Alted
>>
>>
>> ------------------------------------------------------------------------------
>> This SF.net email is sponsored by Windows:
>>
>> Build for Windows Store.
>>
>> http://p.sf.net/sfu/windows-dev2dev
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
> http://p.sf.net/sfu/windows-dev2dev
>
>
>
> _______________________________________________
> Pytables-users mailing lis...@li...://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>