[Pytables-users] Datatype sizes and object headers (was: Error message from hdf5)

[Francesc A. <fa...@py...>]

Hi,

After asking in the HDF5 list, THG developers says that the problem is 
that the default values for the data type definition (your description 
below) are saved in a special area of the HDF5 file called "object 
header" which has a limited size of 64 KB.  This is why PyTables cannot 
create datasets whose default values would take more than 64 KB to keep 
on disk.  They have filed a ticket because they want to fix this in 
HDF5.

Having said that, I've come to implement a workaround that allows to 
create the dataset *whenever* you have no default values, or they are 
zero-based (0, 0., False, 0j or '').  For details see:

http://www.pytables.org/trac/ticket/211

However, I see that you define defaults different from zero in your 
table description, so the above workaround would not work for you, 
unless you decide to set all defaults to zero.

At any rate, having a very large type size in your datasets is not a 
good thing, as you can easily fall into I/O performance problems.  Your 
best bet would be to place fields 'adcI' and 'adcQ' to a couple of 
EArray/CArray objects and keep a reference (the number of row in 
EArray, for example) in your table so as to reach those fields.

HTH,

Francesc

A Thursday 26 February 2009, glenn escrigué:
> I have come across a similar error as this, and I've traced it down a
> bit. My code originally worked fine with HDF 1.8 and Pytables 2.0,
> but now that I'm running HDF 1.8.2 and Pytables 2.1, I see the error
> below with this table description:
> {'Timestamp':tables.Float64Col(),
>                 'A_ID':tables.UInt32Col(),
>                 'ID':tables.UInt64Col(),
>                 'adcI':tables.Int8Col(shape=2**17,)),
>                 'adcQ':tables.Int8Col(shape=2**17,)),
>                 'SynFreq':tables.Float32Col(dflt=-1),
>                 'ReceiverReady':tables.BoolCol(dflt=False),
>                 'ReceiverState':tables.UInt32Col(),
>                 'Noise':tables.UInt8Col(dflt=0),
>                 'TertiaryPosition':tables.Int32Col(dflt=0),
>                 'TelescopeHA':tables.Float32Col(dflt=0),
>                 'TelescopeDec':tables.Float32Col(dflt=0),
>                 'TelescopeLastUpdated':tables.Float64Col(dflt=0)}
>
> The problem is with the 'adcI' and 'adcQ' columns. If I reduce their
> shape to 2**14 or lower, it seems to work fine. I hope this helps
> track down the bug, I really need this to work again.
>
> HDF5-DIAG: Error detected in HDF5 (1.8.2) thread 0:
>   #000: H5Ddeprec.c line 170 in H5Dcreate1(): unable to create
> dataset major: Dataset
>     minor: Unable to initialize object
>   #001: H5Dint.c line 429 in H5D_create_named(): unable to create and
> link to dataset
>     major: Dataset
>     minor: Unable to initialize object
>   #002: H5L.c line 1633 in H5L_link_object(): unable to create new
> link to object major: Links
>     minor: Unable to initialize object
>   #003: H5L.c line 1856 in H5L_create_real(): can't insert link
>     major: Symbol table
>     minor: Unable to insert object
>   #004: H5Gtraverse.c line 877 in H5G_traverse(): internal path
> traversal failed major: Symbol table
>     minor: Object not found
>   #005: H5Gtraverse.c line 703 in H5G_traverse_real(): traversal
> operator failed major: Symbol table
>     minor: Callback failed
>   #006: H5L.c line 1679 in H5L_link_cb(): unable to create object
>     major: Object header
>     minor: Unable to initialize object
>   #007: H5O.c line 2531 in H5O_obj_create(): unable to open object
>     major: Object header
>     minor: Can't open object
>   #008: H5Doh.c line 278 in H5O_dset_create(): unable to create
> dataset major: Dataset
>     minor: Unable to initialize object
>   #009: H5Dint.c line 1139 in H5D_create(): can't update the metadata
> cache major: Dataset
>     minor: Unable to initialize object
>   #010: H5Dint.c line 846 in H5D_update_oh_info(): unable to update
> new fill value header message
>     major: Dataset
>     minor: Unable to initialize object
>   #011: H5Omessage.c line 188 in H5O_msg_append_oh(): unable to
> create new message in header
>     major: Attribute
>     minor: Unable to insert object
>   #012: H5Omessage.c line 228 in H5O_msg_append_real(): unable to
> create new message major: Object header
>     minor: No space available for allocation
>   #013: H5Omessage.c line 1936 in H5O_msg_alloc(): unable to allocate
> space for message
>     major: Object header
>     minor: Unable to initialize object
>   #014: H5Oalloc.c line 972 in H5O_alloc(): object header message is
> too large major: Object header
>     minor: Unable to initialize object
> TimeSeriesH5.prepare_for_writing(): Exception: Problems creating the
> table
>
>
>
>
>
> ---------------------------------------------------------------------
>--------- Open Source Business Conference (OSBC), March 24-25, 2009,
> San Francisco, CA -OSBC tackles the biggest issue in open source:
> Open Sourcing the Enterprise -Strategies to boost innovation and cut
> costs with open source participation -Receive a $600 discount off the
> registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users



-- 
Francesc Alted