Re: [Pytables-users] Variable Length Datatype support

[Francesc A. <fa...@op...>]

A Dimarts 16 Setembre 2003 20:45, Nicola Larosa va escriure:
> Hi Francesc, thanks for PyTables, it's truly impressive.

Thanks for the compliment :-)

>
> A feature request: what would it take to implement the support for the VL
> (Variable Length) Datatype? I would be more than happy to help with that,
> but need some pointers to move around without breaking anything. :^)
>
> I'm asking since I need to build indices pointing to rows containing each
> data value, and from the HDF5 docs the VL Datatype seems the best solution
> for this.
>
> Otherwise I would have to build a new table for each distinct data value,
> and there can be thousand of those. :^(

Yeah, I am thinking lately about how to implement this in a neat way, but
I'm afraid that the solution is not easy. The Tables are entities that needs
a fixed record length in order to use its I/O buffering capabilities and, at
the moment, unbuffered I/O is not supported in Tables (after all, they are
supposed to be a high performance implementation).

A better approach would be to implement enlargeable Arrays (my top priority
now), and make them flexible enough to support unbuffered I/O (or use buffer
with variable length records, who knows!). That way the indexes might be
created quite easily. However, I don't know when I will implement support
for VL datatypes.

If you want doing that yourself, you can start by studying the Table.Table
class as well as its ancestor, hdf5Extension.Table. From there on, you can
start building a new class (call it VLArray, for example) and making the
necessary changes to support the VL datatype. You should make the new class
a descendant of Leaf (as the Table and Array classes are), so that it will
inherit the necessary methods to interact with the object tree. Finally, you
only need to create some methods in File class to create your new objects.

Well, this is a simplified view of what to do. If you get stuck, please,
tell me and I'll help you. Don't be afraid, as long as you don't mess with
other classes, you can be sure that you won't break anything.

Cheers,

-- 
Francesc Alted