Menu
▾
▴
Re: [Pytables-users] variable length strings in tables?
|
From: Jeff R. <jr...@ya...> - 2012-12-03 18:25:46
|
thanks created https://github.com/PyTables/PyTables/issues/198 I can be reached on my cell (917)971-6387 ________________________________ From: Anthony Scopatz <sc...@gm...> To: Jeff Reback <je...@re...>; Discussion list for PyTables <pyt...@li...> Sent: Monday, December 3, 2012 11:15 AM Subject: Re: [Pytables-users] variable length strings in tables? On Sun, Dec 2, 2012 at 2:49 PM, Jeff Reback <jr...@ya...> wrote: Hi, > >Pandas uses pytables as a storage backend and has worked out quite well >fyi ... http://pandas.pydata.org/pandas-docs/dev/io.html#hdf5-pytables > >I have a particular use case where I build a table, then later append to it. >Fixed types are no problem. However, I often index these tables by StringCols, which I pre-allocated >to the largest size I think that i'll need. So, wanted to think about supporting >variable-length string columns in the table. > >any thoughts on these strategies: >1) any way to directly support a variable-length string in a particular column? (e.g. VLStringCol doesn't exist but a stand-alone VLStringAtom does) This is possible as the underlying HDF5 library will support it. However, no one has had the time to write it. Please open an issue (or possibly a pull request related to this.) 2) As an alternative, I could store along with the table a VLArray the same # of rows as the table and keep string data here > -- of course have to keep the synchronization up to date (and this doesn't help with an 'indexing' column, just with 'data' columns) This is what I do in PyTables and HDF5 itself. It works out quite well for me. This has the advantage that the VLString data get compressed separately from the numeric data (if using compression). Yes, it is one more thing to manage, but the file sizes I are much significantly smaller. Be Well Anthony >thanks, > >Jeff >------------------------------------------------------------------------------ >Keep yourself connected to Go Parallel: >DESIGN Expert tips on starting your parallel project right. >http://goparallel.sourceforge.net/ >_______________________________________________ >Pytables-users mailing list >Pyt...@li... >https://lists.sourceforge.net/lists/listinfo/pytables-users > > |
