pytables-users

[Pytables-users] Compression and Indexing in PyTables

[David R. <dav...@gm...>]

Hi there,  I was wondering if there any nice tutorials that show the
different compression options such as zlib, bzo, etc. and how to actually
use them with my tables.

There seems to be a lot of good information describing the performance
increase under the Optimization Tips section, but I don't see any clear way
of actually doing this.

Maybe I'm missing something.

Thanks for the help.

-Dave

Re: [Pytables-users] Compression and Indexing in PyTables

[Andreas H. <li...@hi...>]

Hi David,

Am 28.07.2013 15:24, schrieb David Reed:
> Hi there,  I was wondering if there any nice tutorials that show the
> different compression options such as zlib, bzo, etc. and how to
> actually use them with my tables.  
> 
> There seems to be a lot of good information describing the performance
> increase under the Optimization Tips section, but I don't see any clear
> way of actually doing this.  
> 
> Maybe I'm missing something.  

Maybe you're missing this:

   http://pandas.pydata.org/pandas-docs/stable/io.html#compression

The HDFStore constructor has a "complib" kwarg which you can use to set
the compression library. Also look at "complevel" to set the compression
efficiency.

-- Andreas.

Re: [Pytables-users] Compression and Indexing in PyTables

[David R. <dav...@gm...>]

maybe I wasn't aware of this, but has PANDAS completely wrapped PyTables,
or is PyTables something I should still be using for storing and accessing
scientific data, and PANDAS has an access point to it?


On Sun, Jul 28, 2013 at 9:48 AM, Andreas Hilboll <li...@hi...> wrote:

> Hi David,
>
> Am 28.07.2013 15:24, schrieb David Reed:
> > Hi there,  I was wondering if there any nice tutorials that show the
> > different compression options such as zlib, bzo, etc. and how to
> > actually use them with my tables.
> >
> > There seems to be a lot of good information describing the performance
> > increase under the Optimization Tips section, but I don't see any clear
> > way of actually doing this.
> >
> > Maybe I'm missing something.
>
> Maybe you're missing this:
>
>    http://pandas.pydata.org/pandas-docs/stable/io.html#compression
>
> The HDFStore constructor has a "complib" kwarg which you can use to set
> the compression library. Also look at "complevel" to set the compression
> efficiency.
>
> -- Andreas.
>

Re: [Pytables-users] Compression and Indexing in PyTables

[Francesc A. <fa...@gm...>]

On 7/28/13 10:21 AM, David Reed wrote:
> maybe I wasn't aware of this, but has PANDAS completely wrapped 
> PyTables, or is PyTables something I should still be using for storing 
> and accessing scientific data, and PANDAS has an access point to it?

Yeah, more the later than the former.  PyTables is an standalone 
library, but Pandas uses it as another storage backend.

-- 
Francesc Alted

Re: [Pytables-users] Compression and Indexing in PyTables

[Francesc A. <fa...@gm...>]

On 7/28/13 9:24 AM, David Reed wrote:
> Hi there,  I was wondering if there any nice tutorials that show the 
> different compression options such as zlib, bzo, etc. and how to 
> actually use them with my tables.
>
> There seems to be a lot of good information describing the performance 
> increase under the Optimization Tips section, but I don't see any 
> clear way of actually doing this.
>
> Maybe I'm missing something.

Well, the compression options are part of the more general Filters 
helper class:

http://pytables.github.io/usersguide/libref/helper_classes.html#the-filters-class

This stems from the fact that in HDF5 a compressor is just like another 
data filter.

-- 
Francesc Alted