[Pytables-users] Fast temporary tables

[Francesc A. <fa...@py...>]

A Thursday 02 December 2010 18:19:50 dav...@ub... escrigué:
> The pytables side, once I figured how to use it was simple. I'm
> needing to write a memory cache on top because I need fast temporary
> tables that have the same interface (retrieval) but aren't backed by
> disk. That's the slow bit.

Hmm, in my opinion it is better to trust with the OS filesystem cache.  
If your temporary tables are accessed enough times (2 or 3, typically), 
they will eventually get loaded into memory (OS cache).  You may want to 
use compression on these tables so that the OS can load them more easily 
(and using Blosc will be of great help so as to not loose speed).

For more speed, you may want to consider using the carray package:

https://github.com/FrancescAlted/carray

It contains a special object, called a ctable, that is basically a NumPy 
structured array, but with important differences:

- The data is stored column-wise (not row-wise)
- The data can be compressed

This minimizes a lot memory usage (both bandwidth and consumption).  To 
prove this, I've just added a small benchmark (bench/ctable-query.py) 
that do some query on a table with 1 million rows and 100 columns.  Here 
are the results (run on a 2-core machine):

Querying '(f1>300) & ((f2>3) & (f2<1e4))' with 10^6 rows
Time for plain numpy --> 0.140
Time for numexpr --> 0.114
Time for ctable (uncompressed) --> 0.048 -- size (MB): 770
Time for ctable (compressed) --> 0.048  -- size (MB): 20
Time for PyTables (non-indexed) --> 0.618
PyTables Pro detected!  Indexing f2 column...
Time for PyTables Pro (indexed) --> 0.113

Note how the ctable object reduces the time to compute the query by a 
factor of 2x, while reducing the memory needs by a huge amount (that 
depends on the dataset indeed).  It can even do queries faster than Pro!

I'm in the final stages for releasing carray 0.3 (I still need to finish 
parts of the documentation), but I plan to announce 0.3 next week or so.  
carray will be a nice complement for PyTables (not only for implementing 
fast temporary tables, but also to serve as intermediate buffer for 
improved I/O), and I expect to integrate it for PyTables 3.

-- 
Francesc Alted