[Numpy-discussion] ANNOUNCE: Pytables 0.4

[Francesc A. <fa...@op...>]

Announcing PyTables 0.4
-----------------------

I'm happy to announce the first beta release of PyTables. It is labelled
beta because it has been thoroughly tested, even in production environmen=
ts,
and it is getting fairly mature.

Besides, from now on, the API will remain mostly stable, so you can start
using PyTables now with the guarantee that your code will also work (well=
,
mostly ;-) in the future. The large amount of unit tests included in
PyTables will also ensure the backward compatibility as well as the quali=
ty
of future releases will remain at least as good as it is now (although
hopefully it should increase!).

What's new
-----------

- numarray objects (NumArray, CharArray and RecArray) supported

- As a consequence of a large internal code redesign (numarray is at
  the core of PyTables now), performance has been improved by a factor
  of 10 (see "How Fast Is It?" section)

- It consumes far less memory than previous version

- Support for reading generic HDF5 files added (!)

- Some bugs and memory leaks existing in 0.2 solved

- Updated documentation

- Added more unit tests (more than 200 now!)

What it is
----------

In short, PyTables provides a powerful and very Pythonic interface to
process table and array data.

Its goal is to enable the end user to manipulate easily scientific
data tables and Numerical and numarray Python objects in a persistent
hierarchical structure. The foundation of the underlying hierarchical
data organization is the excellent HDF5 library
(http://hdf.ncsa.uiuc.edu/HDF5).

A table is defined as a collection of records whose values are stored
in fixed-length fields. All records have the same structure and all
values in each field have the same data type.  The terms
"fixed-length" and strict "data types" seems to be quite a strange
requirement for an interpreted language like Python, but they serve a
useful function if the goal is to save very large quantities of data
(such as is generated by many scientific applications, for example) in
an efficient manner that reduces demand on CPU time and I/O resources.

Quite a bit effort has been invested to make browsing the hierarchical
data structure a pleasant experience. PyTables implements just two
(orthogonal) easy-to-use methods for browsing.

What is HDF5?
-------------

For those people who know nothing about HDF5, it is is a general
purpose library and file format for storing scientific data made at
NCSA. HDF5 can store two primary objects: datasets and groups. A
dataset is essentially a multidimensional array of data elements, and
a group is a structure for organizing objects in an HDF5 file. Using
these two basic constructs, one can create and store almost any kind of
scientific data structure, such as images, arrays of vectors, and
structured and unstructured grids. You can also mix and match them in
HDF5 files according to your needs.

How fast is it?
---------------

PyTables can write table records between 20 and 30 times faster than
cPickle and between 3 and 10 times faster than struct (it is a module
present in the Standard Library); and retrieves information around 100
times faster than cPickle and between 8 and 10 times faster than
struct.

When compared with SQLite (http://www.sqlite.org/), one of the fastest
(free) relational databases available, PyTables achieves between a 60%
and 80% the speed of SQLite during selects of dataset sizes that fit
in the O.S. filesystem memory cache. However, when those sizes does
not fit in the cache (i.e. when dealing with large amounts of data),
PyTables beats SQLite by a factor of 2 or even more (depending on the
kind of record selected), and its performance in this case is only
limited by the I/O speed of the disk subsystem.

Go to http://pytables.sourceforge.net/doc/PyCon.html#section4 for a
detailed description on the conducted benchmarks.

Platforms
---------

I'm using Linux as the main development platform, but PyTables should
be easy to compile/install on other UNIX machines. This package has
also passed all the tests on a UltraSparc platform with Solaris 7 and
Solaris 8. It also compiles and passes all the tests on a SGI
Origin2000 with MIPS R12000 processors and running IRIX 6.5.

If you are using Windows and you get the library to work, please let
me know.

An example?
-----------

At the bottom of this message there is some code that shows basic
capabilities of PyTables. You may also look at
http://pytables.sourceforge.net/tut/tutorial1-1.html and=20
http://pytables.sourceforge.net/tut/tutorial1-2.html
for online code.

Web site
--------

Go to the PyTables web site for downloading and more details:

http://pytables.sf.net/

Share your experience
---------------------

Let me know of any bugs, suggestions, gripes, kudos, etc. you may
have.

Have fun!

-- Francesc Alted
fa...@op...


*-*-*-**-*-*-**-*-*-**-*-*- Small code example  *-*-*-**-*-*-**-*-*-**-*-=
*-*
from tables import *

class Particle(IsDescription):
    identity =3D Col("CharType", 22, " ", pos =3D 0)  # character String
    idnumber =3D Col("Int16", 1, pos =3D 1)  # short integer
    speed =3D Col("Float32", 1, pos =3D 1)  # single-precision

# Open a file in "w"rite mode
fileh =3D openFile("objecttree.h5", mode =3D "w")
# Get the HDF5 root group
root =3D fileh.root

# Create the groups:
group1 =3D fileh.createGroup(root, "group1")
group2 =3D fileh.createGroup(root, "group2")

# Now, create a table in "group0" group
array1 =3D fileh.createArray(root, "array1", ["string", "array"], "String=
=20
array")
# Create 2 new tables in group1
table1 =3D fileh.createTable(group1, "table1", Particle)
table2 =3D fileh.createTable("/group2", "table2", Particle)
# Create the last table in group2
array2 =3D fileh.createArray("/group1", "array2", [1,2,3,4])

# Now, fill the tables:
for table in (table1, table2):
    # Get the record object associated with the table:
    row =3D table.row
    # Fill the table with 10 records
    for i in xrange(10):
        # First, assign the values to the Particle record
        row['identity']  =3D 'This is particle: %2d' % (i)
        row['idnumber'] =3D i
        row['speed']  =3D i * 2.
        # This injects the Record values
        row.append()

    # Flush the table buffers
    table.flush()

# Select actual data from table.
# on entries where TDCcount field is greater than 3 and pressure less tha=
n 50
out =3D [ x['identity'] for x in table.iterrows()
        if x['idnumber'] > 3 and 4 < x['speed'] < 10 ]

print out

# Finally, close the file (this also will flush all the remaining buffers=
!)
fileh.close()