pytables-users — PyTables users discussion list

Re: [Pytables-users] Complex numbers

[Francesc A. <fa...@py...>]

A Dimarts 14 Setembre 2004 08:07, Alexey Goldin va escriure:
> Yet another mail from user who wanted to store electromagnetic
> simulation data in pytables. Is there any particular reason (except
> for lack of time) why  pytables does not incorporate patch 928606 from
> http://sourceforge.net/mailarchive/forum.php?thread_id=4324550&forum_id=13760

Yes, a very unfortunate one: I just didn't know nothing about this patch. I
know that it has been published on this list, but I do not remember it at
all (and I should because it is very good stuff indeed). In fact, I've just
looked again at my mail archives and it is not there (!). Well, perhaps I
pressed too fast the delete key when browsing my e-mail or my anti-spam
agent has get crazy. Anyway, thanks for your warning!

I'll try to include it for the new release of PyTables. I've checked out
that there is another patch by Shack Toms in the patches section of PyTables
project. I'll look into it as well, of course.

Mmm, I've checked that there is no way to subscribe to patches submissions.
So please, if anybody wants to send a patch, please, send me a copy to my
private e-mail as well and we will hopefulyy minimize the risk of loosing it
in the cyberspace.

Thanks again and thanks to Tom Hedley for the patch!

-- 
Francesc Alted

[Pytables-users] Complex numbers

[Alexey G. <ale...@gm...>]

Yet another mail from user who wanted to store electromagnetic
simulation data in pytables. Is there any particular reason (except
for lack of time) why  pytables does not incorporate patch 928606 from
http://sourceforge.net/mailarchive/forum.php?thread_id=4324550&forum_id=13760
? I would like not to maintain separate copy of this package on my
Debian system, especially with upcoming release of CStables....

Thanks!

[Pytables-users] Talk at SciPy 2004

[Francesc A. <fa...@py...>]

Hi,

I've just placed the talk I've recently given at the SciPy '04 Conference
(http://www.scipy.org/wikis/scipy04/), where I presented the new features
for the forthcoming PyTables release (0.9) as well as an introduction to the
newcomers to the PyTables family, namely ViTables and CSTables.

You can find it at:

http://pytables.sourceforge.net/doc/SciPy04.pdf

Hope would be useful to you,

-- 
Francesc Alted

[Pytables-users] PyTables talk on SciPy'04

[Francesc A. <fa...@py...>]

Hi everybody,

This is just to comment that I'll be giving a talk the next September, 3rd,
at Caltech (Pasadena, California) during the SciPy '04 Conference
(http://www.scipy.org/wikis/scipy04/).

I'll be speaking not only about PyTables and the new features incorporated
in the forthcoming 0.9 version (modification of values and table
indexation), but also about its new graphical viewer (ViTables) and the
forthcoming client-server implementation (CSTables).

If you are in the Los Angeles area and like to go there, it will be my
pleasure to meet you.

Cheers,

-- 
Francesc Alted

Re: [Pytables-users] Building on OS X

[Francesc A. <fa...@py...>]

A Dimarts 31 Agost 2004 22:36, James Boyle va escriure:
> I have attempted to build pytables 0.8.1 on OS X.
> I am running OS X 10.3.5, using the OS X python (2.3), HDF5 1.6.2, gcc 
> 3.3, numarray 1.0

That seems fine to me...

> Some details are below, in short I get unresolved references:
> 
> _SZ_BufftoBuffCompress
> _SZ_BufftoBuffDecompress
> 
> at the final linkstep ..
> Can anyone tell me where these references should be found?

Mmmm, no idea. They seems like compressor routines (SZIP?). Maybe have you
downloaded a binary version of HDF5 library (that might depend on SZIP
compressor) and you failed to install SZIP?

Right now PyTables does not use SZIP at all. If the problem is this, try
compiling HDF5 by yourself without SZIP support. That *should* work.

Cheers,

-- 
Francesc Alted

[Pytables-users] Building on OS X

[James B. <bo...@ll...>]

I have attempted to build pytables 0.8.1 on OS X.
I am running OS X 10.3.5, using the OS X python (2.3), HDF5 1.6.2, gcc 
3.3, numarray 1.0

Some details are below, in short I get unresolved references:

_SZ_BufftoBuffCompress
_SZ_BufftoBuffDecompress

at the final linkstep ..
Can anyone tell me where these references should be found?
Has anyone built on OS X successfully?

Thanks for any help.

Jim

[macjboyle:IBM1/python/pytables-0.8.1] boyle% python setup.py install 
--hdf5=/Volumes/IBM1/HDF_5-1.6.2-macosx
.
.
.......Many warnings from gcc, but no failures that are apparent in 
hdf5Extension
.
.
.
building 'tables.hdf5Extension' extension
gcc -Wl,-F. -Wl,-F. -bundle -framework Python 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/hdf5Extension.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/calcoffset.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/arraytypes.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/getfieldfmt.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/utils.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/H5Zlzo.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/H5Zucl.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/H5ARRAY.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/H5VLARRAY.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/H5LT.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/H5TB.o 
build/temp.darwin-7.5.0-Power_Macintosh-2.3/src/H5TB-opt.o 
-L/Volumes/IBM1/HDF_5-1.6.2-macosx/lib -lhdf5 -llzo -o 
build/lib.darwin-7.5.0-Power_Macintosh-2.3/tables/hdf5Extension.so
ld: Undefined symbols:
_SZ_BufftoBuffCompress
_SZ_BufftoBuffDecompress
error: command 'gcc' failed with exit status 1

[Pytables-users] Re: PyTables row size limit

[Francesc A. <fa...@py...>]

Hi Russel,

The 8k limit in pytables 0.8 was mainly a limitation due to my relatively
poor understanding of the chunks and the BTree that HDF5 keeps in memory to
be used to map chunks on disk. A bigger rowsize would make reading access
*much* more slower.

However, PyTables 0.8.1 has implemented a new algorithm to compute both
chunks sizes and buffer sizes. As a consequence, the limit has been raised
to 512 KB without a noticeable impact on performance. I hope that would be
enough for you, but that limit can be further increased in case of
necessity.

Cheers,

A Dijous 12 Agost 2004 20:56, vareu escriure:
>     I have just begun experimenting with PyTables for a project analyzing 
> LC/MS data (from this machine 
> <http://www.ionspec.com/Our%20Products/HiResESI/HiResESI%20fs.htm> ). 
> These data sets consist of 1200 mass spectra each consisting of about 
> 1800000 double-precision values.
>     I created a table with each row consisting of some information about 
> each spectrum and the data points for that spectrum.  This results in a row 
> size substantially larger than the 8192 byte limit imposed by pytables.  I 
> could find no such corresponding limit in the HDF5 code, so I took out the 
> row size test in pytables and recompiled.  I generated some files and 
> everything appears fine.  I can read all the data back and manipulate it 
> just as I hoped.  My question is where does this limit come from?  What 
> would I have to do to safely remove (or substantially increase) the row 
> size limit in pytables?
>      Thanks in advance for any hints.
> Russel Howe
> 
> 

-- 
Francesc Alted

Re: [Pytables-users] support for references, mounting, file drivers?

[Francesc A. <fa...@py...>]

Hi Fabian,

A Dilluns 09 Agost 2004 21:53, Fabian Kloosterman va escriure:
> I just started to learn python and I would like to use pyTables for my
> future projects to add HDF5 support. I was wondering if pyTables supports
> (or will support in the near future) any of the following features:
> 

The 0.9 version is around the corner, so my comments are based on this
version.

> - references

Not implement yet. Perhaps for 1.0?

> - combining two or more HDF files (i.e mounting one in the
> other)

You can open two or more files in the same python program and copy or move
datasets or complete hierachies between them. Is that enough for you?
Anyway, mounting files in other existing hierarchy is a nice idea, so I'll
ponder implementing that (provided it won't be very difficult, and I don't
think so).

> - the use of different file drivers (for example to separate metadata and
> data)

Sorry, I don't know exactly what do you mean by that. Can you be more
specific about your needs?

Cheers,

-- 
Francesc Alted

[Pytables-users] support for references, mounting, file drivers?

[Fabian K. <fk...@MI...>]

Hi,

I just started to learn python and I would like to use pyTables for my
future projects to add HDF5 support. I was wondering if pyTables supports
(or will support in the near future) any of the following features:

- references
- combining two or more HDF files (i.e mounting one in the
other)
- the use of different file drivers (for example to separate metadata and
data)

Best,

- Fabian

[Pytables-users] How to store complex metadata?

[Eric J. <jo...@co...>]

Hello! I'm new to pytables and HDF5, and I'm trying to figure out how to
store my data. The issue of nested / complex attributes has come up:
ideally I'd like to store several large arrays in my file (with sizes
between 256 kB and 1 GB) and attach complex metadata to them, consisting
of tuples, nested objects, etc. But I want the metadata to be readable
from non-python tools, and evidently the attributes interface doesn't
currently support the non-scalar primitives. 

I'm curious what sorts of organizational schemes people have adopted for
their files to get around this limitation. Single-row tables? Just
saving data as strings? Or do people not run into problems with the
pickle'd metadata? 

Thanks for the help,
			...Eric 

[Pytables-users] PyTables 0.8.1 released

[Francesc A. <fa...@py...>]

The primary purpose of this release is to incorporate updates to
related to the newly released numarray 1.0. I've taken the opportunity
to backport some improvements added in PyTables 0.9 (in alpha stage)
as well as to fix the known problems

Improvements:

- The logic for computing the buffer sizes has been revamped. As a
  consequence, the performance of writing/reading tables with large
  row sizes has improved by a factor of ten or more, now exceeding 70
  MB/s for writing and 130 MB/s for reading (using compression). See
  http://sf.net/mailarchive/forum.php?thread_id=4963045&forum_id=13760
  for more info.

- The maximum row size for tables has been raised to 512 KB
  (before it was 8 KB, due to some internal limitations)

- Documentation has been improved in minor details. As a result of a
  fix in the underlying documentation system (tbook), chapters start
  now at odd pages, instead of even. So those of you who want to print
  to double side probably will have better luck now when aligning
  pages ;).  Another one is that HTML documentation has improved its
  look as well.

Bug Fixes:

- Indexing of Arrays with list or tuple flavors (#968131)
  When retrieving single elements from an array with 'List' or
  'Tuple' flavors, an error occurred. This has been
  corrected and now you can retrieve fileh.root.array[2] without
  problems for 'List' or 'Tuple' flavored (E, VL)Arrays.
  
- Iterators on Arrays with list or tuple flavors fail (#968132)
  When using iterators with Array objects with 'List' or
  'Tuple' flavors, an error occurred. This has been
  corrected.

- Last Index (-1) of Arrays doesn't work (#968149)
  When accessing to the last element in an Array using the notation
  -1, an empty list (or tuple or array) is returned instead of the
  proper value. This happened in general with all negative
  indices. Fixed.

- Table.read(flavor="List") should return pure lists (#972534)
  However, it used to return a pointer to numarray.records.Record
  instances, as in:

   >>> fileh.root.table.read(1,2,flavor="List") 
    [<numarray.records.Record instance at 0x4128352c>] 
   >>> fileh.root.table.read(1,3,flavor="List") 
    [<numarray.records.Record instance at 0x4128396c>, 
     <numarray.records.Record instance at 0x41283a8c>] 
 
  Now the next records are returned:

   >>> fileh.root.table.read(1,2, flavor=List) 
    [(' ', 1, 1.0)] 
   >>> fileh.root.table.read(1,3, flavor=List) 
    [(' ', 1, 1.0), 
     (' ', 2, 2.0)] 
 
  In addition, when reading a single row of a table, a
  numarray.records.Record pointer was returned:
 
  >>> fileh.root.table[1] 
   <numarray.records.Record instance at 0x4128398c> 
 
  Now, it returns a tuple:

  >>> fileh.root.table[1] 
   (' ', 1, 1.0) 
 
  Which I think is more consistent, and more Pythonic.

- Copy of leaves fails... (#973370)
  Attempting to copy leaves (Table or Array with different flavors) on
  top of themselves caused an internal error in PyTables. This has
  been corrected by silently avoiding the copy and returning the
  original Leaf as a result.

Minor changes:

- When assigning a value to a non-existing field in a table row, now a
  KeyError is raised, instead of the AttributeError that was issued
  before. I think this is more consistent with the type of error.

- Tests have been improved so as to pass the whole suite when compiled
  in 64 bit mode on a Linux/PowerPC machine (namely a dual-G5 Powermac
  running a 64-bit, 2.6.4 Linux kernel and the preview YDL
  distribution for G5, with 64-bit GCC toolchain). Thanks to Ciro
  Cattuto for testing and reporting the modifications that were
  needed.


Let me know of any bugs, suggestions, gripes, kudos, etc. you may
have.

Have fun!

-- 
Francesc Alted

Re: [Pytables-users] Change data already in PyTables array

[Francesc A. <fa...@py...>]

Hi Sigve,

A Dijous 08 Juliol 2004 15:37, Sigve Tjora va escriure:
> Is there a method to change the data in a PyTables array / earray? I want 
> to save a large array of data to disk, and then change only parts of it 
> after some processing. Is this possible with PyTables or is arrays 
> essential write once, read many? If there were a memory mapped array slice 
> similar to the one provided by NumArray, it would do the trick, but I have 
> not located such a feature.
> 

As of pytables 0.8 there is not a way to modify any data in existing
Table/Array/EArray/VLArray objects. I plan to add such a feature for Table
objects in pytables 0.9 (due to September/October). If I have time I'll
consider to add support for more objects as well.

Regards,

-- 
Francesc Alted

[Pytables-users] Change data already in PyTables array

[Sigve T. <sig...@3d...>]

Is there a method to change the data in a PyTables array / earray? I want 
to save a large array of data to disk, and then change only parts of it 
after some processing. Is this possible with PyTables or is arrays 
essential write once, read many? If there were a memory mapped array slice 
similar to the one provided by NumArray, it would do the trick, but I have 
not located such a feature.

Best regards, Sigve Tjora

Re: [Pytables-users] using pytables for saving realtime video?

[Francesc A. <fa...@py...>]

A Dimarts 22 Juny 2004 13:27, Francesc Alted va escriure:
> An small update: I re-made this C benchmark, but using only the zlib
> compressor (i.e. whitout shuffling) and setting all data to zeros, and I've
> obtained 33 MB/s. Without compression, that figure may perfectly grow up to
> 40 MB/s (whenever the hard disk would support this throughput, of course).

More updates ;). This morning I remembered that Table objects has a much
more efficient interface for writing than EArrays (just because I've spend
more time on optimizing Tables than anything else), and besides, I've
recently reworked the algorithm to compute buffer sizes for Tables, in order
to make them still faster. And the good news is that all of this largely
address to this problem :)

So, if you use the lastest CVS and use a Table instead of an EArray, this
small script would be far more efficient than the equivalent using EArrays:

*****************************************************
import tables
import numarray as na

class Test(tables.IsDescription):
    var1 = tables.UInt8Col(shape=(640,480))

nframes = 200
filename = "data.nobackup/test2.h5"
fileh = tables.openFile(filename, mode="w", title="Raw camera stream")
root = fileh.root
filter_args = dict( complevel = 1, complib = 'lzo', shuffle=0)
hdftable = fileh.createTable(root,'images', Test, "Unsigned byte table",
                             tables.Filters(**filter_args),
                             expectedrows=nframes)

frame = na.zeros(type="UInt8", shape=(640,480))
for i in range(nframes):
    hdftable.row["var1"] = frame
    hdftable.row.append()
fileh.close()
*****************************************************

With it, I was able to save frames at a speed of 46.2 MB/s without
compression (this script generates a 60 MB file, so that it fits well on my
laptop cache). Using ZLIB, I've got 36.4 MB/s, with LZO compression 54.5
MB/s and with UCL it drops down to 8.0 MB/s.

I was curious about how much memory would take a long run, and I made a test
with 20000 frames for a total dataset size of 6 GB. I've used LZO compressor
in order to keep the file size small. With that, the run took 1m23s, for a
total throughput of more than 70 MB/s. And the process took 16 MB of memory
during the run, which is quite reasonable. However, there seems to be a
"small" memory leak that develops at a rate of 3 KB/frame. Whether this is
acceptable or not is up to you.

Cheers,

-- 
Francesc Alted

Re: [Pytables-users] using pytables for saving realtime video?

[Francesc A. <fa...@py...>]

A Dimarts 22 Juny 2004 12:47, Francesc Alted va escriure:
> As I'm doing tests with a very slow hard disk, I used very repetitive data
> (all zeros) to bypass the bottleneck, but the results are barely the same.
> So, the bottleneck seems to be in the I/O calls, indeed.

Ops, I forgot to say that this is using compression.

> In order to determine if the problem was PyTables or the HDF5 layer, I used
> a small C program that opens the EArray only once, write all the data, and
> then close the array (PyTables, on its hand, always open and close on every
> append() operation). With that, I was able to achieve 7.7 MB/s, so very
> close of the writing limits of my disk. When using compression (zlib,
> complevel=1) and shuffling, however, I was able to achieve 22 MB/s. So,
> perhaps it would be feasible to reach 30 MB/s or more without using
> compression by using this kind of optimized writing on a system that
> supports faster writing speeds, like yours.

An small update: I re-made this C benchmark, but using only the zlib
compressor (i.e. whitout shuffling) and setting all data to zeros, and I've
obtained 33 MB/s. Without compression, that figure may perfectly grow up to
40 MB/s (whenever the hard disk would support this throughput, of course).

-- 
Francesc Alted

Re: [Pytables-users] using pytables for saving realtime video?

[Francesc A. <fa...@py...>]

A Dimarts 22 Juny 2004 02:04, Andrew Straw va escriure:
> Sorry, the process never completed while writing the email.   Playing 
> around with hdparm, I can now get ~6.5 MB/sec. With no compression, and 
> UCL, LZO, and zlib all reduce that rate.

Mmm, just out of curiosity, I run some benchmarks simulating your scenario
(see attachment). After some runs I can reproduce your numbers (I can get up
to 5.5 MB/s on my laptop, a P4 Mobile @2 GHz, with a hard disk spinning at
4200 RPM, with a maximum throughput of 8 MB/s during writings). However,
using ._append() instead of .append() *do* helped a lot in my case,
improving the output from 2.8 MB/s to 5.5 MB/s (maybe this is because you
have a faster CPU). These figures has been collected without compression.

As I'm doing tests with a very slow hard disk, I used very repetitive data
(all zeros) to bypass the bottleneck, but the results are barely the same.
So, the bottleneck seems to be in the I/O calls, indeed.

In order to determine if the problem was PyTables or the HDF5 layer, I used
a small C program that opens the EArray only once, write all the data, and
then close the array (PyTables, on its hand, always open and close on every
append() operation). With that, I was able to achieve 7.7 MB/s, so very
close of the writing limits of my disk. When using compression (zlib,
complevel=1) and shuffling, however, I was able to achieve 22 MB/s. So,
perhaps it would be feasible to reach 30 MB/s or more without using
compression by using this kind of optimized writing on a system that
supports faster writing speeds, like yours.

So, most probably HDF5 would be able to achieve the speed that you need.
PyTables is quite slower because of the way that it does I/O (i.e. opening
and closing the EArray object on every append). Of course, as I said you in
my firts message, that would be speeded-up by writing a specialized method
that would open first the object, write frame objects and close at the end.

> >If suggestion 2 is not enough (although I'd doubt it), things can be further
                                  ^^^^^^^^^^^^^^^^^^^^^
Ooops, I have a mouth too big ;)

> Finally, as a suggestion, you may want to incorporate the following code 
> into the C source for PyTables, which will allow other Python threads to 
> continue running when performing long-running HDF5 tasks.  See 
> http://docs.python.org/api/threads.html for more information.
> 
>  PyThreadState *_save;
>  _save = PyEval_SaveThread();
> /* Do work accessing HDF5 library which does not touch the Python API */
> PyEval_RestoreThread(_save);
> 
> (The file_write function in Objects/fileobject.c in the Python 
> sourcecode uses the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS 
> macros to acheive to do the same thing, but do to the error handling of 
> PyTables, you'll probably need 2 copies of 
> "PyEval_RestoreThread(_save)": one in the normal return path and one in 
> the error handling path.)

Ok. Thanks for the suggestion. This is very interesting indeed :)

Cheers,

-- 
Francesc Alted

Re: [Pytables-users] using pytables for saving realtime video?

[Andrew S. <str...@as...>]

Francesc Alted wrote:

>A Dissabte 19 Juny 2004 02:49, Andrew Straw va escriure:
>  
>
>>I am trying to save realtime video (640x480x100 fps uint8 grayscale 
>>data) using PyTables for scientific purposes (lossy compression is 
>>bad).  So, first off, is using PyTables for this task a reasonable 
>>idea?  Although I'm no expert, it seems the compression algorithms that 
>>PyTables offers may be ideal.  It also may be nice to use HDF5 to 
>>incorporate some data.
>>    
>>
>
>I've never thought in such an application for PyTables, but I think that for
>your case (provided that you can't afford lossing information) maybe just
>fine.
>
>  
>
>>Using this code, I get approximately 4 MB/sec with no compression, and 
>>MB/sec with complevel=1 UCL.  This is with an XFS filesystem on linux 
>>    
>>
>
>Mmm... How much using UCL?. Anyway, you may want to try LZO and ZLIB (with
>different compression levels) as well in order to see if this improve the
>speed.
>
>  
>
Sorry, the process never completed while writing the email.   Playing 
around with hdparm, I can now get ~6.5 MB/sec. With no compression, and 
UCL, LZO, and zlib all reduce that rate.

>>So, are there any suggestions for getting this to run faster?
>>    
>>
>
>A couple:
>
>1.- Ensure that your bottleneck is really the call to .append() method by
>commenting it out and doing timings again.
>  
>
Actually, I'm timing purely the call to .append(), which often takes 
seconds.

>2.- EArray.append() method do many checks so as to ensure that you pass an
>object compatible with the EArray being saved. If you are going to pass a
>*NumArray* object that you are sure it's compliant with the underlying
>EArray shape, you can save quite time by calling to the
>._append(numarrayObject) instead of .append(numarrayObject).
>
>If suggestion 2 is not enough (although I'd doubt it), things can be further
>speeded-up by optimizing the number of calls to the underlying HDF5 library.
>However, this must be regarded as a commercial service only (but you can
>always do it by yourself, of course!).
>  
>
That does help a little...

Anyhow, I think using PyTables/HDF5 is too slow for this task -- I can 
easily save at ~50 MB/sec using .write on simple File objects.  So I'll 
use that for now.

Finally, as a suggestion, you may want to incorporate the following code 
into the C source for PyTables, which will allow other Python threads to 
continue running when performing long-running HDF5 tasks.  See 
http://docs.python.org/api/threads.html for more information.

 PyThreadState *_save;
 _save = PyEval_SaveThread();
/* Do work accessing HDF5 library which does not touch the Python API */
PyEval_RestoreThread(_save);

(The file_write function in Objects/fileobject.c in the Python 
sourcecode uses the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS 
macros to acheive to do the same thing, but do to the error handling of 
PyTables, you'll probably need 2 copies of 
"PyEval_RestoreThread(_save)": one in the normal return path and one in 
the error handling path.)

Re: [Pytables-users] using pytables for saving realtime video?

[Francesc A. <fa...@py...>]

A Dissabte 19 Juny 2004 02:49, Andrew Straw va escriure:
> I am trying to save realtime video (640x480x100 fps uint8 grayscale 
> data) using PyTables for scientific purposes (lossy compression is 
> bad).  So, first off, is using PyTables for this task a reasonable 
> idea?  Although I'm no expert, it seems the compression algorithms that 
> PyTables offers may be ideal.  It also may be nice to use HDF5 to 
> incorporate some data.

I've never thought in such an application for PyTables, but I think that for
your case (provided that you can't afford lossing information) maybe just
fine.

> Using this code, I get approximately 4 MB/sec with no compression, and 
> MB/sec with complevel=1 UCL.  This is with an XFS filesystem on linux 

Mmm... How much using UCL?. Anyway, you may want to try LZO and ZLIB (with
different compression levels) as well in order to see if this improve the
speed.

> So, are there any suggestions for getting this to run faster?

A couple:

1.- Ensure that your bottleneck is really the call to .append() method by
commenting it out and doing timings again.

2.- EArray.append() method do many checks so as to ensure that you pass an
object compatible with the EArray being saved. If you are going to pass a
*NumArray* object that you are sure it's compliant with the underlying
EArray shape, you can save quite time by calling to the
._append(numarrayObject) instead of .append(numarrayObject).

If suggestion 2 is not enough (although I'd doubt it), things can be further
speeded-up by optimizing the number of calls to the underlying HDF5 library.
However, this must be regarded as a commercial service only (but you can
always do it by yourself, of course!).

Cheers,

-- 
Francesc Alted

[Pytables-users] using pytables for saving realtime video?

[Andrew S. <str...@as...>]

I am trying to save realtime video (640x480x100 fps uint8 grayscale 
data) using PyTables for scientific purposes (lossy compression is 
bad).  So, first off, is using PyTables for this task a reasonable 
idea?  Although I'm no expert, it seems the compression algorithms that 
PyTables offers may be ideal.  It also may be nice to use HDF5 to 
incorporate some data.

At the moment, however, I'm stymied by slow write speeds, and I seek 
suggestions on how to speed this up.  I need to get this working 
approximately 10x faster to be viable.

At first pass, I've started with code like this:

        self.fileh = tables.openFile(filename,
                                     mode="w",
                                     title="Raw camera stream")
        root = self.fileh.root
        a = tables.UInt8Atom((self.cam_height,self.cam_width,0))
        filter_args = dict( complevel = 0,
                            complib = 'ucl',
                            )
        self.hdfarray = self.fileh.createEArray(root,
                                                'images', a,
                                                "Unsigned byte array",
                                                
tables.Filters(**filter_args))

while 1:
                    # other stuff that fills self.grabbed_frames

                    n_frames = len(self.grabbed_frames)
                    # set to rank 3
                    def add_dim(f):
                        f.shape = (self.cam_height,self.cam_width,1)
                    map( add_dim, self.grabbed_frames)
                    frames = na.concatenate( self.grabbed_frames, axis=2)
                    print frames.shape
                    self.hdfarray.append(frames)

Using this code, I get approximately 4 MB/sec with no compression, and 
MB/sec with complevel=1 UCL.  This is with an XFS filesystem on linux 
kernel 2.6.6 using a SerialATA drive which benchmarks writing at 50 
MB/sec using iozone.

So, are there any suggestions for getting this to run faster?

Cheers!
Andrew

Re: [Pytables-users] Update a table.

[Francesc A. <fa...@py...>]

Hi Ahmad,

In addition to what Vicent has (correctly enough) told you, a couple of
suggestions:

- If you want to mix very large strings with small ones, and want to reduce
the amount of space used on disk, try to use compression (see chapter 5 of
User's Manual). In that way, you will be using only the necessary disk space
(or even less, depending on your data compressibility). It's a simple and
efficient way to simulate variable length records.

- If you insist in using true variable length strings, you may use a VLArray
combined with a VLString atom. That will provide you true (and unicode)
variable length strings. You can link the entries of the VLArray to your
tables by adding an integer field in the later where you can put the
reference to the entry in the VLArray.

This will hopefully allow you to find a workaround until I find time to
implement the VLTables, where fully variable length fields will be allowed.

Regards,

Francesc

A Dimecres 16 Juny 2004 17:18, Ahmad Hosseinzadeh va escriure:
> Hi,
> 
> I'm new to PyTables.
> I don't know how to declare a string field with
> variable length in my tables.
> Also I have problem with updating a table or array.
> I want to update a record of my table (PyTables) but I
> don't know how.
> I have the same problem with arrays in PyTables.
> My solution is to remove the previous one and create a
> new record or array but it is not efficient.
> Can anyone help me?
> 
> Thanks.
> Ahmad.
> 
> 
> ______________________________________________________________________ 
> Post your free ad now! http://personals.yahoo.ca
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference
> Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer
> Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA
> REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> 
> 

-- 
Francesc Alted

Re: [Pytables-users] Update a table.

[Vicent M. <uv...@on...>]

Hi again Ahmad,

El Wednesday 16 June 2004 17:18, Ahmad Hosseinzadeh escribio:
> Hi,
>
> I'm new to PyTables.
> I don't know how to declare a string field with
> variable length in my tables.

You cannot. In a table object rows are defined inheriting the IsDescription 
class. This class, in turn, uses Col classes to declare the fields of the 
row. In particular fields of type string are declared using the StringCol 
class. As you can see in the manual (section 4.11.2) the very first parameter 
of the class constructor fixes the length of the string, so no variable 
length strings are available. However they can be simulated setting the 
length parameter to the maximum length you need. The reason is that when 
strings are read they just use its original size, not the maximum size given 
to the constructor.
Hopefully support for tables with variable length of fields (VLTable) will be 
included in the release 1.0 of pytables.

> Also I have problem with updating a table or array.
> I want to update a record of my table (PyTables) but I
> don't know how.
> I have the same problem with arrays in PyTables.

Updates are not yet supported. You will have to wait to release 0.9

> My solution is to remove the previous one and create a
> new record or array but it is not efficient.
> Can anyone help me?
>
> Thanks.
> Ahmad.
>

-- 
Share what you know, learn what you don't

[Pytables-users] Update a table.

[Ahmad H. <ahm...@ya...>]

Hi,

I'm new to PyTables.
I don't know how to declare a string field with
variable length in my tables.
Also I have problem with updating a table or array.
I want to update a record of my table (PyTables) but I
don't know how.
I have the same problem with arrays in PyTables.
My solution is to remove the previous one and create a
new record or array but it is not efficient.
Can anyone help me?

Thanks.
Ahmad.


______________________________________________________________________ 
Post your free ad now! http://personals.yahoo.ca

[Pytables-users] Pytables included in Quantian

[Francesc A. <fa...@py...>]

Maybe this info would be useful to somebody. As you can see, PyTables has
been included in the latest release of Quantian.

Regards,

=46rancesc

=2D---------  Missatge transmes  ----------

Subject: New Quantian release 0.5.9.1 available
Date: Dimarts 15 Juny 2004 04:45
=46rom: Dirk Eddelbuettel <ed...@de...>
To: qua...@li...
Cc: qua...@ed...


[ This email is sent to those whose email addresses are in my quantian
   mail folder due to prior emails, plus LWN and DWN who had run previous
   announcements, and as suggested, the openmosix-general, clusterknoppix
   and debian-knoppix lists. Anybody who considers this unwanted is kindly
   asked to send me a private mail and I will immediately remove the=20
   corresponding alias entry. --edd ]


Announcing Quantian release 0.5.9.1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D


I   What is it?

    Quantian is a remastering of Knoppix, the self-configuring and directly
    bootable cdrom that turns any pc or laptop into a full-featured Linux
    workstation, and clusterKnoppix, which adds support for openMosix.=20
    However, Quantian differs from (cluster)Knoppix by adding a large set=20
    of programs of interest to applied or theoretical workers in=20
    quantitative or data-driven fields.=20

    See http://dirk.eddelbuettel.com/quantian.html for more details.

   =20
II  What is new?

    o First release based on Knoppix 3.4 with many changes such as improved
      hardware detection and support for new hardware, the addition of
      captive-ntfs which may allow write support for ntfs partitions, KDE
      3.2.2 with kdevelop and more;

    o Based on Knoppix 3.4, the clusterKnoppix release from May 10 adds
      kernel 2.4.26 with the 'testing status' openMosix patch as well as
      a non-openMosix kernel 2.6.6; updated openMosix goodies gomd, chpox,
      tyd; support for atheros wireless, cisco mpi 350 wireless, prism54,=20
      host-ap

    o New Quantian software includes many new R / CRAN packages such as
      gregmisc, sm, quadprog and the entire snow suite: snow, rmpi, rpvm,
      rsprng for 'Simple Network of Workstations' distributed computing=20
      using either one of pvm or mpi; the Axiom computer-algebra system;
      python-tables and cernlib which pulls in a large number of packages
      from the CERN particle physics lab

    o 'kitchen sink' size -- this release comes in at 1.1gb and will /not/
      fit onto a cdrom. It runs really well from disk, see the 'lilo howto'
      http://dirk.eddelbuettel.com/quantian/howto_liloboot.html for details
      on how to boot the iso from hard disks.  Burning a dvd with the larger
      iso image should also work.

    o Last but not least, custom artwork thanks to a contributed background
      image provided by Ed Pegg Jr.

    o Mailing lists for Quantian up and running

      Following the 0.4.9.3 release, a Quantian project was opened on
      alioth.debian.org.  So far the only use of the alioth infrastructure
      has been the creation of two mailing lists

        quantian-announce	  for announcements, intended to be low volume
        quantian-general	  for general discussions about Quantian

      Please go to    =20

        http://alioth.debian.org/mail/?group_id=3D1425

      for subscription info etc., and start using the quantian-general lists
      for general questions, comments, suggestions or discussions about=20
      Quantian.

      Quantian-general is subscribed to quantian-announce, so you only need
      to subscribe to one (but can of course subscribe to both).

    o See http://dirk.eddelbuettel.com/quantian/changelog.html for details.


III Where do I get it?

    Downloads are available from the two main hosts both of which also
    provide rsync:

    o U of Washington: =20
      - http://www.analytics.washington.edu/downloads/quantian
      - rsync://www.analytics.washington.edu::quantian

    o European mirrors, bittorrent site and cdrom vendors will hopefully
      catch up over the next few days. See=20

	    http://dirk.eddelbuettel.com/quantian.html
=20
      for download info.


IV  Known Bugs

    o None right now -- so please test away!=20


V   Other items

    o Mailing lists have been created, see above.

    o Feedback / poll on package additions or removal

      As always, I welcome comments and suggestions about programs to be
      added or removed. Existing Debian packages get pushed to the front of
      the line.

      Please send feedback, questions, comments, ... to the=20
     =20
	qua...@li...

      list to maximise the number of eyes glancing at any one question.

Best regards,  Dirk

=2D-=20
=46EATURE:  VW Beetle license plate in California



=2D------------------------------------------------------


=2D---------  Missatge transmes  ----------

Subject: Re: New Quantian release 0.5.9.1 available
Date: Dimarts 15 Juny 2004 13:12
=46rom: Dirk Eddelbuettel <ed...@de...>
To: Francesc Alted <fa...@py...>


=46rancesc,

On Tue, Jun 15, 2004 at 12:29:46PM +0200, Francesc Alted wrote:
> Hi Dirk,
>=20
> Thanks for including PyTables in Quantian. I hope such inclusion would be
> useful to people with a need to deal with large amounts of scientific dat=
a.
>=20
> By the way, for future references, the name of the package should be stat=
ed
> as PyTables (lower case writing should be fine as well), and not
> python-tables, which follows the debian name guidelines, but it is not the
> package name.

My bad -- sorry. I just corrected it in the changelog.wml file from which
the .html is created. It was copy&paste from there to the announcement.

Regards, Dirk

>=20
> Cheers,
>=20
> --=20
> Francesc Alted
>=20
> A Dimarts 15 Juny 2004 04:45, vareu escriure:
> >=20
> > [ This email is sent to those whose email addresses are in my quantian
> >    mail folder due to prior emails, plus LWN and DWN who had run previo=
us
> >    announcements, and as suggested, the openmosix-general, clusterknopp=
ix
> >    and debian-knoppix lists. Anybody who considers this unwanted is kin=
dly
> >    asked to send me a private mail and I will immediately remove the=20
> >    corresponding alias entry. --edd ]
> >=20
> >=20
> > Announcing Quantian release 0.5.9.1
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >=20
> >=20
> > I   What is it?
> >=20
> >     Quantian is a remastering of Knoppix, the self-configuring and dire=
ctly
> >     bootable cdrom that turns any pc or laptop into a full-featured Lin=
ux
> >     workstation, and clusterKnoppix, which adds support for openMosix.=
=20
> >     However, Quantian differs from (cluster)Knoppix by adding a large s=
et=20
> >     of programs of interest to applied or theoretical workers in=20
> >     quantitative or data-driven fields.=20
> >=20
> >     See http://dirk.eddelbuettel.com/quantian.html for more details.
> >=20
> >    =20
> > II  What is new?
> >=20
> >     o First release based on Knoppix 3.4 with many changes such as impr=
oved
> >       hardware detection and support for new hardware, the addition of
> >       captive-ntfs which may allow write support for ntfs partitions, K=
DE
> >       3.2.2 with kdevelop and more;
> >=20
> >     o Based on Knoppix 3.4, the clusterKnoppix release from May 10 adds
> >       kernel 2.4.26 with the 'testing status' openMosix patch as well as
> >       a non-openMosix kernel 2.6.6; updated openMosix goodies gomd, chp=
ox,
> >       tyd; support for atheros wireless, cisco mpi 350 wireless, prism5=
4,=20
> >       host-ap
> >=20
> >     o New Quantian software includes many new R / CRAN packages such as
> >       gregmisc, sm, quadprog and the entire snow suite: snow, rmpi, rpv=
m,
> >       rsprng for 'Simple Network of Workstations' distributed computing=
=20
> >       using either one of pvm or mpi; the Axiom computer-algebra system;
> >       python-tables and cernlib which pulls in a large number of packag=
es
> >       from the CERN particle physics lab
> >=20
> >     o 'kitchen sink' size -- this release comes in at 1.1gb and will /n=
ot/
> >       fit onto a cdrom. It runs really well from disk, see the 'lilo ho=
wto'
> >       http://dirk.eddelbuettel.com/quantian/howto_liloboot.html for det=
ails
> >       on how to boot the iso from hard disks.  Burning a dvd with the l=
arger
> >       iso image should also work.
> >=20
> >     o Last but not least, custom artwork thanks to a contributed backgr=
ound
> >       image provided by Ed Pegg Jr.
> >=20
> >     o Mailing lists for Quantian up and running
> >=20
> >       Following the 0.4.9.3 release, a Quantian project was opened on
> >       alioth.debian.org.  So far the only use of the alioth infrastruct=
ure
> >       has been the creation of two mailing lists
> >=20
> >         quantian-announce	  for announcements, intended to be low volume
> >         quantian-general	  for general discussions about Quantian
> >=20
> >       Please go to    =20
> >=20
> >         http://alioth.debian.org/mail/?group_id=3D1425
> >=20
> >       for subscription info etc., and start using the quantian-general =
lists
> >       for general questions, comments, suggestions or discussions about=
=20
> >       Quantian.
> >=20
> >       Quantian-general is subscribed to quantian-announce, so you only =
need
> >       to subscribe to one (but can of course subscribe to both).
> >=20
> >     o See http://dirk.eddelbuettel.com/quantian/changelog.html for deta=
ils.
> >=20
> >=20
> > III Where do I get it?
> >=20
> >     Downloads are available from the two main hosts both of which also
> >     provide rsync:
> >=20
> >     o U of Washington: =20
> >       - http://www.analytics.washington.edu/downloads/quantian
> >       - rsync://www.analytics.washington.edu::quantian
> >=20
> >     o European mirrors, bittorrent site and cdrom vendors will hopefully
> >       catch up over the next few days. See=20
> >=20
> > 	    http://dirk.eddelbuettel.com/quantian.html
> > =20
> >       for download info.
> >=20
> >=20
> > IV  Known Bugs
> >=20
> >     o None right now -- so please test away!=20
> >=20
> >=20
> > V   Other items
> >=20
> >     o Mailing lists have been created, see above.
> >=20
> >     o Feedback / poll on package additions or removal
> >=20
> >       As always, I welcome comments and suggestions about programs to be
> >       added or removed. Existing Debian packages get pushed to the fron=
t of
> >       the line.
> >=20
> >       Please send feedback, questions, comments, ... to the=20
> >      =20
> > 	qua...@li...
> >=20
> >       list to maximise the number of eyes glancing at any one question.
> >=20
> > Best regards,  Dirk
> >=20
>=20
>=20
>=20

=2D-=20
=46EATURE:  VW Beetle license plate in California



=2D------------------------------------------------------


=2D---------  Missatge transmes  ----------

Subject: Fwd: New Quantian release 0.5.9.1 available
Date: Dimarts 15 Juny 2004 18:07
=46rom: Francesc Alted <fa...@py...>
To: ge...@ca...

Acaben d'incloure pytables en una distribuci=F3 de caire cient=EDfic basada=
 en
knoppix. Estar=E0 be de cara a la galeria :)

=46ins ara,

=2D---------  Missatge transmes  ----------

Subject: New Quantian release 0.5.9.1 available
Date: Dimarts 15 Juny 2004 04:45
=46rom: Dirk Eddelbuettel <ed...@de...>
To: qua...@li...
Cc: qua...@ed...


[ This email is sent to those whose email addresses are in my quantian
   mail folder due to prior emails, plus LWN and DWN who had run previous
   announcements, and as suggested, the openmosix-general, clusterknoppix
   and debian-knoppix lists. Anybody who considers this unwanted is kindly
   asked to send me a private mail and I will immediately remove the=20
   corresponding alias entry. --edd ]


Announcing Quantian release 0.5.9.1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D


I   What is it?

    Quantian is a remastering of Knoppix, the self-configuring and directly
    bootable cdrom that turns any pc or laptop into a full-featured Linux
    workstation, and clusterKnoppix, which adds support for openMosix.=20
    However, Quantian differs from (cluster)Knoppix by adding a large set=20
    of programs of interest to applied or theoretical workers in=20
    quantitative or data-driven fields.=20

    See http://dirk.eddelbuettel.com/quantian.html for more details.

   =20
II  What is new?

    o First release based on Knoppix 3.4 with many changes such as improved
      hardware detection and support for new hardware, the addition of
      captive-ntfs which may allow write support for ntfs partitions, KDE
      3.2.2 with kdevelop and more;

    o Based on Knoppix 3.4, the clusterKnoppix release from May 10 adds
      kernel 2.4.26 with the 'testing status' openMosix patch as well as
      a non-openMosix kernel 2.6.6; updated openMosix goodies gomd, chpox,
      tyd; support for atheros wireless, cisco mpi 350 wireless, prism54,=20
      host-ap

    o New Quantian software includes many new R / CRAN packages such as
      gregmisc, sm, quadprog and the entire snow suite: snow, rmpi, rpvm,
      rsprng for 'Simple Network of Workstations' distributed computing=20
      using either one of pvm or mpi; the Axiom computer-algebra system;
      python-tables and cernlib which pulls in a large number of packages
      from the CERN particle physics lab

    o 'kitchen sink' size -- this release comes in at 1.1gb and will /not/
      fit onto a cdrom. It runs really well from disk, see the 'lilo howto'
      http://dirk.eddelbuettel.com/quantian/howto_liloboot.html for details
      on how to boot the iso from hard disks.  Burning a dvd with the larger
      iso image should also work.

    o Last but not least, custom artwork thanks to a contributed background
      image provided by Ed Pegg Jr.

    o Mailing lists for Quantian up and running

      Following the 0.4.9.3 release, a Quantian project was opened on
      alioth.debian.org.  So far the only use of the alioth infrastructure
      has been the creation of two mailing lists

        quantian-announce	  for announcements, intended to be low volume
        quantian-general	  for general discussions about Quantian

      Please go to    =20

        http://alioth.debian.org/mail/?group_id=3D1425

      for subscription info etc., and start using the quantian-general lists
      for general questions, comments, suggestions or discussions about=20
      Quantian.

      Quantian-general is subscribed to quantian-announce, so you only need
      to subscribe to one (but can of course subscribe to both).

    o See http://dirk.eddelbuettel.com/quantian/changelog.html for details.


III Where do I get it?

    Downloads are available from the two main hosts both of which also
    provide rsync:

    o U of Washington: =20
      - http://www.analytics.washington.edu/downloads/quantian
      - rsync://www.analytics.washington.edu::quantian

    o European mirrors, bittorrent site and cdrom vendors will hopefully
      catch up over the next few days. See=20

	    http://dirk.eddelbuettel.com/quantian.html
=20
      for download info.


IV  Known Bugs

    o None right now -- so please test away!=20


V   Other items

    o Mailing lists have been created, see above.

    o Feedback / poll on package additions or removal

      As always, I welcome comments and suggestions about programs to be
      added or removed. Existing Debian packages get pushed to the front of
      the line.

      Please send feedback, questions, comments, ... to the=20
     =20
	qua...@li...

      list to maximise the number of eyes glancing at any one question.

Best regards,  Dirk

=2D-=20
=46EATURE:  VW Beetle license plate in California



=2D------------------------------------------------------

=2D-=20
=46rancesc Alted




=2D------------------------------------------------------

=2D-=20
=46rancesc Alted

Re: [Pytables-users] Malloc failure

[Francesc A. <fa...@py...>]

Hi,

Um, that's weird, because I'm very used to create regularly tables in the
order of millions of rows so as to check different capabilities, and
normally that should take as much as 10~15 MB of RAM (depending on
platform). Even with more than 10 billion rows, pytables should not take
more than 300 MB (see
http://pytables.sourceforge.net/html/StressTests.html), which is less than
the 518 MB you are reporting.

Could you send me your script so that I can have a look at it?

Cheers,

A Dimecres 26 Maig 2004 20:15, David Sumner va escriure:
> Hello,
> I am receiving a core dump when using pytables. I have a simple script
> that is adding entries to a single table. I am currently attempting to
> add a million rows to a table. The row consists of a float32col and an
> int32col. I am doing this to get some general transaction rates and file
> size estimates. The python process runs along pretty well at about 30-50
> percent cpu utilization. The process grows to about 518MB in size. At
> that point I receive the following message:
>  
> Python in malloc(): error: allocation failed
>  
> The process continues to run for about 10-15 seconds, I then see the
> following message:
>  
> Abort (core dumped)
>  
> At that point the process terminates. I have run this script with
> pymalloc and without pymalloc and it fails the same way. If I run it
> again, I see the same results. Are there limitations inherent to python
> as to how large a python process can become, or is this indicative of a
> limitation in the pytables api? Are there settings I can change to
> prevent this or avoid this. I saw the expectedrows parameter when
> creating a new table. Is this an optimization, or does this also effect
> how many rows can exist reliably in a table. I know pytables is built to
> keep as much data in memory as available and then push data to the hard
> drive once memory becomes scarce. Can I control this aspect of pytables?
>  
> I am running this on a dual 2.4GHz Xeon server, with 1GB of RAM, raid 5
> with 4 36GB 160 SCSI drives. I am running FreeBSD 5.2.1 with
> hyperthreading enabled, so the system is acting as though it had 4 cpus.
>  
> Any help would be appreciated. I am attempting to use pytables to store
> log entries collected from multiple sources. This would be a long
> running process that interacted with pytables and would probably not be
> shut down during regular use. The application will trim the database to
> keep around 30 days of activity stored in the hdf5 file. Is pytables a
> good backend for this type of solution?
>  
> Thank You,
> David Sumner
> 

-- 
Francesc Alted

[Pytables-users] Malloc failure

[David S. <da...@dr...>]

Hello,
I am receiving a core dump when using pytables. I have a simple script
that is adding entries to a single table. I am currently attempting to
add a million rows to a table. The row consists of a float32col and an
int32col. I am doing this to get some general transaction rates and file
size estimates. The python process runs along pretty well at about 30-50
percent cpu utilization. The process grows to about 518MB in size. At
that point I receive the following message:
 
Python in malloc(): error: allocation failed
 
The process continues to run for about 10-15 seconds, I then see the
following message:
 
Abort (core dumped)
 
At that point the process terminates. I have run this script with
pymalloc and without pymalloc and it fails the same way. If I run it
again, I see the same results. Are there limitations inherent to python
as to how large a python process can become, or is this indicative of a
limitation in the pytables api? Are there settings I can change to
prevent this or avoid this. I saw the expectedrows parameter when
creating a new table. Is this an optimization, or does this also effect
how many rows can exist reliably in a table. I know pytables is built to
keep as much data in memory as available and then push data to the hard
drive once memory becomes scarce. Can I control this aspect of pytables?
 
I am running this on a dual 2.4GHz Xeon server, with 1GB of RAM, raid 5
with 4 36GB 160 SCSI drives. I am running FreeBSD 5.2.1 with
hyperthreading enabled, so the system is acting as though it had 4 cpus.
 
Any help would be appreciated. I am attempting to use pytables to store
log entries collected from multiple sources. This would be a long
running process that interacted with pytables and would probably not be
shut down during regular use. The application will trim the database to
keep around 30 days of activity stored in the hdf5 file. Is pytables a
good backend for this type of solution?
 
Thank You,
David Sumner