Re: [Pytables-users] Malloc failure

[Francesc A. <fa...@py...>]

Hi,

Um, that's weird, because I'm very used to create regularly tables in the
order of millions of rows so as to check different capabilities, and
normally that should take as much as 10~15 MB of RAM (depending on
platform). Even with more than 10 billion rows, pytables should not take
more than 300 MB (see
http://pytables.sourceforge.net/html/StressTests.html), which is less than
the 518 MB you are reporting.

Could you send me your script so that I can have a look at it?

Cheers,

A Dimecres 26 Maig 2004 20:15, David Sumner va escriure:
> Hello,
> I am receiving a core dump when using pytables. I have a simple script
> that is adding entries to a single table. I am currently attempting to
> add a million rows to a table. The row consists of a float32col and an
> int32col. I am doing this to get some general transaction rates and file
> size estimates. The python process runs along pretty well at about 30-50
> percent cpu utilization. The process grows to about 518MB in size. At
> that point I receive the following message:
>  
> Python in malloc(): error: allocation failed
>  
> The process continues to run for about 10-15 seconds, I then see the
> following message:
>  
> Abort (core dumped)
>  
> At that point the process terminates. I have run this script with
> pymalloc and without pymalloc and it fails the same way. If I run it
> again, I see the same results. Are there limitations inherent to python
> as to how large a python process can become, or is this indicative of a
> limitation in the pytables api? Are there settings I can change to
> prevent this or avoid this. I saw the expectedrows parameter when
> creating a new table. Is this an optimization, or does this also effect
> how many rows can exist reliably in a table. I know pytables is built to
> keep as much data in memory as available and then push data to the hard
> drive once memory becomes scarce. Can I control this aspect of pytables?
>  
> I am running this on a dual 2.4GHz Xeon server, with 1GB of RAM, raid 5
> with 4 36GB 160 SCSI drives. I am running FreeBSD 5.2.1 with
> hyperthreading enabled, so the system is acting as though it had 4 cpus.
>  
> Any help would be appreciated. I am attempting to use pytables to store
> log entries collected from multiple sources. This would be a long
> running process that interacted with pytables and would probably not be
> shut down during regular use. The application will trim the database to
> keep around 30 days of activity stored in the hdf5 file. Is pytables a
> good backend for this type of solution?
>  
> Thank You,
> David Sumner
> 

-- 
Francesc Alted