Re: [Pytables-users] Having Memory Allocation Failure for Raw Data Chunk

[Anthony S. <sc...@gm...>]

Hi Jacob,

Hmmmm, this shouldn't be happening.  The data isn't large enough.  While
there *may* be a memory leak (use valgrind to find it) there it is likely
that you are just failing to deference something(s).   Is there any place
in your calculation where you might accidentally keep data around?  Note
that PyTables/HDF5 cachesa bunch of stuff behind the scene.  What does `ps
ux` say while you are running the code or right before the fail?  Does the
code break with data that is 1/10th the size?

Be Well
Anthony

On Fri, Jun 29, 2012 at 1:48 PM, Jacob Bennett <jac...@gm...>wrote:

> Hello PyTables Users,
>
> My current implementation works pretty well now and has the write speeds
> that I am looking for; however, around 20 minutes of execution and of a
> file size of around 127MB with level 3 blosc compression I seem to get
> memory allocation errors. Here is my trace that I get, if anybody can shed
> light on this, that will be excellent. Does my implementation hog all of my
> memory? Is there a memory leak?
>
> HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0:
>   #000: ..\..\hdf5-1.8.8\src\H5Dio.c line 266 in H5Dwrite(): can't write
> data
>     major: Dataset
>     minor: Write failed
>   #001: ..\..\hdf5-1.8.8\src\H5Dio.c line 671 in H5D_write(): can't write
> data
>     major: Dataset
>     minor: Write failed
>   #002: ..\..\hdf5-1.8.8\src\H5Dchunk.c line 1861 in H5D_chunk_write():
> unable t
> o read raw data chunk
>     major: Low-level I/O
>     minor: Read failed
>   #003: ..\..\hdf5-1.8.8\src\H5Dchunk.c line 2776 in H5D_chunk_lock():
> memory al
> location failed for raw data chunk
>     major: Resource unavailable
>     minor: No space available for allocation
> Exception in thread bookthread:
> Traceback (most recent call last):
>   File "C:\Python27bit\lib\threading.py", line 551, in __bootstrap_inner
>     self.run()
>   File "../PyTablesInterface\Acceptor.py", line 21, in run
>     BookDataWrapper.acceptDict()
>   File "../PyTablesInterface\BookDataWrapper.py", line 50, in acceptDict
>     tableD.append(dataArray)
>   File "C:\Python27bit\lib\site-packages\tables\table.py", line 2081, in
> append
>     self._saveBufferedRows(wbufRA, lenrows)
>   File "C:\Python27bit\lib\site-packages\tables\table.py", line 2016, in
> _saveBu
> fferedRows
>     self._append_records(lenrows)
>   File "tableExtension.pyx", line 454, in
> tables.tableExtension.Table._append_re
> cords (tables\tableExtension.c:4623)
> HDF5ExtError: Problems appending the records.
> #####################################
> ######THIS IS A LATER ERROR#########
> #####################################
> Exception in thread CME_10_B:
> Traceback (most recent call last):
>   File "C:\Python27bit\lib\threading.py", line 551, in __bootstrap_inner
>     self.run()
>   File
> "C:\Users\jacob.bennett\development\MarketDataReader\IO\__init__.py", lin
> e 19, in run
>     self.socket.rec()
>   File
> "C:\Users\jacob.bennett\development\MarketDataReader\IO\MarketSocket.py",
>  line 33, in rec
>     Parser.parse(self.sock.recv(1024*16), self.exchange)
>   File "../Parser\Parser.py", line 39, in parse
>     SendInBatch.acceptBookData(instrumentId, timestamp, 0, i, bidPrice,
> bidQuant
> , bidOrders, exchange, source)
>   File "../PyTablesInterface\SendInBatch.py", line 28, in acceptBookData
>     maindict[(instrumentId, yearmonthday)] = [(timestamp1, timestamp2,
> side, lev
> el, price, quant, orders, source, 1)]
> MemoryError
>
> Thanks,
> Jacob Bennett
>
>
> --
> Jacob Bennett
> Massachusetts Institute of Technology
> Department of Electrical Engineering and Computer Science
> Class of 2014| ben...@mi...
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>