Re: [Pytables-users] Having Trouble With Threaded Writing to PyTables

[Anthony S. <sc...@gm...>]

Hello Jacob,

This is not surprising.  The HDF5 parallel library requires MPI and comes
with some special restrictions (no compression on write).  As such, the
pain of implementing a parallel write version of PyTables has not been
worth it.  We certainly welcome pull requests and further discussion on
this issue ;).    Often times it is easier (and faster...writing is
expensive) to do the computation in parallel followed by a single write.
 Or you could have a dedicated thread which queues and executes write
commands as they come in.    Just some thoughts on how to avoid this
problem.

Parallel reads are supported.

Let me know if you have further questions or really want to dive into this
issue.

Be Well
Anthony

On Mon, Jun 25, 2012 at 1:33 PM, Jacob Bennett <jac...@gm...>wrote:

> Hello PyTables Users,
>
> I am very new to pytables and if you all could help me out, that would be
> splendid.
>
> I'm currently having trouble with writing to two separate HDF5 files using
> pytables. Each file itself is only accessible by a single thread, so there
> really shouldn't be any threading issues. When I run my python script
> however, it just seems to crash at random time intervals without any error
> messages received or exceptions thrown.
>
> I write data to the HDF5 files as follows. I have two HDF5 files that
> represent book snapshots and trade snapshots. The data of these snapshots
> come in the form of python dictionaries whose values are the data itself in
> an array. Two threads run on each file. One thread controls when to create
> new files and close others based upon the time of day while the other
> thread iterates over each key value pair in the dictionary and loads data
> to the file. When a thread has access to the file, the file is locked.
>
> I have my two datawrappers attached to the email. Please take a look at
> them. One thread runs acceptDict in a loop while the other runs changeFile
> in a loop. This is really frustrating when I don't get any errors and
> python just crashes unexpectedly.
>
> Thanks,
> Jacob
>
> --
> Jacob Bennett
> Massachusetts Institute of Technology
> Department of Electrical Engineering and Computer Science
>  Class of 2014| ben...@mi...
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>