Re: [Pytables-users] multiprocessing and pytables

[Antonio V. <ant...@ti...>]

Hi Anthony,

Il giorno 16/ott/2012, alle ore 02:04, Anthony Scopatz <sc...@gm...> ha scritto:

> Hello Ernesto, 
> 
> So you are actually asking two different questions, one on reading and the other on writing.  In general reading, or querying, with multiprocessing works very well.  Writing to a single file with multiple processes is destined to failure though.  So the strategy that many people have adopted is to have multiple processes create the data and then have a master process which acts as a queue for writing out the data.  Please see the example here for more inspiration [1].  Note that we have been having problems recently with multiprocess writing out to multiple files, but that is not what you want to do. 
> 
> Be Well
> Anthony
> 
> 1. https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py

It seems that the topic PyTables + multiprocessing became very popular since some time.

Probably we should add a FAQ entry and provide a more extended tutorial based on the example provided by Josh.


cheers

--
Antonio Valentino