Re: [Pytables-users] multiprocessing and pytables

[Anthony S. <sc...@gm...>]

Hello Ernesto,

So you are actually asking two different questions, one on reading and the
other on writing.  In general reading, or querying,
with multiprocessing works very well.  Writing to a single file with
multiple processes is destined to failure though.  So the strategy that
many people have adopted is to have multiple processes create the data and
then have a master process which acts as a queue for writing out the data.
 Please see the example here for more inspiration [1].  Note that we have
been having problems recently with multiprocess writing out to multiple
files, but that is not what you want to do.

Be Well
Anthony

1.
https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py

On Mon, Oct 15, 2012 at 11:45 AM, Ernesto Picardi <e.p...@un...>wrote:

> Dear all,
>
> I have a hdf5 file including several tables. To speed up the creation of
> all tables, could I create each individual table by independent processes
> launched by multiprocessing module? Could I employ independent processes to
> query diverse tables of the same hdf5 file?
>
> Thank you very much in advance for whatever answer.
>
> Regards,
>
> Ernesto
>
>
>
>
>
>
>
> **** Riservatezza / Confidentiality ****
> In ottemperanza al D.Lgs. n. 196 del 30/6/2003 in materia di protezione
> dei dati personali, le informazioni contenute in questo messaggio sono
> strettamente riservate ed esclusivamente indirizzate al destinatario
> indicato (oppure alla persona responsabile di  rimetterlo al destinatario).
> Vogliate tener presente che qualsiasi uso, riproduzione o divulgazione di
> questo messaggio e' vietato. Nel caso in cui aveste ricevuto questo
> messaggio per errore, vogliate cortesemente avvertire il mittente e
> distruggere il presente  messaggio.
>
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>