Re: [Pytables-users] multiprocessing and pytables

[Anthony S. <sc...@gm...>]

On Wed, Oct 17, 2012 at 12:58 AM, Antonio Valentino <
ant...@ti...> wrote:

> Hi Anthony,
>
> Il giorno 16/ott/2012, alle ore 02:04, Anthony Scopatz <sc...@gm...>
> ha scritto:
>
> > Hello Ernesto,
> >
> > So you are actually asking two different questions, one on reading and
> the other on writing.  In general reading, or querying, with
> multiprocessing works very well.  Writing to a single file with multiple
> processes is destined to failure though.  So the strategy that many people
> have adopted is to have multiple processes create the data and then have a
> master process which acts as a queue for writing out the data.  Please see
> the example here for more inspiration [1].  Note that we have been having
> problems recently with multiprocess writing out to multiple files, but that
> is not what you want to do.
> >
> > Be Well
> > Anthony
> >
> > 1.
> https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py
>
> It seems that the topic PyTables + multiprocessing became very popular
> since some time.
>
> Probably we should add a FAQ entry and provide a more extended tutorial
> based on the example provided by Josh.
>

I agree.  This is something that we should advertise a little more.

Be Well
Anthony


>
>
> cheers
>
> --
> Antonio Valentino
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>