Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

[Jacob B. <jac...@gm...>]

Awesome, I think this sounds like a very workable solution and the idea is
very neat. I will try to implement this right away. I definitely agree to
putting a small example.

Let you know how this works, thanks guys!

Thanks,
Jacob

On Sat, Jul 14, 2012 at 2:36 AM, Antonio Valentino <
ant...@ti...> wrote:

> Hi all,
> Il 14/07/2012 00:44, Josh Ayers ha scritto:
> > My first instinct would be to handle all access (read and write) to
> > that file from a single process.  You could create two
> > multiprocessing.Queue objects, one for data to write and one for read
> > requests.  Then the process would check the queues in a loop and
> > handle each request serially.  The data read from the file could be
> > sent back to the originating process using another queue or pipe.  You
> > should be able to do the same thing with sockets if the other parts of
> > your application are in languages other than Python.
> >
> > I do something similar to handle writing to a log file from multiple
> > processes and it works well.  In that case the file is write-only -
> > and just a simple text file rather than HDF5 - but I don't see any
> > reason why it wouldn't work for read and write as well.
> >
> > Hope that helps,
> > Josh
> >
>
> I totally agree with Josh.
>
> I don't have a test code to demonstrate it but IMHO parallelizing I/O
> to/from a single file on a single disk do not makes too much sense
> unless you have special HW.  Is this your case Jacob?
>
> IMHO with standard SATA devices you could have a marginal speedup (in
> the best case), but if your bottleneck is the I/O this will not solve
> your problem.
>
> If someone finds the time to implement a toy example of what Josh
> suggested we could put it on the cookbook :)
>
>
> regards
>
> > On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz <sc...@gm...>
> wrote:
> >> On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett <
> jac...@gm...>
> >> wrote:
> >>
> >> [snip]
> >>
> >>>
> >>>  My first implementation was to have a set of current files stay in
> write
> >>> mode and have an overall lock over these files for the current day, but
> >>> (stupidly) I forgot that lock instances cannot be shared over separate
> >>> processes, only threads.
> >>>
> >>> So could you give me any advice in this situation? I'm sure it has
> come up
> >>> before. ;)
> >>
> >>
> >> Hello All, I previously suggested to Jacob a setup where only one proc
> would
> >> have a write handle and all of the other processes would be in read-only
> >> mode.  I am not sure that this would work.
> >>
> >> Francesc, Antonio, Josh, etc or anyone else, how would you solve this
> >> problem where you may want many processors to query the file, while
> >> something else may be writing to it?  I defer to people with more
> >> experience...  Thanks for your help!
> >>
> >> Be Well
> >> Anthony
> >>
> >>>
> >>> Thanks,
> >>> Jacob Bennett
> >>>
>
>
> --
> Antonio Valentino
>
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>



-- 
Jacob Bennett
Massachusetts Institute of Technology
Department of Electrical Engineering and Computer Science
Class of 2014| ben...@mi...