Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

[Anthony S. <sc...@gm...>]

+1 to example of this!

On Sat, Jul 14, 2012 at 1:36 PM, Jacob Bennett <jac...@gm...>wrote:

> Awesome, I think this sounds like a very workable solution and the idea is
> very neat. I will try to implement this right away. I definitely agree to
> putting a small example.
>
> Let you know how this works, thanks guys!
>
> Thanks,
> Jacob
>
>
> On Sat, Jul 14, 2012 at 2:36 AM, Antonio Valentino <
> ant...@ti...> wrote:
>
>> Hi all,
>> Il 14/07/2012 00:44, Josh Ayers ha scritto:
>> > My first instinct would be to handle all access (read and write) to
>> > that file from a single process.  You could create two
>> > multiprocessing.Queue objects, one for data to write and one for read
>> > requests.  Then the process would check the queues in a loop and
>> > handle each request serially.  The data read from the file could be
>> > sent back to the originating process using another queue or pipe.  You
>> > should be able to do the same thing with sockets if the other parts of
>> > your application are in languages other than Python.
>> >
>> > I do something similar to handle writing to a log file from multiple
>> > processes and it works well.  In that case the file is write-only -
>> > and just a simple text file rather than HDF5 - but I don't see any
>> > reason why it wouldn't work for read and write as well.
>> >
>> > Hope that helps,
>> > Josh
>> >
>>
>> I totally agree with Josh.
>>
>> I don't have a test code to demonstrate it but IMHO parallelizing I/O
>> to/from a single file on a single disk do not makes too much sense
>> unless you have special HW.  Is this your case Jacob?
>>
>> IMHO with standard SATA devices you could have a marginal speedup (in
>> the best case), but if your bottleneck is the I/O this will not solve
>> your problem.
>>
>> If someone finds the time to implement a toy example of what Josh
>> suggested we could put it on the cookbook :)
>>
>>
>> regards
>>
>> > On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz <sc...@gm...>
>> wrote:
>> >> On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett <
>> jac...@gm...>
>> >> wrote:
>> >>
>> >> [snip]
>> >>
>> >>>
>> >>>  My first implementation was to have a set of current files stay in
>> write
>> >>> mode and have an overall lock over these files for the current day,
>> but
>> >>> (stupidly) I forgot that lock instances cannot be shared over separate
>> >>> processes, only threads.
>> >>>
>> >>> So could you give me any advice in this situation? I'm sure it has
>> come up
>> >>> before. ;)
>> >>
>> >>
>> >> Hello All, I previously suggested to Jacob a setup where only one proc
>> would
>> >> have a write handle and all of the other processes would be in
>> read-only
>> >> mode.  I am not sure that this would work.
>> >>
>> >> Francesc, Antonio, Josh, etc or anyone else, how would you solve this
>> >> problem where you may want many processors to query the file, while
>> >> something else may be writing to it?  I defer to people with more
>> >> experience...  Thanks for your help!
>> >>
>> >> Be Well
>> >> Anthony
>> >>
>> >>>
>> >>> Thanks,
>> >>> Jacob Bennett
>> >>>
>>
>>
>> --
>> Antonio Valentino
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>
>
>
> --
> Jacob Bennett
> Massachusetts Institute of Technology
> Department of Electrical Engineering and Computer Science
> Class of 2014| ben...@mi...
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>