Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

[Jacob B. <jac...@gm...>]

Wait, is there perhaps a way to simulataneously read and write without any
kind of blocking? Perhaps the "a" mode or the "r+" mode might help for
simultaneous read/write? I am currently implementing the
multithreading.Queue, but I think that a large number of query requests
might put an necessary load on my writing queue since the data comes in
sooooo fast. ;)

Btw, I will submit the example soon.

-Jacob

On Sat, Jul 14, 2012 at 1:39 PM, Anthony Scopatz <sc...@gm...> wrote:

> +1 to example of this!
>
>
> On Sat, Jul 14, 2012 at 1:36 PM, Jacob Bennett <jac...@gm...>wrote:
>
>> Awesome, I think this sounds like a very workable solution and the idea
>> is very neat. I will try to implement this right away. I definitely agree
>> to putting a small example.
>>
>> Let you know how this works, thanks guys!
>>
>> Thanks,
>> Jacob
>>
>>
>> On Sat, Jul 14, 2012 at 2:36 AM, Antonio Valentino <
>> ant...@ti...> wrote:
>>
>>> Hi all,
>>> Il 14/07/2012 00:44, Josh Ayers ha scritto:
>>> > My first instinct would be to handle all access (read and write) to
>>> > that file from a single process.  You could create two
>>> > multiprocessing.Queue objects, one for data to write and one for read
>>> > requests.  Then the process would check the queues in a loop and
>>> > handle each request serially.  The data read from the file could be
>>> > sent back to the originating process using another queue or pipe.  You
>>> > should be able to do the same thing with sockets if the other parts of
>>> > your application are in languages other than Python.
>>> >
>>> > I do something similar to handle writing to a log file from multiple
>>> > processes and it works well.  In that case the file is write-only -
>>> > and just a simple text file rather than HDF5 - but I don't see any
>>> > reason why it wouldn't work for read and write as well.
>>> >
>>> > Hope that helps,
>>> > Josh
>>> >
>>>
>>> I totally agree with Josh.
>>>
>>> I don't have a test code to demonstrate it but IMHO parallelizing I/O
>>> to/from a single file on a single disk do not makes too much sense
>>> unless you have special HW.  Is this your case Jacob?
>>>
>>> IMHO with standard SATA devices you could have a marginal speedup (in
>>> the best case), but if your bottleneck is the I/O this will not solve
>>> your problem.
>>>
>>> If someone finds the time to implement a toy example of what Josh
>>> suggested we could put it on the cookbook :)
>>>
>>>
>>> regards
>>>
>>> > On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz <sc...@gm...>
>>> wrote:
>>> >> On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett <
>>> jac...@gm...>
>>> >> wrote:
>>> >>
>>> >> [snip]
>>> >>
>>> >>>
>>> >>>  My first implementation was to have a set of current files stay in
>>> write
>>> >>> mode and have an overall lock over these files for the current day,
>>> but
>>> >>> (stupidly) I forgot that lock instances cannot be shared over
>>> separate
>>> >>> processes, only threads.
>>> >>>
>>> >>> So could you give me any advice in this situation? I'm sure it has
>>> come up
>>> >>> before. ;)
>>> >>
>>> >>
>>> >> Hello All, I previously suggested to Jacob a setup where only one
>>> proc would
>>> >> have a write handle and all of the other processes would be in
>>> read-only
>>> >> mode.  I am not sure that this would work.
>>> >>
>>> >> Francesc, Antonio, Josh, etc or anyone else, how would you solve this
>>> >> problem where you may want many processors to query the file, while
>>> >> something else may be writing to it?  I defer to people with more
>>> >> experience...  Thanks for your help!
>>> >>
>>> >> Be Well
>>> >> Anthony
>>> >>
>>> >>>
>>> >>> Thanks,
>>> >>> Jacob Bennett
>>> >>>
>>>
>>>
>>> --
>>> Antonio Valentino
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>
>>
>>
>> --
>> Jacob Bennett
>> Massachusetts Institute of Technology
>> Department of Electrical Engineering and Computer Science
>> Class of 2014| ben...@mi...
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>


-- 
Jacob Bennett
Massachusetts Institute of Technology
Department of Electrical Engineering and Computer Science
Class of 2014| ben...@mi...