Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

[Anthony S. <sc...@gm...>]

On Mon, Jul 16, 2012 at 3:30 PM, Jacob Bennett <jac...@gm...>wrote:

> Wait, is there perhaps a way to simulataneously read and write without any
> kind of blocking? Perhaps the "a" mode or the "r+" mode might help for
> simultaneous read/write? I am currently implementing the
> multithreading.Queue, but I think that a large number of query requests
> might put an necessary load on my writing queue since the data comes in
> sooooo fast. ;)


Hmm I'll have to look into it, but I vaguely recall a file access mode that
HDF5 has that PyTables doesn't expose... I may be wrong about this....


> Btw, I will submit the example soon.
>

+1!


>
> -Jacob
>
>
> On Sat, Jul 14, 2012 at 1:39 PM, Anthony Scopatz <sc...@gm...>wrote:
>
>> +1 to example of this!
>>
>>
>> On Sat, Jul 14, 2012 at 1:36 PM, Jacob Bennett <jac...@gm...
>> > wrote:
>>
>>> Awesome, I think this sounds like a very workable solution and the idea
>>> is very neat. I will try to implement this right away. I definitely agree
>>> to putting a small example.
>>>
>>> Let you know how this works, thanks guys!
>>>
>>> Thanks,
>>> Jacob
>>>
>>>
>>> On Sat, Jul 14, 2012 at 2:36 AM, Antonio Valentino <
>>> ant...@ti...> wrote:
>>>
>>>> Hi all,
>>>> Il 14/07/2012 00:44, Josh Ayers ha scritto:
>>>> > My first instinct would be to handle all access (read and write) to
>>>> > that file from a single process.  You could create two
>>>> > multiprocessing.Queue objects, one for data to write and one for read
>>>> > requests.  Then the process would check the queues in a loop and
>>>> > handle each request serially.  The data read from the file could be
>>>> > sent back to the originating process using another queue or pipe.  You
>>>> > should be able to do the same thing with sockets if the other parts of
>>>> > your application are in languages other than Python.
>>>> >
>>>> > I do something similar to handle writing to a log file from multiple
>>>> > processes and it works well.  In that case the file is write-only -
>>>> > and just a simple text file rather than HDF5 - but I don't see any
>>>> > reason why it wouldn't work for read and write as well.
>>>> >
>>>> > Hope that helps,
>>>> > Josh
>>>> >
>>>>
>>>> I totally agree with Josh.
>>>>
>>>> I don't have a test code to demonstrate it but IMHO parallelizing I/O
>>>> to/from a single file on a single disk do not makes too much sense
>>>> unless you have special HW.  Is this your case Jacob?
>>>>
>>>> IMHO with standard SATA devices you could have a marginal speedup (in
>>>> the best case), but if your bottleneck is the I/O this will not solve
>>>> your problem.
>>>>
>>>> If someone finds the time to implement a toy example of what Josh
>>>> suggested we could put it on the cookbook :)
>>>>
>>>>
>>>> regards
>>>>
>>>> > On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz <sc...@gm...>
>>>> wrote:
>>>> >> On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett <
>>>> jac...@gm...>
>>>> >> wrote:
>>>> >>
>>>> >> [snip]
>>>> >>
>>>> >>>
>>>> >>>  My first implementation was to have a set of current files stay in
>>>> write
>>>> >>> mode and have an overall lock over these files for the current day,
>>>> but
>>>> >>> (stupidly) I forgot that lock instances cannot be shared over
>>>> separate
>>>> >>> processes, only threads.
>>>> >>>
>>>> >>> So could you give me any advice in this situation? I'm sure it has
>>>> come up
>>>> >>> before. ;)
>>>> >>
>>>> >>
>>>> >> Hello All, I previously suggested to Jacob a setup where only one
>>>> proc would
>>>> >> have a write handle and all of the other processes would be in
>>>> read-only
>>>> >> mode.  I am not sure that this would work.
>>>> >>
>>>> >> Francesc, Antonio, Josh, etc or anyone else, how would you solve this
>>>> >> problem where you may want many processors to query the file, while
>>>> >> something else may be writing to it?  I defer to people with more
>>>> >> experience...  Thanks for your help!
>>>> >>
>>>> >> Be Well
>>>> >> Anthony
>>>> >>
>>>> >>>
>>>> >>> Thanks,
>>>> >>> Jacob Bennett
>>>> >>>
>>>>
>>>>
>>>> --
>>>> Antonio Valentino
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> will include endpoint security, mobile security and the latest in
>>>> malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Pytables-users mailing list
>>>> Pyt...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>
>>>
>>>
>>>
>>> --
>>> Jacob Bennett
>>> Massachusetts Institute of Technology
>>> Department of Electrical Engineering and Computer Science
>>> Class of 2014| ben...@mi...
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> --
> Jacob Bennett
> Massachusetts Institute of Technology
> Department of Electrical Engineering and Computer Science
> Class of 2014| ben...@mi...
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>