Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

[Antonio V. <ant...@ti...>]

Hi all,
Il 14/07/2012 00:44, Josh Ayers ha scritto:
> My first instinct would be to handle all access (read and write) to
> that file from a single process.  You could create two
> multiprocessing.Queue objects, one for data to write and one for read
> requests.  Then the process would check the queues in a loop and
> handle each request serially.  The data read from the file could be
> sent back to the originating process using another queue or pipe.  You
> should be able to do the same thing with sockets if the other parts of
> your application are in languages other than Python.
> 
> I do something similar to handle writing to a log file from multiple
> processes and it works well.  In that case the file is write-only -
> and just a simple text file rather than HDF5 - but I don't see any
> reason why it wouldn't work for read and write as well.
> 
> Hope that helps,
> Josh
> 

I totally agree with Josh.

I don't have a test code to demonstrate it but IMHO parallelizing I/O
to/from a single file on a single disk do not makes too much sense
unless you have special HW.  Is this your case Jacob?

IMHO with standard SATA devices you could have a marginal speedup (in
the best case), but if your bottleneck is the I/O this will not solve
your problem.

If someone finds the time to implement a toy example of what Josh
suggested we could put it on the cookbook :)


regards

> On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz <sc...@gm...> wrote:
>> On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett <jac...@gm...>
>> wrote:
>>
>> [snip]
>>
>>>
>>>  My first implementation was to have a set of current files stay in write
>>> mode and have an overall lock over these files for the current day, but
>>> (stupidly) I forgot that lock instances cannot be shared over separate
>>> processes, only threads.
>>>
>>> So could you give me any advice in this situation? I'm sure it has come up
>>> before. ;)
>>
>>
>> Hello All, I previously suggested to Jacob a setup where only one proc would
>> have a write handle and all of the other processes would be in read-only
>> mode.  I am not sure that this would work.
>>
>> Francesc, Antonio, Josh, etc or anyone else, how would you solve this
>> problem where you may want many processors to query the file, while
>> something else may be writing to it?  I defer to people with more
>> experience...  Thanks for your help!
>>
>> Be Well
>> Anthony
>>
>>>
>>> Thanks,
>>> Jacob Bennett
>>>


-- 
Antonio Valentino