Menu
▾
▴
Re: [Pytables-users] PyTables Simultaneous Read Write from Current File
|
From: Josh A. <jos...@gm...> - 2012-09-02 18:48:11
|
Jacob, I just put together a small example demonstrating this. You can find it in the develop branch of the PyTables repository. https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py It's somewhat limited, because all the client processes have to be known at the time the file-accessing process is created. It should be possible to create a more flexible implementation (and probably faster as well) using sockets. I'll put together an example of that at some point. Hope that helps, Josh On Mon, Jul 16, 2012 at 10:03 PM, Anthony Scopatz <sc...@gm...> wrote: > On Mon, Jul 16, 2012 at 3:30 PM, Jacob Bennett <jac...@gm...>wrote: > >> Wait, is there perhaps a way to simulataneously read and write without >> any kind of blocking? Perhaps the "a" mode or the "r+" mode might help for >> simultaneous read/write? I am currently implementing the >> multithreading.Queue, but I think that a large number of query requests >> might put an necessary load on my writing queue since the data comes in >> sooooo fast. ;) > > > Hmm I'll have to look into it, but I vaguely recall a file access mode > that HDF5 has that PyTables doesn't expose... I may be wrong about this.... > > >> Btw, I will submit the example soon. >> > > +1! > > >> >> -Jacob >> >> >> On Sat, Jul 14, 2012 at 1:39 PM, Anthony Scopatz <sc...@gm...>wrote: >> >>> +1 to example of this! >>> >>> >>> On Sat, Jul 14, 2012 at 1:36 PM, Jacob Bennett < >>> jac...@gm...> wrote: >>> >>>> Awesome, I think this sounds like a very workable solution and the idea >>>> is very neat. I will try to implement this right away. I definitely agree >>>> to putting a small example. >>>> >>>> Let you know how this works, thanks guys! >>>> >>>> Thanks, >>>> Jacob >>>> >>>> >>>> On Sat, Jul 14, 2012 at 2:36 AM, Antonio Valentino < >>>> ant...@ti...> wrote: >>>> >>>>> Hi all, >>>>> Il 14/07/2012 00:44, Josh Ayers ha scritto: >>>>> > My first instinct would be to handle all access (read and write) to >>>>> > that file from a single process. You could create two >>>>> > multiprocessing.Queue objects, one for data to write and one for read >>>>> > requests. Then the process would check the queues in a loop and >>>>> > handle each request serially. The data read from the file could be >>>>> > sent back to the originating process using another queue or pipe. >>>>> You >>>>> > should be able to do the same thing with sockets if the other parts >>>>> of >>>>> > your application are in languages other than Python. >>>>> > >>>>> > I do something similar to handle writing to a log file from multiple >>>>> > processes and it works well. In that case the file is write-only - >>>>> > and just a simple text file rather than HDF5 - but I don't see any >>>>> > reason why it wouldn't work for read and write as well. >>>>> > >>>>> > Hope that helps, >>>>> > Josh >>>>> > >>>>> >>>>> I totally agree with Josh. >>>>> >>>>> I don't have a test code to demonstrate it but IMHO parallelizing I/O >>>>> to/from a single file on a single disk do not makes too much sense >>>>> unless you have special HW. Is this your case Jacob? >>>>> >>>>> IMHO with standard SATA devices you could have a marginal speedup (in >>>>> the best case), but if your bottleneck is the I/O this will not solve >>>>> your problem. >>>>> >>>>> If someone finds the time to implement a toy example of what Josh >>>>> suggested we could put it on the cookbook :) >>>>> >>>>> >>>>> regards >>>>> >>>>> > On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz <sc...@gm...> >>>>> wrote: >>>>> >> On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett < >>>>> jac...@gm...> >>>>> >> wrote: >>>>> >> >>>>> >> [snip] >>>>> >> >>>>> >>> >>>>> >>> My first implementation was to have a set of current files stay >>>>> in write >>>>> >>> mode and have an overall lock over these files for the current >>>>> day, but >>>>> >>> (stupidly) I forgot that lock instances cannot be shared over >>>>> separate >>>>> >>> processes, only threads. >>>>> >>> >>>>> >>> So could you give me any advice in this situation? I'm sure it has >>>>> come up >>>>> >>> before. ;) >>>>> >> >>>>> >> >>>>> >> Hello All, I previously suggested to Jacob a setup where only one >>>>> proc would >>>>> >> have a write handle and all of the other processes would be in >>>>> read-only >>>>> >> mode. I am not sure that this would work. >>>>> >> >>>>> >> Francesc, Antonio, Josh, etc or anyone else, how would you solve >>>>> this >>>>> >> problem where you may want many processors to query the file, while >>>>> >> something else may be writing to it? I defer to people with more >>>>> >> experience... Thanks for your help! >>>>> >> >>>>> >> Be Well >>>>> >> Anthony >>>>> >> >>>>> >>> >>>>> >>> Thanks, >>>>> >>> Jacob Bennett >>>>> >>> >>>>> >>>>> >>>>> -- >>>>> Antonio Valentino >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Live Security Virtual Conference >>>>> Exclusive live event will cover all the ways today's security and >>>>> threat landscape has changed and how IT managers can respond. >>>>> Discussions >>>>> will include endpoint security, mobile security and the latest in >>>>> malware >>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>>>> _______________________________________________ >>>>> Pytables-users mailing list >>>>> Pyt...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users >>>>> >>>> >>>> >>>> >>>> -- >>>> Jacob Bennett >>>> Massachusetts Institute of Technology >>>> Department of Electrical Engineering and Computer Science >>>> Class of 2014| ben...@mi... >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Live Security Virtual Conference >>>> Exclusive live event will cover all the ways today's security and >>>> threat landscape has changed and how IT managers can respond. >>>> Discussions >>>> will include endpoint security, mobile security and the latest in >>>> malware >>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>>> _______________________________________________ >>>> Pytables-users mailing list >>>> Pyt...@li... >>>> https://lists.sourceforge.net/lists/listinfo/pytables-users >>>> >>>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. Discussions >>> will include endpoint security, mobile security and the latest in malware >>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>> _______________________________________________ >>> Pytables-users mailing list >>> Pyt...@li... >>> https://lists.sourceforge.net/lists/listinfo/pytables-users >>> >>> >> >> >> -- >> Jacob Bennett >> Massachusetts Institute of Technology >> Department of Electrical Engineering and Computer Science >> Class of 2014| ben...@mi... >> >> >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Pytables-users mailing list >> Pyt...@li... >> https://lists.sourceforge.net/lists/listinfo/pytables-users >> >> > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Pytables-users mailing list > Pyt...@li... > https://lists.sourceforge.net/lists/listinfo/pytables-users > > |
