Menu
▾
▴
Re: [Pytables-users] How would you handle file openings for reading by an API?
|
From: Anthony S. <sc...@gm...> - 2012-07-03 00:43:14
|
No worries ;) On Mon, Jul 2, 2012 at 5:41 PM, Jacob Bennett <jac...@gm...>wrote: > Cool, this seems pretty straightforward. Thanks again Anthony! > > -Jacob > > > On Mon, Jul 2, 2012 at 1:10 PM, Anthony Scopatz <sc...@gm...> wrote: > >> Hello Jacob, >> >> It seems like you have answered your own question ;). The thing is that >> the locking doesn't have to be all that hard. You can simply check if the >> file is already in the open files cache (from previous thread). If it >> isn't, the thread will open the file. Or you don't even have to do the >> check yourself because tables.openFile() will do this for you. If the file >> is already there, openFile() will just return you a reference to that File >> instance, which is what you wanted anyways. This takes care of opening. >> >> On the other side of things, don't allow any thread to close a file. >> Simply close all files in the cache when your code is about to exit. >> Keeping the file handles open and available for future reading isn't * >> that* expensive. >> >> So use openFile() for opening and don't close until the end and this >> should be thread safe for reading. Obviously, writing is more difficult. >> >> Be Well >> Anthony >> >> On Mon, Jul 2, 2012 at 5:38 AM, Jacob Bennett <jac...@gm...>wrote: >> >>> Hello PyTables Users, >>> >>> I am developing an API to access the current data stored in my pytables >>> instance. Note at this point that this is only reading, no writing to the >>> files. The big question on my mind at this point is how am I supposed to >>> handle the opening and closing of files on read requests that are >>> multithreaded? PyTables supports multithreading for read only; however, I >>> don't know how to handle two threads opening the same file or one thread >>> closing a file while the other is still reading it, besides putting a lock >>> on it thus disabling the multithreaded operations. >>> >>> Thanks, >>> Jacob >>> >>> -- >>> Jacob Bennett >>> Massachusetts Institute of Technology >>> Department of Electrical Engineering and Computer Science >>> Class of 2014| ben...@mi... >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. Discussions >>> will include endpoint security, mobile security and the latest in malware >>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>> _______________________________________________ >>> Pytables-users mailing list >>> Pyt...@li... >>> https://lists.sourceforge.net/lists/listinfo/pytables-users >>> >>> >> >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Pytables-users mailing list >> Pyt...@li... >> https://lists.sourceforge.net/lists/listinfo/pytables-users >> >> > > > -- > Jacob Bennett > Massachusetts Institute of Technology > Department of Electrical Engineering and Computer Science > Class of 2014| ben...@mi... > > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Pytables-users mailing list > Pyt...@li... > https://lists.sourceforge.net/lists/listinfo/pytables-users > > |
