Re: [Pytables-users] How would you handle file openings for reading by an API?

[Jacob B. <jac...@gm...>]

Cool, this seems pretty straightforward. Thanks again Anthony!

-Jacob

On Mon, Jul 2, 2012 at 1:10 PM, Anthony Scopatz <sc...@gm...> wrote:

> Hello Jacob,
>
> It seems like you have answered your own question ;).  The thing is that
> the locking doesn't have to be all that hard.  You can simply check if the
> file is already in the open files cache (from previous thread).  If it
> isn't, the thread will open the file.  Or you don't even have to do the
> check yourself because tables.openFile() will do this for you.  If the file
> is already there, openFile() will just return you a reference to that File
> instance, which is what you wanted anyways.  This takes care of opening.
>
> On the other side of things, don't allow any thread to close a file.
>  Simply close all files in the cache when your code is about to exit.
>  Keeping the file handles open and available for future reading isn't *
> that* expensive.
>
> So use openFile() for opening and don't close until the end and this
> should be thread safe for reading.  Obviously, writing is more difficult.
>
> Be Well
> Anthony
>
> On Mon, Jul 2, 2012 at 5:38 AM, Jacob Bennett <jac...@gm...>wrote:
>
>> Hello PyTables Users,
>>
>> I am developing an API to access the current data stored in my pytables
>> instance. Note at this point that this is only reading, no writing to the
>> files. The big question on my mind at this point is how am I supposed to
>> handle the opening and closing of files on read requests that are
>> multithreaded? PyTables supports multithreading for read only; however, I
>> don't know how to handle two threads opening the same file or one thread
>> closing a file while the other is still reading it, besides putting a lock
>> on it thus disabling the multithreaded operations.
>>
>> Thanks,
>> Jacob
>>
>> --
>> Jacob Bennett
>> Massachusetts Institute of Technology
>> Department of Electrical Engineering and Computer Science
>> Class of 2014| ben...@mi...
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>


-- 
Jacob Bennett
Massachusetts Institute of Technology
Department of Electrical Engineering and Computer Science
Class of 2014| ben...@mi...