From: Rosenberg, E. <eri...@ng...> - 2003-07-31 16:37:20
|
If the recman is thread safe then calling commit from two threads should work right? I understand that from an application level view point it will lead to weirdness, but I shouldn't get an exception from jdbm code should I? I definitely have an issue within my application, I need to figure out some way to coordinate the requests I'm servicing. I don't want to synchronize everything because then I can't handle any concurrent requests... If I can get to cvs I'll pull the latest code and give it a try. Isn't what you committed essentially the same as when I synchronized my call to recman.commit? That definitely improved things but I was still getting the bogus ID issues. I'll have to look through the code I was testing and see if I was using any iterators. Thanks for all the help. Eric -----Original Message----- From: Alex Boisvert [mailto:boi...@in...] Sent: Thursday, July 31, 2003 1:54 AM To: Rosenberg, Eric; jdb...@li... Subject: Re: [Jdbm-general] Thread safety Eric, The public JDBM APIs are all meant to be thread-safe. If that isn't the case, then it's a bug. Actually, thread support should be qualified since JDBM only support one thread acting on its data structures at a time. This is enforced by synchronizing most top-level methods such as on RecordManager, BTree and HTree. The second note is that you generally don't want to commit() from different threads. In pseudo-code, you would typically do something like this: T1: Thread A: Open RecordManager T2: Thread B,C,D,...: Use data structures to perform a unit of work T3: Thread A: Commit transaction. Since JDBM only support one active transaction at a time, all changes done by threads B, C and D become part of the same atomic unit of work. In other words, when thread A commits, the work of B, C and D are committed. So it is assumed that the application coordinates the work of different threads before commiting, to ensure a consistent state at the time commit() is called. If you look at the test case found in TestBTree.java, you'll see an example of multi-threaded access. Now, looking at the code, I noticed that BaseRecordManager.commit() and rollback() are not currently synchronized. This is probably an overlook. These methods should be synchronized to guarantee thread-safety. Although, provided you use the pattern described above, the result should always be consistent. I suppose the methods of CacheRecordManager should also all be synchronized as well to guarantee thread-safety. I will test and commit those changes in CVS very shortly. An exception to the thread-safety of data structures is the use of iterators. Iterators in JDBM are not meant to be thread-safe nor do they support structural modification during iteration. If you need to iterate over a data structure, you should synchronize on that data structure to prevent concurrent modification and also avoid doing any modfications to it while you are iterating. This may be the cause of the "bogus row id" exception that you are experiencing, but I cannot be certain. Can you use the pattern illustrated above in your application? alex Rosenberg, Eric wrote: >To what extent is jdbm thread safe? > >I'll try to send along some test code if I can reduce what I have to a >reasonable size. I'm having trouble using the same record manager across >multiple threads. Particularly, when I call commit from both threads, I get >a lot of: > > java.lang.ArrayIndexOutOfBoundsException: 1063 > at jdbm.recman.TransactionManager.start(TransactionManager.java:254) > at jdbm.recman.RecordFile.commit(RecordFile.java:242) > at jdbm.recman.PageManager.commit(PageManager.java:230) > at jdbm.recman.BaseRecordManager.commit(BaseRecordManager.java:416) > at >jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:338) > at >com.northgrum.kps.editor.cache.TestMultithreadedBindingCache$TestInserterTh r >ead.commitRecman(TestMultithreadedBindingCache.java:131) > at >com.northgrum.kps.editor.cache.TestMultithreadedBindingCache$TestInserterTh r >ead.testOk(TestMultithreadedBindingCache.java:108) > at >com.mousepushers.junit.ControllableTestThread.run(ControllableTestThread.ja v >a:79) > >Syncronizing my call to commit by doing: > >synchronized(recman){ > recman.commit(); >} > >Eliminated these exceptions but I'm still getting occasional exceptions from >the record manager. These are a lot rarer then when I didn't have the call >to commit synchronized. Usually, the exception is complaining about a block >id. > >java.lang.Error: bogus block id 1441714830711980032 > at jdbm.recman.BlockIo.setBlockId(BlockIo.java:108) > at jdbm.recman.RecordFile.getNewNode(RecordFile.java:356) > at jdbm.recman.RecordFile.get(RecordFile.java:160) > at >jdbm.recman.FreePhysicalRowIdPageManager.get(FreePhysicalRowIdPageManager.j a >va:89) > at >jdbm.recman.PhysicalRowIdManager.alloc(PhysicalRowIdManager.java:173) > at >jdbm.recman.PhysicalRowIdManager.insert(PhysicalRowIdManager.java:81) > at jdbm.recman.BaseRecordManager.insert(BaseRecordManager.java:213) > at >jdbm.recman.CacheRecordManager.insert(CacheRecordManager.java:158) > at >jdbm.recman.CacheRecordManager.insert(CacheRecordManager.java:141) > at jdbm.htree.HashDirectory.put(HashDirectory.java:229) > at jdbm.htree.HTree.put(HTree.java:130) > at >com.northgrum.kps.editor.cache.BindingCache.put(BindingCache.java:153) > at >com.northgrum.kps.editor.cache.TestMultithreadedBindingCache$TestInserterTh r >ead.testOk(TestMultithreadedBindingCache.java:98) > at >com.mousepushers.junit.ControllableTestThread.run(ControllableTestThread.ja v >a:79) > >and occasionally: > >java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextEntry(HashMap.java:762) > at java.util.HashMap$ValueIterator.next(HashMap.java:792) > at jdbm.recman.RecordFile.commit(RecordFile.java:246) > at jdbm.recman.PageManager.commit(PageManager.java:230) > at jdbm.recman.BaseRecordManager.commit(BaseRecordManager.java:416) > at >jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:338) > at >com.northgrum.kps.editor.cache.TestMultithreadedBindingCache$TestInserterTh r >ead.commitRecman(TestMultithreadedBindingCache.java:131) > at >com.northgrum.kps.editor.cache.TestMultithreadedBindingCache$TestInserterTh r >ead.testOk(TestMultithreadedBindingCache.java:113) > at >com.mousepushers.junit.ControllableTestThread.run(ControllableTestThread.ja v >a:79) > >also: > >java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 > at java.util.ArrayList.RangeCheck(ArrayList.java:508) > at java.util.ArrayList.get(ArrayList.java:320) > at jdbm.htree.HashBucket.writeExternal(HashBucket.java:271) > at >java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1265) > at >java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1243 ) > at >java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052) > at >java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:278) > at jdbm.helper.Serialization.serialize(Serialization.java:75) > at >jdbm.helper.DefaultSerializer.serialize(DefaultSerializer.java:83) > at jdbm.recman.BaseRecordManager.update(BaseRecordManager.java:281) > at >jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:335) > at >com.northgrum.kps.editor.cache.TestMultithreadedBindingCache$TestInserterTh r >ead.commitRecman(TestMultithreadedBindingCache.java:131) > at >com.northgrum.kps.editor.cache.TestMultithreadedBindingCache$TestInserterTh r >ead.testOk(TestMultithreadedBindingCache.java:113) > at >com.mousepushers.junit.ControllableTestThread.run(ControllableTestThread.ja v >a:79) > > >------------------------------------------------------- >This SF.Net email sponsored by: Free pre-built ASP.NET sites including >Data Reports, E-commerce, Portals, and Forums are available now. >Download today and enter to win an XBOX or Visual Studio .NET. >http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01 >_______________________________________________ >Jdbm-general mailing list >Jdb...@li... >https://lists.sourceforge.net/lists/listinfo/jdbm-general > > ------------------------------------------------------- This SF.Net email sponsored by: Free pre-built ASP.NET sites including Data Reports, E-commerce, Portals, and Forums are available now. Download today and enter to win an XBOX or Visual Studio .NET. http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01 _______________________________________________ Jdbm-general mailing list Jdb...@li... https://lists.sourceforge.net/lists/listinfo/jdbm-general |
From: Rosenberg, E. <eri...@ng...> - 2003-07-31 22:13:10
|
Alex, Thanks for sending me the new version. So far it seems to work great. I'm not seeing any of the exceptions I was yesterday. I'm surprised since I thought what you did would be equivalent to the external synchronization I did yesterday. Anyhow, thanks for all the help today. I'll let you know if I stumble across any more issues. Eric -----Original Message----- From: Alex Boisvert [mailto:boi...@in...] Sent: Thursday, July 31, 2003 12:21 PM To: Rosenberg, Eric Cc: jdb...@li... Subject: Re: [Jdbm-general] Thread safety Rosenberg, Eric wrote: >If the recman is thread safe then calling commit from two threads should >work right? I understand that from an application level view point it will >lead to weirdness, but I shouldn't get an exception from jdbm code should I? > That's right. >I definitely have an issue within my application, I need to figure out some >way to coordinate the requests I'm servicing. I don't want to synchronize >everything because then I can't handle any concurrent requests... > Understood. We all want more concurrency ;) The way I've personally dealt with this is to have my application lock at a higher-level, work on the object it needs to and then synchronize on the RecordManager during the updates and commit. The update+commit phase becomes your bottleneck but that's unavoidable with the current JDBM design. JDBM was not designed to handle multiple transactions at a time. (And that's one of the reason for its simplicity) >If I can get to cvs I'll pull the latest code and give it a try. Isn't what >you committed essentially the same as when I synchronized my call to >recman.commit? That definitely improved things but I was still getting the >bogus ID issues. I'll have to look through the code I was testing and see if >I was using any iterators. > Yes, what I committed is equivalent to what you were doing. So I definitely haven't solved your problem yet. I just need a reproductible case to figure out what's going wrong. alex |
From: Alex B. <boi...@in...> - 2003-07-31 16:56:48
|
Rosenberg, Eric wrote: >If the recman is thread safe then calling commit from two threads should >work right? I understand that from an application level view point it will >lead to weirdness, but I shouldn't get an exception from jdbm code should I? > That's right. >I definitely have an issue within my application, I need to figure out some >way to coordinate the requests I'm servicing. I don't want to synchronize >everything because then I can't handle any concurrent requests... > Understood. We all want more concurrency ;) The way I've personally dealt with this is to have my application lock at a higher-level, work on the object it needs to and then synchronize on the RecordManager during the updates and commit. The update+commit phase becomes your bottleneck but that's unavoidable with the current JDBM design. JDBM was not designed to handle multiple transactions at a time. (And that's one of the reason for its simplicity) >If I can get to cvs I'll pull the latest code and give it a try. Isn't what >you committed essentially the same as when I synchronized my call to >recman.commit? That definitely improved things but I was still getting the >bogus ID issues. I'll have to look through the code I was testing and see if >I was using any iterators. > Yes, what I committed is equivalent to what you were doing. So I definitely haven't solved your problem yet. I just need a reproductible case to figure out what's going wrong. alex |