From: Adam R. <ad...@ex...> - 2008-08-15 12:45:58
|
The problem with temporary fragments is well known. Temporary fragments have been removed in the 1.3 branch, so you may like to try that? 2008/8/15 Pieter Deelen <pie...@gm...>: > Hi list, > > I am working on an application using eXist (revision 7446 from the > exist-stable-1.2 branch). Every minute, this application collects some > data, stores it in the database and performs some processing on it. > After it is started, things go smoothly for a while. But, sometimes, > at unpredictable moments, the database turns unresponsive and the > transaction log grows to a gigantic size (tens of GBs). In the best > case, the transaction log does not fill up the disk during this event > (which can last more than 30 minutes), and the database continues its > operation afterwards. In the worst case (disk full), the database is > in an inconsistent state and not usable anymore. > > I performed some analysis on this problem. It turns out that the > explosive growth of the log coincides with the following log entries: > > 2008-08-13 21:36:49,724 [DefaultQuartzScheduler_Worker-1] DEBUG > (BFile.java [removeAll]:345) - Found 73 items to remove. > 2008-08-13 21:37:47,743 [DefaultQuartzScheduler_Worker-1] DEBUG > (BFile.java [removeAll]:345) - Found 0 items to remove. > 2008-08-13 21:37:47,743 [DefaultQuartzScheduler_Worker-1] DEBUG > (BFile.java [removeAll]:345) - Found 0 items to remove. > 2008-08-13 21:37:47,744 [DefaultQuartzScheduler_Worker-1] DEBUG > (NGramIndexWorker.java [removeCollection]:353) - Dropping NGram index > for collection /db/system/temp > 2008-08-13 21:37:47,744 [DefaultQuartzScheduler_Worker-1] DEBUG > (BFile.java [removeAll]:345) - Found 0 items to remove. > 2008-08-13 21:37:47,745 [DefaultQuartzScheduler_Worker-1] DEBUG > (BFile.java [removeAll]:345) - Found 0 items to remove. > 2008-08-13 21:37:47,745 [DefaultQuartzScheduler_Worker-1] DEBUG > (BFile.java [removeAll]:345) - Found 0 items to remove. > 2008-08-13 21:37:47,746 [DefaultQuartzScheduler_Worker-1] DEBUG > (NativeBroker.java [removeCollection]:920) - Removing collection > '/db/system/temp' from its parent... > 2008-08-13 21:37:47,749 [DefaultQuartzScheduler_Worker-1] DEBUG > (NativeBroker.java [removeCollection]:939) - Removing children > collections from their parent '/db/system/temp'... > 2008-08-13 21:37:57,456 [DefaultQuartzScheduler_Worker-1] DEBUG > (BFile.java [removeAll]:345) - Found 192525 items to remove. > 2008-08-13 21:38:26,756 [DefaultQuartzScheduler_Worker-1] DEBUG > (NativeBroker.java [removeCollection]:1009) - Removing resources in > '/db/system/temp'... > 2008-08-13 21:38:29,515 [DefaultQuartzScheduler_Worker-1] DEBUG > (DefaultCacheManager.java [requestMem]:181) - Growing cache dom.dbx (a > org.exist.storage.cache.LRDCache) from 324 to 486 > 2008-08-13 21:38:37,547 [DefaultQuartzScheduler_Worker-1] DEBUG > (DefaultCacheManager.java [requestMem]:181) - Growing cache > collections.dbx (a org.exist.storage.cache.LRUCache) from 80 to 100 > 2008-08-13 21:38:46,050 [DefaultQuartzScheduler_Worker-1] DEBUG > (DefaultCacheManager.java [requestMem]:181) - Growing cache dom.dbx (a > org.exist.storage.cache.LRDCache) from 486 to 729 > 2008-08-13 21:39:07,452 [DefaultQuartzScheduler_Worker-1] DEBUG > (DefaultCacheManager.java [requestMem]:181) - Growing cache dom.dbx (a > org.exist.storage.cache.LRDCache) from 729 to 1,093 > 2008-08-13 21:39:15,352 [DefaultQuartzScheduler_Worker-1] DEBUG > (LRDCache.java [cleanup]:153) - totalReferences = 640001; > maxReferences = 640000 > 2008-08-13 21:53:28,538 [DefaultQuartzScheduler_Worker-1] DEBUG > (DefaultCacheManager.java [requestMem]:181) - Growing cache > collections.dbx (a org.exist.storage.cache.LRUCache) from 100 to 125 > 2008-08-13 21:59:08,150 [DefaultQuartzScheduler_Worker-1] DEBUG > (LRDCache.java [cleanup]:153) - totalReferences = 640001; > maxReferences = 640000 > 2008-08-13 22:00:57,388 [DefaultQuartzScheduler_Worker-1] DEBUG > (DefaultCacheManager.java [requestMem]:181) - Growing cache > collections.dbx (a org.exist.storage.cache.LRUCache) from 125 to 156 > 2008-08-13 22:12:42,312 [DefaultQuartzScheduler_Worker-1] DEBUG > (DefaultCacheManager.java [requestMem]:181) - Growing cache > collections.dbx (a org.exist.storage.cache.LRUCache) from 156 to 195 > 2008-08-13 22:23:41,277 [DefaultQuartzScheduler_Worker-1] DEBUG > (LRDCache.java [cleanup]:153) - totalReferences = 640001; > maxReferences = 640000 > 2008-08-13 22:29:04,382 [DefaultQuartzScheduler_Worker-1] DEBUG > (NativeBroker.java [removeCollection]:1070) - Removing collection > '/db/system/temp' took 3134662 > > It looks like eXist is cleaning up its temporary document fragments. > In itself, this is a good thing, but it takes very long and occurs > seemingly at random. So, I began digging in the eXist code and found > the cleanUpTempResources method (from NativeBroker). If I understood > this correctly, this method removes the '/db/system/temp' collection > if all the fragments it contains are older than TEMP_FRAGMENT_TIMEOUT > (60000 ms, i.e., one minute). Because the query we execute every > minute generates temporary document fragments, this rarely happens > (e.g., only when the query execution is delayed). When it happens, the > amount of temporary fragments is so large, that it takes a very long > time. > > My questions are: what is the meaning of the one minute timeout? Is it > a safety margin of some kind? Can it be safely reduced to a lower > value (maybe even 0?), so eXist cleans up its fragments in small > bursts, instead of in one big operation? > > Regards, > Pieter Deelen > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > -- Adam Retter eXist Developer { England } ad...@ex... irc://irc.freenode.net/existdb |