Menu
▾
▴
Re: [gate-users] Gate throwing Out
|
From: Praveen G. <pra...@ya...> - 2007-12-19 18:51:16
|
Thank you Sir, I ignored the fact that Corpus and document are resources and those need to be explicitly deleted.Once i deleted them things are working fine. Thanks Praveen ----- Original Message ---- From: Hamish Cunningham <ha...@dc...> To: Praveen George <pra...@ya...> Cc: gat...@li... Sent: Tuesday, December 18, 2007 2:27:43 AM Subject: Re: [gate-users] Gate throwing Out hi please search the list and docs on memory, this is an faq basically you have to tell gate not to keep all the docs in memory all the time best h Praveen George wrote: > Hi, > I am using gate for content segregation purposes. We do have a gazette > which takes around 5 MBs. The corpus largely consists of internet > resources, primarily text , You can consider that as contents of web > sites. I am setting the corpus to be the content of 20 urls . So the > corpus contains close to 20 individual documents. In chunks of 20 i am > processing closer to 4000 urls . Currently what i have done is this > process as much as i can in 1 hour depending on the heap space, timeout > that process and restart a new one each hour. It requires unix crons. > But my architect does not like this idea he wants GATE to run > continuously for years without any one doing anything to it. What > happens currently is on a heap size of 512MB is processes around 200/300 > articles ie 10-20 corpus instances in my case. Then it throws an > OutOfMemory error. I tried calling garbage collector after each corpus > processing , but no use , then i tried initializing Gate before > processing each chunk, that does not work either. Is any of you guys > facing similar issues and is there any stuff i can tune to avoid this > behaviour. > > Thanks > Praveen. > > ------------------------------------------------------------------------ > Looking for last minute shopping deals? Find them fast with Yahoo! > Search. > <http://us.rd.yahoo.com/evt=51734/*http://tools.search.yahoo.com/newsearch/category.php?category=shopping> > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > SF.Net email is sponsored by: > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services > for just about anything Open Source. > http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace > > > ------------------------------------------------------------------------ > > _______________________________________________ > GATE-users mailing list > GAT...@li... > https://lists.sourceforge.net/lists/listinfo/gate-users -- Hamish http://www.dcs.shef.ac.uk/~hamish/ ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ |