Menu
▾
▴
Re: [Htmlunit-user] problems with java memory when scraping data from different hosts
|
From: Ahmed A. <asa...@ya...> - 2014-02-13 09:16:23
|
Hi David, Done. Ahmed ________________________________ From: David Michael Gang <mic...@gm...> To: htm...@li... Sent: Thursday, February 13, 2014 11:55 AM Subject: Re: [Htmlunit-user] problems with java memory when scraping data from different hosts Hi, Would it be possible to upload this snapshot to maven? I have a jenkins job which generates the project i run through maven and it is hard for me to change these dependencies to look on a static jar. Thanks, David > >------------------------------ > >Message: 5 >Date: Wed, 12 Feb 2014 18:12:00 +0100 (CET) >From: "Ronald Brill" <rb...@rb...> >Subject: Re: [Htmlunit-user] problems with java memory when scraping > data from different hosts. >To: "htm...@li..." > <htm...@li...> >Message-ID: <100...@rb...> >Content-Type: text/plain; charset="iso-8859-15" > >HI David, > >i guess you are facing a memory issue of the current implementation. >I did a first shot trying to solve one part of the problem in SVN commit 9127. >Please give this version an try and report back. > > RBRi > >On Wed, 12 Feb 2014 09:28:46 +0200 David Michael Gang wrote: >> >>Hi all, >> >>I have a script which goes over ~1000 links and fetches the data from them. >>I have javascript switched on. >>The heap size grows up to about 1.5 GB. >>I performed jmap to get a .hprof file. >>Afterwards i analyzed the results with eclipsemat and got the following >>suspects >> >>836,303 instances of >>*"com.gargoylesoftware.htmlunit.javascript.host.NodeList$DomHtmlAttributeChangeListenerImpl"*, >>loaded by *"sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190"* occupy >>*360,618,336 >>(35.36%)* bytes. These instances are referenced from one instance of >>*"java.lang.Object[]"*, loaded by *"<system class loader>"* >> >> >> 44 instances of *"com.gargoylesoftware.htmlunit.html.HtmlPage"*, >>loaded by *"sun.misc.Launcher$AppClassLoader >>@ 0x2b21564bd190"* occupy *187,091,104 (18.34%)* bytes. >> >>Biggest instances: >> >> - com.gargoylesoftware.htmlunit.html.HtmlPage @ >> 0x2b216fad9408 - 70,380,232 (6.90%) bytes. >> - com.gargoylesoftware.htmlunit.html.HtmlPage @ >> 0x2b2162f66988 - 20,142,528 (1.98%) bytes. >> - com.gargoylesoftware.htmlunit.html.HtmlPage @ >> 0x2b216a296c60 - 12,966,560 (1.27%) bytes. >> - com.gargoylesoftware.htmlunit.html.HtmlPage @ >> 0x2b216a26c390 - 12,966,032 (1.27%) bytes. >> - com.gargoylesoftware.htmlunit.html.HtmlPage @ >> 0x2b2168791e60 - 10,687,768 (1.05%) bytes. >> - com.gargoylesoftware.htmlunit.html.HtmlPage @ >> 0x2b215b8b5460 - 10,651,168 (1.04%) bytes. >> - com.gargoylesoftware.htmlunit.html.HtmlPage @ >> 0x2b215a8090d8 - 10,523,280 (1.03%) bytes. >> >> >> >>*Keywords* >>sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190 >>com.gargoylesoftware.htmlunit.html.HtmlPage >> >>What is the meaning of these suspects? >>How can i reduce the usage of the memory? >> >>Thanks, >>David >> > > > > >------------------------------ > >------------------------------------------------------------------------------ >Android apps run on BlackBerry 10 >Introducing the new BlackBerry 10.2.1 Runtime for Android apps. >Now with support for Jelly Bean, Bluetooth, Mapview and more. >Get your Android app in front of a whole new audience. Start now. >http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk > >------------------------------ > >_______________________________________________ >Htmlunit-user mailing list >Htm...@li... >https://lists.sourceforge.net/lists/listinfo/htmlunit-user > > >End of Htmlunit-user Digest, Vol 93, Issue 21 >********************************************* > ------------------------------------------------------------------------------ Android apps run on BlackBerry 10 Introducing the new BlackBerry 10.2.1 Runtime for Android apps. Now with support for Jelly Bean, Bluetooth, Mapview and more. Get your Android app in front of a whole new audience. Start now. http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk _______________________________________________ Htmlunit-user mailing list Htm...@li... https://lists.sourceforge.net/lists/listinfo/htmlunit-user |
