Menu
▾
▴
Re: [Htmlunit-user] problems with java memory when scraping data from different hosts.
|
From: Ronald B. <rb...@rb...> - 2014-02-12 17:12:18
|
HI David, i guess you are facing a memory issue of the current implementation. I did a first shot trying to solve one part of the problem in SVN commit 9127. Please give this version an try and report back. RBRi On Wed, 12 Feb 2014 09:28:46 +0200 David Michael Gang wrote: > >Hi all, > >I have a script which goes over ~1000 links and fetches the data from them. >I have javascript switched on. >The heap size grows up to about 1.5 GB. >I performed jmap to get a .hprof file. >Afterwards i analyzed the results with eclipsemat and got the following >suspects > >836,303 instances of >*"com.gargoylesoftware.htmlunit.javascript.host.NodeList$DomHtmlAttributeChangeListenerImpl"*, >loaded by *"sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190"* occupy >*360,618,336 >(35.36%)* bytes. These instances are referenced from one instance of >*"java.lang.Object[]"*, loaded by *"<system class loader>"* > > > 44 instances of *"com.gargoylesoftware.htmlunit.html.HtmlPage"*, >loaded by *"sun.misc.Launcher$AppClassLoader >@ 0x2b21564bd190"* occupy *187,091,104 (18.34%)* bytes. > >Biggest instances: > > - com.gargoylesoftware.htmlunit.html.HtmlPage @ > 0x2b216fad9408 - 70,380,232 (6.90%) bytes. > - com.gargoylesoftware.htmlunit.html.HtmlPage @ > 0x2b2162f66988 - 20,142,528 (1.98%) bytes. > - com.gargoylesoftware.htmlunit.html.HtmlPage @ > 0x2b216a296c60 - 12,966,560 (1.27%) bytes. > - com.gargoylesoftware.htmlunit.html.HtmlPage @ > 0x2b216a26c390 - 12,966,032 (1.27%) bytes. > - com.gargoylesoftware.htmlunit.html.HtmlPage @ > 0x2b2168791e60 - 10,687,768 (1.05%) bytes. > - com.gargoylesoftware.htmlunit.html.HtmlPage @ > 0x2b215b8b5460 - 10,651,168 (1.04%) bytes. > - com.gargoylesoftware.htmlunit.html.HtmlPage @ > 0x2b215a8090d8 - 10,523,280 (1.03%) bytes. > > > >*Keywords* >sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190 >com.gargoylesoftware.htmlunit.html.HtmlPage > >What is the meaning of these suspects? >How can i reduce the usage of the memory? > >Thanks, >David > |
