Re: [Htmlunit-user] problems with java memory when scraping data from different hosts.

[asashour <asa...@ya...>]

Hi David,

Any feedback? The releasing is pending for this ;)

Ahmed


Ronald Brill wrote
> HI David,
> 
> i guess you are facing a memory issue of the current implementation.
> I did a first shot trying to solve one part of the problem in SVN commit
> 9127.
> Please give this version an try and report back.
> 
> 	RBRi
> 
> On Wed, 12 Feb 2014 09:28:46 +0200 David Michael Gang wrote:
>>
>>Hi all,
>>
>>I have a script which goes over ~1000 links and fetches the data from
them.
>>I have javascript switched on.
>>The heap size grows up to about 1.5 GB.
>>I performed jmap to get a .hprof file.
>>Afterwards i analyzed the results with eclipsemat and got the following
>>suspects
>>
>>836,303 instances of
>>*"com.gargoylesoftware.htmlunit.javascript.host.NodeList$DomHtmlAttributeChangeListenerImpl"*,
>>loaded by *"sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190"* occupy
>>*360,618,336
>>(35.36%)* bytes. These instances are referenced from one instance of
>>*"java.lang.Object[]"*, loaded by *"
> <system class loader>
> "*
>>
>>
>> 44 instances of *"com.gargoylesoftware.htmlunit.html.HtmlPage"*,
>>loaded by *"sun.misc.Launcher$AppClassLoader
>>@ 0x2b21564bd190"* occupy *187,091,104 (18.34%)* bytes.
>>
>>Biggest instances:
>>
>>   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>>   0x2b216fad9408 - 70,380,232 (6.90%) bytes.
>>   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>>   0x2b2162f66988 - 20,142,528 (1.98%) bytes.
>>   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>>   0x2b216a296c60 - 12,966,560 (1.27%) bytes.
>>   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>>   0x2b216a26c390 - 12,966,032 (1.27%) bytes.
>>   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>>   0x2b2168791e60 - 10,687,768 (1.05%) bytes.
>>   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>>   0x2b215b8b5460 - 10,651,168 (1.04%) bytes.
>>   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>>   0x2b215a8090d8 - 10,523,280 (1.03%) bytes.
>>
>>
>>
>>*Keywords*
>>sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190
>>com.gargoylesoftware.htmlunit.html.HtmlPage
>>
>>What is the meaning of these suspects?
>>How can i reduce the usage of the memory?
>>
>>Thanks,
>>David





--
View this message in context: http://htmlunit.10904.n7.nabble.com/problems-with-java-memory-when-scraping-data-from-different-hosts-tp33164p33188.html
Sent from the HtmlUnit - General mailing list archive at Nabble.com.