[Htmlunit-user] problems with java memory when scraping data from different hosts.

[David M. G. <mic...@gm...>]

Hi all,

I have a script which goes over ~1000 links and fetches the data from them.
I have javascript switched on.
The heap size grows up to about 1.5 GB.
I performed jmap to get a .hprof file.
Afterwards i analyzed the results with eclipsemat and got the following
suspects

836,303 instances of
*"com.gargoylesoftware.htmlunit.javascript.host.NodeList$DomHtmlAttributeChangeListenerImpl"*,
loaded by *"sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190"* occupy
*360,618,336
(35.36%)* bytes. These instances are referenced from one instance of
*"java.lang.Object[]"*, loaded by *"<system class loader>"*


 44 instances of *"com.gargoylesoftware.htmlunit.html.HtmlPage"*,
loaded by *"sun.misc.Launcher$AppClassLoader
@ 0x2b21564bd190"* occupy *187,091,104 (18.34%)* bytes.

Biggest instances:

   - com.gargoylesoftware.htmlunit.html.HtmlPage @
   0x2b216fad9408 - 70,380,232 (6.90%) bytes.
   - com.gargoylesoftware.htmlunit.html.HtmlPage @
   0x2b2162f66988 - 20,142,528 (1.98%) bytes.
   - com.gargoylesoftware.htmlunit.html.HtmlPage @
   0x2b216a296c60 - 12,966,560 (1.27%) bytes.
   - com.gargoylesoftware.htmlunit.html.HtmlPage @
   0x2b216a26c390 - 12,966,032 (1.27%) bytes.
   - com.gargoylesoftware.htmlunit.html.HtmlPage @
   0x2b2168791e60 - 10,687,768 (1.05%) bytes.
   - com.gargoylesoftware.htmlunit.html.HtmlPage @
   0x2b215b8b5460 - 10,651,168 (1.04%) bytes.
   - com.gargoylesoftware.htmlunit.html.HtmlPage @
   0x2b215a8090d8 - 10,523,280 (1.03%) bytes.



*Keywords*
sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190
com.gargoylesoftware.htmlunit.html.HtmlPage

What is the meaning of these suspects?
How can i reduce the usage of the memory?

Thanks,
David