Re: [Htmlunit-user] problems with java memory when scraping data from different hosts

[David M. G. <mic...@gm...>]

Hi,

Would it be possible to upload this snapshot to maven?
I have a jenkins job which generates the project i run through maven and it
is hard for me to change these dependencies to look on a static jar.

Thanks,
David




>
> ------------------------------
>
> Message: 5
> Date: Wed, 12 Feb 2014 18:12:00 +0100 (CET)
> From: "Ronald Brill" <rb...@rb...>
> Subject: Re: [Htmlunit-user] problems with java memory when scraping
>         data    from different hosts.
> To: "htm...@li..."
>         <htm...@li...>
> Message-ID: <100...@rb...>
> Content-Type: text/plain; charset="iso-8859-15"
>
> HI David,
>
> i guess you are facing a memory issue of the current implementation.
> I did a first shot trying to solve one part of the problem in SVN commit
> 9127.
> Please give this version an try and report back.
>
>         RBRi
>
> On Wed, 12 Feb 2014 09:28:46 +0200 David Michael Gang wrote:
> >
> >Hi all,
> >
> >I have a script which goes over ~1000 links and fetches the data from
> them.
> >I have javascript switched on.
> >The heap size grows up to about 1.5 GB.
> >I performed jmap to get a .hprof file.
> >Afterwards i analyzed the results with eclipsemat and got the following
> >suspects
> >
> >836,303 instances of
>
> >*"com.gargoylesoftware.htmlunit.javascript.host.NodeList$DomHtmlAttributeChangeListenerImpl"*,
> >loaded by *"sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190"* occupy
> >*360,618,336
> >(35.36%)* bytes. These instances are referenced from one instance of
> >*"java.lang.Object[]"*, loaded by *"<system class loader>"*
> >
> >
> > 44 instances of *"com.gargoylesoftware.htmlunit.html.HtmlPage"*,
> >loaded by *"sun.misc.Launcher$AppClassLoader
> >@ 0x2b21564bd190"* occupy *187,091,104 (18.34%)* bytes.
> >
> >Biggest instances:
> >
> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
> >   0x2b216fad9408 - 70,380,232 (6.90%) bytes.
> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
> >   0x2b2162f66988 - 20,142,528 (1.98%) bytes.
> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
> >   0x2b216a296c60 - 12,966,560 (1.27%) bytes.
> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
> >   0x2b216a26c390 - 12,966,032 (1.27%) bytes.
> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
> >   0x2b2168791e60 - 10,687,768 (1.05%) bytes.
> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
> >   0x2b215b8b5460 - 10,651,168 (1.04%) bytes.
> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
> >   0x2b215a8090d8 - 10,523,280 (1.03%) bytes.
> >
> >
> >
> >*Keywords*
> >sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190
> >com.gargoylesoftware.htmlunit.html.HtmlPage
> >
> >What is the meaning of these suspects?
> >How can i reduce the usage of the memory?
> >
> >Thanks,
> >David
> >
>
>
>
>
> ------------------------------
>
>
> ------------------------------------------------------------------------------
> Android apps run on BlackBerry 10
> Introducing the new BlackBerry 10.2.1 Runtime for Android apps.
> Now with support for Jelly Bean, Bluetooth, Mapview and more.
> Get your Android app in front of a whole new audience.  Start now.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk
>
> ------------------------------
>
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>
> End of Htmlunit-user Digest, Vol 93, Issue 21
> *********************************************
>