Re: [Htmlunit-user] problems with java memory when scraping data from different hosts

[David M. G. <mic...@gm...>]

Hi,

I checked the memory heap after running a massive load of urls and did not
find any memory leak with eclipse mat.

Well Done :-)

I am just curious what the problem was and how you solved it.

Thanks,
David




On Thu, Feb 13, 2014 at 10:55 AM, David Michael Gang
<mic...@gm...>wrote:

> Hi,
>
> Would it be possible to upload this snapshot to maven?
> I have a jenkins job which generates the project i run through maven and
> it is hard for me to change these dependencies to look on a static jar.
>
> Thanks,
> David
>
>
>
>
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Wed, 12 Feb 2014 18:12:00 +0100 (CET)
>> From: "Ronald Brill" <rb...@rb...>
>> Subject: Re: [Htmlunit-user] problems with java memory when scraping
>>         data    from different hosts.
>> To: "htm...@li..."
>>         <htm...@li...>
>> Message-ID: <100...@rb...>
>> Content-Type: text/plain; charset="iso-8859-15"
>>
>> HI David,
>>
>> i guess you are facing a memory issue of the current implementation.
>> I did a first shot trying to solve one part of the problem in SVN commit
>> 9127.
>> Please give this version an try and report back.
>>
>>         RBRi
>>
>> On Wed, 12 Feb 2014 09:28:46 +0200 David Michael Gang wrote:
>> >
>> >Hi all,
>> >
>> >I have a script which goes over ~1000 links and fetches the data from
>> them.
>> >I have javascript switched on.
>> >The heap size grows up to about 1.5 GB.
>> >I performed jmap to get a .hprof file.
>> >Afterwards i analyzed the results with eclipsemat and got the following
>> >suspects
>> >
>> >836,303 instances of
>>
>> >*"com.gargoylesoftware.htmlunit.javascript.host.NodeList$DomHtmlAttributeChangeListenerImpl"*,
>> >loaded by *"sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190"* occupy
>> >*360,618,336
>> >(35.36%)* bytes. These instances are referenced from one instance of
>> >*"java.lang.Object[]"*, loaded by *"<system class loader>"*
>> >
>> >
>> > 44 instances of *"com.gargoylesoftware.htmlunit.html.HtmlPage"*,
>> >loaded by *"sun.misc.Launcher$AppClassLoader
>> >@ 0x2b21564bd190"* occupy *187,091,104 (18.34%)* bytes.
>> >
>> >Biggest instances:
>> >
>> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>> >   0x2b216fad9408 - 70,380,232 (6.90%) bytes.
>> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>> >   0x2b2162f66988 - 20,142,528 (1.98%) bytes.
>> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>> >   0x2b216a296c60 - 12,966,560 (1.27%) bytes.
>> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>> >   0x2b216a26c390 - 12,966,032 (1.27%) bytes.
>> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>> >   0x2b2168791e60 - 10,687,768 (1.05%) bytes.
>> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>> >   0x2b215b8b5460 - 10,651,168 (1.04%) bytes.
>> >   - com.gargoylesoftware.htmlunit.html.HtmlPage @
>> >   0x2b215a8090d8 - 10,523,280 (1.03%) bytes.
>> >
>> >
>> >
>> >*Keywords*
>> >sun.misc.Launcher$AppClassLoader @ 0x2b21564bd190
>> >com.gargoylesoftware.htmlunit.html.HtmlPage
>> >
>> >What is the meaning of these suspects?
>> >How can i reduce the usage of the memory?
>> >
>> >Thanks,
>> >David
>> >
>>
>>
>>
>>
>> ------------------------------
>>
>>
>> ------------------------------------------------------------------------------
>> Android apps run on BlackBerry 10
>> Introducing the new BlackBerry 10.2.1 Runtime for Android apps.
>> Now with support for Jelly Bean, Bluetooth, Mapview and more.
>> Get your Android app in front of a whole new audience.  Start now.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk
>>
>> ------------------------------
>>
>> _______________________________________________
>> Htmlunit-user mailing list
>> Htm...@li...
>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>
>>
>> End of Htmlunit-user Digest, Vol 93, Issue 21
>> *********************************************
>>
>
>