#4521 [HHQ-4487] Unidirectional agent ran out of memory while moni

[Hyperic Team]

http://jira.hyperic.com/browse/HHQ-4487
Configure a unidirectional agent to monitor 1000 http service checks and HQ Tomcat server.


Update collection interval for HTTP service availability and Response Code to 1 minute (this effectively makes the agent collect ~1500 metrics/minute)

After a while (minutes or hours) the agent throws OOM error in agent.log and JSW ends up restarting agent. This keeps on repeating every few minutes. The stack in agent log with OOM varies. Following are 2 versions I have seen.

Note: This issue is not reproducible on bi-directional agent. It only happens on uni-directional agent.

java.lang.OutOfMemoryError: Java heap space


        at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)


        at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:746)


        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)


        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)


        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)


        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)


        at org.hyperic.hq.plugin.netservices.HTTPCollector.collect(HTTPCollector.java:320)


        at org.hyperic.hq.product.Collector.run(Collector.java:563)


        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)


        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)


        at java.lang.Thread.run(Thread.java:619)

2010-11-30 13:26:30,497 ERROR [pool-1-thread-26] [Collector] Error running http collector: java.lang.OutOfMemoryError: GC overhead limit exceeded


java.lang.OutOfMemoryError: GC overhead limit exceeded


        at java.util.Arrays.copyOf(Arrays.java:2786)


        at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)


        at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:87)


        at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)


        at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)


        at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)


        at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)


        at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)


        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)


        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)


        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)


        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)


        at org.hyperic.hq.plugin.netservices.HTTPCollector.collect(HTTPCollector.java:320)


        at org.hyperic.hq.product.Collector.run(Collector.java:563)


        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)


        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)


        at java.lang.Thread.run(Thread.java:619)