#4353 [HHQ-4224] Availability displaying green after a platform do

[Hyperic Team]

http://jira.hyperic.com/browse/HHQ-4224


Situation:


A resource type availability alert on mbeanserver was firing every 6 minutes on 11 agents since ~6 pm of Aug 17th (yesterday).


No other particular load on the server except for modification of the alert type at about ~10 am this morning to not fire.

Symptoms/issues:

1. As of noon, the platforms stopped displaying metrics in the indicator tabs. hq-server's own platform indicator shows the same behavior as the agents' but the agent threw down alert.


The availability icon still appears green hours after it stopped collecting/processing metrics.


The server log during this time repeatedly shows that the data inserter queue is backed up (more than 80,000 entries in the past couple of hours)

2010-08-18 13:46:17,074 WARN [Thread-1726776] [com.hyperic.hq.measurement.server.session.BatchAggregateDataInserter@344] No space left in the DataInserter queue. Agents are sending data in faster than we can process it, queuesize=3999961

Also, the server log has an exception thrown by the backfiller failing at 11:23 am. Server log is attached. Synposis below:

2010-08-18 11:23:28,907 ERROR [scheduler-1] [org.springframework.scheduling.support.MethodInvokingRunnable@68] Invocation of method 'backfill' on target class [class $Proxy83] fai


led


org.hyperic.hq.common.SystemException: org.springframework.transaction.TransactionTimedOutException: Transaction timed out: deadline was Wed Aug 18 11:15:03 PDT 2010


        at org.hyperic.hq.measurement.server.session.AvailabilityCheckServiceImpl.backfill(AvailabilityCheckServiceImpl.java:217)


        at org.hyperic.hq.measurement.server.session.AvailabilityCheckServiceImpl.backfill(AvailabilityCheckServiceImpl.java:164)


        at sun.reflect.GeneratedMethodAccessor855.invoke(Unknown Source)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)


        at java.lang.reflect.Method.invoke(Method.java:597)


        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)


        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)


        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)


        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)


        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)


        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)


        at $Proxy83.backfill(Unknown Source)


        at sun.reflect.GeneratedMethodAccessor854.invoke(Unknown Source)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)


        at java.lang.reflect.Method.invoke(Method.java:597)


        at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)


        at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:65)


        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:51)


        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)


        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)


        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)


        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)


        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)


        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)


        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)


        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)


        at java.lang.Thread.run(Thread.java:619)


Caused by: org.springframework.transaction.TransactionTimedOutException: Transaction timed out: deadline was Wed Aug 18 11:15:03 PDT 2010


        at org.springframework.transaction.support.ResourceHolderSupport.checkTransactionTimeout(ResourceHolderSupport.java:141)


        at org.springframework.transaction.support.ResourceHolderSupport.getTimeToLiveInMillis(ResourceHolderSupport.java:130)


        at org.springframework.transaction.support.ResourceHolderSupport.getTimeToLiveInSeconds(ResourceHolderSupport.java:114)


        at org.springframework.jdbc.datasource.DataSourceUtils.applyTimeout(DataSourceUtils.java:255)


        at org.springframework.jdbc.datasource.DataSourceUtils.applyTransactionTimeout(DataSourceUtils.java:237)


        at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:234)


        at $Proxy20.prepareStatement(Unknown Source)


        at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:505)


        at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:423)


        at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)


        at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)


        at org.hibernate.loader.Loader.doQuery(Loader.java:673) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)


        at org.hibernate.loader.Loader.doList(Loader.java:2213)


        at org.hibernate.loader.Loader.listUsingQueryCache(Loader.java:2136)


        at org.hibernate.loader.Loader.list(Loader.java:2096)


        at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378)


        at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)


        at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)

Load on server increased during this time and CPU usage also increased about 75%

Attaching logs and screenshots