Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

#230 [HHQ-1128] Deleting a Platform is Slow

Improvement
closed
Hyperic Team
None
5
2014-01-06
2008-01-10
Hyperic Team
No

http://jira.hyperic.com/browse/HHQ-1128
We have a customer that is having problems with their HQ instance scheduling metrics on discovered platforms. As part of the triage exercise, we decided to delete a problematic platform to retry the platform import. This operation never succeeded. On debugging this issue locally, I took a thread dump when deleting the platform and saw that the platform removal is spending alot of time invoking MetricAuxLogManagerEJBImpl.metricDeleted(). This seems to be because the lookup for the metric aux log by metric id is on an unindexed field (eam_metric_aux_logs.metric_id). I also noticed that there were many SRN deletes that may be adding to the performance problems.



Before changing the code we should profile this operation to confirm the actual bottlenecks.



Here's the relevant thread dump:



"http-0.0.0.0-7080-2" daemon prio=5 tid=0x006b1eb0 nid=0x1ae2600 runnable [0xb1da8000..0xb1dabd10]


        at java.net.SocketInputStream.socketRead0(Native Method)


        at java.net.SocketInputStream.read(SocketInputStream.java:129)


        at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)


        at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)


        at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)


        at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:255)


        at org.postgresql.core.v2.QueryExecutorImpl.processResults(QueryExecutorImpl.java:408)


        at org.postgresql.core.v2.QueryExecutorImpl.execute(QueryExecutorImpl.java:364)


        at org.postgresql.core.v2.QueryExecutorImpl.execute(QueryExecutorImpl.java:258)


        - locked <0x0fd1ef68> (a org.postgresql.core.v2.QueryExecutorImpl)


        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:452)


        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:351)


        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:255)


        at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:76)


        at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:211)


        at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:186)


        at org.hibernate.loader.Loader.getResultSet(Loader.java:1787)


        at org.hibernate.loader.Loader.doQuery(Loader.java:674)


        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)


        at org.hibernate.loader.Loader.doList(Loader.java:2220)


        at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)


        at org.hibernate.loader.Loader.list(Loader.java:2099)


        at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378)


        at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)


        at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)


        at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)


        at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)


        at org.hyperic.hq.measurement.server.session.MetricAuxLogDAO.find(MetricAuxLogDAO.java:62)


        at org.hyperic.hq.measurement.server.session.MetricAuxLogManagerEJBImpl.metricDeleted(MetricAuxLogManagerEJBImpl.java:107)


        at sun.reflect.GeneratedMethodAccessor812.invoke(Unknown Source)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)


        at java.lang.reflect.Method.invoke(Method.java:585)


        at org.jboss.invocation.Invocation.performCall(Invocation.java:345)


        at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)


        at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)


        at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:154)


        at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:54)


        at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)


        at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)


        at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)


        at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)


        at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:245)


        at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:281)


        at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)


        at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:153)


        at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)


        at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)


        at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)


        at org.jboss.ejb.Container.invoke(Container.java:873)


        at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:415)


        at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:88)


        at $Proxy311.metricDeleted(Unknown Source)


        at org.hyperic.hq.measurement.server.session.MeasurementStartupListener$1.beforeMetricDelete(MeasurementStartupListener.java:81)


        at sun.reflect.GeneratedMethodAccessor811.invoke(Unknown Source)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)


        at java.lang.reflect.Method.invoke(Method.java:585)


        at org.hyperic.util.callback.CallbackType$1.callListeners(CallbackType.java:50)


        at org.hyperic.util.callback.CallbackDispatcher$CallbackHandler.invoke(CallbackDispatcher.java:80)


        at $Proxy354.beforeMetricDelete(Unknown Source)


        at org.hyperic.hq.measurement.server.session.DerivedMeasurementManagerEJBImpl.removeMeasurements(DerivedMeasurementManagerEJBImpl.java:526)


        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)


        at java.lang.reflect.Method.invoke(Method.java:585)


        at org.jboss.invocation.Invocation.performCall(Invocation.java:345)


        at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)


        at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)


        at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:154)


        at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:54)


        at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)


        at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)


        at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)


        at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)


        at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:245)


        at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:281)


        at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)


        at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:153)


        at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)


        at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)


        at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)


        at org.jboss.ejb.Container.invoke(Container.java:873)


        at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:415)


        at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:88)


        at $Proxy309.removeMeasurements(Unknown Source)


        at org.hyperic.hq.bizapp.server.session.AppdefBossEJBImpl.removePlatform(AppdefBossEJBImpl.java:1626)


        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)


        at java.lang.reflect.Method.invoke(Method.java:585)


        at org.jboss.invocation.Invocation.performCall(Invocation.java:345)


        at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)


        at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)


        at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:154)


        at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:54)


        at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)


        at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)


        at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)


        at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)


        at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:245)


        at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:281)


        at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)


        at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:153)


        at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)


        at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)


        at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)


        at org.jboss.ejb.Container.invoke(Container.java:873)


        at sun.reflect.GeneratedMethodAccessor548.invoke(Unknown Source)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)


        at java.lang.reflect.Method.invoke(Method.java:585)


        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)


        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)


        at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)


        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)


        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)


        at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:155)


        at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:104)


        at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:179)


        at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:165)


        at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)


        at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:241)


        at org.hyperic.hq.application.HQApp$Snatcher.invokeProxyNext(HQApp.java:274)


        at org.hyperic.txsnatch.ProxySnatch.invoke(ProxySnatch.java:37)


        at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)


        at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:97)


        at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)


        at $Proxy247.removePlatform(Unknown Source)


        at org.hyperic.hq.ui.action.resource.hub.RemoveResourceAction.removeResources(RemoveResourceAction.java:116)


        at org.hyperic.hq.ui.action.resource.hub.RemoveResourceAction.execute(RemoveResourceAction.java:83)


        at org.hyperic.hq.ui.action.BaseRequestProcessor.processActionPerform(BaseRequestProcessor.java:63)


        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)


        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)


        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)


        at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)


        at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)


        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)


        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)


        at org.hyperic.hq.ui.AuthenticationFilter.doFilter(AuthenticationFilter.java:110)


        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)


        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)


        at org.hyperic.hibernate.filter.SessionFilter$1.run(SessionFilter.java:59)


        at org.hyperic.hq.hibernate.SessionManager.runInSessionInternal(SessionManager.java:79)


        at org.hyperic.hq.hibernate.SessionManager.runInSession(SessionManager.java:68)


        at org.hyperic.hibernate.filter.SessionFilter.doFilter(SessionFilter.java:57)


        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)


        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)


        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)


        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)


        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)


        at org.hyperic.hq.product.servlet.filter.JMXFilter.doFilter(JMXFilter.java:324)


        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)


        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)


        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)


        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)


        at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)


        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:159)


        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)


        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)


        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)


        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)


        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)


        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)


        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)


        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)


        at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)


        at java.lang.Thread.run(Thread.java:613)

Discussion


Anonymous


Cancel   Add attachments