[opennms-discuss] Error creating graph for JVM SNMP monitor

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi all,

I have recently got SNMP monitoring of a JBoss 7 service working, and gathering data as per this link: https://wiki.opennms.org/wiki/JVM_Monitoring_using_SNMP

All the graphs are working except for the JVM Memory Pool graph, which is giving the following errors in the web.log:

2017-04-05 08:44:50,362 ERROR [qtp1718653452-405] o.o.n.r.j.JRobinRrdStrategy: JRobin: exception occurred creating graph
org.jrobin.core.RrdException: Unknown datasource name: jvmMemPoolUsed
        at org.jrobin.core.RrdDb.getDsIndex(RrdDb.java:746) ~[jrobin-1.6.0.jar:1.6.0]
        at org.jrobin.core.Archive.fetchData(Archive.java:304) ~[jrobin-1.6.0.jar:1.6.0]
        at org.jrobin.core.RrdDb.fetchData(RrdDb.java:609) ~[jrobin-1.6.0.jar:1.6.0]
        at org.jrobin.core.FetchRequest.fetchData(FetchRequest.java:184) ~[jrobin-1.6.0.jar:1.6.0]
        at org.jrobin.data.DataProcessor.fetchRrdData(DataProcessor.java:722) ~[jrobin-1.6.0.jar:1.6.0]
        at org.jrobin.data.DataProcessor.processData(DataProcessor.java:564) ~[jrobin-1.6.0.jar:1.6.0]
        at org.jrobin.graph.RrdGraph.fetchData(RrdGraph.java:537) ~[jrobin-1.6.0.jar:1.6.0]
        at org.jrobin.graph.RrdGraph.createGraph(RrdGraph.java:74) ~[jrobin-1.6.0.jar:1.6.0]
        at org.jrobin.graph.RrdGraph.<init>(RrdGraph.java:53) ~[jrobin-1.6.0.jar:1.6.0]
        at org.opennms.netmgt.rrd.jrobin.JRobinRrdStrategy.createGraphReturnDetails(JRobinRrdStrategy.java:381) [opennms-rrd-jrobin-19.0.1.jar:?]
        at org.opennms.netmgt.rrd.jrobin.JRobinRrdStrategy.createGraph(JRobinRrdStrategy.java:359) [opennms-rrd-jrobin-19.0.1.jar:?]
        at org.opennms.netmgt.rrd.QueuingRrdStrategy.createGraph(QueuingRrdStrategy.java:1056) [opennms-rrd-api-19.0.1.jar:?]
        at org.opennms.netmgt.dao.support.DefaultRrdDao.createGraph(DefaultRrdDao.java:218) [opennms-dao-19.0.1.jar:?]
        at org.opennms.web.svclayer.support.DefaultRrdGraphService.getInputStreamForCommand(DefaultRrdGraphService.java:120) [opennms-web-api-19.0.1.jar:?]
        at org.opennms.web.svclayer.support.DefaultRrdGraphService.getPrefabGraph(DefaultRrdGraphService.java:173) [opennms-web-api-19.0.1.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) [org.apache.servicemix.bundles.spring-aop-4.0.7.RELEASE_1.jar:?]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [org.apache.servicemix.bundles.spring-aop-4.0.7.RELEASE_1.jar:?]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [org.apache.servicemix.bundles.spring-aop-4.0.7.RELEASE_1.jar:?]
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98) [org.apache.servicemix.bundles.spring-tx-4.0.7.RELEASE_1.jar:?]
        ...

the jvmMemPoolUsed is the only "Unknown datasource name" being reported in the log.

In datacollection, I have the following defined:

<datacollection-group name="ABBCS-Jboss">

   <resourceType name="jvmMemManagerIndex" label="JVM GC Stats" resourceLabel="${jvmMemManagerName} (index ${index})">
        <persistenceSelectorStrategy class="org.opennms.netmgt.collection.support.PersistAllSelectorStrategy"/>
        <storageStrategy class="org.opennms.netmgt.collection.support.IndexStorageStrategy"/>
   </resourceType>

   <resourceType name="jvmMemPoolIndex" label="JVM Memory Pool Stats" resourceLabel="Memory Pool - ${jvmMemPoolName} (index ${index})">
        <persistenceSelectorStrategy class="org.opennms.netmgt.collection.support.PersistAllSelectorStrategy"/>
        <storageStrategy class="org.opennms.netmgt.collection.support.IndexStorageStrategy"/>
  </resourceType>

      <!-- Jboss MIB's -->

    <group name="ABBCS-Jboss" ifType="ignore">
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.11" instance="0" alias="jvmHeapUsed" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.12" instance="0" alias="jvmHeapCommitted" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.13" instance="0" alias="jvmHeapMax" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.21" instance="0" alias="jvmNonHeapUsed" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.22" instance="0" alias="jvmNonHeapCommitted" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.23" instance="0" alias="jvmNonHeapMax" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.3.1" instance="0" alias="jvmThreadCount" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.4.11" instance="0" alias="jvmRTUptimeMs" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.100.1.2"  instance="jvmMemManagerIndex" alias="jvmMemManagerName" type="string" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.101.1.2"  instance="jvmMemManagerIndex" alias="jvmMemGCCount" type="Counter64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.101.1.3"  instance="jvmMemManagerIndex" alias="jvmMemGCTimeMs" type="Counter64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.2"  instance="jvmMemPoolIndex" alias="jvmMemPoolName" type="string" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.10" instance="jvmMemPoolIndex" alias="jvmMemPoolInitSize" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11" instance="jvmMemPoolIndex" alias="jvmMemPoolUsed" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.12" instance="jvmMemPoolIndex" alias="jvmMemPoolCommit" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.13" instance="jvmMemPoolIndex" alias="jvmMemPoolMaxSize" type="Gauge64" />
    </group>

    <systemDef name="ABBCS-Jboss">
       <sysoidMask>.1.3.6.1.4.1.8072.3.</sysoidMask>
      <collect>
        <includeGroup>ABBCS-Jboss</includeGroup>
      </collect>
    </systemDef>
</datacollection-group>

And my graph definition is as follows:
report.jvm.mempool.name=JVM Memory Pool
report.jvm.mempool.columns=jvmMemPoolInitSize, jvmMemPoolUsed, jvmMemPoolMaxSize, jvmMemPoolCommit
report.jvm.mempool.type=jvmMemPoolIndex
report.jvm.mempool.propertiesValues=jvmMemPoolName
report.jvm.mempool.command=--title="JVM Memory Pool - {jvmMemPoolName}" \
--vertical-label="Bytes" \
--base=1024 \
DEF:used={rrd1}:jvmMemPoolUsed:AVERAGE \
DEF:max={rrd2}:jvmMemPoolMaxSize:AVERAGE \
DEF:init={rrd3}:jvmMemPoolInitSize:AVERAGE \
DEF:commit={rrd4}:jvmMemPoolCommit:AVERAGE \
AREA:used#00a876:"Used     " \
GPRINT:used:AVERAGE:" Average  \\: %5.2lf %s" \
GPRINT:used:MIN:"Min  \\: %5.2lf %s" \
GPRINT:used:MAX:"Max  \\: %5.2lf %s\\n" \
LINE2:max#FF5900:"Max      " \
GPRINT:max:AVERAGE:" Average  \\: %5.2lf %s" \
GPRINT:max:MIN:"Min  \\: %5.2lf %s" \
GPRINT:max:MAX:"Max  \\: %5.2lf %s\\n" \
LINE2:commit#1047a9:"Committed" \
GPRINT:commit:AVERAGE:" Average  \\: %5.2lf %s" \
GPRINT:commit:MIN:"Min  \\: %5.2lf %s" \
GPRINT:commit:MAX:"Max  \\: %5.2lf %s\\n" \
LINE2:init#000000:"Initial  " \
GPRINT:init:AVERAGE:" Average  \\: %5.2lf %s" \
GPRINT:init:MIN:"Min  \\: %5.2lf %s" \
GPRINT:init:MAX:"Max  \\: %5.2lf %s\\n"

jvmMemPoolUsed is definitely being collected:

2017-04-05 08:56:58,659 DEBUG [Collectd-Thread-5-of-50] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute node[19].jvmMemPoolIndex[2].jvmMemPoolUsed [.1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11] = 372296440
2017-04-05 08:56:58,659 DEBUG [Collectd-Thread-5-of-50] o.o.n.c.a.AbstractPersister: Persisting node[19].jvmMemPoolIndex[2].jvmMemPoolUsed [.1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11] = 372296440
2017-04-05 08:56:58,659 DEBUG [Collectd-Thread-5-of-50] o.o.n.c.a.AbstractPersister: Storing attribute node[19].jvmMemPoolIndex[2].jvmMemPoolUsed [.1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11] = 372296440
2017-04-05 08:56:58,659 DEBUG [Collectd-Thread-5-of-50] o.o.n.r.j.JRobinRrdStrategy: createDefinition: filename [/opt/opennms/share/rrd/snmp/19/jvmMemPoolIndex/2/jvmMemPoolUsed.jrb] already exists returning null as definition
2017-04-05 08:56:58,659 DEBUG [Collectd-Thread-5-of-50] o.o.n.r.j.JRobinRrdStrategy: createRRD: skipping RRD file
2017-04-05 08:56:58,659 INFO  [Collectd-Thread-5-of-50] o.o.n.c.p.r.RrdPersistOperationBuilder: updateRRD: updating RRD file /opt/opennms/share/rrd/snmp/19/jvmMemPoolIndex/2/jvmMemPoolUsed.jrb with values '1491346619:372296440'
2017-04-05 08:56:58,659 DEBUG [Collectd-Thread-5-of-50] o.o.n.c.p.r.RrdPersistOperationBuilder: updateRRD: RRD update command completed.

I tried swapping the "DEF" Lines around so that a different data source was 1st, and the error reported was the new 1st DEF entry

DEF:max={rrd2}:jvmMemPoolMaxSize:AVERAGE \
DEF:used={rrd1}:jvmMemPoolUsed:AVERAGE \
DEF:init={rrd3}:jvmMemPoolInitSize:AVERAGE \
DEF:commit={rrd4}:jvmMemPoolCommit:AVERAGE \

With this definition, the undefined datasource was "jvmMemPoolMaxSize", so it seems that the report is erroring out when it hits the 1st DEF line.

Any ideas on why my graph is not working?

Regards,
John Blackburn.

[ABB logo]

John Blackburn

Solution Architect - Cloud Services

Enterprise Software

Level 1, 757 Ann Street

Fortitude Valley, Queensland, 4006
AUSTRALIA
Phone: +61 7 33033219

Mobile: +61 403 344 524
E-Mail:  joh...@au...<mailto:joh...@au...>
www.abb.com<http://www.abb.com/>

[opennms-discuss] Error creating graph for JVM SNMP monitor

A Java based fault and performance management system

[opennms-discuss] Error creating graph for JVM SNMP monitor