Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

/usr/local/nagios/var/perfdata.log file issue

Help
2012-07-12
2013-05-20
  • Sundaramoorthi
    Sundaramoorthi
    2012-07-12

    I have configured nagios graph around 4 months back and it was working fine till 1 week before. Suddenly it stopped working and /usr/local/nagios/var/perfdata.log file is empty most of the time.

    Following is the output from nagios.debug file.

       Expanded Command Output: /usr/local/nagios/bin/insert.pl
       Running command '/usr/local/nagios/bin/insert.pl'…
       Execution time=0.091 sec, early timeout=0, result=0, output=(null)

    following is output from nagiosgraph.log

    hostname:bdhw8k24.debenhams.com
    servicedesc:D:\ Drive Space
    output:d:\ - total: 60.00 Gb - used: 51.44 Gb (86%) - free 8.56 Gb (14%)
    perfdata:d:\ Used Space=51.44Gb;48.00;54.00;0.00;60.00
    Thu Jul 12 08:16:28 2012 insert.pl debug createrrd(1342077371,bdhw8k24.debenhams.com,D:\ Drive Space,d:\ Used Space)
    Thu Jul 12 08:16:28 2012 insert.pl debug createrrd rrdfile is /usr/local/nagios/var/rrd/bdhw8k24.debenhams.com/D%3A%5C%20Drive%20Space___d%3A%5C%20Used%20Space.rrd
    Thu Jul 12 08:16:28 2012 insert.pl debug createrrd step=300 heartbeat=600 xff=0.5 resolutions=4 steps=1 6 24 288
    Thu Jul 12 08:16:28 2012 insert.pl debug labels-> = [
      'data',
      'GAUGE',
      '51.44'
    ];
    Thu Jul 12 08:16:28 2012 insert.pl debug labels-> = [
      'warn',
      'GAUGE',
      '48.00'
    ];
    Thu Jul 12 08:16:28 2012 insert.pl debug labels-> = [
      'crit',
      'GAUGE',
      '54.00'
    ];
    Thu Jul 12 08:16:28 2012 insert.pl debug labels-> = [
      'min',
      'GAUGE',
      '0.00'
    ];
    Thu Jul 12 08:16:28 2012 insert.pl debug labels-> = [
      'max',
      'GAUGE',
      '60.00'
    ];
    Thu Jul 12 08:16:28 2012 insert.pl debug createrrd filenames = [
      'D%3A%5C%20Drive%20Space___d%3A%5C%20Used%20Space.rrd'
    ];
    Thu Jul 12 08:16:28 2012 insert.pl debug createrrd datasets = [
      [
        0,
        1,
        2,
        3,
        4
      ]
    ];
    Thu Jul 12 08:16:28 2012 insert.pl info runupdate dataset = [
      '/usr/local/nagios/var/rrd/bdhw8k24.debenhams.com/D%3A%5C%20Drive%20Space___d%3A%5C%20Used%20Space.rrd',
      '1342077372:51.44:48.00:54.00:0.00:60.00'
    ];
    Thu Jul 12 08:16:28 2012 insert.pl info processed 1 of 1 lines
    Thu Jul 12 08:16:28 2012 insert.pl debug insert.pl processing complete
    Thu Jul 12 08:16:58 2012 insert.pl debug insert.pl processing started
    Thu Jul 12 08:16:58 2012 insert.pl debug getrules(/usr/local/nagios/etc/map)
    Thu Jul 12 08:16:58 2012 insert.pl debug readperfdata: perflog is /usr/local/nagios/var/perfdata.log
    Thu Jul 12 08:16:58 2012 insert.pl info empty perflog /usr/local/nagios/var/perfdata.log
    Thu Jul 12 08:16:58 2012 insert.pl debug insert.pl processing complete

    Please help me to resolve this issue as soon as possible.

     
  • Sundaramoorthi
    Sundaramoorthi
    2012-07-12

    it's me again.

    nagiosgraph is not graphing anything now. I am monitoring more 150 servers and around 1000 services.

     
  • Ivan
    Ivan
    2012-07-12

    Do you have anything interesting in httpd_error.log and nagiosgraph-cgi.log files? Had you changed anything before even occured? Check files and folders permissions, please, check SeLinux, check even a disk space your rrd files keeping. It hard to say anything without info from log files.

     
  • Sundaramoorthi
    Sundaramoorthi
    2012-07-16

    Below are the error log i found it from "nagiosgraph.log"

    Mon Jul 16 05:51:41 2012 insert.pl warn output/perfdata not recognized:
    hostname:bdhw3kmg.debenhams.com
    servicedesc:Meteor Service
    output:MeteorServer.exe: Running
    perfdata:undefined_perfdata
    Mon Jul 16 05:51:41 2012 insert.pl info processed 0 of 1 lines
    Mon Jul 16 05:51:41 2012 insert.pl debug insert.pl processing complete
    Mon Jul 16 05:52:11 2012 insert.pl debug insert.pl processing started
    Mon Jul 16 05:52:11 2012 insert.pl debug getrules(/usr/local/nagios/etc/map)
    Mon Jul 16 05:52:11 2012 insert.pl debug readperfdata: perflog is /usr/local/nagios/var/perfdata.log
    Mon Jul 16 05:52:11 2012insert.pl info empty perflog /usr/local/nagios/var/perfdata.log

    Found the below content from "nagiosgraph-cgi.log"

    Mon Jul 16 06:02:52 2012 showgraph.cgi debug readfile(/usr/local/nagios/etc/labels.conf, 0)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug rrdline params = {
      'geom' => '',
      'db' => [
        'c:\\ Used Space'
      ],
      'graphonly' => '',
      'showtitle' => '',
      'showdesc' => '',
      'group' => '',
      'fixedscale' => '',
      'expand_period' => '',
      'offset' => 0,
      'expand_controls' => '',
      'rrdopts' => ' -snow-3024000 -enow-0',
      'period' => '',
      'hidelegend' => '',
      'service' => 'C:\\ Drive Space',
      'host' => 'bdhw3kmg.debenhams.com',
      'showgraphtitle' => '',
      'label' =>
    };
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug graphinfo: host=bdhw3kmg.debenhams.com service=C:\ Drive Space
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug graphinfo: db = [
      'c:\\ Used Space'
    ];
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug graphinfo: Specified bdhw3kmg.debenhams.com/C%3A%5C%20Drive%20Space___ db files in /usr/local/nagios/var/rrd: bdhw3kmg.debenhams.com/C%3A%5C%20Drive%20Space___c%3A%5C%20Used%20Space.rrd
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug graphinfo: DS bdhw3kmg.debenhams.com/C%3A%5C%20Drive%20Space___c%3A%5C%20Used%20Space.rrd lines: warn, crit, min, max, data
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug graphinfo: rrd = [
      {
        'file' => 'bdhw3kmg.debenhams.com/C%3A%5C%20Drive%20Space___c%3A%5C%20Used%20Space.rrd',
        'line' => {
          'warn' => 1,
          'crit' => 1,
          'min' => 1,
          'max' => 1,
          'data' => 1
        },
        'dbname' => 'c:\\ Used Space'
      }
    ];
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug rrdline: this graphinfo entry = {
      'file' => 'bdhw3kmg.debenhams.com/C%3A%5C%20Drive%20Space___c%3A%5C%20Used%20Space.rrd',
      'line' => {
        'warn' => 1,
        'crit' => 1,
        'min' => 1,
        'max' => 1,
        'data' => 1
      },
      'dbname' => 'c:\\ Used Space'
    };
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug setlabels(bdhw3kmg.debenhams.com, C:\ Drive Space, c:\ Used Space, crit, 19)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug setdata(C:\ Drive Space, c:\ Used Space, crit, , 3024000)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug setlabels(bdhw3kmg.debenhams.com, C:\ Drive Space, c:\ Used Space, data, 19)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug hashcolor(9)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug hashcolor: returning color = 90d080
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug setdata(C:\ Drive Space, c:\ Used Space, data, , 3024000)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug setlabels(bdhw3kmg.debenhams.com, C:\ Drive Space, c:\ Used Space, max, 19)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug hashcolor(9)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug hashcolor: returning color = 30a030
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug setdata(C:\ Drive Space, c:\ Used Space, max, , 3024000)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug setlabels(bdhw3kmg.debenhams.com, C:\ Drive Space, c:\ Used Space, min, 19)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug hashcolor(9)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug hashcolor: returning color = 90c0e0
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug setdata(C:\ Drive Space, c:\ Used Space, min, , 3024000)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug setlabels(bdhw3kmg.debenhams.com, C:\ Drive Space, c:\ Used Space, warn, 19)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug setdata(C:\ Drive Space, c:\ Used Space, warn, , 3024000)
    Mon Jul 16 06:02:52 2012 showgraph.cgi debug RRDs::graph = [
      '-',
      '-a',
      'PNG',
      'DEF:c___Used_Space_crit=/usr/local/nagios/var/rrd/bdhw3kmg.debenhams.com/C%3A%5C%20Drive%20Space___c%3A%5C%20Used%20Space.rrd:crit:AVERAGE',
      'LINE1:c___Used_Space_crit#D05050:Critical           ',
      'GPRINT:c___Used_Space_crit:MAX:Max\\:%7.2lf%s',
      'GPRINT:c___Used_Space_crit:AVERAGE:Avg\\:%7.2lf%s',
      'GPRINT:c___Used_Space_crit:MIN:Min\\:%7.2lf%s\\n',
      'DEF:c___Used_Space_data=/usr/local/nagios/var/rrd/bdhw3kmg.debenhams.com/C%3A%5C%20Drive%20Space___c%3A%5C%20Used%20Space.rrd:data:AVERAGE',
      'LINE2:c___Used_Space_data#90d080:c\\:\\\\ Used Space,data',
      'GPRINT:c___Used_Space_data:MAX:Max\\:%7.2lf%s',
      'GPRINT:c___Used_Space_data:AVERAGE:Avg\\:%7.2lf%s',
      'GPRINT:c___Used_Space_data:MIN:Min\\:%7.2lf%s\\n',
      'DEF:c___Used_Space_max=/usr/local/nagios/var/rrd/bdhw3kmg.debenhams.com/C%3A%5C%20Drive%20Space___c%3A%5C%20Used%20Space.rrd:max:AVERAGE',
      'LINE2:c___Used_Space_max#30a030:Maximum            ',
      'GPRINT:c___Used_Space_max:MAX:Max\\:%7.2lf%s',
      'GPRINT:c___Used_Space_max:AVERAGE:Avg\\:%7.2lf%s',
      'GPRINT:c___Used_Space_max:MIN:Min\\:%7.2lf%s\\n',
      'DEF:c___Used_Space_min=/usr/local/nagios/var/rrd/bdhw3kmg.debenhams.com/C%3A%5C%20Drive%20Space___c%3A%5C%20Used%20Space.rrd:min:AVERAGE',
      'LINE2:c___Used_Space_min#90c0e0:Minimum            ',
      'GPRINT:c___Used_Space_min:MAX:Max\\:%7.2lf%s',
      'GPRINT:c___Used_Space_min:AVERAGE:Avg\\:%7.2lf%s',
      'GPRINT:c___Used_Space_min:MIN:Min\\:%7.2lf%s\\n',
      'DEF:c___Used_Space_warn=/usr/local/nagios/var/rrd/bdhw3kmg.debenhams.com/C%3A%5C%20Drive%20Space___c%3A%5C%20Used%20Space.rrd:warn:AVERAGE',
      'LINE1:c___Used_Space_warn#D0D050:Warning            ',
      'GPRINT:c___Used_Space_warn:MAX:Max\\:%7.2lf%s',
      'GPRINT:c___Used_Space_warn:AVERAGE:Avg\\:%7.2lf%s',
      'GPRINT:c___Used_Space_warn:MIN:Min\\:%7.2lf%s\\n',
      '-w',
      '600 ',
      '-snow-3024000',
      '-enow-0'
    ];

    File/Folder permissions

    /usr/local/nagios/var # ls -ld .
    drwxrwxrwx 6 nagios nagios 472 Jul 16 06:07 .

    /usr/local/nagios/var # ll
    total 459542
    drwxrwxrwx  2 nagios nagios      4368 Jul 16 00:00 archives
    -rw-rw-rw-  1 nagios nagios    934880 Jul 16 06:06 nagios.debug
    -rw-rw-r-  1 nagios nagios   1000046 Jul 16 02:10 nagios.debug.old
    -rw-r-r-  1 nagios nagios         5 Jul 15 15:04 nagios.lock
    -rwxrwxrwx  1 nagios nagios    103447 Jul 16 06:04 nagios.log
    -rwxrwxrwx  1 wwwrun root      299335 Jul 16 06:02 nagiosgraph-cgi.log
    -rw-rw-rw-  1 nagios nagios 465095045 Jul 16 06:06 nagiosgraph.log
    -rwxrwxrwx  1 nagios nagios    692877 Jul 15 15:04 objects.cache
    -rw-rw-r-  1 nagios nagios         0 Jul 16 06:03 perfdata.log
    -rw-----  1 nagios nagios    990101 Jul 16 06:04 retention.dat
    drwxrwxrwx 98 nagios nagios      3888 Jul 10 04:30 rrd
    drwxrwsr-x  2 nagios nagios        80 Jul 15 15:04 rw
    drwxrwxrwx  3 nagios nagios        80 Mar 14 09:30 spool
    -rw-rw-r-  1 nagios nagios    965174 Jul 16 06:06 status.dat

    /usr/local/nagios/var # df -h .
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/system-usr
                           20G  8.7G   12G  44% /usr

     
  • Sundaramoorthi
    Sundaramoorthi
    2012-07-19

    Could some one please help me on this?

     
  • Sundaramoorthi
    Sundaramoorthi
    2012-07-23

    Hi mwall,

    Could you please help in this. It is very urgent situation for me.