[ net-snmp-Bugs-3408398 ] memory leak in 5.7

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Bugs item #3408398, was opened at 2011-09-13 07:10
Message generated for change (Comment added) made by mmartinec
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=112694&aid=3408398&group_id=12694

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: agent
Group: freeBSD
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: drookie (drookie)
Assigned to: Nobody/Anonymous (nobody)
Summary: memory leak in 5.7

Initial Comment:
SNMP Version:

# snmpd -v

NET-SNMP version:  5.7
Web:               http://www.net-snmp.org/
Email:             net...@li...

FreeBSD:
# uname -a
FreeBSD ural85-gw0-alpha 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #0: Sun Feb  6 12:34:52 YEKT 2011     emz@ural85-gw0-alpha:/usr/obj/usr/src/sys/CRYSTAL  amd64

After upgrading from 5.5 to 5.7 got a memory leak.
It is stable and reproduceable. The output below shows it:

# /usr/local/etc/rc.d/snmpd restart
Stopping snmpd.
Waiting for PIDS: 73341.
Starting snmpd.

# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0 73748     1   0  44  0 22860  7144 select S     ??    0:00,21 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d
# snmpwalk localhost dskTable
(6 times, output skipped)
# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0 73748     1   0  44  0 26956 10756 select S     ??    0:00,50 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d
# snmpwalk localhost dskTable
(6 times, output skipped)
# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0 73748     1   0  44  0 29004 14232 select S     ??    0:00,62 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d
# snmpwalk localhost dskTable
(6 times, output skipped)
# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0 73748     1   0  44  0 33100 17804 select S     ??    0:00,80 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d
# snmpwalk localhost dskTable
(6 times, output skipped)
# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0 73748     1   0  44  0 37196 21352 select S     ??    0:00,95 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d
# snmpwalk localhost dskTable
(6 times, output skipped)
# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0 73748     1   0  44  0 41292 24908 select S     ??    0:01,09 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d
# snmpwalk localhost dskTable
(6 times, output skipped)
# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0 73748     1   0  44  0 43340 28440 select S     ??    0:01,25 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d
# snmpwalk localhost dskTable
(6 times, output skipped)
# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0 73748     1   0  44  0 47436 31936 select S     ??    0:01,39 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d
(quite some time after doing snmpwalks)
# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0 73748     1   0  45  0 90444 75092 select S     ??    0:02,06 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d

This can last forever, until the server memory is exausted.

----------------------------------------------------------------------

Comment By: Mark Martinec (mmartinec)
Date: 2011-09-19 17:34

Message:
I just came across this ticket and can confirm the described problem after
upgrading 5.5 to 5.7.
  FreeBSD 7.2-RELEASE-p2
  NET-SNMP version:  5.7 (form FreeBSD ports)

This brought our server to it knees: in four weeks the snmpd has grown to
VSZ 8 GB
and RSS also almost 8 GB, leaving hardly any room for the rest of the
applications.
With 5.5 the memory size of the snmpd process did not grow.

Here is the interesting tail of the valgring report of running snmpd for
about 30 minutes:

==32100== 844,880 bytes in 179 blocks are definitely lost in loss record
294 of 295
==32100==    at 0x24696B: malloc (in
/usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so)
==32100==    by 0xEEB410: netsnmp_fsys_arch_load (in
/usr/local/lib/libnetsnmpmibs.so.30)
==32100==    by 0xEEB008: _fsys_load (in
/usr/local/lib/libnetsnmpmibs.so.30)
==32100==    by 0xEC7719: var_extensible_disk (in
/usr/local/lib/libnetsnmpmibs.so.30)
==32100==    by 0xD20D7A: netsnmp_old_api_helper (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD3134A: netsnmp_call_handlers (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD3AC30: handle_var_requests (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD3C757: handle_pdu (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD3E9B7: netsnmp_handle_request (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD3EE08: handle_snmp_packet (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0x1666575: _sess_process_packet (in
/usr/local/lib/libnetsnmp.so.30)
==32100==    by 0x1667A99: _sess_read (in /usr/local/lib/libnetsnmp.so.30)
==32100== 
==32100== 4,899,360 bytes in 1,038 blocks are definitely lost in loss
record 295 of 295
==32100==    at 0x24696B: malloc (in
/usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so)
==32100==    by 0xEEB410: netsnmp_fsys_arch_load (in
/usr/local/lib/libnetsnmpmibs.so.30)
==32100==    by 0xEEB008: _fsys_load (in
/usr/local/lib/libnetsnmpmibs.so.30)
==32100==    by 0xEC7719: var_extensible_disk (in
/usr/local/lib/libnetsnmpmibs.so.30)
==32100==    by 0xD20D7A: netsnmp_old_api_helper (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD31B22: netsnmp_call_next_handler (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD1CE20: netsnmp_bulk_to_next_helper (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD3134A: netsnmp_call_handlers (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD3AC30: handle_var_requests (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD3BDBF: handle_getnext_loop (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD3E9B7: netsnmp_handle_request (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100==    by 0xD3EE08: handle_snmp_packet (in
/usr/local/lib/libnetsnmpagent.so.30)
==32100== 
==32100== LEAK SUMMARY:
==32100==    definitely lost: 5,819,592 bytes in 1,247 blocks
==32100==    indirectly lost: 17,185 bytes in 108 blocks
==32100==      possibly lost: 141,600 bytes in 30 blocks
==32100==    still reachable: 177,498 bytes in 317 blocks
==32100==         suppressed: 0 bytes in 0 blocks

----------------------------------------------------------------------

Comment By: drookie (drookie)
Date: 2011-09-13 09:02

Message:
Yup.

# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME
COMMAND
    0 19312     1   0  45  0 55628 38792 select S     ??    0:00,58
/usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d
[emz@ural85-gw0-alpha:~]# snmpd -v

NET-SNMP version:  5.7.1.rc1
Web:               http://www.net-snmp.org/
Email:             net...@li...

(10 sec of intensive snmpwalking)

# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME
COMMAND
    0 19312     1   0  45  0 84300 68896 select S     ??    0:01,97
/usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d

(10 sec of intensive snmpwalking)

# ps alx | grep -v egrep | egrep 'RSS|snmp'
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME
COMMAND
    0 19312     1   0  45  0 112972 96996 select S     ??    0:02,56
/usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d

Config:

syslocation "Norma JSC-Co., Perm, Russia"
syscontact XXXX

sysservices 76

trapcommunity XXXX
trap2sink 192.168.3.4

disk  / 200M
disk  /usr 500M
disk  /var 200M

load 12 12 12

proc squid
proc winbindd
proc smbd
proc nmbd
proc httpd
proc postgres
proc named
proc sendmail
proc ospfd
proc zebra
proc dhcpd
proc asterisk

procfix asterisk /usr/local/etc/rc.d/asterisk start

setEvent asterStart prErrFix.12 = 1
notificationEvent milterCbDown prErrorFlag.12

sysobjectid .1.3.6.1.4.1.8072.3.2.255

monitor -I -o dskErrorMsg.1 spaceRoot dskErrorFlag.1 != 0
monitor -I -o dskErrorMsg.2 spaceUsr dskErrorFlag.2 != 0
monitor -I -o dskErrorMsg.3 spaceVar dskErrorFlag.3 != 0

monitor -I -o prErrMessage.1 squiddRun prErrorFlag.1 != 0
monitor -I -o prErrMessage.2 winbindRun prErrorFlag.2 != 0
monitor -I -o prErrMessage.3 smbdRun prErrorFlag.3 != 0
monitor -I -o prErrMessage.4 nmbdRun prErrorFlag.4 != 0
monitor -I -o prErrMessage.5 apacheRun prErrorFlag.5 != 0
monitor -I -o prErrMessage.6 pgsqlRun prErrorFlag.6 != 0
monitor -I -o prErrMessage.7 namedRun prErrorFlag.7 != 0
monitor -I -o prErrMessage.8 sendmailRun prErrorFlag.8 != 0
monitor -I -o prErrMessage.9 ospfdRun prErrorFlag.9 != 0
monitor -I -o prErrMessage.10 zebraRun prErrorFlag.10 != 0
monitor -I -o prErrMessage.11 dhcpdRun prErrorFlag.11 != 0
monitor -e asterDown -I -o prErrMessage.12 -r 12 astDown prErrorFlag.12 !=
0
monitor -e asterStart -I -o prErrMessage.12 -r 12 astStart prErrorFlag.12
!= 0

engineIDType 1
engineIDNic vlan1

createUser snmp SHA XXXX AES
rwuser XXXX
rwcommunity XXXX

agentSecName XXXX
iquerySecName XXXX

authtrapenable 1

----------------------------------------------------------------------

Comment By:  (bvassche)
Date: 2011-09-13 08:10

Message:
Is this reproducible with 5.7.1rc1
(https://sourceforge.net/projects/net-snmp/files/net-snmp/5.7.1-pre-releases/)
? And what is the contents of your snmpd.conf ?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=112694&aid=3408398&group_id=12694