From: SourceForge.net <no...@so...> - 2011-10-21 15:12:39
|
Bugs item #3408398, was opened at 2011-09-13 07:10 Message generated for change (Comment added) made by bvassche You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=112694&aid=3408398&group_id=12694 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: agent Group: freeBSD Status: Open Resolution: None Priority: 5 Private: No Submitted By: drookie (drookie) Assigned to: Nobody/Anonymous (nobody) Summary: memory leak in 5.7 Initial Comment: SNMP Version: # snmpd -v NET-SNMP version: 5.7 Web: http://www.net-snmp.org/ Email: net...@li... FreeBSD: # uname -a FreeBSD ural85-gw0-alpha 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #0: Sun Feb 6 12:34:52 YEKT 2011 emz@ural85-gw0-alpha:/usr/obj/usr/src/sys/CRYSTAL amd64 After upgrading from 5.5 to 5.7 got a memory leak. It is stable and reproduceable. The output below shows it: # /usr/local/etc/rc.d/snmpd restart Stopping snmpd. Waiting for PIDS: 73341. Starting snmpd. # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 73748 1 0 44 0 22860 7144 select S ?? 0:00,21 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d # snmpwalk localhost dskTable (6 times, output skipped) # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 73748 1 0 44 0 26956 10756 select S ?? 0:00,50 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d # snmpwalk localhost dskTable (6 times, output skipped) # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 73748 1 0 44 0 29004 14232 select S ?? 0:00,62 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d # snmpwalk localhost dskTable (6 times, output skipped) # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 73748 1 0 44 0 33100 17804 select S ?? 0:00,80 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d # snmpwalk localhost dskTable (6 times, output skipped) # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 73748 1 0 44 0 37196 21352 select S ?? 0:00,95 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d # snmpwalk localhost dskTable (6 times, output skipped) # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 73748 1 0 44 0 41292 24908 select S ?? 0:01,09 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d # snmpwalk localhost dskTable (6 times, output skipped) # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 73748 1 0 44 0 43340 28440 select S ?? 0:01,25 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d # snmpwalk localhost dskTable (6 times, output skipped) # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 73748 1 0 44 0 47436 31936 select S ?? 0:01,39 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d (quite some time after doing snmpwalks) # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 73748 1 0 45 0 90444 75092 select S ?? 0:02,06 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d This can last forever, until the server memory is exausted. ---------------------------------------------------------------------- >Comment By: (bvassche) Date: 2011-10-21 17:12 Message: Valgrind output that shows which memory has not been freed (in a new bug ticket) is still welcome. ---------------------------------------------------------------------- Comment By: drookie (drookie) Date: 2011-10-21 13:26 Message: Million apologies. After a day of running, it leaked. Although, I should say, that after patching it DOES NOT leak when issuing massive dskTable snmpwalking, and that was the reason of my previous 'it does'. Obviously, this patch fixes the issue, but, unfortunately, only partially. At this time I cannot localize the actions leading to the leak. I will continue to try. Thanks again. ---------------------------------------------------------------------- Comment By: (bvassche) Date: 2011-10-21 12:47 Message: Thanks for testing - applied the attached patch as commit 6ef6907642247c663b9b8964b9fd44a211dbcaa7 on the v5.7 and the master branches. ---------------------------------------------------------------------- Comment By: drookie (drookie) Date: 2011-10-21 06:16 Message: it does. thanks a lot. ---------------------------------------------------------------------- Comment By: (bvassche) Date: 2011-10-20 20:58 Message: Does the attached patch help ? ---------------------------------------------------------------------- Comment By: drookie (drookie) Date: 2011-10-20 20:19 Message: I got no leak at Solaris 10 x86. Similar (if not identical) config. Seems like this is FreeBSD-specific. ---------------------------------------------------------------------- Comment By: Mark Martinec (mmartinec) Date: 2011-09-19 17:34 Message: I just came across this ticket and can confirm the described problem after upgrading 5.5 to 5.7. FreeBSD 7.2-RELEASE-p2 NET-SNMP version: 5.7 (form FreeBSD ports) This brought our server to it knees: in four weeks the snmpd has grown to VSZ 8 GB and RSS also almost 8 GB, leaving hardly any room for the rest of the applications. With 5.5 the memory size of the snmpd process did not grow. Here is the interesting tail of the valgring report of running snmpd for about 30 minutes: ==32100== 844,880 bytes in 179 blocks are definitely lost in loss record 294 of 295 ==32100== at 0x24696B: malloc (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so) ==32100== by 0xEEB410: netsnmp_fsys_arch_load (in /usr/local/lib/libnetsnmpmibs.so.30) ==32100== by 0xEEB008: _fsys_load (in /usr/local/lib/libnetsnmpmibs.so.30) ==32100== by 0xEC7719: var_extensible_disk (in /usr/local/lib/libnetsnmpmibs.so.30) ==32100== by 0xD20D7A: netsnmp_old_api_helper (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD3134A: netsnmp_call_handlers (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD3AC30: handle_var_requests (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD3C757: handle_pdu (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD3E9B7: netsnmp_handle_request (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD3EE08: handle_snmp_packet (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0x1666575: _sess_process_packet (in /usr/local/lib/libnetsnmp.so.30) ==32100== by 0x1667A99: _sess_read (in /usr/local/lib/libnetsnmp.so.30) ==32100== ==32100== 4,899,360 bytes in 1,038 blocks are definitely lost in loss record 295 of 295 ==32100== at 0x24696B: malloc (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so) ==32100== by 0xEEB410: netsnmp_fsys_arch_load (in /usr/local/lib/libnetsnmpmibs.so.30) ==32100== by 0xEEB008: _fsys_load (in /usr/local/lib/libnetsnmpmibs.so.30) ==32100== by 0xEC7719: var_extensible_disk (in /usr/local/lib/libnetsnmpmibs.so.30) ==32100== by 0xD20D7A: netsnmp_old_api_helper (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD31B22: netsnmp_call_next_handler (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD1CE20: netsnmp_bulk_to_next_helper (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD3134A: netsnmp_call_handlers (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD3AC30: handle_var_requests (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD3BDBF: handle_getnext_loop (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD3E9B7: netsnmp_handle_request (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== by 0xD3EE08: handle_snmp_packet (in /usr/local/lib/libnetsnmpagent.so.30) ==32100== ==32100== LEAK SUMMARY: ==32100== definitely lost: 5,819,592 bytes in 1,247 blocks ==32100== indirectly lost: 17,185 bytes in 108 blocks ==32100== possibly lost: 141,600 bytes in 30 blocks ==32100== still reachable: 177,498 bytes in 317 blocks ==32100== suppressed: 0 bytes in 0 blocks ---------------------------------------------------------------------- Comment By: drookie (drookie) Date: 2011-09-13 09:02 Message: Yup. # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 19312 1 0 45 0 55628 38792 select S ?? 0:00,58 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d [emz@ural85-gw0-alpha:~]# snmpd -v NET-SNMP version: 5.7.1.rc1 Web: http://www.net-snmp.org/ Email: net...@li... (10 sec of intensive snmpwalking) # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 19312 1 0 45 0 84300 68896 select S ?? 0:01,97 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d (10 sec of intensive snmpwalking) # ps alx | grep -v egrep | egrep 'RSS|snmp' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 19312 1 0 45 0 112972 96996 select S ?? 0:02,56 /usr/local/sbin/snmpd -p /var/run/snmpd.pid -Ls d Config: syslocation "Norma JSC-Co., Perm, Russia" syscontact XXXX sysservices 76 trapcommunity XXXX trap2sink 192.168.3.4 disk / 200M disk /usr 500M disk /var 200M load 12 12 12 proc squid proc winbindd proc smbd proc nmbd proc httpd proc postgres proc named proc sendmail proc ospfd proc zebra proc dhcpd proc asterisk procfix asterisk /usr/local/etc/rc.d/asterisk start setEvent asterStart prErrFix.12 = 1 notificationEvent milterCbDown prErrorFlag.12 sysobjectid .1.3.6.1.4.1.8072.3.2.255 monitor -I -o dskErrorMsg.1 spaceRoot dskErrorFlag.1 != 0 monitor -I -o dskErrorMsg.2 spaceUsr dskErrorFlag.2 != 0 monitor -I -o dskErrorMsg.3 spaceVar dskErrorFlag.3 != 0 monitor -I -o prErrMessage.1 squiddRun prErrorFlag.1 != 0 monitor -I -o prErrMessage.2 winbindRun prErrorFlag.2 != 0 monitor -I -o prErrMessage.3 smbdRun prErrorFlag.3 != 0 monitor -I -o prErrMessage.4 nmbdRun prErrorFlag.4 != 0 monitor -I -o prErrMessage.5 apacheRun prErrorFlag.5 != 0 monitor -I -o prErrMessage.6 pgsqlRun prErrorFlag.6 != 0 monitor -I -o prErrMessage.7 namedRun prErrorFlag.7 != 0 monitor -I -o prErrMessage.8 sendmailRun prErrorFlag.8 != 0 monitor -I -o prErrMessage.9 ospfdRun prErrorFlag.9 != 0 monitor -I -o prErrMessage.10 zebraRun prErrorFlag.10 != 0 monitor -I -o prErrMessage.11 dhcpdRun prErrorFlag.11 != 0 monitor -e asterDown -I -o prErrMessage.12 -r 12 astDown prErrorFlag.12 != 0 monitor -e asterStart -I -o prErrMessage.12 -r 12 astStart prErrorFlag.12 != 0 engineIDType 1 engineIDNic vlan1 createUser snmp SHA XXXX AES rwuser XXXX rwcommunity XXXX agentSecName XXXX iquerySecName XXXX authtrapenable 1 ---------------------------------------------------------------------- Comment By: (bvassche) Date: 2011-09-13 08:10 Message: Is this reproducible with 5.7.1rc1 (https://sourceforge.net/projects/net-snmp/files/net-snmp/5.7.1-pre-releases/) ? And what is the contents of your snmpd.conf ? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=112694&aid=3408398&group_id=12694 |