Menu
▾
▴
[ net-snmp-Bugs-3165949 ] snmpd crashes on AIX 6.1.6.2
|
From: SourceForge.net <no...@so...> - 2012-05-16 18:26:35
|
Bugs item #3165949, was opened at 2011-01-26 07:04 Message generated for change (Comment added) made by blentz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=112694&aid=3165949&group_id=12694 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: agent Group: aix Status: Open Resolution: None Priority: 5 Private: No Submitted By: blentz (blentz) Assigned to: Nobody/Anonymous (nobody) Summary: snmpd crashes on AIX 6.1.6.2 Initial Comment: We're experiencing core dumps in snmpd on AIX 6.1 TL6 SP2. Looks like an issue with var_ipRouteEntry. Can someone take a look and advise? I will gladly provide a test platform if the developers don't have one (I'm aware that AIX isn't prevalent). load_rtentries(0x0) at 0xd4b2d3f8 load_rtentries(0x0) at 0xd4b2d3f8 Route_Scan_Reload() at 0xd4b2d998 var_ipRouteEntry(0x2ff1c068, 0x2ff1c27c, 0x2ff1c47c, 0x0, 0x2ff1c278, 0x2ff1c274) at 0xd4b2cbb0 netsnmp_old_api_helper(0x2001b808, 0x2001bca8, 0x20127468, 0x20127488) at 0xd4c301a0 netsnmp_call_handler(0x2001b808, 0x2001bca8, 0x20127468, 0x20127488) at 0xd4c5069c netsnmp_call_handlers(0x2001bca8, 0x20127468, 0x20127488) at 0xd4c50a9c handle_var_requests(0x20126df8) at 0xd4c5eedc handle_getnext_loop(0x20126df8) at 0xd4c5fd40 handle_pdu(0x20126df8) at 0xd4c60954 netsnmp_handle_request(0x20126df8, 0x0) at 0xd4c603e0 handle_snmp_packet(0x1, 0x20115a78, 0x63e6e980, 0x20126af8, 0x0) at 0xd4c5cef0 _sess_process_packet(0x20115a18, 0x20115a78, 0x20115a38, 0x201158c8, 0x201159c8, 0x14, 0x20116ab8, 0x2c) at 0xd4cb089c _sess_read(0x20115a18, 0x2ff1c9d4) at 0xd4cb1fa0 snmp_sess_read(0x20115a18, 0x2ff1c9d4) at 0xd4cb2044 snmp_read(0x2ff1c9d4) at 0xd4cb0a90 receive() at 0x10002c2c main(0x5, 0x2ff22bb8) at 0x100025f0 (dbx) ---------------------------------------------------------------------- >Comment By: blentz (blentz) Date: 2012-05-16 11:26 Message: Problem also exists on AIX 6.1 TL6 SP7. We've tried upgrading 5.4.2.1 to 5.5, 5.6, 5.7.1, tried SVN nightlies, RCs and PREs, but because of the missing HOST-RESOURCES data for the process table and storage table, we've continued to exclude .1.3.6.1.2.1.4.21 in most of our configs to prevent them from crashing. I appreciate any help anyone can provide. ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2012-05-16 11:23 Message: Are there any updates? This still continues to plague over 50% of our systems. I managed to find another system, recently installed, and took a backtrace: $ /opt/local/net-snmp/sbin/snmpd --version NET-SNMP version: 5.4.2.1 Web: http://www.net-snmp.org/ Email: net...@li... $ sudo /opt/local/net-snmp/sbin/snmpd -Le -p /opt/local/net-snmp/var/run/snmpd.pid -a -f 0.0.0.0:16161 nlist err: neither proc nor _proc found. NET-SNMP version 5.4.2.1 Received SNMP packet(s) from UDP: [0.0.0.0]->[10.0.17.14]:-5505 Memory fault(coredump) $ sudo file core core: AIX core file fulldump 32-bit, snmpd $ sudo dbx /opt/local/net-snmp/sbin/snmpd core Type 'help' for help. [using memory image in core] reading symbolic information ...warning: no source compiled with -g Segmentation fault in read at 0xd013910c ($t1) 0xd013910c (read+0x2c) 9421ff20 stwu r1,-224(r1) (dbx) where read(0x3, 0x2df23128, 0x80000000) at 0xd013910c klread(0x2df23128, 0x18) at 0xd1ad5efc klookup(0x0, 0x2df23128, 0x18) at 0xd1ad5fdc load_rtentries(0x0) at 0xd19d135c load_rtentries(0x0) at 0xd19d13f8 load_rtentries(0x0) at 0xd19d13f8 load_rtentries(0x0) at 0xd19d13f8 <Thousands of these> load_rtentries(0x0) at 0xd19d13f8 load_rtentries(0x0) at 0xd19d13f8 load_rtentries(0x0) at 0xd19d13f8 Route_Scan_Reload() at 0xd19d1998 var_ipRouteEntry(0x2ff1c1b8, 0x2ff1c3cc, 0x2ff1c5cc, 0x0, 0x2ff1c3c8, 0x2ff1c3c4) at 0xd19d0bb0 netsnmp_old_api_helper(0x2001bd98, 0x2001c238, 0x2012ab58, 0x2012ab78) at 0xd1a991a0 netsnmp_call_handler(0x2001bd98, 0x2001c238, 0x2012ab58, 0x2012ab78) at 0xd1ab969c netsnmp_call_handlers(0x2001c238, 0x2012ab58, 0x2012ab78) at 0xd1ab9a9c handle_var_requests(0x2011a008) at 0xd1ac7edc handle_getnext_loop(0x2011a008) at 0xd1ac8d40 handle_pdu(0x2011a008) at 0xd1ac9954 netsnmp_handle_request(0x2011a008, 0x0) at 0xd1ac93e0 handle_snmp_packet(0x1, 0x2011a0e8, 0x65654897, 0x2012a258, 0x0) at 0xd1ac5ef0 _sess_process_packet(0x2011a088, 0x2011a0e8, 0x2011a0a8, 0x20119f38, 0x20119f88, 0x14, 0x2011a248, 0x2c) at 0xd1b1989c _sess_read(0x2011a088, 0x2ff1cb24) at 0xd1b1afa0 snmp_sess_read(0x2011a088, 0x2ff1cb24) at 0xd1b1b044 snmp_read(0x2ff1cb24) at 0xd1b19a90 receive() at 0x10002c2c main(0x7, 0x2ff22d00) at 0x100025f0 (dbx) ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2011-09-02 08:18 Message: This is 5.4.2.1. ---------------------------------------------------------------------- Comment By: Niels Baggesen (nba) Date: 2011-05-29 04:59 Message: What version of Net-SNMP are you testing? ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2011-05-29 04:49 Message: Okay, still dumps core with the same errors even while compiled using --without-kmem-usage ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2011-05-29 04:45 Message: I recompiled on AIX 5.3 TL 12 SP 2 using --without-kmem-usage and make test reports the following: 58: testing if snmpnetstat finds the running agent in udpTable... FAIL 59: testing full snmpwalk (SNMPv2c) against agent (may take time)... ..ok Summary: 58 / 59 succeeded. make: 1254-004 The error code from the last command is 1. Stop. make: 1254-004 The error code from the last command is 2. Stop. ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2011-05-29 04:28 Message: It seems both the iproute and the hrstorage issues are similar. I can run net-snmp on my affected AIX systems if I use: view systemview included .1 view systemview excluded .1.3.6.1.2.1.4 view systemview excluded .1.3.6.1.2.1.25.2.3.1 My compile is still running. The vast majority of my systems (175+) work fine with this same copy of binaries, on the same version of AIX, at the same TL and SP levels. It's very odd. ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2011-05-29 04:20 Message: -DALL gives: wtest: oid in: HOST-RESOURCES-MIB::hrStorageType.39 trace: netsnmp_subtree_find_prev(): agent_registry.c, 1529: wtest: oid cmp: HOST-RESOURCES-MIB::hrStorageDescr --- off = 11, in off = 0 test = 1 trace: netsnmp_add_varbind_to_cache(): snmp_agent.c, 1870: snmp_agent: add_vb_to_cache(2012d8b8, 1, HOST-RESOURCES-MIB::hrStorageType.39, 2006e668) trace: _callback_lock(): callback.c, 130: 9:callback:lock: locked (APP,null) trace: snmp_call_callbacks(): callback.c, 322: callback: START calling callbacks for maj=1 min=12 trace: snmp_call_callbacks(): callback.c, 336: callback: calling a callback for maj=1 min=12 trace: vacm_check_view(): mibgroup/mibII/vacm_conf.c, 1300: mibII/vacm_vars: vacm_in_view: ver=1, community=aix2011 trace: netsnmp_udp_getSecName(): snmpUDPDomain.c, 1259: netsnmp_udp_getSecName: resolve <"aix2011", 0x0a00110e> trace: netsnmp_udp_getSecName(): snmpUDPDomain.c, 1264: netsnmp_udp_getSecName: compare <"aix2011", 0x00000000/0x00000000>... SUCCESS trace: netsnmp_subtree_find_first(): agent_registry.c, 156: subtree: looking for subtree for context: "" trace: netsnmp_subtree_find_first(): agent_registry.c, 160: subtree: found one for: "" trace: vacm_check_view(): mibgroup/mibII/vacm_conf.c, 1427: mibII/vacm_vars: vacm_in_view: sn=notConfigUser, gn=notConfigGroup, vn=systemview trace: netsnmp_view_subtree_check(): vacm.c, 532: 9:vacm:checkSubtree: view systemview trace: netsnmp_view_subtree_check(): vacm.c, 569: 9:vacm:checkSubtree: systemview matched? trace: netsnmp_view_subtree_check(): vacm.c, 630: 9:vacm:checkSubtree: matched trace: netsnmp_view_subtree_check(): vacm.c, 653: vacm:checkSubtree: , included trace: snmp_call_callbacks(): callback.c, 348: callback: END calling callbacks for maj=1 min=12 (1 called) trace: _callback_unlock(): callback.c, 164: 9:callback:lock: unlocked (APP,null) trace: netsnmp_add_varbind_to_cache(): snmp_agent.c, 1941: snmp_agent: tp->start HOST-RESOURCES-MIB::hrStorageType, tp->end HOST-RESOURCES-MIB::hrStorageDescr, trace: netsnmp_add_varbind_to_cache(): snmp_agent.c, 1961: verbose:asp: asp 2012d8b8 reqinfo 2012df28 assigned to request trace: netsnmp_add_varbind_to_cache(): snmp_agent.c, 1968: verbose:asp: asp 2012d8b8 reqinfo 2012df28 assigned to request trace: netsnmp_call_handlers(): agent_handler.c, 510: handler:calling: main handler bulk_to_next trace: netsnmp_call_handler(): agent_handler.c, 430: handler:calling: calling handler bulk_to_next for mode GETNEXT trace: netsnmp_call_handler(): agent_handler.c, 438: handler:returned: handler bulk_to_next returned 0 trace: netsnmp_call_handler(): agent_handler.c, 430: handler:calling: calling handler old_api for mode GETNEXT trace: header_hrstoreEntry(): host/hr_storage.c, 373: host/hr_storage: var_hrstoreEntry: request HOST-RESOURCES-MIB::hrStorageType.39 exact=0 trace: header_hrstoreEntry(): host/hr_storage.c, 381: host/hr_storage: var_hrstoreEntry: compare HOST-RESOURCES-MIB::hrStorageType => 1 (index 31 ....HOST-RESOURCES-MIB::hrStorageType.31 (index 32 ....HOST-RESOURCES-MIB::hrStorageType.32 (index 33 ....HOST-RESOURCES-MIB::hrStorageType.33 (index 34 ....HOST-RESOURCES-MIB::hrStorageType.34 (index 35 ....HOST-RESOURCES-MIB::hrStorageType.35 (index 36 ....HOST-RESOURCES-MIB::hrStorageType.36 (index 37 ....HOST-RESOURCES-MIB::hrStorageType.37 (index 38 ....HOST-RESOURCES-MIB::hrStorageType.38 (index 39 ....HOST-RESOURCES-MIB::hrStorageType.39 (index 40 ....HOST-RESOURCES-MIB::hrStorageType.40 trace: header_hrstoreEntry(): host/hr_storage.c, 476: host/hr_storage: var_hrstoreEntry: process HOST-RESOURCES-MIB::hrStorageType.40 (ffffffff) Memory fault(coredump) ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2011-05-29 04:20 Message: -DALL gives: wtest: oid in: HOST-RESOURCES-MIB::hrStorageType.39 trace: netsnmp_subtree_find_prev(): agent_registry.c, 1529: wtest: oid cmp: HOST-RESOURCES-MIB::hrStorageDescr --- off = 11, in off = 0 test = 1 trace: netsnmp_add_varbind_to_cache(): snmp_agent.c, 1870: snmp_agent: add_vb_to_cache(2012d8b8, 1, HOST-RESOURCES-MIB::hrStorageType.39, 2006e668) trace: _callback_lock(): callback.c, 130: 9:callback:lock: locked (APP,null) trace: snmp_call_callbacks(): callback.c, 322: callback: START calling callbacks for maj=1 min=12 trace: snmp_call_callbacks(): callback.c, 336: callback: calling a callback for maj=1 min=12 trace: vacm_check_view(): mibgroup/mibII/vacm_conf.c, 1300: mibII/vacm_vars: vacm_in_view: ver=1, community=aix2011 trace: netsnmp_udp_getSecName(): snmpUDPDomain.c, 1259: netsnmp_udp_getSecName: resolve <"aix2011", 0x0a00110e> trace: netsnmp_udp_getSecName(): snmpUDPDomain.c, 1264: netsnmp_udp_getSecName: compare <"aix2011", 0x00000000/0x00000000>... SUCCESS trace: netsnmp_subtree_find_first(): agent_registry.c, 156: subtree: looking for subtree for context: "" trace: netsnmp_subtree_find_first(): agent_registry.c, 160: subtree: found one for: "" trace: vacm_check_view(): mibgroup/mibII/vacm_conf.c, 1427: mibII/vacm_vars: vacm_in_view: sn=notConfigUser, gn=notConfigGroup, vn=systemview trace: netsnmp_view_subtree_check(): vacm.c, 532: 9:vacm:checkSubtree: view systemview trace: netsnmp_view_subtree_check(): vacm.c, 569: 9:vacm:checkSubtree: systemview matched? trace: netsnmp_view_subtree_check(): vacm.c, 630: 9:vacm:checkSubtree: matched trace: netsnmp_view_subtree_check(): vacm.c, 653: vacm:checkSubtree: , included trace: snmp_call_callbacks(): callback.c, 348: callback: END calling callbacks for maj=1 min=12 (1 called) trace: _callback_unlock(): callback.c, 164: 9:callback:lock: unlocked (APP,null) trace: netsnmp_add_varbind_to_cache(): snmp_agent.c, 1941: snmp_agent: tp->start HOST-RESOURCES-MIB::hrStorageType, tp->end HOST-RESOURCES-MIB::hrStorageDescr, trace: netsnmp_add_varbind_to_cache(): snmp_agent.c, 1961: verbose:asp: asp 2012d8b8 reqinfo 2012df28 assigned to request trace: netsnmp_add_varbind_to_cache(): snmp_agent.c, 1968: verbose:asp: asp 2012d8b8 reqinfo 2012df28 assigned to request trace: netsnmp_call_handlers(): agent_handler.c, 510: handler:calling: main handler bulk_to_next trace: netsnmp_call_handler(): agent_handler.c, 430: handler:calling: calling handler bulk_to_next for mode GETNEXT trace: netsnmp_call_handler(): agent_handler.c, 438: handler:returned: handler bulk_to_next returned 0 trace: netsnmp_call_handler(): agent_handler.c, 430: handler:calling: calling handler old_api for mode GETNEXT trace: header_hrstoreEntry(): host/hr_storage.c, 373: host/hr_storage: var_hrstoreEntry: request HOST-RESOURCES-MIB::hrStorageType.39 exact=0 trace: header_hrstoreEntry(): host/hr_storage.c, 381: host/hr_storage: var_hrstoreEntry: compare HOST-RESOURCES-MIB::hrStorageType => 1 (index 31 ....HOST-RESOURCES-MIB::hrStorageType.31 (index 32 ....HOST-RESOURCES-MIB::hrStorageType.32 (index 33 ....HOST-RESOURCES-MIB::hrStorageType.33 (index 34 ....HOST-RESOURCES-MIB::hrStorageType.34 (index 35 ....HOST-RESOURCES-MIB::hrStorageType.35 (index 36 ....HOST-RESOURCES-MIB::hrStorageType.36 (index 37 ....HOST-RESOURCES-MIB::hrStorageType.37 (index 38 ....HOST-RESOURCES-MIB::hrStorageType.38 (index 39 ....HOST-RESOURCES-MIB::hrStorageType.39 (index 40 ....HOST-RESOURCES-MIB::hrStorageType.40 trace: header_hrstoreEntry(): host/hr_storage.c, 476: host/hr_storage: var_hrstoreEntry: process HOST-RESOURCES-MIB::hrStorageType.40 (ffffffff) Memory fault(coredump) ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2011-05-29 04:18 Message: I am recompiling now and will run make test again. My development environment was moved by another admin, so I don't have the same build as before. I am also getting cores while walking HOST-RESOURCES-TYPES::hrStorageFixedDisk $ sudo dbx /opt/local/net-snmp/sbin/snmpd core Type 'help' for help. [using memory image in core] reading symbolic information ...warning: no source compiled with -g Segmentation fault in . at 0xd410 ($t1) 0x0000d410 warning: Unable to access address 0xd410 from core (dbx) where .() at 0xd410 hasmntopt(??, ??) at 0xd03e4350 var_hrstore(0x2ff1c1c8, 0x2ff1c3dc, 0x2ff1c5dc, 0x0, 0x2ff1c3d8, 0x2ff1c3d4) at 0xd48472f4 netsnmp_old_api_helper(0x2006f688, 0x2006f508, 0x2011f048, 0x2011f068) at 0xd48a41a0 netsnmp_call_handler(0x2006f688, 0x2006f508, 0x2011f048, 0x2011f068) at 0xd48c469c netsnmp_call_handlers(0x2006f508, 0x2011f048, 0x2011f068) at 0xd48c4a9c handle_var_requests(0x2011e9d8) at 0xd48d2edc handle_pdu(0x2011e9d8) at 0xd48d4914 netsnmp_handle_request(0x2011e9d8, 0x0) at 0xd48d43e0 handle_snmp_packet(0x1, 0x2011d1f8, 0xc4a525, 0x2011e6d8, 0x0) at 0xd48d0ef0 _sess_process_packet(0x2011d198, 0x2011d1f8, 0x2011d1b8, 0x2011d048, 0x2011d178, 0x14, 0x20214ee8, 0x2f) at 0xd4a2189c _sess_read(0x2011d198, 0x2ff1cad4) at 0xd4a22fa0 snmp_sess_read(0x2011d198, 0x2ff1cad4) at 0xd4a23044 snmp_read(0x2ff1cad4) at 0xd4a21a90 receive() at 0x10002c2c main(0x9, 0x2ff22cb8) at 0x100025f0 (dbx) ---------------------------------------------------------------------- Comment By: Thomas Anders (tanders) Date: 2011-01-30 04:04 Message: Please try again with --without-kmem-usage and report back. Also: - Does "make test" succeed? - Please attach the full output of "snmpwalk ... localhost .1". If you feel if contains sensitive information,please send it privately to me. ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2011-01-29 23:22 Message: Regarding README.aix: - I have my own workaround for -brtl using: export CC="gcc -Wl,-blibpath:$WHERE/lib:/usr/lib:/lib,-brtl" - I built using GNU grep and GNU sed - I didn't use --without-kmem-usage, maybe that's my issue? - I don't use or need IPv6 - I don't need the perl module (at this time) Thanks for your help!! ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2011-01-29 23:19 Message: /opt/local/net-snmp/bin/net-snmp-config --configure-options '--prefix=/opt/local/net-snmp' '--enable-shared=yes' '--enable-static=no' '--with-defaults' 'CC=gcc -Wl,-blibpath:/opt/local/net-snmp/lib:/usr/lib:/lib,-brtl' 'CFLAGS=-I/opt/local/net-snmp/include' 'CPPFLAGS=-I/opt/local/net-snmp/include' 'LDFLAGS=-L/opt/local/net-snmp/lib' ---------------------------------------------------------------------- Comment By: Thomas Anders (tanders) Date: 2011-01-29 22:33 Message: What configure options did you use when building net-snmp on AIX 6.1? ("net-snmp-config --configure-options") Did you read and follow the notes in README.aix? ---------------------------------------------------------------------- Comment By: blentz (blentz) Date: 2011-01-26 07:10 Message: Excluding .1.3.6.1.2.1.4.21.1 appears to provide a workaround. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=112694&aid=3165949&group_id=12694 |