RE: Core dump with net-snmp-5.8

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi,

> I will check, but it will be tomorrow
>I don't know how I read the code last time, I was referring to the wrong commit where the code was introduced, it was introduced with
>"commit b7b50bbac7f21a924149d03da26ff0a44b25ec60

Thank you for your response.

I share steps to reproduce the issue just to be sure.

0.      extract files from reproducer.tar.gz

1.      deploy snmpd.conf

2.      start snmpd

3.      make (to generate sub_agent)

4.      ./sub_agent &

5.      pkill -STOP -x sub_agent

6.      snmpwalk -v2c -c "public" 127.0.0.1 .1.3.6.1.4.1.99999 -Ona

Regards,
Shogo Matsumoto

From: Anders Wallin [mailto:wal...@gm...]
Sent: Thursday, April 4, 2019 8:37 PM
To: sho...@jp...<mailto:sho...@jp...>
Cc: net...@li...<mailto:net...@li...>
Subject: Re: Core dump with net-snmp-5.8

I will check, but it will be tomorrow
I don't know how I read the code last time, I was referring to the wrong commit where the code was introduced, it was introduced with
"commit b7b50bbac7f21a924149d03da26ff0a44b25ec60
Author: VMwareDev Randy <snm...@vm...<mailto:snm...@vm...>>
Date:   Mon Jun 22 22:20:43 2015 -0400

    snmp_send callback updates

    - add new NETSNMP_CALLBACK_OP_RESEND
    - add missing calls for NETSNMP_CALLBACK_OP_SEND_FAILED

    Signed-off-by: Robert Story <rs...@fr...<mailto:rs...@fr...>>"

Regards
Anders Wallin

On Thu, Apr 4, 2019 at 10:48 AM Matsumoto, Shogo <sho...@jp...<mailto:sho...@jp...>> wrote:
Hi,

The issue also occurs with the following patches.

    NEWS: snmplib: PATCH: 1349: Fix perl/other crash against bad SNMPv3

 0001-agentx-logging-to-late-responses.patch
 0002-agentx-do-not-shut-down-all-sessions-when-one-sessio.patch

The issue occurs with the following patch (2914) too but I found
the cause of this issue.

 https://sourceforge.net/p/net-snmp/bugs/2914/
 0001-BUG2914-Agent-master-needs-to-treat-resend-as-normal.patch

With the patch 2914, netsnmp_free_delegated_cache is called
several times for the same object as follows:

 1. snmp_resend_request calls agentx_got_response with NETSNMP_CALLBACK_OP_RESEND,
 2. agentx_got_response's NETSNMP_CALLBACK_OP_RESEND handler do nothing
 3. snmp_resend_request calls agentx_got_response with NETSNMP_CALLBACK_OP_SEND_FAILED,
 4. agentx_got_response's NETSNMP_CALLBACK_OP_SEND_FAILED handler calls netsnmp_free_delegated_cache,
 5. snmp_sess_close calls agentx_got_response with NETSNMP_CALLBACK_OP_TIMED_OUT,
 6. agentx_got_response's NETSNMP_CALLBACK_OP_TIMED_OUT handler calls netsnmp_free_delegated_cache
    (double free)

gdb
--------------------------------------------------------------------------
Breakpoint 2, snmp_resend_request (slp=slp@entry=0x564eec5df000, rp=rp@entry=0x564eec5eb160, incr_retries=1) at snmp_api.c:6747
6747                rp->callback(NETSNMP_CALLBACK_OP_RESEND, sp,
(gdb) c
Continuing.

Breakpoint 3, netsnmp_free_delegated_cache (dcache=0x564eec5f2ec0) at agent_handler.c:929
929     {
(gdb) bt
#0  netsnmp_free_delegated_cache (dcache=0x564eec5f2ec0) at agent_handler.c:929
#1  0x00007fab254d5363 in agentx_got_response (operation=<optimized out>, session=0x564eec4ad560, reqid=2, pdu=0x564eec5e3050, magic=<optimized out>)
    at mibgroup/agentx/master.c:262
#2  0x00007fab24c6b58f in snmp_sess_timeout (sessp=sessp@entry=0x564eec5df000) at snmp_api.c:6813
#3  0x00007fab24c6b710 in snmp_timeout () at snmp_api.c:6660
#4  0x0000564eeb4c0f58 in receive () at snmpd.c:1347
#5  0x0000564eeb4c066e in main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1126
(gdb) c
Continuing.

Breakpoint 1, snmp_resend_request (slp=slp@entry=0x564eec5df000, rp=rp@entry=0x564eec5f3e50, incr_retries=1) at snmp_api.c:6735
6735                rp->callback(NETSNMP_CALLBACK_OP_SEND_FAILED, sp,
(gdb) c
Continuing.

Breakpoint 3, netsnmp_free_delegated_cache (dcache=0x564eec5f3730) at agent_handler.c:929
929     {
(gdb) bt
#0  netsnmp_free_delegated_cache (dcache=0x564eec5f3730) at agent_handler.c:929
#1  0x00007fab254d541a in agentx_got_response (operation=3, session=0x564eec4ad560, reqid=4, pdu=0x564eec5e54a0, magic=0x564eec5f3730)
    at mibgroup/agentx/master.c:223
#2  0x00007fab24c69325 in snmp_resend_request (slp=slp@entry=0x564eec5df000, rp=rp@entry=0x564eec5f3e50, incr_retries=1) at snmp_api.c:6735
#3  0x00007fab24c6b5db in snmp_sess_timeout (sessp=sessp@entry=0x564eec5df000) at snmp_api.c:6826
#4  0x00007fab24c6b710 in snmp_timeout () at snmp_api.c:6660
#5  0x0000564eeb4c0f58 in receive () at snmpd.c:1347
#6  0x0000564eeb4c066e in main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1126
(gdb) c
Continuing.

Breakpoint 3, netsnmp_free_delegated_cache (dcache=0x564eec5f3730) at agent_handler.c:929
929     {
(gdb) bt
#0  netsnmp_free_delegated_cache (dcache=0x564eec5f3730) at agent_handler.c:929
#1  0x00007fab254d541a in agentx_got_response (operation=2, session=0x564eec4ad560, reqid=4, pdu=0x564eec5e54a0, magic=0x564eec5f3730)
    at mibgroup/agentx/master.c:223
#2  0x00007fab24c69586 in snmp_sess_close (sessp=0x564eec5df000) at snmp_api.c:1975
#3  0x00007fab24c6afea in snmp_sess_select_info2_flags (sessp=0x0, numfds=0x7fff68db3694, fdset=0x7fff68db36b0, timeout=0x7fff68db36a0,
    block=0x7fff68db369c, flags=0) at snmp_api.c:6556
#4  0x0000564eeb4c0e95 in receive () at snmpd.c:1263
#5  0x0000564eeb4c066e in main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1126
(gdb) c
Continuing.

Program received signal SIGABRT, Aborted.
0x00007fab2335f93f in raise () from /lib64/libc.so.6
--------------------------------------------------------------------------

On the other hand, without the patch 2914 netsnmp_free_delegated_cache is called
several times for the same object as follows:

 1. snmp_resend_request calls agentx_got_response with NETSNMP_CALLBACK_OP_RESEND,
 2. agentx_got_response's "Unknown operation" handler calls netsnmp_free_delegated_cache,
 3. (retry) snmp_resend_request calls agentx_got_response AGAIN with NETSNMP_CALLBACK_OP_RESEND,
 4. agentx_got_response's "Unknown operation" handler calls netsnmp_free_delegated_cache
    (double free)

gdb
------------------------------------------------------
(gdb) b netsnmp_free_delegated_cache
Breakpoint 1 at 0x7f03bd437250: file agent_handler.c, line 929.
(gdb) c
Continuing.

Breakpoint 1, netsnmp_free_delegated_cache (dcache=0x558981be9110) at agent_handler.c:929
929     {
(gdb) bt
#0  netsnmp_free_delegated_cache (dcache=0x558981be9110) at agent_handler.c:929
#1  0x00007f03bd44a2df in agentx_got_response (operation=6, session=0x558981aa1540, reqid=12, pdu=0x558981be9370, magic=<optimized out>)
    at mibgroup/agentx/master.c:292
#2  0x00007f03bcbde0e6 in snmp_resend_request (slp=slp@entry=0x558981bd2e90, rp=rp@entry=0x558981be67a0, incr_retries=1) at snmp_api.c:6747
#3  0x00007f03bcbe05db in snmp_sess_timeout (sessp=sessp@entry=0x558981bd2e90) at snmp_api.c:6826
#4  0x00007f03bcbe0710 in snmp_timeout () at snmp_api.c:6660
#5  0x0000558980c5df58 in receive () at snmpd.c:1347
#6  0x0000558980c5d66e in main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1126
(gdb) c
Continuing.

Breakpoint 1, netsnmp_free_delegated_cache (dcache=0x558981be9110) at agent_handler.c:929
929     {
(gdb) bt
#0  netsnmp_free_delegated_cache (dcache=0x558981be9110) at agent_handler.c:929
#1  0x00007f03bd44a5ce in agentx_got_response (operation=6, session=0x558981aa1540, reqid=12, pdu=0x558981be9370, magic=0x558981be9110)
    at mibgroup/agentx/master.c:223
#2  0x00007f03bcbde0e6 in snmp_resend_request (slp=slp@entry=0x558981bd2e90, rp=rp@entry=0x558981be67a0, incr_retries=1) at snmp_api.c:6747
#3  0x00007f03bcbe05db in snmp_sess_timeout (sessp=sessp@entry=0x558981bd2e90) at snmp_api.c:6826
#4  0x00007f03bcbe0710 in snmp_timeout () at snmp_api.c:6660
#5  0x0000558980c5df58 in receive () at snmpd.c:1347
#6  0x0000558980c5d66e in main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1126
(gdb) c
Continuing.

Program received signal SIGABRT, Aborted.
------------------------------------------------------

I am not sure it's acceptable but attach a patch to fix the issue.

Regards,

Shogo Matsumoto
Fujitsu Ltd.

> The introduction of that code fixes another issue;
> "commit 56c30b11f3616ea4f0c38a21e08e78f050096020
> Author: Bill Fenner <fenner@...<mailto:fenner@...>>
> Date:   Wed Dec 20 21:52:10 2017 +0000
>
>     NEWS: snmplib: PATCH: 1349: Fix perl/other crash against bad SNMPv3
> agent
>
>     With the patch in 1214, the snmp_api code assumed that if magic was
>     set, it was the "struct synch-state" from snmp_client.  Of course,
>     magic belongs to the caller, and the perl library uses it differently,
>     so reaching into it is verboten.  Introduce a new callback (that
>     was already introduced in 5.8) to report this "retries exceeded"
>     state, and use it in snmp_client."
>
> I think the problem is really about shutting down the agentx connection
> when one(1) response is to late. I have
> done 2 patches (one that only write a better log message and one that
> removes the "bad" code.
> With these patches I don't get any crash. I think that 5.7.3 has this issue
> as well, but it can not be crashed with the agentofdead code
>
> Can you please try this?
>
> Regards
> Anders Wallin

_______________________________________________
Net-snmp-coders mailing list
Net...@li...<mailto:Net...@li...>
https://lists.sourceforge.net/lists/listinfo/net-snmp-coders