Menu

#2923 5.8.0: linux seeing snmpd snmpusb.c:usm_rgenerate_out_msg() core dump with double free error

linux
closed
nobody
None
4
2019-08-02
2019-02-14
Sam Tannous
No

Merged code from V5-8-patches (Nov 2018) running on Linux Debian Jessie based system is seeing
regular core dumps.

The problems appears to be a double free in snmplib/snmpusm.c: 1936 usm_free_usmStateReference(secStateRef)

Would an acceptable fix be to just check this pointer before freeing? I don't see how this is
can be triggered (since I can't recreate the problem).

syslogs and full backtrace is attached.

1 Attachments

Related

Bugs: #2923

Discussion

  • Sam Tannous

    Sam Tannous - 2019-02-14

    For some reason, I couldn't paste into chrome: Here's the syslog:

    2018-11-30T01:16:21.559681-06:00 system1 systemd[1]: Started Simple Network Management Protocol (SNMP) Daemon..
    2018-11-30T01:16:23.638656-06:00 system1 snmptrapd[1746]: unknown snmp version 193
    2018-11-30T01:16:23.639248-06:00 system1 snmptrapd[1746]: NET-SNMP version 5.8 AgentX subagent connected
    2018-11-30T01:17:21.151819-06:00 system1 ip[18579]: *** Error in `/usr/sbin/snmpd': double free or corruption (fasttop): 0x00000000022ad5c0 ***
    2018-11-30T01:17:21.161948-06:00 system1 systemd[1]: Starting Cleanup core files to avoid filling filesystem...
    

    and the backtrace:

    (gdb) bt full
    #0  0x00007fe91aa1a067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
    No symbol table info available.
    #1  0x00007fe91aa1b448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
    No symbol table info available.
    #2  0x00007fe91aa581b4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
    No symbol table info available.
    #3  0x00007fe91aa5d98e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
    No symbol table info available.
    #4  0x00007fe91aa5e696 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
    No symbol table info available.
    #5  0x00007fe91b119048 in usm_rgenerate_out_msg (msgProcModel=18579, globalData=0x4893 <error: Cannot access memory at address 0x4893>, globalDataLen=19, maxMsgSize=446799975, secModel=1630679600, 
        secEngineID=0x6f6974707572726f <error: Cannot access memory at address 0x6f6974707572726f>, secEngineIDLen=1, secName=0x22abe80 "userxxxxx", secNameLen=9, secLevel=0, scopedPdu=0x2297ee0 "02\004\021\200", scopedPduLen=52, 
        secStateRef=0x22ad5c0, wholeMsg=0x7ffdc69dc438, wholeMsgLen=0x7ffdc69dc448, offset=0x7ffdc69dc3b0) at snmpusm.c:1936
            boots_uint = 12
            time_uint = 60
            boots_long = 12
            time_long = 60
            thePrivProtocolLength = 1
            salt_length = 0
            salt = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\000"
            authParams = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\000"
            iv = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\000"
            __func__ = "usm_rgenerate_out_msg"
    #6  0x00007fe91b11aa05 in usm_secmod_rgenerate_out_msg (parms=<optimized out>) at snmpusm.c:1416
    No locals.
    #7  0x00007fe91b0cc374 in snmpv3_packet_realloc_rbuild (pkt=0x7ffdc69dc438, pkt_len=0x7ffdc69dc448, offset=0x7ffdc69dc3b0, session=0x2296990, pdu=0x22ad150, pdu_data=<optimized out>, pdu_data_len=0) at snmp_api.c:2681
            parms = {msgProcModel = 0, globalData = 0x22aac80 "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\003", globalDataLen = 19, maxMsgSize = 1472, secModel = 3, secEngineID = 0x22b2e70 "\200", 
              secEngineIDLen = 17, secName = 0x22abe80 "userxxxxx", secNameLen = 9, secLevel = 3, scopedPdu = 0x2297ee0 "02\004\021\200", scopedPduLen = 52, secStateRef = 0x22ad5c0, secParams = 0x0, secParamsLen = 0x3400000000000, 
              wholeMsg = 0x7ffdc69dc438, wholeMsgLen = 0x7ffdc69dc448, wholeMsgOffset = 0x7ffdc69dc3b0, pdu = 0x22ad150, session = 0x2296990}
            scoped_pdu = 0x2297ee0 "02\004\021\200"
            hdrbuf = <optimized out>
            hdr = <optimized out>
            hdrbuf_len = 51
            hdr_offset = 19
            spdu_offset = 52
            body_end_offset = <optimized out>
            body_len = <optimized out>
            sptr = 0x21c8c60
            rc = <optimized out>
            __func__ = "snmpv3_packet_realloc_rbuild"
    #8  0x00007fe91b0d0219 in snmpv3_build (pdu=<optimized out>, session=<optimized out>, offset=<optimized out>, pkt_len=<optimized out>, pkt=<optimized out>) at snmp_api.c:2268
            ret = 36360528
    #9  _snmp_build (pdu=<optimized out>, session=<optimized out>, offset=<optimized out>, pkt_len=<optimized out>, pkt=<optimized out>) at snmp_api.c:2853
            h0e = 0x0
            start_offset = 0
            length = 0
            version = 140638704107575
    #10 snmp_build (pkt=0x7ffdc69dc438, pkt_len=0x7ffdc69dc448, offset=0x7ffdc69dc3b0, pss=0x2296990, pdu=0x22ad150) at snmp_api.c:3192                                                       
            rc = 0
    #11 0x00007fe91b0d0924 in netsnmp_build_packet (isp=<optimized out>, sp=<optimized out>, pdu=<optimized out>, pktbuf_p=0x7ffdc69dc438, pktbuf_len_p=0x7ffdc69dc448, pkt_p=0x7ffdc69dc440, len_p=0x7ffdc69dc450) at snmp_api.c:4946
            offset = 108
            result = <optimized out>
    #12 0x00007fe91b0d0b55 in _build_initial_pdu_packet (slp=0x0, pdu=0x22ad150, bulk=6, bulk@entry=0) at snmp_api.c:5114
            session = 0x22ad150
            isp = 0x2296380
            transport = 0x0
            pktbuf = 0x2297d30 "xxxxxxxxxxxxxxxxxxxxxxx7"
            packet = 0x2297d30 "xxxxxxxxxxxxxxxxxxxxxxx7"
            pktbuf_len = 484
            length = 0
            result = -962739120
            __func__ = "_build_initial_pdu_packet"
    #13 0x00007fe91b0d2d3f in _sess_async_send (cb_data=0x0, callback=<optimized out>, pdu=<optimized out>, sessp=0x2296350) at snmp_api.c:5241
            slp = 0x2296350
            transport = 0x2296260
            reqid = <optimized out>
            session = 0x2296990
            isp = 0x2296380
            result = <optimized out>
    #14 snmp_sess_async_send (sessp=0x2296350, pdu=0x22ad150, callback=callback@entry=0x0, cb_data=cb_data@entry=0x0) at snmp_api.c:5398
    No locals.
    #15 0x00007fe91b0d5187 in snmp_async_send (session=<optimized out>, pdu=<optimized out>, callback=callback@entry=0x0, cb_data=cb_data@entry=0x0) at snmp_api.c:5217
            sessp = <optimized out>
    #16 0x00007fe91b0d5199 in snmp_send (session=<optimized out>, pdu=<optimized out>) at snmp_api.c:5203
    No locals.
    #17 0x00007fe91b8e7f74 in netsnmp_wrap_up_request (asp=asp@entry=0x22abdd0, status=<optimized out>, status@entry=0) at snmp_agent.c:2089
            __func__ = "netsnmp_wrap_up_request"
    #18 0x00007fe91b8eb45b in netsnmp_handle_request (asp=asp@entry=0x22abdd0, status=status@entry=0) at snmp_agent.c:3753
            __func__ = "netsnmp_handle_request"
    #19 0x00007fe91b8eba42 in handle_snmp_packet (op=<optimized out>, session=<optimized out>, reqid=<optimized out>, pdu=0x22ab1c0, magic=<optimized out>) at snmp_agent.c:2293
            status = 0
            access_ret = <optimized out>
            asp = 0x22abdd0
            rc = <optimized out>
            reqid = <optimized out>
            pdu = 0x22ab1c0
    #20 0x00007fe91b0d8b7f in _sess_process_packet_handle_pdu (isp=0x2296380, isp=0x2296380, transport=0x2296260, pdu=0x22ab1c0, sp=0x2296990, sessp=0x2296350) at snmp_api.c:5807
            slp = 0x2296350
            rp = <optimized out>
            orp = 0x0
            handled = 1
    #21 _sess_process_packet (sessp=0x2296350, sp=0x2296990, isp=0x2296380, transport=0x2296260, opaque=<optimized out>, olength=<optimized out>, 
        packetptr=0x2299120 "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx200", length=140) at snmp_api.c:5863
            slp = 0x2296350
            rc = 0
            __func__ = "_sess_process_packet"
    #22 0x00007fe91b0da0e2 in _sess_read (sessp=0x2296350, fdset=0x7ffdc69dc780) at snmp_api.c:6124
            rcvp = {packet = 0x2299120 "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx200", packet_len = 140, opaque = 0x22d9d30, olength = 60}
            sp = 0x2296990
            isp = 0x2296380
            transport = 0x2296260
            pdulen = 36266592
            length = 0
            olength = 0
            opaque = 0x0
            __func__ = "_sess_read"
    #23 0x00007fe91b0da6b9 in snmp_sess_read2 (sessp=sessp@entry=0x2296350, fdset=fdset@entry=0x7ffdc69dc780) at snmp_api.c:6397
            psl = <optimized out>
            pss = <optimized out>
            rc = 0
    #24 0x00007fe91b0da70b in snmp_read2 (fdset=0x7ffdc69dc780) at snmp_api.c:5912
            slp = 0x2296350
    #25 0x0000000000404bd0 in ?? ()
    No symbol table info available.
    #26 0x0000000000403e75 in main ()
    No symbol table info available.
    (gdb) 
    
     
  • Sam Tannous

    Sam Tannous - 2019-02-14

    This looks very similar to a bug reported on RedHat systems:

    https://bugzilla.redhat.com/show_bug.cgi?id=1663027

     
  • Bill Fenner

    Bill Fenner - 2019-02-14

    Hi Sam! Does requesting a small messageMaxSize cause this more often, like,

    snmpbulkget --sendMessageMaxSize=%d -Cn%d -Cr%d ...

    We had some problems in this area and I forget if this was the cause. I don't remember the details of when the crash happened for us, the regression test uses

    for i in [ None, 65530, 32768, 32767, 9000, 1600, 1472 ] + range( 1400, 40, -40):

    and I have a vague memory of it starting at 1472.

    Bill

     
  • Sam Tannous

    Sam Tannous - 2019-02-14

    Yes, that helps a lot. I can crash it at will now. An snmpbulkget like this
    will crash a recent (V5-8-patches as of dc3194eaecb4 Dec 28, 2019) debug build.

    snmpbulkget -v3 -Cn1 -Cr1472 -l authPriv -u testuser -a SHA -A testshashasha -x AES -X testaesaesaes localhost 1.3.6.1.2.1.1.5 1.3.6.1.2.1.1.7

    A full backtrace looks like the following:

    (gdb) bt full                                                                                                                                                                                                                                                                                                                                     
    #0  0x00007ffff6a5b067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
    No symbol table info available.
    #1  0x00007ffff6a5c448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
    No symbol table info available.
    #2  0x00007ffff6a991b4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
    No symbol table info available.
    #3  0x00007ffff6a9e98e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
    No symbol table info available.
    #4  0x00007ffff6a9f696 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
    No symbol table info available.
    #5  0x00007ffff7391647 in usm_free_usmStateReference (old=0x97caf0) at snmpusm.c:302
            old_ref = 0x97caf0
    #6  0x00007ffff7392f48 in usm_generate_out_msg (msgProcModel=0, globalData=0xa02510 "0\202", globalDataLen=27, maxMsgSize=1472, secModel=3, secEngineID=0x929320 "\200", secEngineIDLen=17, secName=0x993390 "testuser", secNameLen=8, secLevel=3, scopedPdu=0x7fffffffd790 "0\202\005\274\004\021\200", scopedPduLen=1472, secStateRef=0x97caf0, 
        secParams=0xa0252b ' ' <repeats 200 times>..., secParamsLen=0x7fffffffd6b0, wholeMsg=0x7fffffffd6c0, wholeMsgLen=0x7fffffffdf58) at snmpusm.c:1146
            otstlen = 36
            seq_len = 32
            msgAuthParmLen = 0
            msgPrivParmLen = 0
            msgSecParmLen = 38
            authParamsOffset = 63
            privParamsOffset = 65
            datalen = 1472
            dataOffset = 65
            theTotalLength = 1537
            ptr = 0xa02510 "0\202"
            ptr_len = 140737340833347
            remaining = 0
            offSet = 140737488348736
            boots_uint = 56
            time_uint = 341
            boots_long = 56
            time_long = 341
            theName = 0x97a260 ""
            theNameLength = 0
            theEngineID = 0x929320 "\200"
            theEngineIDLength = 17
            theAuthKey = 0x0
            theAuthKeyLength = 0
            theAuthProtocol = 0x0
            theAuthProtocolLength = 0
            thePrivKey = 0x0
            thePrivKeyLength = 0
            thePrivProtocol = 0x0
            thePrivProtocolLength = 0
            theSecLevel = 0
            __func__ = "usm_generate_out_msg"
    #7  0x00007ffff73926b9 in usm_secmod_generate_out_msg (parms=0x7fffffffd6f0) at snmpusm.c:868
    No locals.
    #8  0x00007ffff733f162 in snmpv3_packet_build (session=0x9c1d30, pdu=0x97b310, packet=0xa02510 "0\202", out_length=0x7fffffffdf58, pdu_data=0x0, pdu_data_len=0) at snmp_api.c:2795
            parms = {msgProcModel = 0, globalData = 0xa02510 "0\202", globalDataLen = 27, maxMsgSize = 1472, secModel = 3, secEngineID = 0x929320 "\200", secEngineIDLen = 17, secName = 0x993390 "testuser", secNameLen = 8, secLevel = 3, scopedPdu = 0x7fffffffd790 "0\202\005\274\004\021\200", scopedPduLen = 1472, secStateRef = 0x97caf0, 
              secParams = 0xa0252b ' ' <repeats 200 times>..., secParamsLen = 0x7fffffffd6b0, wholeMsg = 0x7fffffffd6c0, wholeMsgLen = 0x7fffffffdf58, wholeMsgOffset = 0xa, pdu = 0x97b310, session = 0x9c1d30}
            global_data = 0xa02510 "0\202"
            sec_params = 0xa0252b ' ' <repeats 200 times>...
            spdu_hdr_e = 0x7fffffffd794 "\004\021\200"
            global_data_len = 27
            sec_params_len = 1437
            spdu_buf = "0\202\005\274\004\021\200\000\037\210\200\222\245\240\001\272H1\\\000\000\000\000\004\000\242\202\005\243\002\004w\333o\354\002\001\000\002\001\000\060\202\005\223\060\202\000\030\006\b+\006\001\002\001\001\005\000\004\fcel-redxp-010\202\000\r\006\b+\006\001\002\001\001\a\000\002\001H0\202\000\r\006\b+\006\001\002\001\001\b\000
    C\001\003\060\202\000\027\006\n+\006\001\002\001\001\t\001\002\001\006\t+\006\001\006\003\n\003\001\001\060\202\000\027\006\n+\006\001\002\001\001\t\001\002\002\006\t+\006\001\006\003\v\003\001\001\060\202\000\027\006\n+\006\001\002\001\001\t\001\002\003\006\t+\006\001\006\003\017\002\001\001\060\202\000\024\006\n+\006\001\002\001\001"...
            spdu_buf_len = 1468
            spdu_len = 1472
            cp = 0xa02510 "0\202"
            result = 0
            sptr = 0x8b7590
            __func__ = "snmpv3_packet_build"
    #9  0x00007ffff733c879 in snmpv3_build (pkt=0x7fffffffdf40, pkt_len=0x7fffffffdf58, offset=0x7fffffffdee0, session=0x9c1d30, pdu=0x97b310) at snmp_api.c:2272
            ret = 0
            __func__ = "snmpv3_build"
    #10 0x00007ffff733f3e8 in _snmp_build (pkt=0x7fffffffdf40, pkt_len=0x7fffffffdf58, offset=0x7fffffffdee0, session=0x9c1d30, pdu=0x97b310) at snmp_api.c:2853
            h0e = 0x0
            start_offset = 0
            version = 140737488346688
            rc = 0
            length = 140737349626967
            cp = 0x9fb6e0 "\001"
            __func__ = "_snmp_build"
    #11 0x00007ffff73406b6 in snmp_build (pkt=0x7fffffffdf40, pkt_len=0x7fffffffdf58, offset=0x7fffffffdee0, pss=0x9c1d30, pdu=0x97b310) at snmp_api.c:3192
            rc = 0
    #12 0x00007ffff7347e39 in netsnmp_build_packet (isp=0x977150, sp=0x9c1d30, pdu=0x97b310, pktbuf_p=0x7fffffffdf40, pktbuf_len_p=0x7fffffffdf50, pkt_p=0x7fffffffdf48, len_p=0x7fffffffdf58) at snmp_api.c:4953
            offset = 0
            result = 0
    #13 0x00007ffff73484c5 in _build_initial_pdu_packet (slp=0x98e250, pdu=0x97b310, bulk=1) at snmp_api.c:5114
            session = 0x9c1d30
            isp = 0x977150
            transport = 0x976340
            pktbuf = 0xa02510 "0\202"
            packet = 0xa02510 "0\202"
            pktbuf_len = 1472
            offset = 0
            length = 1472
            orig_length = 2437
            result = 0
            orig_count = 99
            curr_count = 99
            __func__ = "_build_initial_pdu_packet"
    #14 0x00007ffff7b9a9ff in netsnmp_wrap_up_request (asp=0x9697d0, status=0) at snmp_agent.c:2007
            slp = 0x98e250
            __func__ = "netsnmp_wrap_up_request"
    #15 0x00007ffff7b9f271 in netsnmp_handle_request (asp=0x9697d0, status=0) at snmp_agent.c:3743
            __func__ = "netsnmp_handle_request"
    #16 0x00007ffff7b9b309 in handle_snmp_packet (op=1, session=0x9c1d30, reqid=2010869740, pdu=0x993f00, magic=0x0) at snmp_agent.c:2283
            asp = 0x9697d0
            status = 0
            access_ret = 0
            rc = 60
            __func__ = "handle_snmp_packet"
    #17 0x00007ffff734a282 in _sess_process_packet_handle_pdu (sessp=0x98e250, sp=0x9c1d30, isp=0x977150, transport=0x976340, pdu=0x993f00) at snmp_api.c:5807
            slp = 0x98e250
            rp = 0x7ffff7380390 <netsnmp_transport_recv+268>
            orp = 0x0
            handled = 1
            __func__ = "_sess_process_packet_handle_pdu"
    #18 0x00007ffff734a409 in _sess_process_packet (sessp=0x98e250, sp=0x9c1d30, isp=0x977150, transport=0x976340, opaque=0x97b130, olength=60, packetptr=0x9d6e20 "0\201\231\002\001\003\060\020\002\004C5\362\024\002\002\005\300\004\001\a\002\001\003\004>0<\004\021\200", length=156) at snmp_api.c:5863
            slp = 0x98e250
            pdu = 0x993f00
            rc = 0
            __func__ = "_sess_process_packet"
    #19 0x00007ffff734afc6 in _sess_read (sessp=0x98e250, fdset=0x7fffffffe2d0) at snmp_api.c:6124
            rcvp = {packet = 0x9d6e20 "0\201\231\002\001\003\060\020\002\004C5\362\024\002\002\005\300\004\001\a\002\001\003\004>0<\004\021\200", packet_len = 156, opaque = 0x97b130, olength = 60}
            slp = 0x98e250
            sp = 0x9c1d30
            isp = 0x977150
            transport = 0x976340
            pdulen = 0
            rxbuf_len = 65536
            rxbuf = 0x0
            length = 0
            olength = 0
            rc = 0
            opaque = 0x0
            __func__ = "_sess_read"
    #20 0x00007ffff734beaf in snmp_sess_read2 (sessp=0x98e250, fdset=0x7fffffffe2d0) at snmp_api.c:6397
            psl = 0x962f10
            pss = 0x9c1f20
            rc = 0
    #21 0x00007ffff734a5eb in snmp_read2 (fdset=0x7fffffffe2d0) at snmp_api.c:5912
            slp = 0x98e250
    #22 0x00000000004051f4 in receive () at snmpd.c:1353
            numfds = 13
            readfds = {lfs_setsize = 1024, lfs_setptr = 0x7fffffffe2e0, lfs_set = {fds_bits = {0 <repeats 16 times>}}}
            writefds = {lfs_setsize = 1024, lfs_setptr = 0x7fffffffe370, lfs_set = {fds_bits = {0 <repeats 16 times>}}}
            exceptfds = {lfs_setsize = 1024, lfs_setptr = 0x7fffffffe400, lfs_set = {fds_bits = {0 <repeats 16 times>}}}
            timeout = {tv_sec = 4, tv_usec = 6470}
            tvp = 0x7fffffffe2c0
            count = 1
            block = 0
            i = 32
            sd = 32767
            __func__ = "receive"
    #23 0x000000000040489f in main (argc=11, argv=0x7fffffffe648) at snmpd.c:1137
            options = "aAc:CdD::fhHI:l:L:m:M:n:p:P:qrsS:UvV-:Y:yg:u:x:X"
            arg = -1
            i = 11
            ret = 0
            exit_code = 1
            dont_fork = 1
            do_help = 0
            log_set = 0
            agent_mode = -1
            pid_file = 0x7fffffffe8d8 "/run/snmpd.pid"
            option_compatability = "-Le"
            prepared_sockets = 0
            fd = 13
            PID = 0x9c2110
            __func__ = "main"
    (gdb) 
    
     
  • Sam Tannous

    Sam Tannous - 2019-02-14

    It looks like the pdu->securityStateRef is never set correctly in this case.
    At least all the up to frame 18:

    (gdb) frame 16
    #16 0x00007ffff7b9b309 in handle_snmp_packet (op=1, session=0x9c1d30, reqid=2010869740, pdu=0x993f00, magic=0x0) at snmp_agent.c:2283
    2283        rc = netsnmp_handle_request(asp, status);
    (gdb) p *asp->pdu
    $22 = {version = 3, command = 162, reqid = 2010869740, msgid = 1127608852, transid = 53, sessid = 0, errstat = 0, errindex = 0, time = 0, flags = 98308, securityModel = 3, securityLevel = 3, msgParseModel = 0, msgMaxSize = 1472, transport_data = 0x973280, transport_data_length = 60, tDomain = 0x7ffff75cf140 <netsnmpUDPDomain>, tDomainLen = 7, 
      variables = 0x9ccec0, community = 0x0, community_len = 0, enterprise = 0x0, enterprise_length = 0, trap_type = 0, specific_type = 0, agent_addr = "\000\000\000", contextEngineID = 0x97b430 "\200", contextEngineIDLen = 17, contextName = 0x97c9e0 "", contextNameLen = 0, securityEngineID = 0x929320 "\200", securityEngineIDLen = 17, 
      securityName = 0x993390 "testuser", securityNameLen = 8, priority = 0, range_subid = 0, securityStateRef = 0x97caf0}
    (gdb) p *pdu
    $23 = {version = 3, command = 165, reqid = 2010869740, msgid = 1127608852, transid = 53, sessid = 0, errstat = 1, errindex = 1472, time = 0, flags = 4, securityModel = 3, securityLevel = 3, msgParseModel = 0, msgMaxSize = 1472, transport_data = 0x97b130, transport_data_length = 60, tDomain = 0x7ffff75cf140 <netsnmpUDPDomain>, tDomainLen = 7, 
      variables = 0x9c2520, community = 0x0, community_len = 0, enterprise = 0x0, enterprise_length = 0, trap_type = 0, specific_type = 0, agent_addr = "\000\000\000", contextEngineID = 0x97eea0 "\200", contextEngineIDLen = 17, contextName = 0x98f400 "", contextNameLen = 0, securityEngineID = 0x978250 "\200", securityEngineIDLen = 17, 
      securityName = 0x97cca0 "testuser", securityNameLen = 8, priority = 0, range_subid = 0, securityStateRef = 0x97caf0}
    (gdb) frame 17
    #17 0x00007ffff734a282 in _sess_process_packet_handle_pdu (sessp=0x98e250, sp=0x9c1d30, isp=0x977150, transport=0x976340, pdu=0x993f00) at snmp_api.c:5807
    5807          sp->callback(NETSNMP_CALLBACK_OP_RECEIVED_MESSAGE,
    (gdb) p *pdu
    $24 = {version = 3, command = 165, reqid = 2010869740, msgid = 1127608852, transid = 53, sessid = 0, errstat = 1, errindex = 1472, time = 0, flags = 4, securityModel = 3, securityLevel = 3, msgParseModel = 0, msgMaxSize = 1472, transport_data = 0x97b130, transport_data_length = 60, tDomain = 0x7ffff75cf140 <netsnmpUDPDomain>, tDomainLen = 7, 
      variables = 0x9c2520, community = 0x0, community_len = 0, enterprise = 0x0, enterprise_length = 0, trap_type = 0, specific_type = 0, agent_addr = "\000\000\000", contextEngineID = 0x97eea0 "\200", contextEngineIDLen = 17, contextName = 0x98f400 "", contextNameLen = 0, securityEngineID = 0x978250 "\200", securityEngineIDLen = 17, 
      securityName = 0x97cca0 "testuser", securityNameLen = 8, priority = 0, range_subid = 0, securityStateRef = 0x97caf0}
    (gdb) p  *(struct usmStateReference *)pdu->secStateRef
    There is no member named secStateRef.
    (gdb) p  *(struct usmStateReference *)pdu->securityStateRef
    $25 = {usr_name = 0x97a260 "", usr_name_length = 0, usr_engine_id = 0x0, usr_engine_id_length = 0, usr_auth_protocol = 0x0, usr_auth_protocol_length = 0, usr_auth_key = 0x0, usr_auth_key_length = 0, usr_priv_protocol = 0x0, usr_priv_protocol_length = 0, usr_priv_key = 0x0, usr_priv_key_length = 0, usr_sec_level = 0}
    (gdb) frame 18
    #18 0x00007ffff734a409 in _sess_process_packet (sessp=0x98e250, sp=0x9c1d30, isp=0x977150, transport=0x976340, opaque=0x97b130, olength=60, packetptr=0x9d6e20 "0\201\231\002\001\003\060\020\002\004C5\362\024\002\002\005\300\004\001\a\002\001\003\004>0<\004\021\200", length=156) at snmp_api.c:5863
    5863            rc = _sess_process_packet_handle_pdu(sessp, sp, isp, transport, pdu);
    (gdb) p  *(struct usmStateReference *)pdu->securityStateRef
    $26 = {usr_name = 0x97a260 "", usr_name_length = 0, usr_engine_id = 0x0, usr_engine_id_length = 0, usr_auth_protocol = 0x0, usr_auth_protocol_length = 0, usr_auth_key = 0x0, usr_auth_key_length = 0, usr_priv_protocol = 0x0, usr_priv_protocol_length = 0, usr_priv_key = 0x0, usr_priv_key_length = 0, usr_sec_level = 0}
    
     
  • Sam Tannous

    Sam Tannous - 2019-02-14

    deleted...this will never work.

     

    Last edit: Sam Tannous 2019-02-15
  • Sam Tannous

    Sam Tannous - 2019-02-15

    Valgrind shows that memory is freed twice:

    ==8200== Invalid free() / delete / delete[] / realloc()
    ==8200==    at 0x4C29E90: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==8200==    by 0x56147F0: usm_free_usmStateReference (snmpusm.c:322)
    ==8200==    by 0x5619747: usm_rgenerate_out_msg (snmpusm.c:1941)
    ==8200==    by 0x56172DF: usm_secmod_rgenerate_out_msg (snmpusm.c:1421)
    ==8200==    by 0x55C1A2E: snmpv3_packet_realloc_rbuild (snmp_api.c:2681)
    ==8200==    by 0x55BF848: snmpv3_build (snmp_api.c:2268)
    ==8200==    by 0x55C23E7: _snmp_build (snmp_api.c:2853)
    ==8200==    by 0x55C36B5: snmp_build (snmp_api.c:3192)
    ==8200==    by 0x55CADCB: netsnmp_build_packet (snmp_api.c:4946)
    ==8200==    by 0x55CB4C4: _build_initial_pdu_packet (snmp_api.c:5114)
    ==8200==    by 0x55CBBCF: _sess_async_send (snmp_api.c:5241)
    ==8200==    by 0x55CC0F7: snmp_sess_async_send (snmp_api.c:5398)
    ==8200==  Address 0x8e2e150 is 0 bytes inside a block of size 104 free'd
    ==8200==    at 0x4C29E90: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==8200==    by 0x56147F0: usm_free_usmStateReference (snmpusm.c:322)
    ==8200==    by 0x5619747: usm_rgenerate_out_msg (snmpusm.c:1941)
    ==8200==    by 0x56172DF: usm_secmod_rgenerate_out_msg (snmpusm.c:1421)
    ==8200==    by 0x55C1A2E: snmpv3_packet_realloc_rbuild (snmp_api.c:2681)
    ==8200==    by 0x55BF848: snmpv3_build (snmp_api.c:2268)
    ==8200==    by 0x55C23E7: _snmp_build (snmp_api.c:2853)
    ==8200==    by 0x55C36B5: snmp_build (snmp_api.c:3192)
    ==8200==    by 0x55CADCB: netsnmp_build_packet (snmp_api.c:4946)
    ==8200==    by 0x55CB4C4: _build_initial_pdu_packet (snmp_api.c:5114)
    ==8200==    by 0x4E779FE: netsnmp_wrap_up_request (snmp_agent.c:2007)
    ==8200==    by 0x4E7C270: netsnmp_handle_request (snmp_agent.c:3743)
    ==8200==  Block was alloc'd at
    ==8200==    at 0x4C2AD10: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==8200==    by 0x5614602: usm_malloc_usmStateReference (snmpusm.c:288)
    ==8200==    by 0x561ADEE: usm_process_in_msg (snmpusm.c:2465)
    ==8200==    by 0x561AC0E: usm_secmod_process_in_msg (snmpusm.c:2369)
    ==8200==    by 0x55C75A9: snmpv3_parse (snmp_api.c:3937)
    ==8200==    by 0x55C8AA6: _snmp_parse (snmp_api.c:4347)
    ==8200==    by 0x55C92F6: snmp_parse (snmp_api.c:4527)
    ==8200==    by 0x55CCAEB: _sess_process_packet_parse_pdu (snmp_api.c:5618)
    ==8200==    by 0x55CD3D0: _sess_process_packet (snmp_api.c:5852)
    ==8200==    by 0x55CDFC5: _sess_read (snmp_api.c:6124)
    ==8200==    by 0x55CEEAE: snmp_sess_read2 (snmp_api.c:6397)
    ==8200==    by 0x55CD5EA: snmp_read2 (snmp_api.c:5912)
    ==8200== 
    

    I don't quite understand how this can happen. It looks like the calloc address is slightly different then the two freed addresses.

     
  • Sam Tannous

    Sam Tannous - 2019-02-15

    Just noticed this " * FIX Memory leaks if secStateRef ..." I guess we need to fix this ;-)

    /*******************************************************************-o-******
     * usm_process_in_msg
     *
     * Parameters:
     *      (See list below...)
     *      
     * Returns:
     *      SNMPERR_SUCCESS                 On success.
     *      SNMPERR_USM_AUTHENTICATIONFAILURE
     *      SNMPERR_USM_DECRYPTIONERROR
     *      SNMPERR_USM_GENERICERROR
     *      SNMPERR_USM_PARSEERROR
     *      SNMPERR_USM_UNKNOWNENGINEID
     *      SNMPERR_USM_PARSEERROR
     *      SNMPERR_USM_UNKNOWNSECURITYNAME
     *      SNMPERR_USM_UNSUPPORTEDSECURITYLEVEL
     *
     *
     * ASSUMES size of decrypt_buf will always be >= size of encrypted sPDU.
     *
     * FIX  Memory leaks if secStateRef is allocated and a return occurs
     *      without cleaning up.  May contain secrets...
     */
    int
    usm_process_in_msg(int msgProcModel,    /* (UNUSED) */
                       size_t maxMsgSize,   /* IN     - Used to calc maxSizeResponse.  */
                       u_char * secParams,  /* IN     - BER encoded securityParameters. */
                       int secModel,        /* (UNUSED) */
                       int secLevel,        /* IN     - AuthNoPriv, authPriv etc.      */
                       u_char * wholeMsg,   /* IN     - Original v3 message.           */
                       size_t wholeMsgLen,  /* IN     - Msg length.                    */
                       u_char * secEngineID,        /* OUT    - Pointer snmpEngineID.          */
                       size_t * secEngineIDLen,     /* IN/OUT - Len available, len returned.   */
                       /*
                        * NOTE: Memory provided by caller.      
                        */
                       char *secName,       /* OUT    - Pointer to securityName.       */
                       size_t * secNameLen, /* IN/OUT - Len available, len returned.   */
                       u_char ** scopedPdu, /* OUT    - Pointer to plaintext scopedPdu. */
                       size_t * scopedPduLen,       /* IN/OUT - Len available, len returned.   */
                       size_t * maxSizeResponse,    /* OUT    - Max size of Response PDU.      */
                       void **secStateRf,   /* OUT    - Ref to security state.         */
                       netsnmp_session * sess,      /* IN     - session which got the message  */
                       u_char msg_flags)
    {                               /* IN     - v3 Message flags.              */
    
     
    • Bill Fenner

      Bill Fenner - 2019-02-17

      Our analysis ended up being that this was an API flaw and the code added in
      5.8 could never have worked - when rebuilding the pdu forward, we need the
      info that was freed while trying to build the pdu backward. I think the
      right fix is to move the functionality into a new function whose API is to
      not free on failure, and use that when building if you think you might try
      to rebuild the pdu forward.

      Bill

      On Fri, Feb 15, 2019 at 3:46 PM Sam Tannous stannous-cn@users.sourceforge.net wrote:

      Just noticed this " * FIX Memory leaks if secStateRef ..." I guess we need
      to fix this ;-)

      /**********-o- * usm_process_in_msg * Parameters: * (See list below...) * * Returns: * SNMPERR_SUCCESS On success. * SNMPERR_USM_AUTHENTICATIONFAILURE * SNMPERR_USM_DECRYPTIONERROR * SNMPERR_USM_GENERICERROR * SNMPERR_USM_PARSEERROR * SNMPERR_USM_UNKNOWNENGINEID * SNMPERR_USM_PARSEERROR * SNMPERR_USM_UNKNOWNSECURITYNAME * SNMPERR_USM_UNSUPPORTEDSECURITYLEVEL * * ASSUMES size of decrypt_buf will always be >= size of encrypted sPDU. * FIX Memory leaks if secStateRef is allocated and a return occurs * without cleaning up. May contain secrets... /
      int
      usm_process_in_msg(int msgProcModel, /
      (UNUSED) / size_t maxMsgSize, / IN - Used to calc maxSizeResponse. / u_char * secParams, / IN - BER encoded securityParameters. / int secModel, / (UNUSED) / int secLevel, / IN - AuthNoPriv, authPriv etc. / u_char * wholeMsg, / IN - Original v3 message. / size_t wholeMsgLen, / IN - Msg length. / u_char * secEngineID, / OUT - Pointer snmpEngineID. / size_t * secEngineIDLen, / IN/OUT - Len available, len returned. / / * NOTE: Memory provided by caller. / char secName, / OUT - Pointer to securityName. / size_t * secNameLen, / IN/OUT - Len available, len returned. / u_char
      scopedPdu, / OUT - Pointer to plaintext scopedPdu. / size_t * scopedPduLen, / IN/OUT - Len available, len returned. / size_t * maxSizeResponse, / OUT - Max size of Response PDU. / void secStateRf, / OUT - Ref to security state. / netsnmp_session * sess, / IN - session which got the message / u_char msg_flags){ / IN - v3 Message flags. /


      Status: open
      Group: linux
      Created: Thu Feb 14, 2019 12:22 AM UTC by Sam Tannous
      Last Updated: Fri Feb 15, 2019 09:29 PM UTC
      Owner: nobody
      Attachments:

      Merged code from V5-8-patches (Nov 2018) running on Linux Debian Jessie
      based system is seeing
      regular core dumps.

      The problems appears to be a double free in snmplib/snmpusm.c: 1936
      usm_free_usmStateReference(secStateRef)

      Would an acceptable fix be to just check this pointer before freeing? I
      don't see how this is
      can be triggered (since I can't recreate the problem).

      syslogs and full backtrace is attached.

      Sent from sourceforge.net because you indicated interest in
      https://sourceforge.net/p/net-snmp/bugs/2923/

      To unsubscribe from further messages, please visit
      https://sourceforge.net/auth/subscriptions/

       

      Related

      Bugs: #2923

  • Sam Tannous

    Sam Tannous - 2019-02-21

    (v3doublefree2.patch is slightly better in that it creates an snmp_api function
    snmp_free_securityStateRef() that handles only the securityStateRef freeing.)

    This patch avoids the free on failures. It also avoids the double free problem if
    free_securityStateRef() is done in snmp_free_pdu().

    Instead, the free for pdu->securityStateRef is done in free_agent_snmp_session() just before
    the pdu(s) are freed. As a bonus, it does not appear to leak memory ;-) I had to remove the
    "static" from free_securityStateRef in snmp_api so I could call it from snmp_agent.c.
    I'm not sure if this is the right thing to do here so comments are welcome.

    --Sam Tannous

     

    Last edit: Sam Tannous 2019-02-25
  • Bart Van Assche

    Bart Van Assche - 2019-04-10

    Version 3 of this patch has been applied on the v5.8 and master branches. Thanks for the patch!

     
  • Bart Van Assche

    Bart Van Assche - 2019-04-10
    • status: open --> closed
     
  • Ming Chen

    Ming Chen - 2019-06-05

    Hi Bart / Sam,

    There is a minor bug in this version 3 patch.

    +        if (old_ref->usr_auth_protocol_length)
    +            SNMP_FREE(old_ref->usr_auth_protocol);
    +        if (old_ref->usr_auth_protocol_length)
    +            SNMP_FREE(old_ref->usr_priv_protocol);
    

    The variable usr_auth_protocol_length should be usr_priv_protocol_length.

     

    Last edit: Ming Chen 2019-06-05
    • Bart Van Assche

      Bart Van Assche - 2019-06-06

      This patch has been checked in on the v5.8 and master branches. Thanks for the patch!

       
      • Ming Chen

        Ming Chen - 2019-06-06

        Thanks. Do you think we need to keep the condition style consistent with the original patch? Or update the orginal to yours. Please see the incline comment as below. But I'm OK if you don't want to make another change.

        void
        usm_free_usmStateReference(void *old)
        {
            struct usmStateReference *old_ref = (struct usmStateReference *) old;
        
            if (old_ref) {
        
                if (old_ref->usrnamelength)
                    SNMP_FREE(old_ref->usr_name);
                if (old_ref->usr_engine_id_length)
                    SNMP_FREE(old_ref->usr_engine_id);
                if (old_ref->usr_auth_protocol_length) 
                    SNMP_FREE(old_ref->usr_auth_protocol);
                 // Ming's comment, using usr_priv_protocol_length?
                if (old_ref->usr_priv_protocol)
                    SNMP_FREE(old_ref->usr_priv_protocol);
        
                if (old_ref->usr_auth_key_length && old_ref->usr_auth_key) {
                    SNMP_ZERO(old_ref->usr_auth_key, old_ref->usr_auth_key_length);
                    SNMP_FREE(old_ref->usr_auth_key);
                }
                if (old_ref->usr_priv_key_length && old_ref->usr_priv_key) {
                    SNMP_ZERO(old_ref->usr_priv_key, old_ref->usr_priv_key_length);
                    SNMP_FREE(old_ref->usr_priv_key);
                }
        
                SNMP_ZERO(old_ref, sizeof(*old_ref));
                SNMP_FREE(old_ref);
        
            }
        
        }  
        
         

        Last edit: Ming Chen 2019-06-06
        • Bart Van Assche

          Bart Van Assche - 2019-06-06

          It seems like I misread your patch. I have replaced the patch that had been pushed out by a version that uses usr_priv_protocol_length instead of usr_priv_protocol.

           
          • Ming Chen

            Ming Chen - 2019-06-06

            Thanks for your patience , Bart.
            Also, there is another change for formatting the code, as part of the original patch doesn't comply with the net-snmp spacing style. Please see the attched patch file.

             
            • Bart Van Assche

              Bart Van Assche - 2019-06-06

              That patch has been applied. Thanks for the patch.

               
  • Bart Van Assche

    Bart Van Assche - 2019-07-31

    Sam, it would help a lot if a reproducer for this issue would be added to the Net-SNMP source tree in the form of a regression test. That will help to prevent that this issue gets reintroduced. Do you perhaps know how to reproduce this issue easily?

     
  • Sam Tannous

    Sam Tannous - 2019-07-31

    With a hint from Bill, see 4th comment above, I could reliably reproduce it with something like this:

    snmpbulkget  -v3 -Cn1 -Cr1472  -lauthPriv -u testuser -a SHA -A testsha1234 -x AES -X testaes1234 localhost    1.3.6.1.2.1.1.5 1.3.6.1.2.1.1.7
    
     
  • Bart Van Assche

    Bart Van Assche - 2019-08-02

    My attempts so far to reproduce the double free with the following test were unsuccessful:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    #!/bin/sh
    
    . ../support/simple_eval_tools.sh
    
    HEADER SNMPv3 bulkget ...
    
    SKIPIFNOT USING_MIBII_SYSTEM_MIB_MODULE
    
    snmp_version=v3
    . ./Sv3usmconfigagent
    
    STARTAGENT
    
    CAPTURE "snmpbulkget $SNMP_FLAGS -On -Cn1 -Cr1472 $TESTPRIVARGS $SNMP_TRANSPORT_SPEC:$SNMP_TEST_DEST$SNMP_SNMPD_PORT sysUpTime sysUpTime sysUpTime sysUpTime sysUpTime .1 .1"
    
    STOPAGENT
    
    CHECKCOUNT atleastone "^\.1\.3\.6\.1\.2\.1\.1\.3\.0 = Timeticks: "
    CHECKCOUNT 97 "^\.1\."
    
    FINISHED
    
     

Log in to post a comment.