#2360 snmpnetstat shows wrong data

linux
open
apps (205)
5
2014-01-30
2012-05-01
dxr
No

- Server -
* Operation System: Debian Squeeze
# cat /etc/debian_version && uname -a
6.0.4
Linux c1 2.6.32-5-amd64 #1 SMP Mon Jan 9 20:49:59 UTC 2012 x86_64 GNU/Linux

* SNMPD (apt installed):
# snmpd -v

NET-SNMP version: 5.4.3
Web: http://www.net-snmp.org/
Email: net-snmp-coders@lists.sourceforge.net

- Client -
* Operation System: Debian Squeeze
# cat /etc/debian_version && uname -a
6.0.4
Linux monitor.xxxxxxxx.com 2.6.32-5-amd64 #1 SMP Mon Jan 16 16:22:28 UTC 2012 x86_64 GNU/Linux

* SNMPD (apt installed):
# snmpnetstat -V
NET-SNMP version: 5.4.3

After several tests we can see the data is very diferent from real scene:

Test from the client:

monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -t 10 -Cn -Cp tcp -v 2c -c changed|wc -l
10
monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -t 10 -Cn -Cp tcp -v 2c -c changed|wc -l
10
monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -t 10 -Cn -Cp tcp -v 2c -c changed|wc -l
10
monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -t 10 -Cn -Cp tcp -v 2c -c changed|wc -l
10
monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -t 10 -Cn -Cp tcp -v 2c -c changed|wc -l
10
monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -t 10 -Cn -Cp tcp -v 2c -c changed|wc -l
10
monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -t 10 -Cn -Cp tcp -v 2c -c changed|wc -l
10
monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -t 10 -Cn -Cp tcp -v 2c -c changed|wc -l
10
monitor:~#

"Simultaneous" in the server:

root@c1:~# ss -n --tcp|wc -l
466
root@c1:~# ss -n --tcp|wc -l
471
root@c1:~# ss -n --tcp|wc -l
473
root@c1:~# ss -n --tcp|wc -l
481
root@c1:~# ss -n --tcp|wc -l
486
root@c1:~# ss -n --tcp|wc -l
488
root@c1:~# ss -n --tcp|wc -l
492
root@c1:~# ss -n --tcp|wc -l
490
root@c1:~# ss -n --tcp|wc -l
465
root@c1:~#

I tested it in some servers that has < 5 concurrent conections and it looks OK, but in servers with more traffic, it's different. I can see some strange things:

- Snmpnetstat doesn't show 127.0.0.1 connections (if i ignore it in ss command, it doesn't resolve the problem)
- Snmpnetstats show this states:

monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -t 10 -Cn -Cp tcp -v 2c -c changed
Active Internet (tcp) Connections
Proto Local Address Remote Address (state)
tcp XX.XX.XX.130.25 XXX.XX.X.174.55778 TIMEWAIT
tcp XX.XX.XX.130.25 XXX.XX.X.174.55964 TIMEWAIT
tcp XX.XX.XX.130.22 XX.XX.XX.XX.36800 ESTABLISHED
tcp XX.XX.XX.133.80 X.X.X.X.45.51679 SYNRECEIVED
tcp XX.XX.XX.136.80 X.X.X.X.2084 SYNRECEIVED
tcp XX.XX.XX.136.80 XX.X.X.X.2090 SYNRECEIVED
tcp XX.XX.XX.136.80 XX.X.X.X.2091 SYNRECEIVED
tcp XX.XX.XX.144.80 X.XXX.XXX.X.7845 SYNRECEIVED
tcp XX.XX.XX.144.80 X.XXX.XXX.X.15808 SYNRECEIVED
tcp XX.XX.XX.144.80 X.XXX.XXX.X.34080 SYNRECEIVED
tcp XX.XX.XX.144.80 X.XXX.XXX.X.49823 SYNRECEIVED
tcp XX.XX.XX.166.80 XX.XX.XX.X.1292 SYNRECEIVED
tcp XX.XX.XX.171.80 XX.XXX.XX.X.56590 SYNRECEIVED
tcp XX.XX.XX.188.80 XXX.XX.X.XX.52352 SYNRECEIVED
tcp XX.XX.XX.188.80 XXX.XX.X.XX.52355 SYNRECEIVED
tcp XX.XX.XX.188.80 XXX.XX.X.XX.52356 SYNRECEIVED
tcp XX.XX.XX.188.80 XXX.XX.X.XX.52358 SYNRECEIVED
tcp XX.XX.XX.188.80 XXX.XX.X.XX.52371 SYNRECEIVED
tcp XX.XX.XX.188.80 XXX.XX.XXX.X.49463 SYNRECEIVED

root@c1:~# ss -n --tcp|grep ESTA|wc -l
493
root@c1:~# ss -n --tcp|grep ESTA|wc -l
467
root@c1:~# ss -n --tcp|grep ESTA|wc -l
469
root@c1:~#

Thanks

Discussion

  • Niels Baggesen
    Niels Baggesen
    2012-05-02

    How many of your connections are (ipv4 mapped) ipv6?
    What do you see from snmpwalk XX.XX.XX.130 tcpConnectionState? It should give you the full story.
    I guess the problem is that the current snmpnetstat does not support the current tcpConnectionTable, but only the oldfashioned ipv4-only tcpConnTable

     
  • dxr
    dxr
    2012-05-02

    I use ipv4 only. I show some data:

    monitor:~# /usr/bin/snmpwalk XX.XX.XX.130 -v 2c -c changed tcpConnTable|wc -l
    82
    monitor:~# /usr/bin/snmpwalk XX.XX.XX.130 -v 2c -c changed tcpConnectionState|wc -l
    15
    monitor:~# /usr/bin/snmpwalk 77.240.119.130 -v 2c -c cvcact3r tcpConnTable|wc -l
    85
    monitor:~# /usr/bin/snmpwalk 77.240.119.130 -v 2c -c cvcact3r tcpConnectionState|wc -l
    15
    monitor:~# /usr/bin/snmpwalk XX.XX.XX.130 -v 2c -c changed tcpConnTable|wc -l
    85
    monitor:~# /usr/bin/snmpwalk XX.XX.XX.130 -v 2c -c changed tcpConnectionState|wc -l
    15
    monitor:~# /usr/bin/snmpwalk XX.XX.XX.130 -v 2c -c changed ipv6TcpConnTable
    IPV6-TCP-MIB::ipv6TcpConnTable = No Such Object available on this agent at this OID
    monitor:~#

    Thanks

     
  • Niels Baggesen
    Niels Baggesen
    2012-05-02

    Even though you don't use ipv6, some sockets may be. For instance I have (from ss)

    ESTAB 0 318 ::ffff:10.0.0.2:imap ::ffff:130.225.26.35:jmq-daemon-2
    ESTAB 0 318 ::ffff:10.0.0.2:imap ::ffff:130.225.26.35:35240

    which are ipv4-mapped ipv6, running ipv4, but dont show up in tcpConnTable.

    Are there any message from snmpd on the server side?
    Any correlation between the sessions you see and their place in /proc/net/tcp?

     
  • Niels Baggesen
    Niels Baggesen
    2012-05-02

    One more thing ... what happens if you use SNMPv1?

     
  • dxr
    dxr
    2012-05-03

    ss show some ip4-mapped

    root@c1:~# ss -n|grep --color ffff|wc -l
    18
    root@c1:~#

    With v 1 the result is the same:

    monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -v 2c -c changed tcpConnTablea|wc -l
    15
    monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -v 1 -c changed tcpConnTablea|wc -l
    18
    monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -v 2c -c changed tcpConnTablea|wc -l
    18
    monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -v 1 -c changed tcpConnTablea|wc -l
    17
    monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -v 2c -c changed tcpConnTablea|wc -l
    17
    monitor:~# /usr/bin/snmpnetstat XX.XX.XX.130 -v 1 -c changed tcpConnTablea|wc -l
    15
    monitor:~#

    root@c1:~# cat /proc/net/tcp|wc -l
    12
    root@c1:~# cat /proc/net/tcp|wc -l
    12
    root@c1:~# cat /proc/net/tcp|wc -l
    11
    root@c1:~# cat /proc/net/tcp|wc -l
    14
    root@c1:~# cat /proc/net/tcp|wc -l
    10
    root@c1:~# cat /proc/net/tcp|wc -l
    9
    root@c1:~# cat /proc/net/tcp|wc -l
    9
    root@c1:~# cat /proc/net/tcp|wc -l
    10
    root@c1:~#

     
  • Niels Baggesen
    Niels Baggesen
    2012-05-07

    That sounds very strange. Would you mind mailing me (privately) the complete output from snmpwalk of tcpConnectionTable, ss --tcp, and a cat of /pro/net/tcp and /proc/net/tcp6?

     
  • dxr
    dxr
    2012-05-15

    I sent you privately some days ago.

     
  • Niels Baggesen
    Niels Baggesen
    2012-05-16

    Hmm, I have not received it, and I cannot immediately find anything in my spam filters.
    Could you try sending it again and then to my private mail (niels@baggesen.net)