Python/CRASH API Wiki

Brought to you by: alexsid, mooremar

xportshow

Xportshow

Overview

xportshow is a tool to extract and print the networking transport related information from linux kernel dumps. It is implemented as an extension to crash dump-analysis tool using PyKdump framework.

crash already has a builtin net command, but it can do just three things:

print information based on pid, socket, or filedescriptor
print ARP-table
print networking devices

xportshow can print many more things than net. It should be useful for anybody with basic networking knowledge, even though some output will be meaningful only for those who understand the details of Linux TCP/IP implementation. Most short options are the same as in standard netstat command.

Printing all available options

crash64> xportshow -h
Usage: xportshow.py [options]

Options:
  -h, --help            show this help message and exit
  -a                    print all sockets
  -v                    verbose output
  -r                    Print routing table. Adding -v prints all routing
                        tables and policies
  --program=PROGRAM     print sockets for cmdname
  --pid=PID             print sockets for PID
  --netfilter           Print Netfilter Hooks
  --softnet             Print Softnet Queues
  --summary             Print A Summary
  -s, --statistics      Print Statistics
  -i                    Print Interface Info
  --interface=IF1       Limit output to the specified interface only
  --decode=DECODE       Decode iph/th/uh
  --port=PORT           Limit output to the specified port (src or dst)
  -l, --listening       Print LISTEN sockets only
  -t, --tcp             Print TCP Info
  -u, --udp             Print UDP Info
  -w, --raw             Print RAW Info
  -x, --unix            Print UNIX Info
  --sysctl              Print sysctl info for net.
  --devpack             Print dev_pack info
  --arp                 Print ARP & Neighbouring info
  --rtcache             Print the routing cache
  --skbuffhead=SKBUFFHEAD
                        Print sk_buff_head
  --version             Print program version and exit
  --everything          Run all functions available for regression testing

  Experimental Options:
    Caution: this is work in progress, not fully supported for all kernels
    yet.

    --sport=SPORT       Limit output to the specified sport
    --dport=DPORT       Limit output to the specified dport
    --ipsec             Print IPSEC stuff
    --profile           Run with profiler

'-v' option (verbose) can be used several times (e.g. '-vv'); this increases the amount of output for some subcommands.

Please note that options can be short (one letter preceded with one dash) and long (a word preceded with two dashes). The options parser can identify a long option by several first characters as soon as there is no ambiguity. For example, if you have two long options --stacksummary and --stdout, you cannot use --st, but --stack and --std should be good enough. With short options, the option argument (if any) should be separated by space. For long options, it can be either space or equals '=' sign. Examples:

-d 1 --decode 1 --decode=1

IPv6 support is not tested as extensively as IPv4, but it should mostly work. The program has been tested for many kernels starting from 2.4.21 up to 3.2.0 on Debian, RH, SLES and Ubuntu distributions.

TCP Sockets

crash32> xportshow -at
tcp   0.0.0.0:42691              0.0.0.0:*                   LISTEN
tcp   127.0.0.1:9161             0.0.0.0:*                   LISTEN
tcp   0.0.0.0:8010               0.0.0.0:*                   LISTEN
tcp   127.0.0.1:9165             0.0.0.0:*                   LISTEN
tcp   0.0.0.0:111                0.0.0.0:*                   LISTEN
tcp6  :::22                      :::*                        LISTEN
tcp   127.0.0.1:631              0.0.0.0:*                   LISTEN
tcp   15.236.177.25:52414        16.236.16.79:5223           ESTABLISHED
tcp   15.236.177.25:35016        15.37.113.20:143            ESTABLISHED
tcp   15.236.177.25:60457        16.236.31.2:143             ESTABLISHED
tcp   15.236.177.25:53004        69.159.122.174:22           ESTABLISHED
tcp   15.236.177.25:35015        15.37.113.20:143            ESTABLISHED
tcp   127.0.0.1:47939            127.0.0.1:9165              ESTABLISHED
tcp   127.0.0.1:9165             127.0.0.1:47939             ESTABLISHED
tcp   127.0.0.1:9161             127.0.0.1:54388             TIME_WAIT
tcp   127.0.0.1:9161             127.0.0.1:54387             TIME_WAIT
 ** Execution took   0.03s (real)   0.02s (CPU)

TCP Sockets Verbose

Here we show what can be printed for TCP connections in different states

crash32> xportshow -atv
------------------------------------------------------------------------------
<struct tcp_sock 0xf62c8000> TCP
tcp 0.0.0.0:42691 0.0.0.0:* LISTEN
family=PF_INET
backlog=0(16)
max_qlen_log=5 qlen=0 qlen_young=0
------------------------------------------------------------------------------
<struct tcp_sock 0xf7580980>            TCP
tcp   15.236.177.171:51095       16.236.16.79:5223           ESTABLISHED
windows: rcv=63480, snd=32767  advmss=1398 rcv_ws=0 snd_ws=0
nonagle=0 sack_ok=0 tstamp_ok=0
rmem_alloc=0, wmem_alloc=0
rx_queue=0, tx_queue=0
rcvbuf=87380, sndbuf=16384
rcv_tstamp=7.8 s, lsndtime=10.2 s  ago
------------------------------------------------------------------------------
<struct tcp_sock 0xf7954e40> TCP
tcp 127.0.0.1:9161               127.0.0.1:54393             TIME_WAIT
tw_timeout=15000, ttd=1730

The names of printed variables are mostly the same as in kernel sources and can be interpreted easily; for example, for LISTEN state backlog=0(16) means that we have 0 connections queued and the maximum is 16 (this is the value specified in the listen() system call).

If you specify '-vv', we print the contents of socket queues, e.g.

<struct sock 0xd5c6c600>                TCP
tcp   0.0.0.0:7778               0.0.0.0:*                   LISTEN
         family=PF_INET
         backlog=129(128)
         max_qlen_log=10 qlen=69 qlen_young=1
    --- Accept Queue <struct open_request 0xf001e600>
          laddr=128.8.61.4 raddr=10.148.6.13
          laddr=128.8.61.4 raddr=10.148.2.101
          laddr=128.8.61.4 raddr=10.149.6.7
    --- SYN-Queue
          laddr=128.8.61.4           raddr=128.8.11.24
          laddr=128.8.61.4           raddr=10.148.16.12
          laddr=128.8.61.4           raddr=10.152.0.45
          laddr=128.8.61.4           raddr=10.149.4.8

Sockets used by several subsystems (such as networking FS) do not have a filedescriptor and are not associated with any PID. These sockets have non-NULL sk_user_data field. If we see that this field is non-NULL, we print its value with some additional information (at this moment the support is limited to NFS/RPC mainly). For example:

------------------------------------------------------------------------------
<struct tcp_sock 0xffff81006e6bf380>            TCP
tcp   15.37.113.237:2049         15.236.176.36:1020          ESTABLISHED
        windows: rcv=6880, snd=5632  advmss=1448 rcv_ws=5 snd_ws=7
        nonagle=1 sack_ok=3 tstamp_ok=1
        rmem_alloc=0, wmem_alloc=0
        rx_queue=0, tx_queue=0
        rcvbuf=1597440, sndbuf=5857280
        rcv_tstamp=184 hours, lsndtime=184 hours  ago
         user_data 0xffff81002fa6ecc0  -> 'struct svc_sock'

UDP Sockets

crash32> xportshow -ua
udp   0.0.0.0:8010               0.0.0.0:*                  st=7
udp   0.0.0.0:9164               0.0.0.0:*                  st=7
udp   0.0.0.0:111                0.0.0.0:*                  st=7
udp   15.236.177.25:500          0.0.0.0:*                  st=7
udp   192.168.0.5:500            0.0.0.0:*                  st=7
udp   15.236.177.25:123          0.0.0.0:*                  st=7
udp6  fe80::200:ff:fe00:0:123    :::*                       st=7
udp   192.168.0.5:123            0.0.0.0:*                  st=7
udp   127.0.0.1:123              0.0.0.0:*                  st=7
udp6  fe80::216:d4ff:fee6:bcbf:123:::*                       st=7
udp6  ::1:123                    :::*                       st=7
udp6  :::123                     :::*                       st=7
udp   0.0.0.0:123                0.0.0.0:*                  st=7

UDP Sockets Verbose

crash32> xportshow -uav
------------------------------------------------------------------------------
<struct udp_sock 0xf791b280>            UDP
udp6  ::1:123                    :::*                       st=7
        rx_queue=0, tx_queue=0
        rcvbuf=110592, sndbuf=110592
        pending=0, corkflag=0, len=0
------------------------------------------------------------------------------
<struct udp_sock 0xf791b000>            UDP
udp6  :::123                     :::*                       st=7
        rx_queue=0, tx_queue=0
        rcvbuf=110592, sndbuf=110592
        pending=0, corkflag=0, len=0
------------------------------------------------------------------------------

Unix Sockets

crash32> xportshow -ax
unix   State          I-node  Path
----------------------------------
unix   LISTEN         17667   /var/run/acpid.socket
unix   LISTEN         17996   @/var/run/hald/dbus-eYQQ7ZQwSxe
unix   LISTEN         17928   /var/run/dbus/system_bus_socket
unix   LISTEN         19733   /dev/gpmctl

Raw Sockets

crash32> xportshow -awv
------------------------------------------------------------------------------
<struct raw_sock 0xe7678600>            RAW
raw   0.0.0.0:1                  0.0.0.0:*                  st=7
        rx_queue=0, tx_queue=0
        rcvbuf=131072, sndbuf=2048

Sockets Info For PID or Command

crash32> xportshow --pid 1
-----PID=1  COMM=init
 fd     file              socket
 --     ----              ------
  3  0xdfe12780          0xdf8baa80          PF_FILE  SOCK_DGRAM   UNIX
crash32> xportshow --program sshd
-----PID=5827  COMM=sshd
 fd     file              socket
 --     ----              ------
  3  0xf74c13c0          0xf7b8f900          PF_INET6 SOCK_STREAM  TCPv6
      tcp6  :::22                      :::*                        LISTEN

Port Filtering

For UDP and TCP you can limit output to those connections that use a specific port (either src or dst):

crash64> xportshow -tavv --port 2049
------------------------------------------------------------------------------
<struct tcp_sock 0xffff810070f95300>            TCP
tcp   0.0.0.0:2049               0.0.0.0:*                   LISTEN
         family=PF_INET
         backlog=0(64)
         max_qlen_log=7 qlen=0 qlen_young=0
         user_data 0xffff81007084f180  -> 'struct svc_sock'
------------------------------------------------------------------------------
<struct tcp_sock 0xffff81006e6bf380>            TCP
tcp   15.37.113.237:2049         15.236.176.36:1020          ESTABLISHED
        windows: rcv=6880, snd=5632  advmss=1448 rcv_ws=5 snd_ws=7
        nonagle=1 sack_ok=3 tstamp_ok=1
        rmem_alloc=0, wmem_alloc=0
        rx_queue=0, tx_queue=0
        rcvbuf=1597440, sndbuf=5857280
        rcv_tstamp=184 hours, lsndtime=184 hours  ago
         user_data 0xffff81002fa6ecc0  -> 'struct svc_sock'

Print A Summary

A summary of networking connections is printed, if there are any unusual/suspicious situations, the corresponding output lines are marked with WARNING

crash32> xportshow --summary
TCP Connection Info
-------------------
        ESTABLISHED      7
          TIME_WAIT      2
             LISTEN      7
                        NAGLE disabled (TCP_NODELAY):     1
UDP Connection Info
-------------------
  13 UDP sockets, 0 in ESTABLISHED
Unix Connection Info
------------------------
        ESTABLISHED    331
              CLOSE     12
             LISTEN     21
Raw sockets info
--------------------
              CLOSE      1

Routing Info

crash32> xportshow -r
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
15.236.176.4    15.236.177.25   255.255.255.255 UGH   0      0      0   nlv0
192.168.0.1     192.168.0.5     255.255.255.255 UGH   0      0      0   eth0
15.236.177.25   127.0.0.1       255.255.255.255 UGH   0      0      0   lo
156.152.224.149 192.168.0.1     255.255.255.255 UGH   0      0      0   eth0
192.168.0.0     15.236.177.25   255.255.255.0   UG    0      0      0   nlv0
192.168.0.0     0.0.0.0         255.255.255.0   U     1      0      0   eth0
169.254.0.0     15.236.177.25   255.255.0.0     UG    0      0      0   nlv0
169.254.0.0     0.0.0.0         255.255.0.0     U     1001   0      0   eth0
0.0.0.0         15.236.177.25   0.0.0.0         UG    0      0      0   nlv0
0.0.0.0         192.168.0.1     0.0.0.0         UG    101    0      0   eth0

Statistics

This is similar to 'netstat -s'

crash32> xportshow -s
-------------------- ip_statistics --------------------
                  InReceives               244579
                 InHdrErrors                    0
                InAddrErrors                   64
               ForwDatagrams                    0
             InUnknownProtos                    0
                  InDiscards                    0
                  InDelivers               145692
                 OutRequests               154084
...
-------------------- tcp_statistics --------------------
                RtoAlgorithm                    1
                      RtoMin                  200
                      RtoMax               120000
                     MaxConn                   -1
                 ActiveOpens                 6307
                PassiveOpens                 2077
                AttemptFails                    0
                 EstabResets                  124
                   CurrEstab                    8
...

ARP Cache

crash32> xportshow --arp
=== <struct neigh_table 0xf8a9df40> PF_INET6 nd_tbl
IP ADDRESS        HW TYPE    HW ADDRESS           DEVICE  STATE
----------        -------    ----------           ------  -----
::                LOOPBACK   00:00:00:00:00:00    lo      NOARP
=== <struct neigh_table 0xc03c8e60> PF_INET arp_tbl
IP ADDRESS        HW TYPE    HW ADDRESS           DEVICE  STATE
----------        -------    ----------           ------  -----
16.94.111.232     ETHER      00:00:00:00:00:00    nlv0    NOARP
0.0.0.0           LOOPBACK   00:00:00:00:00:00    lo      NOARP
16.89.16.131      ETHER      00:00:00:00:00:00    nlv0    NOARP
206.47.244.133    ETHER      00:00:00:00:00:00    nlv0    NOARP
16.94.111.234     ETHER      00:00:00:00:00:00    nlv0    NOARP
199.212.17.21     ETHER      00:00:00:00:00:00    nlv0    NOARP
16.94.111.233     ETHER      00:00:00:00:00:00    nlv0    NOARP
16.236.16.79      ETHER      00:00:00:00:00:00    nlv0    NOARP
192.168.0.1       ETHER      00:1d:5a:c0:94:59    eth0    REACHABLE
198.235.216.135   ETHER      00:00:00:00:00:00    nlv0    NOARP
128.100.56.135    ETHER      00:00:00:00:00:00    nlv0    NOARP
15.37.113.20      ETHER      00:00:00:00:00:00    nlv0    NOARP
16.94.111.231     ETHER      00:00:00:00:00:00    nlv0    NOARP
16.94.111.235     ETHER      00:00:00:00:00:00    nlv0    NOARP
16.236.31.2       ETHER      00:00:00:00:00:00    nlv0    NOARP

Similar to 'sysctl -a' with only networking-related tunables left in output

crash32> xportshow --sysctl
net.core.dev_weight                           64
net.core.message_burst                        10
net.core.message_cost                         1250
net.core.netdev_max_backlog                   1000
net.core.optmem_max                           10240
net.core.rmem_default                         110592
net.core.rmem_max                             131071
...
net.netfilter.nf_conntrack_udp_timeout_stream 45000
net.nf_conntrack_max                          65536
net.token-ring.rif_timeout                    150000
net.unix.max_dgram_qlen                       10

Interfaces Info

Similar to 'ifconfig' command

crash32> xportshow -i
============================== lo ============================================
lo 127.0.0.1/8 mtu=16436 LOOPBACK
inet6 addr: ::1/128
flags=<IFF_UP|IFF_LOOPBACK>
features=<FRAGLIST|SG|NO_CSUM|HIGHDMA|LLTX>
================================ eth0 ========================================
eth0 192.168.0.5/24 mtu=1500 00:16:d4:e6:bc:bf ETHER
inet6 addr: fe80::216:d4ff:fee6:bcbf/64
flags=<IFF_UP|IFF_BROADCAST|IFF_MULTICAST>
features=<SG|HW_VLAN_TX|HIGHDMA|HW_VLAN_RX|IP_CSUM>
================================ eth1 ========================================
eth1 mtu=1500 00:19:d2:b1:83:08 ETHER
flags=<IFF_UP|IFF_BROADCAST|IFF_MULTICAST>
================================ nlv0 ========================================
nlv0 15.236.177.25/24 mtu=1438 00:00:00:00:00:00 ETHER
inet6 addr: fe80::200:ff:fe00:0/64
flags=<IFF_UP|IFF_NOARP|IFF_BROADCAST>

Interface Info Verbose

crash32> xportshow -iv
================================ lo ==========================================
lo             127.0.0.1/8  mtu=16436                        LOOPBACK
  inet6 addr: ::1/128
    flags=<IFF_UP|IFF_LOOPBACK>
    features=<FRAGLIST|SG|NO_CSUM|HIGHDMA|LLTX>
    LINK_STATE   6 (START|PRESENT)
    open=<None>, stats=<get_stats> mtu=16436 promisc=0
        last_rx    5.64 s ago
            RX                -= Stats =-            TX
     -----------------------                ------------------------
   --CPU 0
    rx_packets           21531             tx_packets           21531
    rx_bytes             5452117           tx_bytes             5452117
   --CPU 1
    rx_packets           28538             tx_packets           28538
    rx_bytes             7363135           tx_bytes             7363135
    .............................................................
    <struct Qdisc 0xc03c6920> qlen=0
        enqueue=<None> dequeue=<noop_dequeue>
        qlen=0 backlog=0 drops=0 requeues=0 overlimits=0
================================ eth0 ========================================
eth0        192.168.0.5/24  mtu=1500      00:16:d4:e6:bc:bf  ETHER
  inet6 addr: fe80::216:d4ff:fee6:bcbf/64
    flags=<IFF_UP|IFF_BROADCAST|IFF_MULTICAST>
    features=<SG|HW_VLAN_TX|HIGHDMA|HW_VLAN_RX|IP_CSUM>
    LINK_STATE   6 (START|PRESENT)
    open=<tg3_open>, stats=<tg3_get_stats> mtu=1500 promisc=0
        last_rx    0.14 s ago
        trans_start    0.10 s ago
    !!!INFO: to print tg3 stats you need to install debuginfo version
             of tg3 module
    .............................................................
    <struct Qdisc 0xdfb1f700> qlen=0
        enqueue=<pfifo_fast_enqueue> dequeue=<pfifo_fast_dequeue>
        qlen=0 backlog=0 drops=0 requeues=0 overlimits=0
        == Bands ==
          sk_buff_head=0xdfb1f780 len=0
          sk_buff_head=0xdfb1f790 len=0
          sk_buff_head=0xdfb1f7a0 len=0
...

Routing Cache

crash64> xportshow --rtcache
dev      rt_src            rt_dst          fl4_src         fl4_dst   sec ago
---   -------------    -------------    -------------    -----------  --------
lo    15.37.118.191    15.37.119.255    15.37.118.191    15.37.119.255    129
lo    127.0.0.1        127.0.0.1        0.0.0.0          0.0.0.0          30
lo    15.37.117.81     15.37.119.255    15.37.117.81     15.37.119.255    160
lo    15.37.113.242    255.255.255.255  15.37.113.242    255.255.255.255  184
lo    16.212.4.117     15.37.113.237    16.212.4.117     15.37.113.237    0
lo    15.37.116.204    15.37.119.255    15.37.116.204    15.37.119.255    1
lo    15.37.114.221    15.37.119.255    15.37.114.221    15.37.119.255    197
eth0  15.37.113.237    224.0.0.251      15.37.113.237    224.0.0.251      25
...
71 entries

Netfilter Hooks Info

crash32> xportshow --netfilter
NPROTO=34, NF_MAX_HOOKS=8
=====PROTO= PF_INET
    NF_IP_PRE_ROUTING
        prio=-400,  hook=ipv4_conntrack_defrag
        prio=-200,  hook=ipv4_conntrack_in
    NF_IP_LOCAL_IN
        prio=0,  hook=ipt_hook
        prio=2147483645,  hook=ipv4_conntrack_help
        prio=2147483647,  hook=ipv4_confirm
    NF_IP_FORWARD
        prio=0,  hook=ipt_hook
    NF_IP_LOCAL_OUT
        prio=-400,  hook=ipv4_conntrack_defrag
        prio=-200,  hook=ipv4_conntrack_local
        prio=0,  hook=ipt_local_out_hook
    NF_IP_POST_ROUTING
        prio=2147483645,  hook=ipv4_conntrack_help
        prio=2147483647,  hook=ipv4_confirm

Packet Handlers Stack

Protocol handlers stack, as added by dev_add_pack() kernel function

crash32> xportshow --devpack
--------ptype_all-------------------------------------------
<struct packet_type 0xe79b5f8c>
        type=0x0003 dev=0xdfe0b000 func=packet_rcv
            pid=13197, command=tcpdump
--------ptype_base-------------------------------------------
<struct packet_type 0xc03c97c0>  (bucket=0)
        type=0x0800 dev=0x0 func=ip_rcv
<struct packet_type 0xc03c6700>  (bucket=1)
        type=0x0011 dev=0x0 func=llc_rcv
<struct packet_type 0xc03c66e0>  (bucket=4)
        type=0x0004 dev=0x0 func=llc_rcv
<struct packet_type 0xc03c9000>  (bucket=6)
        type=0x0806 dev=0x0 func=arp_rcv
<struct packet_type 0xf8b15ba0>  (bucket=7)
        type=0x0017 dev=0x0 func=irlap_driver_rcv
<struct packet_type 0xf8a9df20>  (bucket=13)
        type=0x86dd dev=0x0 func=ipv6_rcv

Wiki: Home