Xportshow
From pykdump
Xportshow
Overview
xportshow is a tool to extract and print the networking transport related information from linux kernel dumps. It is implemented as an extension to crash dump-analysis tool using PyKdump framework.
crash already has a builtin net command, but it can do just three things:
- print information based on pid, socket, or filedescriptor
- print ARP-table
- print networking devices
xportshow can print many more things than net. It should be useful for anybody with basic networking knowledge, even though some output will be meaningful only for those who understand the details of Linux TCP/IP implementation. Most short options are the same as in standard netstat command.
Printing all available options
crash32> xportshow -h
Usage: xportshow [options]
Options:
-h, --help show this help message and exit
-a print all sockets
-v verbose output
-r print routing table
--program=PROGRAM print sockets for cmdname
--pid=PID print sockets for PID
--netfilter Print Netfilter Hooks
--softnet Print Softnet Queues
--summary Print A Summary
-s, --statistics Print Statistics
-i Print Interface Info
--interface=IF1 Limit output to the specified interface only
--decode=DECODE Decode iph/th/uh
--port=PORT Limit output to the specified port (src or dst)
-l, --listening Print LISTEN sockets only
-t, --tcp Print TCP Info
-u, --udp Print UDP Info
-w, --raw Print RAW Info
-x, --unix Print UNIX Info
--sysctl Print sysctl info for net.
--devpack Print dev_pack info
--arp Print ARP & Neighbouring info
--rtcache Print the routing cache
--skbuffhead=SKBUFFHEAD
Print sk_buff_head
--everything Run all functions available
--version Print 'xportshow' version and exit
** Execution took 0.54s (real) 0.52s (CPU)
'-v' option (verbose) can be used several times (e.g. '-vv'); this increases the amount of output for some subcommands.
Please note that options can be short (one letter preceded with one dash) and long (a word preceded with two dashes). The options parser can identify a long option by several first characters as soon as there is no ambiguity. For example, if you have two long options --stacksummary and --stdout, you cannot use --st, but --stack and --std should be good enough. With short options, the option argument (if any) should be separated by space. For long options, it can be either space or equals '=' sign. Examples:
-d 1 --decode 1 --decode=1
IPv6 support is not tested as extensively as IPv4, but it should mostly work. The program has been tested for many kernels starting from 2.4.21 up to 2.6.22 on Debian, RH, SLES and Ubuntu distributions.
TCP Sockets
crash32> xportshow -at tcp 0.0.0.0:42691 0.0.0.0:* LISTEN tcp 127.0.0.1:9161 0.0.0.0:* LISTEN tcp 0.0.0.0:8010 0.0.0.0:* LISTEN tcp 127.0.0.1:9165 0.0.0.0:* LISTEN tcp 0.0.0.0:111 0.0.0.0:* LISTEN tcp6 :::22 :::* LISTEN tcp 127.0.0.1:631 0.0.0.0:* LISTEN tcp 15.236.177.25:52414 16.236.16.79:5223 ESTABLISHED tcp 15.236.177.25:35016 15.37.113.20:143 ESTABLISHED tcp 15.236.177.25:60457 16.236.31.2:143 ESTABLISHED tcp 15.236.177.25:53004 69.159.122.174:22 ESTABLISHED tcp 15.236.177.25:35015 15.37.113.20:143 ESTABLISHED tcp 127.0.0.1:47939 127.0.0.1:9165 ESTABLISHED tcp 127.0.0.1:9165 127.0.0.1:47939 ESTABLISHED tcp 127.0.0.1:9161 127.0.0.1:54388 TIME_WAIT tcp 127.0.0.1:9161 127.0.0.1:54387 TIME_WAIT ** Execution took 0.03s (real) 0.02s (CPU)
TCP Sockets Verbose
Here we show what can be printed for TCP connections in different states
crash32> xportshow -atv ------------------------------------------------------------------------------ <struct tcp_sock 0xf62c8000> TCP tcp 0.0.0.0:42691 0.0.0.0:* LISTEN family=PF_INET backlog=0(16) max_qlen_log=5 qlen=0 qlen_young=0 ------------------------------------------------------------------------------ <struct tcp_sock 0xf7580980> TCP tcp 15.236.177.171:51095 16.236.16.79:5223 ESTABLISHED windows: rcv=63480, snd=32767 advmss=1398 rcv_ws=0 snd_ws=0 nonagle=0 sack_ok=0 tstamp_ok=0 rmem_alloc=0, wmem_alloc=0 rx_queue=0, tx_queue=0 rcvbuf=87380, sndbuf=16384 rcv_tstamp=7.8 s, lsndtime=10.2 s ago ------------------------------------------------------------------------------ <struct tcp_sock 0xf7954e40> TCP tcp 127.0.0.1:9161 127.0.0.1:54393 TIME_WAIT tw_timeout=15000, ttd=1730
The names of printed variables are mostly the same as in kernel sources and can be interpreted easily; for example, for LISTEN state backlog=0(16) means that we have 0 connections queued and the maximum is 16 (this is the value specified in the listen() system call).
If you specify '-vv', we print the contents of socket queues, e.g.
<struct sock 0xd5c6c600> TCP
tcp 0.0.0.0:7778 0.0.0.0:* LISTEN
family=PF_INET
backlog=129(128)
max_qlen_log=10 qlen=69 qlen_young=1
--- Accept Queue <struct open_request 0xf001e600>
laddr=128.8.61.4 raddr=10.148.6.13
laddr=128.8.61.4 raddr=10.148.2.101
laddr=128.8.61.4 raddr=10.149.6.7
--- SYN-Queue
laddr=128.8.61.4 raddr=128.8.11.24
laddr=128.8.61.4 raddr=10.148.16.12
laddr=128.8.61.4 raddr=10.152.0.45
laddr=128.8.61.4 raddr=10.149.4.8
Sockets used by several subsystems (such as networking FS) do not have a filedescriptor and are not associated with any PID. These sockets have non-NULL sk_user_data field. If we see that this field is non-NULL, we print its value with some additional information (at this moment the support is limited to NFS/RPC mainly). For example:
------------------------------------------------------------------------------
<struct tcp_sock 0xffff81006e6bf380> TCP
tcp 15.37.113.237:2049 15.236.176.36:1020 ESTABLISHED
windows: rcv=6880, snd=5632 advmss=1448 rcv_ws=5 snd_ws=7
nonagle=1 sack_ok=3 tstamp_ok=1
rmem_alloc=0, wmem_alloc=0
rx_queue=0, tx_queue=0
rcvbuf=1597440, sndbuf=5857280
rcv_tstamp=184 hours, lsndtime=184 hours ago
user_data 0xffff81002fa6ecc0 -> 'struct svc_sock'
UDP Sockets
crash32> xportshow -ua udp 0.0.0.0:8010 0.0.0.0:* st=7 udp 0.0.0.0:9164 0.0.0.0:* st=7 udp 0.0.0.0:111 0.0.0.0:* st=7 udp 15.236.177.25:500 0.0.0.0:* st=7 udp 192.168.0.5:500 0.0.0.0:* st=7 udp 15.236.177.25:123 0.0.0.0:* st=7 udp6 fe80::200:ff:fe00:0:123 :::* st=7 udp 192.168.0.5:123 0.0.0.0:* st=7 udp 127.0.0.1:123 0.0.0.0:* st=7 udp6 fe80::216:d4ff:fee6:bcbf:123:::* st=7 udp6 ::1:123 :::* st=7 udp6 :::123 :::* st=7 udp 0.0.0.0:123 0.0.0.0:* st=7
UDP Sockets Verbose
crash32> xportshow -uav
------------------------------------------------------------------------------
<struct udp_sock 0xf791b280> UDP
udp6 ::1:123 :::* st=7
rx_queue=0, tx_queue=0
rcvbuf=110592, sndbuf=110592
pending=0, corkflag=0, len=0
------------------------------------------------------------------------------
<struct udp_sock 0xf791b000> UDP
udp6 :::123 :::* st=7
rx_queue=0, tx_queue=0
rcvbuf=110592, sndbuf=110592
pending=0, corkflag=0, len=0
------------------------------------------------------------------------------
Unix Sockets
crash32> xportshow -ax unix State I-node Path ---------------------------------- unix LISTEN 17667 /var/run/acpid.socket unix LISTEN 17996 @/var/run/hald/dbus-eYQQ7ZQwSxe unix LISTEN 17928 /var/run/dbus/system_bus_socket unix LISTEN 19733 /dev/gpmctl
Raw Sockets
crash32> xportshow -awv
------------------------------------------------------------------------------
<struct raw_sock 0xe7678600> RAW
raw 0.0.0.0:1 0.0.0.0:* st=7
rx_queue=0, tx_queue=0
rcvbuf=131072, sndbuf=2048
Sockets Info For PID or Command
crash32> xportshow --pid 1
-----PID=1 COMM=init
fd file socket
-- ---- ------
3 0xdfe12780 0xdf8baa80 PF_FILE SOCK_DGRAM UNIX
crash32> xportshow --program sshd
-----PID=5827 COMM=sshd
fd file socket
-- ---- ------
3 0xf74c13c0 0xf7b8f900 PF_INET6 SOCK_STREAM TCPv6
tcp6 :::22 :::* LISTEN
Port Filtering
For UDP and TCP you can limit output to those connections that use a specific port (either src or dst):
crash64> xportshow -tavv --port 2049
------------------------------------------------------------------------------
<struct tcp_sock 0xffff810070f95300> TCP
tcp 0.0.0.0:2049 0.0.0.0:* LISTEN
family=PF_INET
backlog=0(64)
max_qlen_log=7 qlen=0 qlen_young=0
user_data 0xffff81007084f180 -> 'struct svc_sock'
------------------------------------------------------------------------------
<struct tcp_sock 0xffff81006e6bf380> TCP
tcp 15.37.113.237:2049 15.236.176.36:1020 ESTABLISHED
windows: rcv=6880, snd=5632 advmss=1448 rcv_ws=5 snd_ws=7
nonagle=1 sack_ok=3 tstamp_ok=1
rmem_alloc=0, wmem_alloc=0
rx_queue=0, tx_queue=0
rcvbuf=1597440, sndbuf=5857280
rcv_tstamp=184 hours, lsndtime=184 hours ago
user_data 0xffff81002fa6ecc0 -> 'struct svc_sock'
Print A Summary
A summary of networking connections is printed, if there are any unusual/suspicious situations, the corresponding output lines are marked with WARNING
crash32> xportshow --summary
TCP Connection Info
-------------------
ESTABLISHED 7
TIME_WAIT 2
LISTEN 7
NAGLE disabled (TCP_NODELAY): 1
UDP Connection Info
-------------------
13 UDP sockets, 0 in ESTABLISHED
Unix Connection Info
------------------------
ESTABLISHED 331
CLOSE 12
LISTEN 21
Raw sockets info
--------------------
CLOSE 1
Routing Info
crash32> xportshow -r Destination Gateway Genmask Flags Metric Ref Use Iface 15.236.176.4 15.236.177.25 255.255.255.255 UGH 0 0 0 nlv0 192.168.0.1 192.168.0.5 255.255.255.255 UGH 0 0 0 eth0 15.236.177.25 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 156.152.224.149 192.168.0.1 255.255.255.255 UGH 0 0 0 eth0 192.168.0.0 15.236.177.25 255.255.255.0 UG 0 0 0 nlv0 192.168.0.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0 169.254.0.0 15.236.177.25 255.255.0.0 UG 0 0 0 nlv0 169.254.0.0 0.0.0.0 255.255.0.0 U 1001 0 0 eth0 0.0.0.0 15.236.177.25 0.0.0.0 UG 0 0 0 nlv0 0.0.0.0 192.168.0.1 0.0.0.0 UG 101 0 0 eth0
Statistics
This is similar to 'netstat -s'
crash32> xportshow -s
-------------------- ip_statistics --------------------
InReceives 244579
InHdrErrors 0
InAddrErrors 64
ForwDatagrams 0
InUnknownProtos 0
InDiscards 0
InDelivers 145692
OutRequests 154084
...
-------------------- tcp_statistics --------------------
RtoAlgorithm 1
RtoMin 200
RtoMax 120000
MaxConn -1
ActiveOpens 6307
PassiveOpens 2077
AttemptFails 0
EstabResets 124
CurrEstab 8
...
ARP Cache
crash32> xportshow --arp === <struct neigh_table 0xf8a9df40> PF_INET6 nd_tbl IP ADDRESS HW TYPE HW ADDRESS DEVICE STATE ---------- ------- ---------- ------ ----- :: LOOPBACK 00:00:00:00:00:00 lo NOARP === <struct neigh_table 0xc03c8e60> PF_INET arp_tbl IP ADDRESS HW TYPE HW ADDRESS DEVICE STATE ---------- ------- ---------- ------ ----- 16.94.111.232 ETHER 00:00:00:00:00:00 nlv0 NOARP 0.0.0.0 LOOPBACK 00:00:00:00:00:00 lo NOARP 16.89.16.131 ETHER 00:00:00:00:00:00 nlv0 NOARP 206.47.244.133 ETHER 00:00:00:00:00:00 nlv0 NOARP 16.94.111.234 ETHER 00:00:00:00:00:00 nlv0 NOARP 199.212.17.21 ETHER 00:00:00:00:00:00 nlv0 NOARP 16.94.111.233 ETHER 00:00:00:00:00:00 nlv0 NOARP 16.236.16.79 ETHER 00:00:00:00:00:00 nlv0 NOARP 192.168.0.1 ETHER 00:1d:5a:c0:94:59 eth0 REACHABLE 198.235.216.135 ETHER 00:00:00:00:00:00 nlv0 NOARP 128.100.56.135 ETHER 00:00:00:00:00:00 nlv0 NOARP 15.37.113.20 ETHER 00:00:00:00:00:00 nlv0 NOARP 16.94.111.231 ETHER 00:00:00:00:00:00 nlv0 NOARP 16.94.111.235 ETHER 00:00:00:00:00:00 nlv0 NOARP 16.236.31.2 ETHER 00:00:00:00:00:00 nlv0 NOARP
Network-related SYSCTL Tunables
Similar to 'sysctl -a' with only networking-related tunables left in output
crash32> xportshow --sysctl net.core.dev_weight 64 net.core.message_burst 10 net.core.message_cost 1250 net.core.netdev_max_backlog 1000 net.core.optmem_max 10240 net.core.rmem_default 110592 net.core.rmem_max 131071 ... net.netfilter.nf_conntrack_udp_timeout_stream 45000 net.nf_conntrack_max 65536 net.token-ring.rif_timeout 150000 net.unix.max_dgram_qlen 10
Interfaces Info
Similar to 'ifconfig' command
crash32> xportshow -i ============================== lo ============================================ lo 127.0.0.1/8 mtu=16436 LOOPBACK inet6 addr: ::1/128 flags=<IFF_UP|IFF_LOOPBACK> features=<FRAGLIST|SG|NO_CSUM|HIGHDMA|LLTX> ================================ eth0 ======================================== eth0 192.168.0.5/24 mtu=1500 00:16:d4:e6:bc:bf ETHER inet6 addr: fe80::216:d4ff:fee6:bcbf/64 flags=<IFF_UP|IFF_BROADCAST|IFF_MULTICAST> features=<SG|HW_VLAN_TX|HIGHDMA|HW_VLAN_RX|IP_CSUM> ================================ eth1 ======================================== eth1 mtu=1500 00:19:d2:b1:83:08 ETHER flags=<IFF_UP|IFF_BROADCAST|IFF_MULTICAST> ================================ nlv0 ======================================== nlv0 15.236.177.25/24 mtu=1438 00:00:00:00:00:00 ETHER inet6 addr: fe80::200:ff:fe00:0/64 flags=<IFF_UP|IFF_NOARP|IFF_BROADCAST>
Interface Info Verbose
crash32> xportshow -iv
================================ lo ==========================================
lo 127.0.0.1/8 mtu=16436 LOOPBACK
inet6 addr: ::1/128
flags=<IFF_UP|IFF_LOOPBACK>
features=<FRAGLIST|SG|NO_CSUM|HIGHDMA|LLTX>
LINK_STATE 6 (START|PRESENT)
open=<None>, stats=<get_stats> mtu=16436 promisc=0
last_rx 5.64 s ago
RX -= Stats =- TX
----------------------- ------------------------
--CPU 0
rx_packets 21531 tx_packets 21531
rx_bytes 5452117 tx_bytes 5452117
--CPU 1
rx_packets 28538 tx_packets 28538
rx_bytes 7363135 tx_bytes 7363135
.............................................................
<struct Qdisc 0xc03c6920> qlen=0
enqueue=<None> dequeue=<noop_dequeue>
qlen=0 backlog=0 drops=0 requeues=0 overlimits=0
================================ eth0 ========================================
eth0 192.168.0.5/24 mtu=1500 00:16:d4:e6:bc:bf ETHER
inet6 addr: fe80::216:d4ff:fee6:bcbf/64
flags=<IFF_UP|IFF_BROADCAST|IFF_MULTICAST>
features=<SG|HW_VLAN_TX|HIGHDMA|HW_VLAN_RX|IP_CSUM>
LINK_STATE 6 (START|PRESENT)
open=<tg3_open>, stats=<tg3_get_stats> mtu=1500 promisc=0
last_rx 0.14 s ago
trans_start 0.10 s ago
!!!INFO: to print tg3 stats you need to install debuginfo version
of tg3 module
.............................................................
<struct Qdisc 0xdfb1f700> qlen=0
enqueue=<pfifo_fast_enqueue> dequeue=<pfifo_fast_dequeue>
qlen=0 backlog=0 drops=0 requeues=0 overlimits=0
== Bands ==
sk_buff_head=0xdfb1f780 len=0
sk_buff_head=0xdfb1f790 len=0
sk_buff_head=0xdfb1f7a0 len=0
...
Routing Cache
crash64> xportshow --rtcache dev rt_src rt_dst fl4_src fl4_dst sec ago --- ------------- ------------- ------------- ----------- -------- lo 15.37.118.191 15.37.119.255 15.37.118.191 15.37.119.255 129 lo 127.0.0.1 127.0.0.1 0.0.0.0 0.0.0.0 30 lo 15.37.117.81 15.37.119.255 15.37.117.81 15.37.119.255 160 lo 15.37.113.242 255.255.255.255 15.37.113.242 255.255.255.255 184 lo 16.212.4.117 15.37.113.237 16.212.4.117 15.37.113.237 0 lo 15.37.116.204 15.37.119.255 15.37.116.204 15.37.119.255 1 lo 15.37.114.221 15.37.119.255 15.37.114.221 15.37.119.255 197 eth0 15.37.113.237 224.0.0.251 15.37.113.237 224.0.0.251 25 ... 71 entries
Netfilter Hooks Info
crash32> xportshow --netfilter
NPROTO=34, NF_MAX_HOOKS=8
=====PROTO= PF_INET
NF_IP_PRE_ROUTING
prio=-400, hook=ipv4_conntrack_defrag
prio=-200, hook=ipv4_conntrack_in
NF_IP_LOCAL_IN
prio=0, hook=ipt_hook
prio=2147483645, hook=ipv4_conntrack_help
prio=2147483647, hook=ipv4_confirm
NF_IP_FORWARD
prio=0, hook=ipt_hook
NF_IP_LOCAL_OUT
prio=-400, hook=ipv4_conntrack_defrag
prio=-200, hook=ipv4_conntrack_local
prio=0, hook=ipt_local_out_hook
NF_IP_POST_ROUTING
prio=2147483645, hook=ipv4_conntrack_help
prio=2147483647, hook=ipv4_confirm
Packet Handlers Stack
Protocol handlers stack, as added by dev_add_pack() kernel function
crash32> xportshow --devpack
--------ptype_all-------------------------------------------
<struct packet_type 0xe79b5f8c>
type=0x0003 dev=0xdfe0b000 func=packet_rcv
pid=13197, command=tcpdump
--------ptype_base-------------------------------------------
<struct packet_type 0xc03c97c0> (bucket=0)
type=0x0800 dev=0x0 func=ip_rcv
<struct packet_type 0xc03c6700> (bucket=1)
type=0x0011 dev=0x0 func=llc_rcv
<struct packet_type 0xc03c66e0> (bucket=4)
type=0x0004 dev=0x0 func=llc_rcv
<struct packet_type 0xc03c9000> (bucket=6)
type=0x0806 dev=0x0 func=arp_rcv
<struct packet_type 0xf8b15ba0> (bucket=7)
type=0x0017 dev=0x0 func=irlap_driver_rcv
<struct packet_type 0xf8a9df20> (bucket=13)
type=0x86dd dev=0x0 func=ipv6_rcv
