Python 3 Network Packet Sniffer Files

RootWire adopts NETProtocols 2.9: the library now decodes some twenty more protocols, and RootWire renders each of them on screen, adds --decode-as, pcapng replay and conversation statistics, and hardens what reaches the terminal. It is a major release for three breaking changes, each marked BREAKING below: the NDJSON wire type field of ICMPv4/ICMPv6/IGMP/BGP/OSPF/PIM/ VRRP layers moves to type_; rootwire.pcap.read_pcap() is renamed read_captures(); and DecodedFrame.payload no longer carries the Ethernet padding or captured FCS past the frame's declared lengths, which move to DecodedFrame.trailer. Existing CLI options and exit codes are unchanged; --decode-as is new.

Security

• LLDP's System Name, Port Description and System Description are now shown at most four times each, with any further duplicates counted as (+N more). Every duplicate used to get its own line, so one crafted LLDPDU of minimal TLVs (about 500 of them fit in a frame) could flood the terminal and bury real output.
• Library diagnostics are sanitized before they reach the terminal. The [!] Malformed header: ... line and every [!] ... malformed line printed for a lazily parsed section now pass through the same control-character neutralization as payload text: NETProtocols' error messages can quote wire bytes, and an attacker-chosen byte string must never reach the analyst's terminal raw. The new [~] best-effort-miss line (below) is sanitized the same way.
• The new DNS renderer (below) now neutralizes terminal control characters in question/answer names and RDATA, the same way -d's payload display already does — a DNS name is fully attacker-controlled (whatever a queried server answers), and rendering it unsanitized would let a crafted response inject ANSI escape sequences into the analyst's terminal. Caught by review before merge (#75).
• DNS question names, answer names and RDATA are now also kept on one line: a label holding a newline used to split the record's line, so a crafted name could forge a RootWire [!] line under it (#94).
• The IPv4 and TCP Options: lines now list at most 16 options and a (+N more) marker, so a header full of one-byte No-Operation options can no longer stretch the line without bound (#94).

Changed

• NETProtocols floor raised to 2.9 (netprotocols>=2.9,<3; lock 2.2.1 → 2.9.0). The library now decodes LLDP, CDP, STP, LLC/SNAP, MPLS, VXLAN, IP-in-IP, IPsec ESP/AH, OSPF, PIM, VRRP, SCTP, BGP, TLS, STUN, NTP, RADIUS, RIP, HSRP and TFTP on its default dispatch, so frames that used to end at Ethernet, IP or the transport now carry those layers, in full in --json and through a renderer of its own on screen (see Added). An LLDP frame (EtherType 0x88CC) is no longer "Ethernet plus payload".
• Best-effort application decoders are no longer mistaken for malformed frames. 2.3+ dispatches TLS (443), BGP (179), STUN (3478), NTP (123), RADIUS (1812/1813), HSRP (1985), TFTP (69), RIP (520) and VXLAN (4789) on the port alone, and calls the decoder on whatever the segment holds; every bare ACK and continuation segment on those ports then raised, which RootWire recorded as frame.error and counted as malformed. A stop inside one of those classes — derived from the library's registry at import, not hardcoded — is now recorded as the new DecodedFrame.l7_miss: no [!], not counted as malformed, the undecoded bytes left as payload, and a [~] Payload left undecoded note only when there are bytes to explain. DNS, DNS-over-TCP and DHCP are deliberately excluded: their malformations predate the upgrade and remain errors.
• Layer cap raised from 16 to 32, the library's own default. BGP, STUN over TCP and TLS decode one layer per message or record, and a valid segment holds dozens; at 16, ordinary dense BGP/TLS traffic was diagnosed as malformed. An overflow where the next layer is another of the same port-dispatched class as the last is a best-effort miss; any other overflow (an extension-header chain, a VXLAN nesting bomb) is still an error, now reading "chain still going after 32 headers".
• A lying length now stops the walk. The library bounds every layer by the declared IPv4 Total Length / UDP Length and rejects one shorter than its own header, so for an IPv4 total_length below ihl * 4 the IPv4 layer still renders but the layers inside it are no longer decoded (previously they were, off ihl), and frame.error is set alongside frame.malformed_length. The two are one fault: stats count it once and the screen prints the single specific line, now ending "upper layers not decoded". A UDP Length of 1 to 7 is likewise an error at UDP. total_length == 0 (TSO) stays exempt. Restoring the inner layers for display is left for later.
• BREAKING (library API): Ethernet padding and a captured FCS are split out as DecodedFrame.trailer (the library's packet.trailer) instead of being part of frame.payload, so -d no longer prints padding as payload. NDJSON reports its length as the new trailer_len. The split needs a declared length to cut at (IPv4 Total Length, UDP Length, an 802.3 Length), so behind an Ethernet II EtherType that nothing decodes, a captured FCS stays in payload.
• NDJSON records gain l7_miss and trailer_len.
• decoder.py now delegates its chain walk to netprotocols.decode_frame() instead of hand-rolling the same loop the library has shipped since 2.0. No behavior change for well-formed frames. The layer-cap diagnostic's wording changed to match the library's own MaxDepthExceededError message (DecodedFrame.error now reads "chain still going after N headers (...)" instead of "decode chain exceeded N layers"); every other error message is unchanged, since each already names its own protocol in its text (#74).
• Replay's non-Ethernet-linktype rejection (previously classic-pcap only) now also covers pcapng, checked against every declared Interface Description Block before any frame is decoded — an Enhanced Packet Block can reference any interface a preceding IDB declared, not just the first one, so checking only the first would miss a crafted file mixing an Ethernet IDB with a non-Ethernet one. Nothing else stops a wrong link type from silently decoding into nonsense, since Ethernet accepts any 14+ bytes structurally (#77).

Added

• --decode-as TABLE=KEY:CLASS, repeatable: decode what a NETProtocols dispatch table maps KEY to as CLASS for this capture (tcp.port=8080:HTTP, udp.port=40000:TFTP, udp.port=5353:DNS), or as nothing with none (udp.port=1812:none). The overrides derive one registry for the whole capture. A class mapped to a port is best-effort like the library's own port decoders, so a bare ACK or a response body on a port told to be HTTP is a [~] miss, not a malformed frame; a class a type field also reaches (IPv4 on a port) is not. A bad value -- malformed, an unknown table or decoder, a key out of its field's range, a table and key given twice -- is a usage error (exit 2).
• BREAKING (library API): -r/--read now replays pcapng captures, not just classic pcap. RootWire's own reader was classic-pcap-only (an explicit descope of the original pcap/replay work); the read path now delegates to NETProtocols' read_captures(), which auto-detects the format from its magic bytes. rootwire.pcap.read_pcap() is renamed read_captures() to match — a break only for code embedding RootWire as a library and importing that function by name, not for any CLI usage. -w/PcapWriter are unaffected: NETProtocols ships no writer, so -w still emits classic pcap only (#77).
• --filter accepts sctp, IP protocol 132 in both stacks, exactly as tcpdump -dd sctp compiles it (pinned against tcpdump 4.99.4 bytecode over the whole corpus). port still matches TCP and UDP only, as before.
• Checksum verification rendering (closes the README's own long-standing Roadmap item). Every IPv4/ICMPv4/ICMPv6/TCP/UDP checksum is now verified against a recomputation via netprotocols.checksum.verify(), flagging a mismatch inline (e.g. Checksum: 0x0000 [!] mismatch (expected 0x1a2b)) instead of just displaying the on-wire value. Silent (no false positive) for a non-reassembled IP fragment — the wire checksum covers the reassembled datagram, which RootWire never rebuilds, so it can't be verified against a single fragment's bytes without lying (#78). IGMP and GRE checksums are verified the same way (GRE only when its Checksum-Present bit is set), and the renderers added below verify their own: SCTP's CRC32c, VRRP, PIM, OSPF and its LSAs, CDP, and the STUN FINGERPRINT.
• Decoded IPv4 and TCP options, replacing the previous Options: N bytes byte count. Kinds the library understands (IPv4's Record Route, Timestamp, Router Alert; TCP's MSS, Window Scale, SACK, Timestamps) render their actual value; every other kind still shows its name with a raw byte count. Malformed options (a declared length running past the buffer) surface as a [!] diagnostic rather than raising (#79).
• ICMPv6 Neighbor Discovery detail. A Router/Neighbor Solicitation or Advertisement (or Redirect) now shows its target address and every option TLV (source/target link-layer address rendered as a MAC, others as their raw byte count) instead of a bare Type/Code line — IPv6's ARP-equivalent was previously indistinguishable from any other ICMPv6 message on screen.
• Screen renderers for DNS, DNS-over-TCP, and DHCP. These already decoded correctly (chain dispatch is registry-driven, not a RootWire whitelist) but printed as the generic "(no renderer)" placeholder on screen. DNS now shows the query/response direction, opcode/RCODE, and every question and answer (name, type, TTL, decoded RDATA); DHCP shows the message type, client MAC, the four address fields, and a parsed-option count. Both surface a [!] diagnostic instead of raising if their on-demand-parsed sections (DNS records, DHCP options) turn out malformed — these are parsed lazily on first access, unlike every previously-rendered protocol, which validates fully at decode time (#75).
• Screen renderers for GRE and IGMP. Both already decoded correctly (chain dispatch is registry-driven, not a RootWire whitelist) but printed as the generic "(no renderer)" placeholder on screen. GRE now shows the encapsulated EtherType and which optional checksum/key/sequence fields are present; IGMP shows the message type, and either the multicast group (query/v1/v2 report/leave) or every group record (a v3 report), each with its record type and source-address list. IGMP's body-parsing accessors surface a [!] diagnostic instead of raising if malformed, matching the DNS/DHCP renderers' contract for the same reason: parsed on demand, not validated at decode time (#76).
• Screen renderers for TLS, SCTP and NTP. TLS renders one line per record (content type, handshake message, record version, length) and, for a ClientHello, its version, SNI host name, cipher-suite count and extension list. SCTP shows its ports, verification tag, a verified CRC32c checksum and its chunk types with flags. NTP shows version, mode and stratum, leap/poll/precision/root delay and dispersion, the Reference ID or a stratum-0 kiss code, the transmit time, NTPv4 extension fields and the MAC. Attacker-chosen text (the SNI, which the library only checks is ASCII — ESC is ASCII — and the Reference ID or kiss code) is sanitized for the terminal; on-demand sections (SCTP chunks, the NTP body) render a [!] line instead of raising when malformed, and a Handshake record that does not parse as the ClientHello its first byte announces renders an informational [~] line, since an encrypted TLS 1.2 Finished begins that way one time in 256; and list lines cap at 16 items with "(+N more)", since one packet can carry hundreds of 4-byte SCTP chunks.
• SNI of a ClientHello split across segments. A ClientHello too large for one segment (Chromium's) is a TLS best-effort miss, but its first segment usually holds the host name: the decoder now reads it with NETProtocols' peek_server_name() into the new DecodedFrame.sni_peek, rendered as [+] TLS ClientHello (continues in a later segment) SNI: ... and emitted as NDJSON sni_peek. Only a found name is kept: "not reached" (the name is in a later segment) is not "no SNI", and shows nothing.
• Screen renderers for VRRP, HSRP, RIP, OSPF, PIM and BGP, the routing and first-hop-redundancy protocols an analyst triages a segment by. VRRP (v2/v3, IPv4/IPv6) shows the VRID, priority (0 a master resigning, 255 the address owner), virtual addresses, interval, a verified checksum and VRRPv2's authentication; HSRP the op code, group, state, priority, virtual IP and MAC, timers and the Authentication Data, flagged when it is the default cisco; RIP (v1/v2) a request for the whole table, or each route with its prefix, next hop and metric (16 shown as unreachable), and a RIPv2 password or RFC 4822 keyed-digest entry; OSPFv2 the router and area, the authentication, the packet checksum (or that cryptographic authentication replaces it), and per message type the Hello's timers, DR/BDR and neighbors, a Database Description's MTU, flags and LSA headers, a request's LSAs, an update's LSAs with each LS checksum verified, and an acknowledgment's headers; PIM a verified checksum and the Hello options, a Register's inner flow (or Null-Register), Register-Stop, every Join/Prune group with its joined and pruned sources and their WC/RPT bits, and Assert metrics; BGP one section per message — an OPEN's version, AS, hold time, ID and parameters, an UPDATE's withdrawn routes, attributes and NLRI (or End-of-RIB), a NOTIFICATION's error and RFC 9003 shutdown communication. Clear-text passwords (VRRPv2 PASS, RIP, OSPF AuType 1), HSRP's Authentication Data and BGP's shutdown communication are sanitized for the terminal and kept on one line (a newline shows as \n, so a peer cannot forge a frame header or diagnostic); a raw BGP attribute value shows its first 32 bytes and how many more it holds; BGP and PIM bodies parse on demand and render a [!] line instead of raising when malformed; every list caps at 16 items. A PIM Register's inner datagram is read from its IP header alone, not walked, so the frame keeps a single layer budget.
• Screen renderers for LLDP, CDP, STP, MPLS, LLC and SNAP, the layer-2 neighbor-discovery, bridging and label-switching headers. LLDP shows the Chassis ID (a MAC as a MAC), Port ID and TTL (0 a shutdown LLDPDU), flags an LLDPDU that does not open with them as 802.1AB requires, and shows the System Name, Port Description and System Description, then every other TLV by name or, for an Organizationally Specific one, by OUI and subtype; CDP its version, TTL and verified checksum, the Device ID, Port ID, Platform and Software Version, the capabilities and addresses; STP (802.1D and RSTP) the root and bridge identifiers, root path cost, port, flags (an RST BPDU's port role and proposal/agreement handshake among them) and timers, or a TCN's type alone; MPLS each label of the stack with its traffic class and TTL, special-purpose labels by name and the bottom marked, and the payload the stack implies; LLC its DSAP and SSAP by name, group and command/response bits and control format, or Novell's raw 802.3; SNAP its OUI and protocol ID, named as an EtherType where the OUI makes it one. The neighbor-chosen text of LLDP and CDP is sanitized for the terminal, kept on one line and cut at 128 characters.
• Screen renderers for VXLAN, ESP, AH, TFTP, RADIUS and STUN, the tunnels, IPsec headers and UDP services. VXLAN shows its VNI and flags, and notes reserved bits that are set; ESP its SPI, sequence number and the size of the protected rest (its Next Header is encrypted, so the walk stops there); AH, under IPv4 or IPv6, its SPI, sequence number and what it protects, tunnel mode named, and the ICV's size (never verified: that needs the SA's key). IP-in-IP needs no renderer of its own: the inner IPv4 or IPv6 header renders as one. TFTP shows a request's filename, mode and options, a DATA's or ACK's block, an ERROR's code, name and message, and an OACK's options; RADIUS its code, identifier, length and Authenticator, each text attribute (User-Name, Reply-Message, the station IDs, Acct-Session-Id) on its own line, and every other attribute by name with its address or number, a hidden password by size only; STUN its method and class, Transaction ID and verified FINGERPRINT (a misplaced one flagged), the mapped addresses, ERROR-CODE, SOFTWARE, UNKNOWN-ATTRIBUTES and whether MESSAGE-INTEGRITY is present. All sender-chosen text is sanitized for the terminal, kept on one line and cut at 128 characters, and a malformed RADIUS or STUN attribute list is reported as [!] ... malformed without hiding the header.
• HTTP/1.x screen renderer, for HTTP reached through --decode-as (NETProtocols registers it on no port): the start line; Host, User-Agent, Referer, Content-Type, Server and Location, then every field by name (credentials by name only); the body's framing and size, never its bytes, and a chunked body's trailer count; the anomalies that make a message's end ambiguous -- Content-Length with Transfer-Encoding, a repeated or invalid Content-Length, a second Host, ... -- flagged [!] as request-smuggling signal, the rest noted [~]; and a count of the bytes after the message, since the walk does not chain pipelined messages. Every value of the head is sanitized for the terminal (read as latin-1, a lone 0x9B is a C1 CSI), kept on one line and cut.
• SCTP has its own bucket in the capture statistics.
• Conversation statistics in the end-of-run summary: how many TCP/UDP/SCTP conversations the capture held and the ten largest by bytes, each with its frame and byte counts, both directions together (netprotocols.flow.flow_key). A tunnelled frame counts under the conversation inside the tunnel, VXLAN's underlay UDP never counts as one, and at most 65,536 conversations are tracked, the frames of any past that counted as untracked.
• The corpus grows from 65 frames in 12 captures to 617 in 42, vendored from NETProtocols 2.9.0 (see tests/fixtures/MANIFEST.md), including the BGP, TLS and RFC 3489 STUN session captures upstream keeps outside its own corpus: their bare ACKs and split records are exactly what the best-effort rule above must not call malformed. Every frame is decoded, rendered and serialized, and no rendered checksum may read as a mismatch.
• tests/fixtures/http/http.pcap, vendored byte-for-byte from NETProtocols 2.9.0: 98 frames of seven HTTP/1.1 connections, outside the corpus glob as upstream keeps it.

Fixed

• The NDJSON encoder's last-resort str() fallback is no longer silent: it warns once per type on stderr, so a field type a future NETProtocols release adds is noticed instead of shipping as an opaque string.
• --json crashed on LLDP and CDP frames ("Object of type bytes is not JSON serializable"): bytes were hex-encoded only at a layer's top level, but LLDP/CDP keep theirs inside TLV tuples, and LLDP is decoded by default. Encoding now goes through a json.dumps default= hook that handles bytes, enums, ipaddress objects and sets at any depth, and stringifies anything else rather than abort the capture.
• BREAKING (NDJSON): NDJSON lost the layer name of ICMPv4, ICMPv6 and IGMP (and now BGP, OSPF, PIM, VRRP): spreading a layer's fields after its "type" discriminator let the layer's own type field overwrite it, so an echo request read "type": 8 instead of "type": "ICMPv4". "type" is now always the class name and the wire field is emitted as "type_". Underscored decode-context fields (BGP's _followed) are no longer emitted.
• False checksum mismatches inside tunnels. TCP/UDP/ICMP checksums were verified against the outermost IP header, which is wrong for every tunnelled segment (VXLAN, GRE, IP-in-IP, AH tunnel mode): the corpus' inner UDP in VXLAN and TCP behind AH tunnel mode both read as mismatches. They now use the nearest enclosing IP header, as Packet.with_checksums() does.
• False checksum mismatches behind an IPv6 Routing header. The pseudo-header now carries the route's final destination when a Routing header with segments left sits between the IP header and the transport (RFC 8200 §8.1), via verify(..., routing=).
• False checksum mismatches on padded frames. A TCP/UDP/ICMPv6 checksum input included Ethernet padding and any captured FCS (a 54-byte bare ACK padded to 60 bytes summed 6 bytes too many); it now ends at that layer's own declared end, the tightest IPv4/IPv6/UDP/802.3 length among the headers enclosing it. Per layer rather than the frame's trailer, so a VXLAN tunnel's outer UDP checksum still covers the inner frame's padding. ICMPv4 now gets its IP header too, so the library bounds its sum the same way.
• Checksums were skipped, or wrongly checked, around tunnelled fragments. Fragmentation was judged frame-wide from the outermost IPv4 header and any IPv6 Fragment header anywhere, so an inner IPv4 fragment got a false mismatch and an inner IPv6 fragment hid the outer UDP's. A checksum is now skipped only when a fragment header encloses its own layer.
• Truncation was missed behind a VLAN tag. The truncation check put the IP header at a fixed offset of 14 bytes, so a capture short by up to a VLAN tag's (or QinQ pair's, MPLS stack's, LLC/SNAP header's) length read as complete. It now uses the outermost IP header's true offset.
• A lying inner IPv4 length was not flagged as malformed_length (only the first IPv4 header was checked); every IPv4 header is checked now, and the screen line quotes the one that lied.