gsalsbery - 2014-09-27

EXD

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | EXAMPLES
DEPENDENCIES | SEE ALSO | BUGS | AUTHOR | COPYRIGHT


NAME

exd − Locate and display or replace bytes in hex dumps or packet captures

SYNOPSIS

exd

[−i <beg[,:]end>] [−b <byte replacement or checkvalues[bitmasks]] [−w] [−R x|o|b|o0] [−H <header len>] [−C] [−m [−z]] [−t bin|pcap] [−l <beg[,:]end>] [−c <min[,:]max>] [−k <fields>] [−n <fields>] [−S <field separator>] [−W <field widths>] [−K <fields>] [−F <field list>] [−I] [−x] [−B <lines>] [−A <lines>] [−N] [−M] [−V] [−L] [−p <packets>] [−r] [−o <protocol>] ... [−s <src address>] ... [−d <dst regexp>] ... [−f <fne,expr>] [−f <te,expr>] [−f <fle,expr>] [−e <packet end separator>] [−a <regexp>] [−Z <regexp separator>] [−U <regexp>] ... [−J <regexp>] ... [−D <regexp>] ... [−E <regexp>] ... [−j <regexp>] ... [−u <[OPT,]expr>] ... [−g <capture group number>] [−T] [−P] [−O <tshark options>] ... [−X 0|1|l|r] [−G] [−Y] [−y] [−q] [−v] [−h] <file>

DESCRIPTION

Examine/edit hex dump tool − Locates hex, octal or binary bytes at specified indices in a binary or text file, or stdin. If byte values are specified using the −b option, then they will be compared to the ones found at the indices. If they agree, the exit status will be zero. If the write option (−w) is specified, then the bytes at the indices will be replaced with the ones specified using the −b option. If the file is a packet capture, then the bytes may be examined or replaced for every packet or for specific packets. The edit hex dump tool, exd, is a bash-wrapped AWK script which aids in examining and/or editing hex, octal or binary bytes in hex dumps and packet capture files. It is meant to be used in conjunction with hex dump tools such as xxd, od, hexdump and hd, and with packet capture dump tools such as tshark and text2pcap. The exd script may be used directly with binary files and pcap files, but it will call xxd or tshark/text2pcap to do the heavy lifting.

OPTIONS

−i <idx1,idx2,...>

List of indices

−i <beg:end>

Range of indices.

If the end index is less than the beginning index, then the bytes will be displayed or replaced in reverse order. Index -1 indicates the last byte of the file or frame. If the beginning index is omitted, then it defaults to 1. If the end index is omitted, then it defaults to -1.

−b <bytevals[bitmasks]>

Replacement/check byte values.

If the write option (−w) is specified, then replace the bytes at the indices with the bytevals. Otherwise, compare the bytes at the indices with the bytevals. If all agree, then the return value will be zero. If multiple values, then ,- or :-separate, or space-separate and quote. If the number of indices exceeds the number of bytevals, then cycle through the values specified. If the byte values have radix prefixes, such as 0x or 16#, then the values will be converted to the radix of the target bytes. If they do not have prefixes, then the byte values must be in the correct radix. A byte bitmask, which will be applied to both the check value and the byte at the corresponding index before comparison, may be specified by enclosing it in parentheses () or square brackets [] immediately following the check value, with no spaces. If the bitmask has the same radix as the check value, then no radix prefix is required.

−w

Replace the bytes at the indices with the values following the −b option.

If a binary or pcap file is specified, then the file will be rewritten with the changes. Otherwise the changes will be written to stdout.

−R x|o|b|o0

Radix of the bytes - he[x], [o]ctal or [b]inary.

Defaults to hex. If octal, and the bytes include the value 0001, is that a byte 001 with a leading prefix of 0, or is that two bytes, 000 and 001? If the former, then specify octal using o0 rather than o to enable the single-zero octal prefix.

−H <header len>

Length of the header.

Number of bytes to skip before indices increment. Defaults to 0.

−C

Put exd in count mode.

Display a count the number of matched bytes rather than a list of the found bytes. If no check bytes are supplied then all bytes trivially match. If packets are specified, then byte count of each packet will be displayed, followed by a colon and a space separated list of pairs of matched packets and their counts, #-separated.

Has no effect if the write (−w) option is specified.

−m

Put exd in match mode.

If check bytes are present:

  • The return value will be zero if any matches occur, rather than only if all match.
  • If context is on, and the check bytes agree with the bytes at the indices, display that there is a match immediately before the display of the found bytes.
  • If context if off, and packets are specified, suppress the display of bytes found.
  • If packets are specified, list the packets which matched.

    If count mode is enabled and packets are specified:

  • Suppress the display of counts for each packet, and suppress the colon which separates the individual packet counts from the final list.

    Has no effect if the write (−w) option is specified.

−z

Reverse the sense of match mode.

If check bytes are present:

  • The return value will be non-zero if all match.
  • If context is on, and the check bytes do not all agree with the bytes at the indices, display that there is a match immediately before the display of the found bytes.
  • If packets are specified, list the packets which survive the filter but do not match all the check bytes.

    Has no effect if match mode is not set.

−t bin|pcap

Set file type to binary or packet capture. If binary, sets −k to 1, −n to 8, −c to :50, and runs xxd for conversions. If packet capture, sets −k to 1, −n to 16, −c to :56, and runs tshark and text2pcap for conversions. If you’re planning to run exd many times on the same binfile, then it would probably be a better idea to convert once using xxd or tshark, run exd multiple times on the resulting hex dump, and then convert once back using xxd −r or text2pcap, rather than calling exd many times with the −t option doing a conversion out and back in every time.

−l <beg[,:]end>

Restrict the search to the given range of lines.

options for dealing with non-byte fields in hex dumps:

−c <min[,:]max>

Restrict the search or changes to the range of columns.

−k <fields>

The number of initial fields to skip in every line before searching for bytes. If xxd is used to create a hex dump, the first column is the file offset, which ends with a colon; exd recognizes this and skips the field. However, if od is used, the file offset column does not end with a colon, and therefore is not distinguishable from data. In this case, the −k option allows one to skip the field.

−n <fields>

The number of fields in a line (after skipped) to parse.

−S <field separator>

Regular expression which separate fields One may use regexp character class [:blank:] or [:space:] for whitespace - not supported for older awks, or one may use the exd-only character classes [:s:] for a single space and [:t:] for a single tab.
E.g.

-S [,[:blank:]]+ or equivalently
-S [,[:s:]\\t]+ or
-S [,[:s:][:t:]]+ for comma- or whitespace-separated fields

−W <field widths>

Comma-separated list of field widths to split lines based on fixed-width data - overrides field separators. This will only work if the AWK on your system supports fieldwidths and if the value of the variable awk_fieldwidths, located near the top of the script, is set to 1.

−K <fields>

The number of fields to skip between byte fields.

−F <field list>

Comma-separated list of valid byte fields.

−I

Ignore non-byte fields.

The standard behavior when exd encounters a field that is not a byte field is to stop processing the line. If this option is selected, then fields which do not contain bytes will simply be skipped.

options for context display - ignored if the write (−w) option is selected:

−x

Show a number of context lines before and a number of context lines after the target bytes. Default is 20 before and four after.

Highlight the location of the target bytes using pointers. For hex and binary data, the pointers will match up exactly with the bytes. For octal, the pointers will come as close as possible to highlighting the bytes.

−B <count>

The number of lines of context to show before the target byte. Default 20. A negative value means show all.

−A <count>

The number of lines of context to show after the target byte. Default 4. A negative value means show all.

−N

Do not show a running count of bytes at the end of each context line before the first target byte is reached.

−M

Do not display pointers for bytes in the middle of the range, only pointers for the endpoints.

−V

Verify pointers.

Inspect the contents of the byte string highlighted by the byte pointers. If they do not agree with the expected byte value, print an error message and return non-zero to indicate failure. This option should really only be used during Verification and Validation tests of the tool.

options for dealing with endian issues:

−L

If the memory architecture is little endian, and the tool used to extract the hex dump does not convert the output to big endian (e.g. od −A x −t x2 or od −A x −t x4), then this flag may be used to correct the selection of the desired bytes.

−LP

If the -L and -P options are combined, print whether the current architecture is little endian or big endian and exit.

options for packet capture files or their hex dumps:

If the file is a packet capture or the output of tshark with the −x option, then each packet will be parsed for bytes.

−p <pkt1,pkt2,...>

List of packets to process.

−p <beg:end>

Range of packets to process.

Packet numbers are determined by position and not by any numbers found in packet summary lines.

If the file includes packet summary lines, such as produced with the −P option of tshark, or contains a packet tree, or packet detail lines, such as produced with the −V option of tshark, then the packets may be filtered using this information.

−r

Packet numbering is relative. A packet is counted only if it survives the filter.

The standard packet summary lines begin with a frame number, followed by time (in any of the standard tshark formats), source, destination, protocol and frame length, as in the line

    4  0.000003 192.168.0.1 -> 192.168.0.2 TCP 56 ...

−o <protocol regexp> ...

Skip packets which do not match the protocol(s).

−s <src address regexp> ...

Skip packets which do not match the source address(es).

−d <dst address regexp> ...

Skip packets which do not match the destination address(es).

The next three filters work with comparison operators and coordinating conjunction operators.

The comparison operators are:

            <, <=, >, >=, ==, !=, ~, !~, or
         :LT:, :LE:, :GT:, :GE:, :EQ:, or :NE:
     all but ~ and !~ are numeric or time-specific

And the coordinating conjunction operators are:

              && or :AND: or || or :OR:

−f fne,<expr>

Skip packets which do not satisfy the simple comparison expression expr against the frame number. The comparison expression must be of the form [fn]opVal[\<conj>[fn]opVal], where Val is an integer or is a regular expression for an integer, op is a comparison operator and conj is a coordinating conjunction operator.
E.g.

            >5&&<31   or   ~[34][0-9] or
          fn>5&&fn<31 or fn~[34][0-9]

−f te,<expr>

Skip packets which do not satisfy the simple comparison expression expr against the time. The comparison expression must be of the form [t]opVal[\<conj>[t]opVal], where Val fits a valid tshark time format or is a regular expression for a valid tshark time format,op is a comparison operator and conj is a coordinating conjunction operator.
E.g.

           >21.24 or  ==2014-08-16 13:09:34.51 or
          t>21.24 or t==2014-08-16 13:09:34.51

−f fle,<expr>

Skip packets which do not satisfy the simple comparison expression expr against the frame length. The comparison expression must be of the form [fl]opVal[\<conj>[fl]opVal], where Val is an integer or is a regular expression for an integer,op is a comparison operator and conj is a coordinating conjunction operator.
E.g.

            >1400 or   ==56 or
          fl>1400 or fl==56

The < or > may need to be escaped so that they are not mistaken for redirection operators, and the &s and |s may need to be escaped to prevent their being mistaken for background or pipe operators.

The above six filters require either a packet summary line or a packet tree. Packet summary lines are preferred, and if both are present, the packet summary lines will be used. If using packet details, then some of the values may differ from their corresponding packet summary values.

−e <packet end separator>

Regular expression for the separator marking the end of a packet (cf. the −S option in tshark).

−a <packet summary/detail regexp>

Alternate packet summary or detail line regular expression. This may be an alternative regular expression for a standard packet summary line, or it may be for a custom packet summary line.

−Z <separator>

Separator for passing multiple user-defined regular expressions in one string, for when more than one −U option, −u option, −J option, −D option, −E option, or −j option is specified. Defaults to ///.

−U <packet summary/detail regexp> ...

User-defined regular expression to parse the packet summary or detail line for extracting the value which will be matched against the corresponding user-defined regular expression passed with the −u option.

−J <packet summary/detail regexp> ...

User-defined primary auxilliary regular expression to parse the packet summary or detail line after it has been parsed by either the corresponding regular expression from the −U option or by the regular expression passed with the −a option. Used after any capture groups.

−D <gsub regexp> ...

User-defined auxilliary regular expression used with gsub to delete the matching portion from the results. Used after the corresponding -J option.

−E <sub regexp> ...

User-defined auxilliary regular expression used with sub to delete the matching portion from the results. Used after the corresponding -D option.

−j <packet summary/detail regexp> ...

User-defined secondary auxilliary regular expression to parse the packet summary or detail line. Used after the corresponding -J -D and -E options.

−u <[OPT,]user-defined> ...

User-defined simple comparison expressions to match against the string found by parsing the packet summary line or any packet tree lines using the regular expression defined with the −U option or with the −a option and any auxilliary regular expressions. If prepended with an "OPT,", then the expression is not mandatory.

−T

If combined with the −u option, then test the regular expressions used to determine the value for the comparison. This will print to stderr the intermediate strings after each of the options −U, −J, −D, −E and −j are applied.

−g <group numbers>

Comma- or colon-separated group numbers for matching parenthesized subexpressions in the user-defined regular expression specified with the −U option or −a option. This will only work if the awk on your system supports the third argument to match, if the value of the variable awk_third_arg_match, located near the top of the exd script, is set to 1, and if the function call_match has been modified according to the instructions accompanying the function.

−T

Unless combined with the −u option, tidy a packet capture hex dump for easier processing and exit.
Removes any packet lines after matching /\^Reassembled/ or /\^Uncompressed entity body/. Removes anything after column 56 of every hexdump line to remove the possibility of confusing the ASCII text dump with bytes in the hexdump. It leaves the standard packet summary lines intact so that they may be used for filtering. If the −V option is used, tidying includes removing the packet tree or packet details (from −V option of tshark).

−P

Unless combined with the −L option, prep a packet capture hex dump for conversion for conversion to a pcap file using text2pcap and exit.
Removes packet summary lines, but leaves the time portion of the summary line alone so that text2pcap can retain the time stamp information. If the time stamp is not desired, run using the −P option twice.
Removes any packet lines after matching /\^Reassembled/ or /\^Uncompressed entity body/. Removes anything after column 56 of every hexdump line to remove the possibility of confusing the ASCII text dump with bytes in the hexdump. This last means that the text2pcap −a option should not be used with a file prepared using the −P option.

−V

If a file is being tidied using the −T option, remove the packet tree or packet details (from −V option of tshark).

−O <tshark options> ...

If using exd directly on a pcap file, then tshark is used for the conversion to a hex dump. This option allows you to set tshark options. The default options are "-o tcp.desegment_tcp_streams:FALSE -o http.decompress_body:FALSE".

other options:

−X 0|1|l|r

Allow shifting to the left or right into field separators or whitespace if the replacement byte is larger than the original. Default is not to allow shifting unless fieldwidths are used. If enabling shifting, then one may use l or r to indicate preferred direction of shift. The default preferred shift direction is left.

−G

Allow byte fields to include other garbage data or characters at the end of the field.

−Y

Verbose display of target byte.

−y

Echo the command line.

−q

Suppress output of found bytes.

−v

Show the version and exit.

EXAMPLES

Locate and display the 32nd byte in a hex dump.

exd −i 32

Locate and display, with context, the 32nd byte in a hex dump.

exd −i 32 −x

Locate and display the 32nd byte in a hex dump, return 0 if it is 0a.

exd −i 32 −b 0a

Locate the 32nd byte in a hex dump, return 0 if it is 0a.

exd −i 32 −b 0a −q

Replace the 32nd byte in a hex dump with 0a.

exd −i 32 −b 0a −w

Locate and display the 32nd byte in an octal dump on a little endian machine, return 0 if it is 012.

exd −i 32 −b 012 −R o −L

Replace the 32nd byte in an octal dump on a little endian machine with 0x0a converted to octal 012.

exd −i 32 −b 0x0a −R o −L −w

Locate and display the 32nd byte in a binary dump, return 0 if it is 00001010.

exd −i 32 −b 1010 −R b

Replace the 32nd byte in a binary dump with 0x0a converted to binary 00001010.

exd −i 32 −b 0x0a −R b −w

Replace the 32nd byte in a little endian octal dump with binary 2#1010 converted to octal 012.

exd −i 32 −b 2#1010 −R o −L −w

Locate and display the 32nd byte in a binary file.

exd −i 32 −t bin

Locate and display, with context, the 32nd byte in a binary file.

exd −i 32 −x −t bin

Locate and display the 32nd byte in a binary file, return 0 if it is 0a.

exd −i 32 −b 0a −t bin

Replace the 32nd byte in a binary file with 0a.

exd −i 32 −b 0a −t bin −w

Overwrite a binary file with 0xdeadbeef.

exd −i 1: −b 0xde:0xad:0xbe:0xef −t bin −w

examples with packets:

Locate and display the 3rd and 4th bytes after a 14 byte header in every TCP packet of a pcap file.

exd −p −1 −H 14 −i 3:4 −o TCP −t pcap

Locate and display the 3rd and 4th bytes after a 14 byte header in every TCP packet of a tshark hex dump.

exd −p −1 −k 1 −n 16 −c :56 −H 14 −i 3:4 −o TCP

Locate and display, with context, the 3rd and 4th bytes after a 14 byte header in every TCP packet of a pcap file.

exd −x −p −1 −H 14 −i 3:4 −o TCP −t pcap

Locate and display, with context, the 3rd and 4th bytes after a 14 byte
header in every TCP or HTTP packet of a pcap file.

exd −x −p −1 −H 14 −i 3:4 −o TCP −o HTTP −t pcap

Display a space-separated list of every HTTP packet of a pcap file.

exd −p −1 −o HTTP −m −t pcap

For a pcap file, display a space-separated list of every packet with protocol exactly HTTP, excluding protocols with HTTP in the name, such as HTTP/XML

exd −p −1 −o \^HTTP\$ −m −t pcap

For a pcap file, locate and display, with context, the 3rd and 4th bytes after a 14 byte header in every TCP packet with source 65.254.160.237.

exd −x −p −1 −H 14 −i 3:4 −o TCP −s \^65.254.160.237\$ −t pcap

           Note that without the leading caret (^), the regular
            expression for the source would also match 165.254.160.237.
            Also note that in regular expressions . means any character,
            but in this case a dot is a character, and we are unlikely
            to encounter other characters in that position, so it fits.
            If one wanted to be absolutely sure to match 65.254.160.237,
            and not something like 65x254_160p237, then one should use
            ^65[.]254[.]160[.]237$.

For a pcap file, locate and display, with context, the 3rd and 4th bytes after a 14 byte header in every TCP packet with destinations 192.168.20.41, 192.168.20.42, 192.168.20.43, or 192.168.20.47.

exd −x −p −1 −H 14 −i 3:4 −o TCP −d \^192.168.20.4[1-37]\$ −t pcap

Locate and change the 2nd byte after a 14 byte header to 0x10 in packets 2, 3, 7 and 19 of a pcap file.

exd −p 2,3,7,19 −H 14 −i 2 −b 0x10 −w −t pcap

For a tshark hex dump, of the first 200 packets, list the TCP packets.

exd −p :200 −k 1 −n 16 −c :56 −o TCP −m

For a tshark hex dump, of the first 200 packets, list the TCP packets whose 2nd byte after a 14 byte header is 00.

exd −p :200 −k 1 −n 16 −c :56 −o TCP −H 14 −i 2 −b 0x00 −m

For a tshark hex dump, of the first 200 packets, list the TCP packets whose 2nd byte after a 14 byte header is not 00.

exd −p :200 −k 1 −n 16 −c :56 −o TCP −H 14 −i 2 −b 0x00 −m −z

For a tshark hex dump, locate and display the 3rd and 4th bytes after a 14 byte header in the fourth packet, but only if it is a TCP packet.

exd −p 4 −k 1 −n 16 −c :56 −H 14 −i 3:4 −o TCP

For a tshark hex dump, locate and display the 3rd and 4th bytes after a 14 byte header in the fourth TCP packet.

exd −p 4 −r −k 1 −n 16 −c :56 −H 14 −i 3:4 −o TCP

For a tshark hex dump, count the number of bytes in the 4th packet.

exd −p 4 −k 1 −n 16 −c :56 −C

For a tshark hex dump, count the number of bytes in every packet.

exd −p −1 −k 1 −n 16 −c :56 −C

For a tshark hex dump, count the number of bytes in every packet, but suppress displaying individual counts and only display the final space-separated list of packet#count pairs.

exd −p −1 −k 1 −n 16 −c :56 −C −m

For a tshark hex dump, locate and display, with full context, the first byte after a 20 byte header in every ICMP packet captured between the first and third seconds, and display ===MATCH=== if its least significant nybble is 8. List the packets which matched.

    exd −x −B −1 −A −1 −p −1 −k 1 −n 16 −c :56 −o ICMP -H 20
        −i 1 −b 08[0f] −m −f te,"t>=1.0&&t<=3.0"
                           or
    exd −x −B −1 −A −1 −p −1 −k 1 −n 16 −c :56 −o ICMP -H 20
        −i 1 −b 0x08[0x0f] −m −f te,t:GE:1.0:AND:t:LE:3.0

For a tshark hex dump, locate and display, with context, the 1st byte in every packet with packet detail "Acknowledgment number: 480". Test the regular expressions by printing to stderr the intermediate matches. Show a list of the packets which matched.

    exd −x −k 1 −n 16 −c :56 −i 1 −p −1    
        −U "^[[:s:][:t:]]*Acknowledgment number:[0-9]+[[:s:][:t:]]+[(]relative ack number[)]$"
        −J "[0-9]+[[:s:][:t:]]+[(]relative ack number[)]$"
        −E "[[:s:][:t:]]+[(]relative ack number[)]$" −u "u==480" −T −m

Locate and display, with context, the 1st byte in every packet with packet detail "Acknowledgment number: 480" or in which there is no Acknowledgment number detail.

    exd −x −k 1 −n 16 −c :56 −i 1 −p −1
        −U "^[[:s:][:t:]]*Acknowledgment number: [0-9]+[[:s:][:t:]]+[(]relative ack number[)]$"    
        −J "[0-9]+[[:s:][:t:]]+[(]relative ack number[)]$"
        −E "[[:s:][:t:]]+[(]relative ack number[)]$"
        −u "OPT,u==480"

DEPENDENCIES

If used with a binary file, xxd is used for conversions. If used with a packet capture (pcap) file, tshark and text2pcap are used for conversions. If using capture groups or fieldwidths, then an AWK which supports a three-argument match function and/or fieldwidths, such as gawk, must be used.

SEE ALSO

xxd(1), hexdump(1), od(1), pcap(3), wireshark(1), tshark(1), text2pcap(1)

BUGS

No known bugs.

AUTHOR

Gary Salsbery <garysalsbery[AT]mac[DOT]com>

Copyright © 2014 Gary Salsbery

Licensed under the Apache License, Version 2.0 or later

apache.org licenses ⟨ URL: http://www.apache.org/licenses/LICENSE-2.0

This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.


 

Last edit: gsalsbery 2014-09-27