command-output-to-html-table Wiki
A shell script to convert any file or command output into a html table
Brought to you by:
nathan-sr
Menu
▾
▴
collection of linux one-liner commands
Table of Contents
- General Usage
- File Spacing
- Numbering and Calculations
- Conversion and Substitution
- Selective Printing
- Selective Deletion
- Insertion
- String Creation
- Special Applications for sed
- Thanks
General Usage
Most of the following commands can be used both on files directly and in pipes, e.g. the command
sed expression can be used in both of the following ways:
... | sed expression sed expression path/to/file
File Spacing
Double space:
sed G awk '1;{print ""}' awk 'BEGIN{ORS="\n\n"};1'
Double space which already has blank lines in it. Output should contain no more than one blank line
between lines of text.
sed '/^$/d;G'
awk 'NF{print $0 "\n"}'
Triple space:
sed 'G;G'
awk '1;{print "\n"}'
Undo double-spacing (assumes even-numbered lines are always blank):
sed 'n;d'
Insert a blank line above every line which matches regex:
sed '/regex/{x;p;x;}'
Insert a blank line below every line which matches regex:
sed '/regex/G'
Insert a blank line above and below every line which matches the regular expression:
sed '/regex/{x;p;x;G;}'
Appends trailing newline:
sed '$a\'
Numbering and Calculations
Number each line (simple left alignment), using a tab:
sed = filename | sed 'N;s/\n/\t/'
awk '{print FNR "\t" $0}'
Precede each line by its line number FOR ALL FILES TOGETHER, using a tab:
awk '{print NR "\t" $0}' files*
Number each line (number on left, right-aligned):
nl
sed = filename | sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /'
awk '{printf("%5d : %s\n", NR,$0)}'
Number each line, but only print numbers if line is not blank:
sed '/./=' filename | sed '/./N; s/\n/ /' awk 'NF{$0=++a " :" $0};1' awk '{print (NF? ++a " :" :"") $0}'
Count lines:
wc -l sed -n '$=' awk 'END{print NR}'
Print the sums of the fields of every line:
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
Add all fields in all lines and print the sum:
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'
Print every line after replacing each field with its absolute value:
awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }' awk '{for (i=1; i<=NF; i++) $i = ($i < 0) ? -$i : $i; print }'
Print the total number of fields ("words") in all lines:
awk '{ total = total + NF }; END {print total}' file
Print the total number of lines that contain "Beth":
awk '/Beth/{n++}; END {print n+0}'
Print the largest first field and the line that contains it (intended for finding the longest string
in field 1):
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'
Print the number of fields in each line, followed by the line:
awk '{ print NF ":" $0 } '
Print the last field of each line:
awk '{ print $NF }'
Print all except the first field:
awk '{ $1 = ""; print substr($0, 2) }'
Print the last field of the last line:
awk '{ field = $NF }; END{ print field }'
Print every line with more than 4 fields:
awk 'NF > 4'
Print every line where the value of the last field is > 4
awk '$NF > 4'
Add first and last column in each line:
perl -lane 'print $F[0] + $F[-1]'
Increments all numbers by 1:
perl -pe 's/(\d+)/ 1 + $1 /ge'
Sum up a column:
awk '{ sum+=$1 } END { print sum }'
Conversion and Substitution
Convert newline to space:
tr '\n' ' ' perl -pe 's/\n/ /' sed ':a;N;$!ba;s/\n/ /g' # will not convert last newline!
Convert CRLF to LF format:
sed 's/.$//' # assumes that all lines end with CRLF sed 's/^M$//' # in bash/tcsh, press Ctrl-V then Ctrl-M sed 's/\x0D$//' # works on ssed, gsed 3.02.80 or higher tr -d \r # GNU tr version 1.22 or higher awk '{sub(/\r$/,"")};1' # assumes EACH line ends with Ctrl-M perl -p -i -e 's/\012?\015/\n/g'
Delete leading whitespace (spaces, tabs) from front of each line, aligns all text, flush left:
sed 's/^[ \t]*//'
awk '{sub(/^[ \t]+/, "")};1'
Delete trailing whitespace (spaces, tabs) from end of each line:
sed 's/[ \t]*$//'
awk '{sub(/[ \t]+$/, "")};1'
Delete both leading and trailing whitespace from each line:
sed 's/^[ \t]*//;s/[ \t]*$//' awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1' awk '{$1=$1};1' # also removes extra space between fields
Insert 5 blank spaces at beginning of each line (make page offset):
sed 's/^/ /'
awk '{sub(/^/, " ")};1'
Align all text, flush right on a 79-column width:
sed -e :a -e 's/^.\{1,78\}$/ &/;ta'
awk '{printf "%79s\n", $0}'
Center all text in the middle of 79-column width. In method 1, spaces at the beginning of the line
are significant, and trailing spaces are appended at the end of the line. In method 2, spaces at the
beginning of the line are discarded in centering the line, and no trailing spaces appear at the end
of lines.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # method 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' # method 2
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}'
Substitute (find and replace) foo with bar on each line:
sed 's/foo/bar/' # replaces only 1st instance in a line awk '{sub(/foo/,"bar")}; 1' # replaces only 1st instance in a line sed 's/foo/bar/4' # replaces only 4th instance in a line gawk '{$0=gensub(/foo/,"bar",4)}; 1' # replaces only 4th instance in a line sed 's/foo/bar/g' # replaces ALL instances in a line awk '{gsub(/foo/,"bar")}; 1' # replaces ALL instances in a line sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # replaces the next-to-last case sed 's/\(.*\)foo/\1bar/' # replaces only the last case perl -p -i.bak -e 's/foo/bar/g' *.c # also keeps backups
Substitute foo with bar ONLY for lines which contain baz:
sed '/baz/s/foo/bar/g'
awk '/baz/{gsub(/foo/, "bar")}; 1'
Substitute foo with bar except for lines which contain baz:
sed '/baz/!s/foo/bar/g'
awk '!/baz/{gsub(/foo/, "bar")}; 1'
Change scarlet or ruby or puce to red:
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # most seds
gsed 's/scarlet\|ruby\|puce/red/g' # GNU sed only
awk '{gsub(/scarlet|ruby|puce/, "red")}; 1'
Reverse order of lines:
tac sed '1!G;h;$!d' sed -n '1!G;h;$p' awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' perl -e 'print reverse <>'
Reverses paragraphs:
perl -00 -e 'print reverse <>'
Reverse each character on the line:
rev sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//' perl -nle 'print scalar reverse $_'
Join pairs of lines side-by-side:
paste sed '$!N;s/\n/ /'
If a line ends with a backslash, append the next line to it:
sed -e :a -e '/\\$/N; s/\\\n//; ta'
If a line begins with an equal sign, append it to the previous line and replace the "=" with a
single space:
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'
Add commas to numeric strings, changing "1234567" to "1,234,567":
gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' # other seds
Add commas to numbers with decimal points and minus signs:
gsed -r ':a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta'
Add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.):
gsed '0~5G' # GNU sed only sed 'n;n;n;n;G;' # other seds
Print and sort the login names of all users:
awk -F ":" '{print $1 | "sort" }' /etc/passwd
Print the first 2 fields, in opposite order, of every line:
awk '{print $2, $1}'
Switch the first 2 fields of every line:
awk '{temp = $1; $1 = $2; $2 = temp}'
Print every line, deleting the first field of that line:
cut -d' ' -f2-
awk '{ $1 = ""; print }'
Print every line, deleting the second field of that line:
awk '{ $2 = ""; print }'
Print in reverse order the fields of every line:
awk '{for (i=NF; i>0; i--) printf("%s ",$i);print ""}'
Concatenate every 5 lines of input, using a comma separator between fields:
awk 'ORS=NR%5?",":"\n"'
Rename in each file name the string aaa by bbb:
ls | perl -ne 'chomp; next unless -e; $o = $_; s/aaa/bbb/; next if -e; rename $o, $_';
Selective Printing
Print first 10 lines:
head sed 10q awk 'NR < 11'
Print first line:
head -1 sed q awk 'NR>1{exit};1'
Print last 10 lines:
tail sed -e :a -e '$q;N;11,$D;ba'
Print last 2 lines:
tail -2 sed '$!N;$!D' awk '{y=x "\n" $0; x=$0};END{print y}'
Print last line:
tail -1 sed '$!d' sed -n '$p' awk 'END{print}'
Print next-to-the-last line:
sed -e '$!{h;d;}' -e x # for 1-line, print blank line sed -e '1{$q;}' -e '$!{h;d;}' -e x # for 1-line, print the line sed -e '1{$d;}' -e '$!{h;d;}' -e x # for 1-line, print nothing
Print only lines which match regular expression:
grep 'regex' sed -n '/regex/p' # method 1 sed '/regex/!d' # method 2 awk '/regex/'
Print only lines which do not match regular expression:
grep -v regex sed -n '/regex/!p' # method 1, corresponds to above sed '/regex/d' # method 2, simpler syntax awk '!/regex/'
Print the line immediately before a regular expression, but not the line containing it:
sed -n '/regex/{g;1!p;};h' awk '/regex/{print x};{x=$0}' awk '/regex/{print (NR==1 ? "match on line 1" : x)};{x=$0}'
Print the line immediately after a regular expression, but not the line containing it:
sed -n '/regex/{n;p;}' awk '/regex/{getline;print}'
Print 1 line of context before and after a regular expression, with line numbers:
grep -A1 -B1 -n regex sed -n -e '/regex/{=;x;1!p;g;$!N;p;D;}' -e h
Search for AAA and BBB and CCC (in any order):
sed '/AAA/!d; /BBB/!d; /CCC/!d' awk '/AAA/ && /BBB/ && /CCC/'
Search for AAA and BBB and CCC (in that order):
sed '/AAA.*BBB.*CCC/!d' awk '/AAA.*BBB.*CCC/'
Search for AAA or BBB or CCC:
egrep "AAA|BBB|CCC" grep -E "AAA|BBB|CCC" sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # most seds gsed '/AAA\|BBB\|CCC/!d' # GNU sed only
Print paragraph if it contains AAA (blank lines separate paragraphs):
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'
Print paragraph if it contains AAA and BBB and CCC (in any order):
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'
Print paragraph if it contains AAA or BBB or CCC:
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d' # GNU sed only
Print only lines of 65 characters or longer:
sed -n '/^.\{65\}/p'
awk 'length > 64'
Print only lines of less than 65 characters:
sed -n '/^.\{65\}/!p' # method 1, corresponds to above
sed '/^.\{65\}/d' # method 2, simpler syntax
awk 'length < 65'
Print section from regular expression to end:
sed -n '/regex/,$p' awk '/regex/,0' awk '/regex/,EOF'
Print section based on line numbers (lines 8-12, inclusive):
sed -n '8,12p' sed '8,12!d' awk 'NR==8,NR==12' perl -ne 'print if 8 .. 12' perl -pe 'exit if 8<$. && $.<12'
Print line number 52:
sed -n '52p' sed '52!d' sed '52q;d' # efficient on large files awk 'NR==52' awk 'NR==52 {print;exit}' # more efficient on large files
Beginning at line 3, print every 7th line:
gsed -n '3~7p' sed -n '3,${p;n;n;n;n;n;n;}'
Print section between two regular expressions (inclusive):
sed -n '/START/,/END/p' awk '/START/,/END/' perl -ne 'print if /START/ .. /END/' perl -ne 'print if m{START} .. m{END}'
Print all except section between 2 regular expressions:
sed '/START/,/END/d' perl -i.old -ne 'print unless /START/ .. /END/'
Match a field against a regular expression:
awk '$7 ~ /^[a-f]/' # print line if field #7 matches regex awk '$7 !~ /^[a-f]/' # print line if field #7 does NOT match regex
Print any line where field 5 is equal to abc123:
awk '$5 == "abc123"'
Print only those lines where field 5 is NOT equal to abc123 (also prints lines with less than 5
fields):
awk '$5 != "abc123"' awk '!($5 == "abc123")'
Print palindromes:
perl -lne 'print if $_ eq reverse'
Print duplicated words in a line:
perl -0777 -ne 'print "$.: doubled $_\n" while /\b(\w+)\b\s+\b\1\b/gi'
Extract from pattern:
awk 'match($0, /(.+)/, a) { print a[1] }'
grep -oP "queue name='\K.+?(?=')"
Filter with context and print fields from context:
grep -A1 'to filter' /etc/multipath.conf | awk 'BEGIN { RS="--" } ; { print $2, $4 }'
Print from capture group:
awk 'match($0, /h_vmem=([0-9]+[kKmMgGtT])/, a) { print a[1] }'
Selective Deletion
Delete duplicate, consecutive lines, first line in a set of duplicate lines is kept, rest are
deleted:
uniq
sed '$!N; /^\(.*\)\n\1$/!P; D'
awk 'a !~ $0; {a=$0}'
Delete duplicate, nonconsecutive lines:
sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
awk '!a[$0]++' # most concise script
awk '!($0 in a){a[$0];print}' # most efficient script
Delete all lines except duplicate lines:
uniq -d sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'
Delete the first line:
tail -n +2 awk 'NR > 1'
Delete the first 10 lines:
sed '1,10d' awk 'NR > 10' perl -ne 'print unless 1 .. 10'
Delete the 5th line:
awk 'NR != 5' sed '5d'
Delete a range of lines, e.g. 5-10:
awk 'NR < 5 || NR > 10' sed '5,10d'
Delete the last line:
sed '$d'
Delete the last 2 lines:
sed 'N;$!P;$!D;$d'
Delete the last 10 lines:
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # method 2
Delete every 8th line:
gsed '0~8d' # GNU sed only sed 'n;n;n;n;n;n;n;d;' # other seds
Delete lines matching pattern:
sed '/pattern/d'
Delete all blank lines:
grep '.' sed '/^$/d' # method 1 sed '/./!d' # method 2 awk NF awk '/./'
Delete all consecutive blank lines except the first, also deletes all blank lines from top and end:
cat -s sed '/./,/^$/!d' # method 1, allows 0 blanks at top, 1 at EOF sed '/^$/N;/\n$/D' # method 2, allows 1 blank at top, 0 at EOF
Delete all consecutive blank lines except the first 2:
sed '/^$/N;/\n$/N;//D'
Delete all leading blank lines at top:
sed '/./,$!d'
Delete all trailing blank lines at end:
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # works on all seds sed -e :a -e '/^\n*$/N;/\n$/ba' # dito, except for gsed 3.02.*
Delete the last line of each paragraph:
sed -n '/^$/{p;h;};/./{x;/./p;}'
Insertion
Insert as first line:
sed '1 i foo
Insert after first line (as second line):
sed '1 a foo'
Insert line with BBB above lines containing AAA:
sed '/AAA/i BBB'
String Creation
Create a string of a specific length (e.g., generate 513 spaces)
awk 'BEGIN{while (a++<513) s=s " "; print s}'
Insert a string of specific length at a certain character position (example inserts 49 spaces after
column 6 of each input line):
gawk --re-interval 'BEGIN{while(a++<49)s=s " "};{sub(/^.{6}/,"&" s)};1'
Special Applications for sed
Remove nroff overstrikes (char, backspace) from man pages. The 'echo' command may need an -e switch
if you use Unix System V or bash shell.
sed "s/.`echo \\\b`//g" # double quotes required for Unix environment sed 's/.^H//g' # in bash/tcsh, press Ctrl-V and then Ctrl-H sed 's/.\x08//g' # hex expression for sed 1.5, GNU sed, ssed
Get Usenet/e-mail message header:
sed '/^$/q' # deletes everything after first blank line
Get Usenet/e-mail message body:
sed '1,/^$/d' # deletes everything up to first blank line
Get Subject header, but remove initial "Subject: " portion:
sed '/^Subject: */!d; s///;q'
Get return address header:
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'
Parse out the address proper. Pulls out the e-mail address by itself from the 1-line return address
header (see preceding script)
sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'
Add a leading angle bracket and space to each line (quote a message):
sed 's/^/> /'
Delete leading angle bracket & space from each line (unquote a message):
sed 's/^> //'
Remove most HTML tags (accommodates multiple-line tags):
sed -e :a -e 's/<[^>]*>//g;/</N;//ba'
Extract multi-part uuencoded binaries, removing extraneous header info, so that only the uuencoded
portion remains. Files passed to sed must be passed in the proper order. Version 1 can be entered
from the command line; version 2 can be made into an executable Unix shell script.
sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode # vers. 1 sed '/^end/,/^begin/d' "$@" | uudecode # vers. 2
Sort paragraphs of file alphabetically. Paragraphs are separated by blank lines. GNU sed uses \v for
vertical tab, or any unique char will do.
sed '/./{H;d;};x;s/\n/={NL}=/g' file | sort | sed '1s/={NL}=//;s/={NL}=/\n/g'
gsed '/./{H;d};x;y/\n/\v/' file | sort | sed '1s/\v//;y/\v/\n/'
TYPICAL USE: Sed takes one or more editing commands and applies all of them, in sequence, to each
line of input. After all the commands have been applied to the first input line, that line is output
and a second input line is taken for processing, and the cycle repeats. The preceding examples
assume that input comes from the standard input device (i.e, the console, normally this will be
piped input). One or more filenames can be appended to the command line if the input does not come
from stdin. Output is sent to stdout (the screen). Thus:
cat filename | sed '10q' # uses piped input sed '10q' filename # same effect, avoids a useless "cat" sed '10q' filename > newfile # redirects output to disk
For additional syntax instructions, including the way to apply editing commands from a disk file
instead of the command line, consult "sed & awk, 2nd Edition," by Dale Dougherty and Arnold Robbins
(O'Reilly, 1997; http://www.ora.com), "UNIX Text Processing," by Dale Dougherty and Tim O'Reilly
(Hayden Books, 1987) or the tutorials by Mike Arst distributed in U-SEDIT2.ZIP (many sites). To
fully exploit the power of sed, one must understand "regular expressions." For this, see "Mastering
Regular Expressions" by Jeffrey Friedl (O'Reilly, 1997). The manual ("man") pages on Unix systems
may be helpful (try "man sed", "man regex", or the subsection on regular expressions in "man ed"),
but man pages are notoriously difficult. They are not written to teach sed use or regexs to
first-time users, but as a reference text for those already acquainted with these tools.
QUOTING SYNTAX: The preceding examples use single quotes ('...') instead of double quotes ("...") to
enclose editing commands, since sed is typically used on a Unix platform. Single quotes prevent the
Unix shell from intrepreting the dollar sign ($) and backquotes (...), which are expanded by the
shell if they are enclosed in double quotes. Users of the "csh" shell and derivatives will also need
to quote the exclamation mark (!) with the backslash (i.e., !) to properly run the examples listed
above, even within single quotes. Versions of sed written for DOS invariably require double quotes
("...") instead of single quotes to enclose editing commands.
USE OF '\t' IN SED SCRIPTS: For clarity in documentation, we have used the expression '\t' to
indicate a tab character (0x09) in the scripts. However, most versions of sed do not recognize the
'\t' abbreviation, so when typing these scripts from the command line, you should press the TAB key
instead. '\t' is supported as a regular expression metacharacter in awk, perl, and HHsed, sedmod,
and GNU sed v3.02.80.
VERSIONS OF SED: Versions of sed do differ, and some slight syntax variation is to be expected. In
particular, most do not support the use of labels (:name) or branch instructions (b,t) within
editing commands, except at the end of those commands. We have used the syntax which will be
portable to most users of sed, even though the popular GNU versions of sed allow a more succinct
syntax. When the reader sees a fairly long command such as this:
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
It is heartening to know that GNU sed will let you reduce it to:
sed '/AAA/b;/BBB/b;/CCC/b;d' # or even sed '/AAA\|BBB\|CCC/b;d'
In addition, remember that while many versions of sed accept a command like "/one/ s/RE1/RE2/", some
do NOT allow "/one/! s/RE1/RE2/", which contains space before the 's'. Omit the space when typing
the command.
OPTIMIZING FOR SPEED: If execution speed needs to be increased (due to large input files or slow
processors or hard disks), substitution will be executed more quickly if the "find" expression is
specified before giving the "s/.../.../" instruction. Thus:
sed 's/foo/bar/g' filename # standard replace command sed '/foo/ s/foo/bar/g' filename # executes more quickly sed '/foo/ s//bar/g' filename # shorthand sed syntax
On line selection or deletion in which you only need to output lines from the first part of the
file, a "quit" command (q) in the script will drastically reduce processing time for large files.
Thus:
sed -n '45,50p' filename # print line nos. 45-50 of a file sed -n '51q;45,50p' filename # same, but executes much faster
If you have any additional scripts to contribute or if you find errors in this document, please send
e-mail to the compiler. Indicate the version of sed you used, the operating system it was compiled
for, and the nature of the problem. To qualify as a one-liner, the command line must be 65
characters or less. Various scripts in this file have been written or contributed by:
Thanks
Special thanks to the authors of sed1line and
awk1line for providing such awesome documents.
