drm_tools 1.1.34 OCT 14, 2020
Author: David Mathog, mathog@caltech.edu
This is a collection of small tools that are
sometimes helpful in specific instances. Each
is provided as source code with a man page and an
html version of the man page. Included programs are:
accudate: An extended version of the unix "date" program.
If supported by the compiler and OS it has subsecond
accuracy and can easily be used to measure delta
times in scripts.
binformat: A utility for formatting complex binary data into text.
Convert a mixture of data types within a
single fixed length record to text, optionally inserting
delimiters and labels, correcting for byte order,
and/or inserting a record number count.
binload: Load data into a binary file using simple commands from
the input.
binorder: Sort, merge, search, retrieve or generate test data
consisting of fixed size binary records.
binreplace: Command line utility that may be used in scripts to find
or find/replace in binary files. Initial release supports
only fixed length binary sequences for FIND and REPLACE
patterns.
binsplit: Split test data consisting of fixed size binary records
into one or more output streams. Split can be either
by record order or by key order, where a key is a defined
range of bytes in each record.
chardiff: Find changes between two files at the character level.
Unlike diff, it lists just the characters that differ, so
if the 40,000th character is different only that one character
will be shown, not the entire line.
columnadd: Add columns of integers, decimals, and/or times.
datasniffer A utility for formatting binary data dumps. Handy
for reverse engineering binary data files if the
contents are either already partially known or are
expected to be intelligible upon careful examination.
dmath Double precision interactive command line math calculator.
(Same math functions as in extract.)
extract: A utility that is designed to extract and
emit data from text files based on character
or token position. In many instances it can
replace tr, sed, or awk.
execinput: A utility that reads from stdin and executes
each line as a command in a subprocess.
indexed_text:
A utility for rapid retrieval of text by line
numbers, in any order, from a text file.
manServer_107.pl
A (modified) utility from elsewhere used to generate html
versions of manpages with:
ls -1 *.1 \
| extract -fmt './manServer_107.pl [1,] > /tmp/pages/[1,-2]html' \
| execinput
mdump: A utility for formatting binary data. Handy
for decoding foreign binary data files.
msgqueue: A command line interface to create message queues
and send/receive messages.
mbin: Multiple buffer in and out. Used for buffering a lot
mbout: of data between a slow device and a fast device.
Mostly for buffering streaming tape drives for use
with slower network connections, so that streaming
is maintained as much as possible to minimize wear
on the tape device.
pockmark: Corrupt data streams - useful for testing error correction
and data recovery.
tarsieve: Filter, list, or split a tar file.
Other files:
CMakeLists.txt Cmake build.
dna_ctable.txt Example data file to use with -table option in binload or binformat.
math_funcs.c Common math functions for extract and dmath
math_funcs.h
sol8.h A few bits and pieces to work around missing things
in the Solaris 8 system libraries. (Note, there
still minor incompatibilities, some NAN on other
systems are -INF here.)
test_all.sh Script for running all of the tests.
test_binformat.sh Script for testing binformat.
test_binload.sh Script for testing binload.
test_binorder.sh Script for testing binorder.
test_binsplit.sh Script for testing binformat.
test_chardiff.sh Script for testing chardiff.
test_dmath.sh Script for testing dmath
test_extract_increment.sh Script for comparing incremental versions of extract.
DEPRECATED.
test_extract_features.sh Script for testing new features in 1.1.0 and up. (Also
contains many scripting examples.)
Note - slight numeric differences in gcc trig libraries
may cause test_extract_features.sh to fail test 33 on
some platforms. (16331239353195370.000000 vs.
16331778728383844.000000)
test_extract_features2.sh Script for testing extract features. Replaces
test_extract_increment.sh (deprecated): does
not require an older extract binary for comparison.
test_indexed_text.sh Script for testing indexed_text.
test_mdump.sh Script for testing mdump.
test_tarsieve.sh Script for testing tarsieve.
windows-mmap.c Windows implementation of mmap functions from
windows-mmap.c University of Stuttgart.
Build instructions:
Option 1:
Use cmake. Inspect the CMakeLists.txt and verify that it will install
to the directory you want. Then do:
mkdir build
cd build
cmake ..
make
(in Mingw[32|64] on Windows use instead:
cmake -G "MSYS Makefiles" -DMSYS_ROOT=/ ..
)
(The path relative to whatever "/" is may be specified on the command
line instead of modifying CMakeLists.txt. On linux/unix this just means
removing the first character, but on Windows it might be "C:\blah\blah"
which is omitted. Linux example (to /usr/common/foo, note the missing
first slash!)
cmake -DCMAKE_RELATIVE_INSTALL_PREFIX:PATH=usr/common/foo ..
)
Option 2:
Edit the build_(OSNAME) scripts as desired and run them. Supported targets
are linux, solaris, and mingw.
The test scripts must use gnu diff which is not the default on Solaris -
edit that path before running. There are up to 2 failures in test_dmath.sh
and test_extract_features.sh which are due to slight
differences in the values calculated by the math libraries on different platforms
and clib versions. There is also a 3rd failure in test_extract_features.sh
unless test_dmath is compiled with -DUSETOK. That is not the default as that
is just a testing mode for some code only used in extract.
Testing:
Assuming target directory is /tmp/testinstall.
1. Build and install with USETOK defined (else a large block of dmath fails)
2. cp test_*sh /tmp/testinstall/bin
3. cd /tmp/testinstall/bin
4. ./test_all.sh 2>&1 | tee test_all.log
Rebuild the dmath program without USETOK, which is only needed for testing.
Some of the scripts use preexisting (/usr/local/bin or wherever) binaries
to test the current batch. If they have not been previously installed add
"." to PATH so that the current versions are used instead.
Dependencies:
The required libraries are most likely already installed in a Linux environment,
but some pieces may be missing on Solaris. These are:
Install with (Solaris)
libpcre pkg-get -i pcre
libsigsegv pkg-get -i libsigsegv
libintl pkg-get -i libintl
Windows notes:
The mingw programs will run in a DOS (cmd.exe) window. All are linked to a Microsoft
C RTL on the host system and consequently any \n in an output line will
actually emit \r\n, which is the default end of line marker on Windows.
The extract -b switch will leave \n as \n, but only if the binary was built with
a C99 compatible compiler or -DMSCRTL was used (for instance, on mingw).
As of version 1.1.6 all of these programs are covered by the
GNU General Public License, Version 2. A copy of the license
terms is found in the LICENSE file.
Program versions in this distribution:
accudate.c 1.0.12 08-JAN-2020
binformat.c 0.0.11 14-OCT-2020
binload.c 0.0.9 14-OCT-2020
binorder.c 0.0.22 02-JAN-2020
binreplace.c 1.0.5 13-JAN-2018
binsplit.c 0.0.4 02-MAY-2019
chardiff.c 0.0.6 08-JAN-2020
columnadd.c 1.0.4 08-JAN-2020
datasniffer.c 1.0.5 08-JAN-2020
execinput.c 1.0.3 10-MAY-2019
extract.c 1.1.23 20-MAY-2020
indexed_text.c 1.0.6 29-APR-2020
math_funcs.c 0.0.4 29-APR-2020
mbin.c 1.0.5 29-APR-2020
mbout.c 1.0.4 02-MAY-2019
mdump.c 1.0.8 08-JAN-2020
msgqueue.c 1.0.2 23-OCT-2006
pockmark.c 1.0.6 02-JAN-2020
tarsieve.c 1.0.3 07-JAN-2020
test_math.c 0.0.12 29-APR-2020
manServer_107.pl 1.07* 20-MAY-2020
Distribution History
1.1.34 Added bit vector type (LSB and MSB), and bit complement to binload.
Added bit vector type (LSB and MSB), bit complement, and
-no_rec_EOL, to binformat.
Added example file dna_ctable.txt (notes for usage at bottom).
Regenerate all html files.
1.1.33 Modified CMakeLists.txt to respect CMAKE_RELATIVE_INSTALL_PREFIX
if defined on command line. (NOT CMAKE_INSTALL_PREFIX!)
Fixed minor compiler warnings on newest gcc compilers in:
math_funcs.c, test_math.c, mbin.c, text_indexed.c
Cleaned up syntax for test_dmath.c and changed documentation
regarding the use of a variable like "A" on the right side
of a math operation.
Cleaned up some syntax warnings in match_funcs.c and caught a buffer
overflow on some nonsense math strings.
Fixed off by 1 values in HREF in table of contents for manServer_107.pl.
1.1.32 Fixed binorder, pockmark portability issue - rand() 32 bit on linux
but 16bit on Windows. Replace with Mersenne twister from
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/CODES/mt19937ar.c
Fixed binformat build problem on Windows.
Changed tan(rd2(90)) result to match gcc/linux 64 bit (was Solaris.)
Updated test for tarsieve to take into account "new" information headers.
Rerrange code in tarsieve.
Fixed single Walloc-size-larger-than warning warning in extract.c
Removed mingw specific test_*sh scripts, general ones now work there
(other than some MS specific float formatting quirks.)
Fixed -O3 warnings in columnadd, tarsieve, chardiff, accudate.
Fixed fopen() with "rb" for mdump.c, datasniffer.c, for mingw64.
Fixed compatibility issue for some mmap() calls on Windows.
Added 'string' mode to binload.
Added -pre,-post to binreplace
Added test scripts for binreplace, indexed_text.
Include manServer_107.pl, used for making html versions of manpages.
1.1.31 Fixed warning in binformat.
Fixed execinput build on windows - no fork()/wait() on that OS,
now supports threads. Added -env,-unenv, -quote, -exectype,
and -execwith.
Added \. for extract so that \DIGITS -> character can be followed
by a digit.
Fixed CMakeLists.txt so that it works again on windows
Fixed warning related to TYPE_INSERT in binformat.
Modified build* scripts to all use std=c99.
1.1.30 Added -bs switch to tarsieve.
Changed 64 bit integer print statements for better portability.
Better portability for 64 bit fseeko/ftello/mmap.
1.1.29 Fixed bug in tok() function.
Fixed formatting errors in extract man and html pages.
Updated test scripts.
Made execinput multi-threaded.
1.1.28 Added -dm, -dh, -dd, and -tref to accudate.
Added -iffmt, -iftermfmt, -ifaccept, -ifreject
and operators scmn, scmd, sccn, sccd,
lgct, lgcf.
Added operators scmn, scmd,
sccn, sccd, lgct, lgcf to test_math.
Reduced the number of differences between the linux and mingw
variants of the test scripts.
1.1.27 Fixed a bug in extract.
Fixed an unitialized field in extract.
Fixed output and record position bugs in binorder.
Fixed obscure errno issue in binorder, binload, binformat.
Added -sat, {, and } to binload.
Added CMakeLists.txt for cmake build.
Added test_mdump.sh, test_extract_features2.sh (replaces
test_extract_increment.sh)
1.1.26 Moved common code from test_math.c and extract.c into math_funcs.
Tweaks to a few programs.
1.1.25 Added binload, binsplit, binorder, binformat, indexed_text and
corresponding test scripts.
pockmark added minrun, mingap, changed start state.
1.1.24 Added binmerge. Fixed new compiler warnings: test_math.c, extract.c
1.1.23 Fixed stupid bug in pockmark.
1.1.22 Modified pockmark, updated its documentation.
1.1.21 Added chardiff.
1.1.20 Fixed a bug in extract that could result in a crash. Updated
its test scripts and documentation. Slight modification of
behavior of extract -template.
1.1.19 Fixed a bug in extract that could result in a crash.
1.1.18 Added tarsieve. Updated link to drm_tools download site in man
and html pages which had not already been corrected.
1.1.17 Fixed a regression in extract which was affecting -fmt with consecutive
[] clauses. Modified documentation and test scripts.
Modified documentation following suggestions by "Tanya".
1.1.16 Added bitwise operators and integer formats to dmath and extract.
Fixed some bugs in extract. Updated documentation for dmath and extract.
1.1.15.2 Fixed minor bug in extract.
1.1.15.1 Changed all programs to use -DMSCRTL, some were using -DMSCLIB for
the same thing.
1.1.15 Document some solaris dependencies.
Slight changes to sol8.h.
Fix compiler warnings on newer gcc.
Changed build scripts, moved -l(lib) after name.c.
1.1.14 (Re)ported to mingw. All run except msgqueue, which is too
Unix specific. Fixed a couple of minor bugs in various programs.
Added binary output mode for extract (for systems other than Linux/Unix.)
1.1.13 fixed a couple of bugs in new version of extract.
1.1.12 major changes to extract (uses and requires PCRE,
added scripting and math capabilities). Added dmath.
1.1.11 added binreplace, two minor buf fixes to pockmark, slight change
in build scripts to suppress a warning for mbout.
1.1.10 added pockmark
1.1.9 bug fixes in extract.
1.1.8 bug fixes in extract, datasniffer, mbin, and mbout.
Added sorts for multitoken fields to extract.
1.1.7 modified extract to support adding delimiters between single
letter fields (expanded existing -bs function).
1.1.6 Added mbin/mbout. Changed the license from proprietary to GPL 2
on all programs. Adjusted version numbers and documentation to
reflect the new license and update copyrights.
1.1.5 Extract version 1.0.33, fixes one minor bug and one potential bug.
1.1.4.1 Removed unneeded unistd.h include in some programs. No functional
changes.
1.1.4 Added columnadd. Slightly modified the other man and html pages.
1.1.3 Modified extract slightly to simplify iftermeol
1.1.1 Modified extract slightly so that it could also emit line lengths.
1.1.0 Added datasniffer. Removed include of unistd.h from all but
msgqueue.c.
1.0.0 Initial release
Download Latest Version
drm_tools-1.1.34.tar.gz (567.9 kB)
