Development

doall POSIX shell script

This document is intended for developers and hackers who wish to modify or extend the doall script. User information is in the doall.1.man page file, derived from README.md markdown.

Execution Environment

doall should run properly under any POSIX shell interpreter such as dash, busybox, or bash. It will perform best under dash.

In "pretend" mode, it creates a file of the potential commands, as there may be many matching files. For this, it uses common GNU utilities, rather than just pure POSIX shell scripting. It calls the GNU 'mktemp' program and whatever program is specified by the PAGER environment variable, typically 'less', or 'cat' if PAGER is unset. Unless the "-c never" option is used, it calls 'tput' to create terminal ANSI colour codes.

In non-pretend modes, doall is pure shell script with the only external program being the one named in the command specified by the user.

doall originally had an "ask" mode, but that was messy, as it ran the code once to display a list of commands (using the pretend mode functions) and then again to execute them. This is a race condition waiting to happen, so it was removed. Instead there's "interactive" mode, which is tedious but safe.

Special Characters in Filenames and Patterns

This is the big challenge in any POSIX shell script, since so many shell operations result in strings being parsed according to delimiters in IFS, and the various expansions that will, without care, modify embedded special characters. The only safe place to store an arbitrary number of strings containing special characters in POSIX scripts is the parameter list "$@". (Bash scripts have it much easier with arrays; POSIX allows only the one array "$@".) Accordingly the various functions within doall act as a sort of pipe, taking the parameters one at a time off "$@" and appending a derived value to the end of "$@".

The end of the the original parameter list can be identified either by count, or by including a unique value that cannot naturally occur. Thus when processing the tokens in the original command list passed in to doall, a token cannot be null, so a null string is used. Later, when processing the list of substrings forming the wildcard match, a bare asterisk is impossible, and used as the delimiter.

Path Components and Hidden Files

POSIX specifies that globbing wildcards will not match '/' path delimiters, nor the leading '.' in hidden files. That means that doall needn't worry about either;
if a path matches a pattern, the pattern must explicitly contain the '/'s and '.'s, and as explained below, doall will explicitly match those, and therefore the wildcard matches that it identifies cannot contain '/' or leading '.'.

Evaluating Tilde Expansions

Tilde expansions, such as '~user/Desktop/*' are a particular problem. The need to be expanded by the shell, which means using 'eval', which is a security minefield. To prevent security holes, doall rejects embedded ';'s in separators it needs to expand, and it performs the expansion in an assignment, thus:
eval "expanded=$separator"
Similarly, when it iterates over the files matching the pattern, doall performs the expansion within a command, thus
eval "for name in $pathname; do ... done"

Security and expansions

It turns out that dodgy filenames such as "$(dangerouscommand)" are harmless, as the quoting process prevents them being executed.

Dodgy patterns such as
doall 'ls $(dangerouscommand)'
get no more security than shell provides against someone typing:
ls $(dangerouscommand)
which is, of course, none; if you insist on both pointing a gun at your feet and pulling the trigger, you get what you deserve.

Wildcards

doall only allows '*' and '?' in wildcard patterns. There's no theoretical problem with handling '[...]' patterns either, but parsing them is messy and would significantly increase the amount of code in the parsing function for a relatively small increase in function. As more code probably means more bugs, doall doesn't do it. If you want that sort of function, you can do that, and more, with some combination of grep, find and xargs.

Identifying Matching Substrings in Wildcard Patterns

At first sight, this would be easier using grep or bash which allow matching groups in regular expression matches. But it turns out to be fairly easy in POSIX script language.
Consider a string S such as 'ab-cdefg' being matched with a pattern P such as '-???'
The important point to notice is that if P can be split as '*' followed by P1 '-??*?',
then if P matches S, when we remove the tail of S matching P1, call it S1, '-cdefg',
what's left, 'ab', must match the first '*'.
Similarly, we can split off the leading '-' from P1 and S1, and S2, 'cdefg' matches P2 '??*?'.
Now we can handle the leading '??' from P2 which matches 'cd', leaving S3 'efg' and P3 '*?'.
Since they match, we can split the leading '*' from P3, leaving P4, '?', and if we remove the tail of S3 that matches, 'g', the '*' must match the remainder 'ef'.

The parsing is not, in general, unique - consider 'abc-def-ghi' and a pattern '-'.
That could be parsed 'abc-def' and 'ghi' or 'abc' and 'def-ghi'. C'est la vie.
doall parses from the right, so its parse would be the first of those choices.

Pattern Expansion

The trick here is to replace a token containing wildcards with one containing parameter substitutions, and evaluate them in an assignment within a function
which has a parameter list containing the matching strings.
Thus a token such as "*-." becomes "$1-$2.$3", and if the matches are 'todo', 'list' and 'txt',
the token evaluates to 'todo-list.txt'.
Again this requires 'eval' in an assignment
eval "parm=$item"
and since "$item" was prevented from containing ';'s or spaces, this assignment is safe.

doallutils.sh

This is used to declutter the doall and other of the author's scripts, with functions such as Die, Tempfile, SetColours and Ask. It contains several other functions not used in doall, such as Cmd, which colourises stderr output, and support for copying output to syslog.