In reading the CMUCL source code for stream I found this simple
implementation for peeking...
peek-type eof-value char
(read-char stream eof-errorp eof-value)
(unread-char char stream))
...and instantly realised that peeking a character invalidates the use of
unread character. As my code contains this algorithm in one place...
;;use up #\.
(read-char stream t)
(let ((nextchar (peek-char nil stream)))
(t ;;replace #\. if peeked char was not a delimiter
(unread-char firstchar stream)))))
...I was essentially using unread-char twice in a row.
This implementation of PEEK-CHAR is permitted by the Hyperspec: "The
consequences of invoking unread-char on any character preceding that which
is returned by peek-char (including those passed over by peek-char that
has a non-nil peek-type) are unspecified. In particular, the consequences
of invoking unread-char after peek-char are unspecified."
This means peek-char may be implemented as syntactic sugar that merely
obscures the dangerous use of unread-char. My implementation has to be
rewritten and it's going to be messy as the only way to know if a . makes
up the start of a symbol or number is by reading the next character. But
after the next character has been read it's no longer possible to replace
the . before calling read. Which means (a wrapper for) read now has to be
implemented as well.
I can now appreciate the design decisions behind CMUCL's reader