[sicsh-develop] Additional evaluation layer

[Gary V. V. <ga...@or...>]

I have been thinking (dangerous, I know) and it occurs to me that the
syntax mechanism could be extended to recognise tokens and mark them
in the Tokens* generated by tokenize() -- either in addition to, or
probably as a replacement for the string that is generated currently.

A third type of module would then be possible to implement higher
level semantic extensions.  For example a loadable module could
install a syntax handler that recognises "if", "then", "else" and
"fi", and registers (enumerated) tokens `SYNTAX_IF', `SYNTAX_THEN',
`SYNTAX_ELSE' and `SYNTAX_FI'.  The tokenize() function would be
changed to recognise these tokens while it is building the Tokens* and
insert the enumerated values rather than the raw strings.

Finally, the new type of module entry point (in addition to syntax and
builtin entry points) would determine allowable enumerated token
orderings.  In this case, it would check that the above enumerated
tokens were well formed and correctly nested -- generating a SIC_ERROR
for semantic errors, SIC_INCOMPLETE if more lookahead is required, or
SIC_OKAY if the evaluator should search for a suitable builtin handler.

The builtin initialiser would also need extending to allow builtins to
be registered against tokens rather than just single characters.

Thoughts?

Gary.
-- 
  ___              _   ___   __              _ mailto:ga...@or...
 / __|__ _ _ ___ _| | / / | / /_ _ _  _ __ _| |_  __ _ ___       ga...@gn... 
| (_ / _` | '_|// / |/ /| |/ / _` | || / _` | ' \/ _` | _ \      
 \___\__,_|_|\_, /|___(_)___/\__,_|\_,_\__, |_||_\__,_|//_/
home page:  /___/                      /___/                  gpg public key:
http://www.oranda.demon.co.uk           http://www.oranda.demon.co.uk/key.asc