Re: [sicsh-develop] Additional evaluation layer

[Ezra P. <epe...@zi...>]

>I have been thinking (dangerous, I know) and it occurs to me that the
>syntax mechanism could be extended to recognise tokens and mark them
>in the Tokens* generated by tokenize() -- either in addition to, or
>probably as a replacement for the string that is generated currently.



>A third type of module would then be possible to implement higher
>level semantic extensions.  For example a loadable module could
>install a syntax handler that recognises "if", "then", "else" and
>"fi", and registers (enumerated) tokens `SYNTAX_IF', `SYNTAX_THEN',
>`SYNTAX_ELSE' and `SYNTAX_FI'.  The tokenize() function would be
>changed to recognise these tokens while it is building the Tokens* and
>insert the enumerated values rather than the raw strings.

I like the idea. One of the concerns I had in writing the equals
handler is understanding the context that the equals sign is to be
used.


Clearly "if a=1 then" is very different from "a=1" vs "typo a=1". The
first a conditional, the second an assignment, the third a syntax error.

It would be nice if the recognition could happen durring the
syntax_handler phase - before generation of Tokens *. Perhaps a syntax
handler can indicate that it is okay to create tokens for everything
to where I am. This would allow differentiation of the above test cases.


>Finally, the new type of module entry point (in addition to syntax and
>builtin entry points) would determine allowable enumerated token
>orderings.  In this case, it would check that the above enumerated
>tokens were well formed and correctly nested -- generating a SIC_ERROR
>for semantic errors, SIC_INCOMPLETE if more lookahead is required, or
>SIC_OKAY if the evaluator should search for a suitable builtin handler.
I like the idea, but I am not sure how you plan to describe orderings.
You need to be able to describe optional  orderings, vs required ones.

IF ... THEN ... ENDIF
IF ....THEN ... ELSE ... ENDIF

are straight forward with ELSE being optional - but would you ever
need syntax that is more complex. (we can say no and be done with it)

A complex example might be:
a ... b ... c
a ... c ... b ... d

Where d or c comes after b depending on if b comes after c or a. (a 
bit contrived - but makes a description more difficult"


>The builtin initialiser would also need extending to allow builtins to
>be registered against tokens rather than just single characters.
Logically...


I think it is a good idea that deserves merit. The big win can be for
the syntax handlers knowing the context. It also allows for real
structure.

Long term, we eventuall need a way to disable builtin syntax handlers
that have side affects.

Consider:
if a=1 then a=2 else b=3

currently in parsing the line a and b will be assigned. We need a way
to say - recognize, but don't execute. The same with $(eval).

   Ezra