Based on this discussion, I checked in some changes to semantic.
Specifically, I used `define-override' to create
`semantic-bovinate-region' which is loosely based on
`semantic-edits-bovinate-region'. It does not take a table, lex
stream nor anything else. Just start, end, and a parser symbol. I
found that it didn't need a table.
Anyway, the overall changes are quite small, and mostly consists of
deleting a lot of stuff from `semantic-bovinate-toplevel'.
I also change semantic-edit to use the new bovinate region function,
and defined a bovinate-region function for semantic-bovine.el.
David, you will need to create a `wisent-bovinate-region' or
similarly named function which your wisent setup code can install via
`semantic-install-function-overrides'. Then again, perhaps the
default will work for you.
I'll leave additional items from below for a later check in. Note
comments regarding bugs with unmatched syntax with the new code.
Unmatched syntax may need similar treatment to returned tags.
>>> David Ponce <david@...> seems to think that:
> > Currently the function `semantic-bovinate-toplevel' performs the
> > lexical step, followed by calling the active parser.
> > I'd like to change this so that it only calls the active parser, and
> > it is the responsibility of those parsers to perform their own
> > lexical analysis.
>IMO, this is the good design to adopt. I completely agree with you
>that it is the parser responsibility to perform their own lexical
>analysis. Does it mean that `semantic-bovinate-nonterminal'
>(STREAM TABLE &optional NONTERMINAL)
>will be changed to:
>(START END TABLE &optional NONTERMINAL)
> > This would let wisent call a different lexical routine, instead of
> > having to post-process the existing lexical output. (Some
> > discussion on that occurred already.)
>In fact, with the new semantic-lex toolkit, the Wisent's post-lexer is
>very simple. It just pops the next token from the lexical output stream,
>previously built by the actual lexer, and changes the semantic-lex
>token format to Wisent format (see `wisent-lex' in wisent-flex.el).
> From a performance point of view, it is a good implementation. The
>more consuming step (building the lexical output stream) is done is
>one pass avoiding the overhead of calling the lexer for each token. I
>verified with ELP that the global elapsed time to call the post-lexer
>is negligible, even when there are a lot of tokens to be read. In
>fact this implementation is the one used for buffered input stream ;-)
>So, I don't think I would eliminate the Wisent's post-lexer. But just
>move the true lexer in `wisent-bovinate-nonterminal'.
>Of course other parsers are free to adopt different implementations to
>perform fast lexical analysis. This is clearly an advantage of the new
>design you propose for `semantic-bovinate-toplevel'!
> > Another benefit would be that we could eliminate
> > semantic-bovinate-toplevel-override as an option, and such parsers
> > could use the new mechanism that wisent takes advantage of.
>More generally, if the lexical analysis step is removed from
>`semantic-bovinate-toplevel', `semantic-bovinate-parser' becomes a
>more powerful alternative than `semantic-bovinate-toplevel-override'.
>So the later can be eliminated.
>As I am writing that, I just realize that the incremental parser
>quasi already obeys the new architecture ;-)
> > Thoughts?
>This is a great idea!
[ ... ]