Re[2]: [CEDET-devel] semantic-bovinate-toplevel & lexical analysis

[Eric M. L. <er...@si...>]

Hi,

  Based on this discussion, I checked in some changes to semantic.
Specifically, I used `define-override' to create
`semantic-bovinate-region' which is loosely based on
`semantic-edits-bovinate-region'.  It does not take a table, lex
stream nor anything else.  Just start, end, and a parser symbol.  I
found that it didn't need a table.

  Anyway, the overall changes are quite small, and mostly consists of
deleting a lot of stuff from `semantic-bovinate-toplevel'.

  I also change semantic-edit to use the new bovinate region function,
and defined a bovinate-region function for semantic-bovine.el.

  David, you will need to create a `wisent-bovinate-region' or
similarly named function which your wisent setup code can install via
`semantic-install-function-overrides'.  Then again, perhaps the
default will work for you.

  I'll leave additional items from below for a later check in.  Note
comments regarding bugs with unmatched syntax with the new code.
Unmatched syntax may need similar treatment to returned tags.

Have fun
Eric

>>> David Ponce <da...@dp...> seems to think that:
>Hi Eric,
>
> > Currently the function `semantic-bovinate-toplevel' performs the
> > lexical step, followed by calling the active parser.
> >
> > I'd like to change this so that it only calls the active parser, and
> > it is the responsibility of those parsers to perform their own
> > lexical analysis.
>
>IMO, this is the good design to adopt.  I completely agree with you
>that it is the parser responsibility to perform their own lexical
>analysis.  Does it mean that `semantic-bovinate-nonterminal'
>arguments:
>
>(STREAM TABLE &optional NONTERMINAL)
>
>will be changed to:
>
>(START END TABLE &optional NONTERMINAL)
>
> > This would let wisent call a different lexical routine, instead of
> > having to post-process the existing lexical output.  (Some
> > discussion on that occurred already.)
>
>In fact, with the new semantic-lex toolkit, the Wisent's post-lexer is
>very simple.  It just pops the next token from the lexical output stream,
>previously built by the actual lexer, and changes the semantic-lex
>token format to Wisent format (see `wisent-lex' in wisent-flex.el).
>
> From a performance point of view, it is a good implementation.  The
>more consuming step (building the lexical output stream) is done is
>one pass avoiding the overhead of calling the lexer for each token.  I
>verified with ELP that the global elapsed time to call the post-lexer
>is negligible, even when there are a lot of tokens to be read.  In
>fact this implementation is the one used for buffered input stream ;-)
>
>So, I don't think I would eliminate the Wisent's post-lexer.  But just
>move the true lexer in `wisent-bovinate-nonterminal'.
>
>Of course other parsers are free to adopt different implementations to
>perform fast lexical analysis.  This is clearly an advantage of the new
>design you propose for `semantic-bovinate-toplevel'!
>
> > Another benefit would be that we could eliminate
> > semantic-bovinate-toplevel-override as an option, and such parsers
> > could use the new mechanism that wisent takes advantage of.
>
>More generally, if the lexical analysis step is removed from
>`semantic-bovinate-toplevel', `semantic-bovinate-parser' becomes a
>more powerful alternative than `semantic-bovinate-toplevel-override'.
>So the later can be eliminated.
>
>As I am writing that, I just realize that the incremental parser
>quasi already obeys the new architecture ;-)
>
> > Thoughts?
>
>This is a great idea!
  [ ... ]