CEDET

Hi Eric,

 >   I added a new `framework' file into semantic: semantic-fw.el.
 > This file contains the overlay compatibility code, and other bits
 > that are used in several files that isn't really specific to
 > parsing.  Eventually, I hope to remove some circular dependencies in
 > the byte compilation phase.

This looks very good!

 >   Anyway, one of the things that moved into the framework file is
 > the code for override methods.  I'd like to see the lex and parser
 > variables replaced by override methods instead.  The only bit that
 > worries me is that if we turn the lexer into such a method, it will
 > add a hash operation which might slow things down.  On the other
 > hand, it will specify what is really wanted which is a method for
 > lexing or parsing.

I agree.  Maybe we could try a first implementation and if it really
slows things down too much, keep the current mechanism?

 >   In preparing for this (eventual) change, there is a new
 > `semantic-defoverload'.  I'm not sure I like the name though.
 > Anyway, I used this macro for all the token->text functions in
 > semantic-util.  Creating a new override method looks like this:
[...]

I definitively like this idea to have macros to define things!  For
now we have `define-lex', `define-lex-analyzer' and so on, and
`define-wisent-lexer'.  To be consistent, why not just
`define-overload'?

 >   The macro not only fills in the boiler plate, but will leave a tag
 > behind.  It will then be possible to write a function or two that
 > lets you list all functions that can be overridden.  This in
 > combination with some code to browse those functions that are
 > overriding may help new language authors figure out what to do.

Nice!

 >   I'd like to have a second macro for overriding a function.  I'm
 > not really sure how that would work, in that I'd like the language
 > setup functions to say "set-up-overrides" with no arguments, and
 > have it figure it out on it's own based on something from the macro.

I don't understand well your idea here, could you give an example?

 >   Lastly, I'd like these override tables set up once per mode,
 > instead of once per file.  I thought it would be needed to override
 > some function in one buffer only, but I haven't come up with a use
 > for this.
[...]

Yes this would be definitively better to setup overrides per mode.
Maybe we could use a global override table and prefix symbol names
with mode symbols, both when installing (explicit) and fetching
(implicit).  Something like this:

(defun semantic-install-function-overrides (modes overrides
                                                  &optional transient)
 ...)

where modes will be a list of `major-mode' symbols (to take care of
derived modes ;)  Of course installation will be done once instead
of for each new buffer!

Calling `semantic-fetch-overload' will not change, but it will prefix
the symbol argument with current `major-mode'.

 >   Well, I'm off for a week and won't be checking email, so I'll
 > catch up with everyone later.

Vacation?  If so, enjoy!

David

>>> David Ponce <david@...> seems to think that:
  [ ... ]
> >   In preparing for this (eventual) change, there is a new
> > `semantic-defoverload'.  I'm not sure I like the name though.
> > Anyway, I used this macro for all the token->text functions in
> > semantic-util.  Creating a new override method looks like this:
>[...]
>
>I definitively like this idea to have macros to define things!  For
>now we have `define-lex', `define-lex-analyzer' and so on, and
>`define-wisent-lexer'.  To be consistent, why not just
>`define-overload'?

That makes sense.

> >   The macro not only fills in the boiler plate, but will leave a tag
> > behind.  It will then be possible to write a function or two that
> > lets you list all functions that can be overridden.  This in
> > combination with some code to browse those functions that are
> > overriding may help new language authors figure out what to do.
>
>Nice!
>
> >   I'd like to have a second macro for overriding a function.  I'm
> > not really sure how that would work, in that I'd like the language
> > setup functions to say "set-up-overrides" with no arguments, and
> > have it figure it out on it's own based on something from the macro.
>
>I don't understand well your idea here, could you give an example?

For example, someone might write:

(define-overload short-name-of-override (args)
   "Doc about why specific"
   spiffy code)

and that's it.  It has overridden `short-name-of-override' and
installed it in the right override table.

As you say, perhaps an extra argument of the major mode is needed.

> >   Lastly, I'd like these override tables set up once per mode,
> > instead of once per file.  I thought it would be needed to override
> > some function in one buffer only, but I haven't come up with a use
> > for this.
>[...]
>
>Yes this would be definitively better to setup overrides per mode.
>Maybe we could use a global override table and prefix symbol names
>with mode symbols, both when installing (explicit) and fetching
>(implicit).  Something like this:
>
>(defun semantic-install-function-overrides (modes overrides
>                                                  &optional transient)
> ...)
>
>where modes will be a list of `major-mode' symbols (to take care of
>derived modes ;)  Of course installation will be done once instead
>of for each new buffer!
>
>Calling `semantic-fetch-overload' will not change, but it will prefix
>the symbol argument with current `major-mode'.

This is similar to what I was thinking.  I was contemplating hanging
mode-specific override tables of the major mode symbol's plist.  I'm
not sure which is faster to use.

What would be nice is not to have to pass the major mode down for
every call to an override method.  Perhaps there would be an active
override table that could be locally bound when operating on
buffer-less token lists.

> >   Well, I'm off for a week and won't be checking email, so I'll
> > catch up with everyone later.
>
>Vacation?  If so, enjoy!

Yes, a week away from everything.  It should be fun.  Packing is
taking a long time though.

Later.
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Eric,

Attached you will find a patch to semantic-fw.el, where I
re-implemented `semantic-install-function-overrides' and
`semantic-fetch-overload', and the macro `semantic-defoverload' as
`define-overload'.  There is also a patch to semantic-util.el to use
`define-overload', plus small fixes.  And another minor patch to
semantic.el to inline some functions  (see change log at end).

As you will see `semantic-install-function-overrides' and
`semantic-fetch-overload' are fully compatible with previous
versions.  But they can also work on mode basis if requested :-)

`semantic-install-function-overrides' accepts a new optional symbol
argument specifying a major mode where to statically install function
overrides.  By default it dynamically (via mode hook) installs the
overrides in `semantic-override-table'.  It is even possible to mix
mode and local installation!  When doing a buffer local installation,
the function checks if a table exists for the current `major-mode' (or
its parents) and, if so, installs the local overrides into a local
copy of that table (a sort of poor man inheritance mechanism ;-)

I provided two new functions: `semantic-current-overrides' and
`semantic-new-overrides', to respectively find and return the current
active overrides table, and to create a new one, maybe by doing a deep
copy of an existing table.

To improve performance when used on mode basis,
`semantic-fetch-overload' set `semantic-override-table' with the mode
override table it found when first called.  So next calls just fetch
values from the buffer local table.  The nice thing is that when
`semantic-override-table' was already setup before the first call, it
will be used like in previous version ;-)

Finally, I followed your idea and used a 'semantic-override-table
property of the major mode symbol to store the mode override table.  I
found it easy to implement, taking care of derived modes.

I did some tests and the new code seems to works very well with
existing language stuff.  The performance is not so different of what
I measured with the old version.  I also hacked semantic-java.el to
install function overrides in java-mode.  And had no problem with the
JDEE which locally installs its own version of 'prototype-nonterminal!

What's your opinion?  Can I check my changes in?

Thanks! Hope you had a nice week ;-)

David

----------------------------- Change Log -----------------------------

* semantic-fw.el:

(semantic-current-overrides, semantic-new-overrides): New functions.
(semantic-install-function-overrides): Use them.  New optional
argument MODE to install into.
(semantic-fetch-overload): Use `semantic-current-overrides'.
(semantic-defoverload): Renamed to...
(define-overload): New macro.

* semantic-util.el:

(semantic-name-nonterminal, semantic-abbreviate-nonterminal)
(semantic-summarize-nonterminal, semantic-prototype-nonterminal)
(semantic-concise-prototype-nonterminal)
(semantic-uml-abbreviate-nonterminal)
(semantic-uml-prototype-nonterminal)
(semantic-uml-concise-prototype-nonterminal)
(semantic-nonterminal-static): Use `define-overload' instead of
`semantic-defoverload'.

(semantic-find-dependency, semantic-find-nonterminal)
(semantic-find-documentation): Put the right property
'semantic-overload.

* semantic.el:

(semantic-bovinate-from-nonterminal)
(semantic-bovinate-from-nonterminal-full)
(semantic-bovinate-region-until-error): Inlined to avoid an
unnecessary extra call.

>>> David Ponce <david@...> seems to think that:
>Hi Eric,
>
>Attached you will find a patch to semantic-fw.el, where I
>re-implemented `semantic-install-function-overrides' and
>`semantic-fetch-overload', and the macro `semantic-defoverload' as
>`define-overload'.  There is also a patch to semantic-util.el to use
>`define-overload', plus small fixes.  And another minor patch to
>semantic.el to inline some functions  (see change log at end).
  [ ... ]
>
>What's your opinion?  Can I check my changes in?

This seems like a great idea.  Please check them in.

>Thanks! Hope you had a nice week ;-)
  [ ... ]

The weather was pleasant, and my wife and I got some free
babysitting.  Can't go wrong there.

I also re-wrote semantic-edit.el on my laptop.  The new incremental
parser has some nice new features, and helps prevent full-reparses
more often, including insertion and deletion of tokens.  Yay!  On the
bad side, I had to also build a couple new minor modes to replace the
old show-dirty minor mode, and the auto-parse mode needs some more
adjusting I think.

My next task is getting that code off my laptop as the CVS install on
it doesn't seem to work very well. :(

If the wisent parser was working with the old style rebovinate token,
then this new incremental parser should also work well with it.

Have fun
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Eric,

[...]
 > > What's your opinion?  Can I check my changes in?
 >
 > This seems like a great idea.  Please check them in.

Thanks! Done.

[...]
 > I also re-wrote semantic-edit.el on my laptop.  The new incremental
 > parser has some nice new features, and helps prevent full-reparses
 > more often, including insertion and deletion of tokens.  Yay!  On
 > the bad side, I had to also build a couple new minor modes to
 > replace the old show-dirty minor mode, and the auto-parse mode needs
 > some more adjusting I think.
[...]

You really did a great work!  I tried to use the new incremental
parser and found some bugs I fixed (checked-in).  I also added the new
minor modes to Senator menu.  Finally I used different hooks in
semantic-imenu that seems to work far better that the previous one
(no more messages "Imenu is out of date, try again. (internal bug)").

It remains a problem when `semantic-edits-bovinate-region' just
returns nil when the parser fails.  In that case
`semantic-edits-splice-remove' fails with the following error because
cache-list is nil:

  setcdr(nil nil)
  (if (eq first (car chil)) (progn (setq cacheend chil) (while ...
  ...) (setcar chil ...) (setcdr chil ...)) (while (and cachestart
  ...) (setq cachestart ...)) (setq cacheend cachestart) (while (and
  cacheend ...) (setq cacheend ...)) (setcdr cachestart (cdr
  cacheend))) (let* ((first ...) (last ...) (chil ...)  (cachestart
  cachelist) (cacheend nil) (tmp oldtokens)) (if (eq first ...) (progn
  ... ... ... ...) (while ... ...) (setq cacheend cachestart) (while
  ... ...) (setcdr cachestart ...)) (while oldtokens
  (semantic-deoverlay-token ...) (setq oldtokens ...)))
  semantic-edits-splice-remove((("v[]" variable "int" nil (...) nil
  (...) #<overlay from 2489 to 2547 in Essai.java>)) nil nil)

In fact I got this problem on a statement like this (Java):

  private int v[
  ] = {1,
       2,
       3,
       4},
    w = 0;

that results in two tokens, one for `v' from `private' to the comma,
and a second one for `w' from `w' to after the semicolon.  In such
case the region covered by each token can't be parsed.  Only the
whole statement is valid.  So when you change something in that
statement the incremental parser always fails.  Unfortunately I don't
found yet how to get both incremental parse and token differentiation
(useful for navigation, completion, etc.) work together.

Maybe we need a new property telling the true bounds of such tokens?

Have you any other idea?

Sincerely,
David

P.S.:  Here is the change log of what I checked-in:

-------------------------------------------------------------------
* semantic-edit.el:

(semantic-edits-bovinate-region): Added missing parameter
reparse-symbol.
(semantic-edits-incremental-parser): Set reparse-symbol.
(semantic-edits-splice-remove): Fixed typo.

* semantic-fw.el:

(semantic-current-overrides, semantic-new-overrides): New functions.
(semantic-install-function-overrides): Use them.  New optional
argument MODE to install into.
(semantic-fetch-overload): Use `semantic-current-overrides'.
(semantic-defoverload): Renamed to...
(define-overload): New macro.

* semantic-imenu.el:

(semantic-imenu-flush-fcn, semantic-create-imenu-index):  Use
`semantic-after-toplevel-cache-change-hook' and
`semantic-after-partial-cache-change-hook' instead of
`semantic--before-toplevel-cache-flush-hook' and
`semantic-clean-token-hooks'.

* semantic-util.el:

(semantic-name-nonterminal, semantic-abbreviate-nonterminal)
(semantic-summarize-nonterminal, semantic-prototype-nonterminal)
(semantic-concise-prototype-nonterminal)
(semantic-uml-abbreviate-nonterminal)
(semantic-uml-prototype-nonterminal)
(semantic-uml-concise-prototype-nonterminal)
(semantic-nonterminal-static): Use `define-overload' instead of
`semantic-defoverload'.

(semantic-find-dependency, semantic-find-nonterminal)
(semantic-find-documentation): Put the right property
'semantic-overload.

* semantic.el:

(semantic-bovinate-toplevel): Added missing `working-status-form'
around incremental parse.
(semantic-bovinate-from-nonterminal)
(semantic-bovinate-from-nonterminal-full)
(semantic-bovinate-region-until-error): Inlined to avoid an
unnecessary extra call.

* senator.el:

(senator-register-mode-menu-entry ["Highlight Dirty Tokens"]):
Commented out.
(senator-register-mode-menu-entry ["Highlight changes"])
(senator-register-mode-menu-entry ["Show parser state"]): Added.

>>> David Ponce <david@...> seems to think that:
>Hi Eric,
>
>[...]
> > > What's your opinion?  Can I check my changes in?
> >
> > This seems like a great idea.  Please check them in.
>
>Thanks! Done.
>
>[...]
> > I also re-wrote semantic-edit.el on my laptop.  The new incremental
> > parser has some nice new features, and helps prevent full-reparses
> > more often, including insertion and deletion of tokens.  Yay!  On
> > the bad side, I had to also build a couple new minor modes to
> > replace the old show-dirty minor mode, and the auto-parse mode needs
> > some more adjusting I think.
>[...]
>
>You really did a great work!  I tried to use the new incremental
>parser and found some bugs I fixed (checked-in).  I also added the new
>minor modes to Senator menu.  Finally I used different hooks in
>semantic-imenu that seems to work far better that the previous one
>(no more messages "Imenu is out of date, try again. (internal bug)").

Thanks.  I did a lot of testing in situations without the
parent/child relationship.

There are many other issues and missing features from the incremental
parser at the moment, and it is nice to have some verification in
Java.

I have a change for senator also.  When adding
senator-completion-cache-flush-fcn to some parser hooks, the hook
semantic-after-partial-cache-change-hook should be used instead of
`semantic-clean-token-hooks'.

>It remains a problem when `semantic-edits-bovinate-region' just
>returns nil when the parser fails.  In that case
>`semantic-edits-splice-remove' fails with the following error because
>cache-list is nil:
>
>  setcdr(nil nil)
>  (if (eq first (car chil)) (progn (setq cacheend chil) (while ...
>  ...) (setcar chil ...) (setcdr chil ...)) (while (and cachestart
>  ...) (setq cachestart ...)) (setq cacheend cachestart) (while (and
>  cacheend ...) (setq cacheend ...)) (setcdr cachestart (cdr
>  cacheend))) (let* ((first ...) (last ...) (chil ...)  (cachestart
>  cachelist) (cacheend nil) (tmp oldtokens)) (if (eq first ...) (progn
>  ... ... ... ...) (while ... ...) (setq cacheend cachestart) (while
>  ... ...) (setcdr cachestart ...)) (while oldtokens
>  (semantic-deoverlay-token ...) (setq oldtokens ...)))
>  semantic-edits-splice-remove((("v[]" variable "int" nil (...) nil
>  (...) #<overlay from 2489 to 2547 in Essai.java>)) nil nil)
>
>In fact I got this problem on a statement like this (Java):
>
>  private int v[
>  ] = {1,
>       2,
>       3,
>       4},
>    w = 0;
>
>that results in two tokens, one for `v' from `private' to the comma,
>and a second one for `w' from `w' to after the semicolon.  In such
>case the region covered by each token can't be parsed.  Only the
>whole statement is valid.  So when you change something in that
>statement the incremental parser always fails.  Unfortunately I don't
>found yet how to get both incremental parse and token differentiation
>(useful for navigation, completion, etc.) work together.
>
>Maybe we need a new property telling the true bounds of such tokens?
>
>Have you any other idea?

I would be loath to change how overlays are used for explicit token
boundaries.  This is how a user wants to think about their text.

As this is only for the incremental parser, perhaps the notion of a
parsable unit, or a tag segment, or other useful name is needed.
Something to represent the minimum text needed for the rule in
question to generate the tag list.

Functions for -start and -end for this unit could mime
`semantic-token-start' and `semantic-token-end' in most cases, but if
a token had an extra-specifier or a property indicating which
user-visible tag started or ended the parsable unit the tag was
generated from.

Unfortunately, this information would require special care when saving
and restoring a tags database on disk unless it was not a reference,
but a key of some sort (such as a name, or index forward and backward
through the current cache.)

I would like to add support for more types of overlays in semantic
set into the extra-specifier or property list.  In particular, I'd
like tokens to be able to specify a segment within themselves that
can be edited w/out concern for the cache. (ei, if you edit within
that space, don't bother parsing.)  This too could speed things up.
The assumption is that even if parens go out of sync, the user will
try to fix it up later.  With some generic support for such overlay
based properties, a solution for the above problem would be solved.

Of course, it could be as simple as creating overloadable functions
and you could use some clever regexp based hack to get the correct
bounds.

Of course, short term, the fail should automatically revert to
requiring a full reparse.  If not, a condition-case or lex
configuration may be needed.

Well, enough of that for tonight.
Have fun
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Richard,

>I got back from my one week vacation Sunday.
>
Welcome back!  I hope you enjoyed your vacation.
Eric is on vacation this week.

>I will try to catup with all the emails you two ASAP.
>  
>
Good luck  ;-)

David

>>> David Ponce <david@...> seems to think that:
>Hi Eric,
>
>As I am currently on vacation, I got some free time to work on
>Semantic 2.0 :-)
>
>First, I submit you a patch to semantic-lex.el and semantic.el to
>simplify and improve some code.  See the attached file semantic.patch
>for details.

This patch looks good.  Is there a reason that `semantic-init' and
`semantic-lex' was moved lower?  I could not divine that.

>My second point is the following proposal for an enhanced API to
>manage nonterminal expansion (see the code at end).
>
>* The new API can work like the old one.
>
>   In mode hook, instead of using:
>
>     (setq semantic-expand-nonterminal 'my-expander)
>
>   you must write:
>
>     (semantic-install-nonterminal-expanders
>        '((t . my-expander)))
>
>* To avoid unnecessary call to the expander on every tokens (and
>   save CPU!), it is now also possible to do something like this:
>
>     (semantic-install-nonterminal-expanders
>       '((variable . my-expander)
>         (type     . my-expander)))
>
>* Even better it is possible to use specific expanders, like this:
>
>     (semantic-install-nonterminal-expanders
>       '((variable . my-variable-expander)
>         (type     . my-type-expander)))
>
>What do you think?

This is an interesting idea.  On one side, we save a function call.
On the other, you cannot write an expander that does the same thing
to two different types of tokens without adding more layers of
indirection.

Of course, I'm not sure when you would want to do that, but you never
know.  I think the simplicity this adds outweighs a possible future
need.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Eric,

[...]
 > This patch looks good.  Is there a reason that `semantic-init' and
 > `semantic-lex' was moved lower?  I could not divine that.

In fact only `semantic-lex' needed to be moved lower to satisfy
`defsubst' dependencies.  For consistency I moved `semantic-lex-init'
too.

I checked it in plus Wisent's changes ;-)

 > > My second point is the following proposal for an enhanced API to
 > > manage nonterminal expansion (see the code at end).
[...]

 > This is an interesting idea.  On one side, we save a function call.
 > On the other, you cannot write an expander that does the same thing
 > to two different types of tokens without adding more layers of
 > indirection.
 >
 > Of course, I'm not sure when you would want to do that, but you
 > never know.  I think the simplicity this adds outweighs a possible
 > future need.

I investigated a little more on this new mechanism...

I did performance measurements (Emacs 21.3.50.1 & Emacs 20.7.3) with
the current version and the new one, and surprisingly not noticed any
ameliorations!  In most cases the current implementation performs a
little faster than the new one!  So, for now, I don't plan to work
more on that.

Thanks for your time!
David

>>> David Ponce <david@...> seems to think that:
>Hi Eric,
>
>[...]
> > This patch looks good.  Is there a reason that `semantic-init' and
> > `semantic-lex' was moved lower?  I could not divine that.
>
>In fact only `semantic-lex' needed to be moved lower to satisfy
>`defsubst' dependencies.  For consistency I moved `semantic-lex-init'
>too.
>
>I checked it in plus Wisent's changes ;-)

Aha, thanks.

> > > My second point is the following proposal for an enhanced API to
> > > manage nonterminal expansion (see the code at end).
>[...]
>
> > This is an interesting idea.  On one side, we save a function call.
> > On the other, you cannot write an expander that does the same thing
> > to two different types of tokens without adding more layers of
> > indirection.
> >
> > Of course, I'm not sure when you would want to do that, but you
> > never know.  I think the simplicity this adds outweighs a possible
> > future need.
>
>I investigated a little more on this new mechanism...
>
>I did performance measurements (Emacs 21.3.50.1 & Emacs 20.7.3) with
>the current version and the new one, and surprisingly not noticed any
>ameliorations!  In most cases the current implementation performs a
   ^  Cool word.

>little faster than the new one!  So, for now, I don't plan to work
>more on that.
  [ ... ]

Makes sense.  Thanks for clever hack.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

>>> David Ponce <david@...> seems to think that:
>Eric,
>
>While writing new Semantic lexers for Wisent I implemented the
>following high-level macro to define a lexical analyzer for paired
>delimiters block:
>
>(defmacro define-lex-block-analyzer (name doc regexp opensym blocksym)
>  "Create a lexical analyzer with NAME and DOC, that match REGEXP.
  [ ... ]
>(define-lex-block-analyzer wisent-java-lex-open-paren
>  "Detect and create a paren block or open paren token."
>  "(" 'LPAREN 'PAREN_BLOCK)
>
>(define-lex-block-analyzer wisent-java-lex-open-brace
>  "Detect and create a brace block or open brace token."
>  "{" 'LBRACE 'BRACE_BLOCK)
>
>It could be used for semantic-list too, like this:
>
>(define-lex-block-analyzer semantic-lex-paren-or-list
>  "Detect open parenthesis.
>Return either a paren token or a semantic list token depending on
>`current-depth'."
>  "\\s(" 'open-paren 'semantic-list)
>
>If you find it convenient I would like to add it in semantic-lex.el.
>What do you think?
  [ ... ]

Hi,

  This seems like a good idea.  I used the short-cut in semantic
because of regexp matches in the rules.

  To simplify list rule generation, perhaps something like this would
be good:

(define-lex-block-analyzer semantic-lex-blah-blah-list
   '( ( "{" "}" brace )
      ( "(" ")" paren ) )
  )

It would auto-generate brace-open, brace-close, and brace-list.
(Sorry, I'm a fan of lower case, but am open to suggestions)

This is because it is faster (I suspect) to do this:

(cond ...
  ((looking-at "\\s(")
   (let ((text (...)))
      (cond ((string= "(" text) ...)
             (string= "{" text) ...))))))
  ...
)

than several looking-at regexps.  (Only one regexp to compile).

It might be nice to pre-name many of them, such as {, [, (, <.  Such
naming will help keep language files looking the same.  While
compiling the macro, it could search the syntax table for which ones
are available.  Hmm, the syntax table may not be available.  Perhaps
we can specify what the syntax table is somewhere since we already
offer syntax table deltaas.  Then only languages with really strange
characters would need to worry about the specific macro you have
proposed.

Of course, this then begs the macro that detects "BEGIN", and finds
the matching "END" keyword for languages like Ada, pascal, and OCTAVE.
What fun.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Eric,

[...]
 >  To simplify list rule generation, perhaps something like this would
 >  be good:
 >
 > (define-lex-block-analyzer semantic-lex-blah-blah-list
 >    '( ( "{" "}" brace )
 >       ( "(" ")" paren ) )
 >   )
 >
 > It would auto-generate brace-open, brace-close, and brace-list.
 > (Sorry, I'm a fan of lower case, but am open to suggestions)

In fact, I use upper case for terminal symbols to follow the
recommendation from the Bison's manual:

   "The Bison representation for a terminal symbol is also called a
"token type".  Token types as well can be represented as C-like
identifiers.  By convention, these identifiers should be upper case to
distinguish them from nonterminals: for example, `INTEGER',
`IDENTIFIER', `IF' or `RETURN'.  A terminal symbol that stands for a
particular keyword in the language should be named after that keyword
converted to upper case."

Maybe if the prefix is given in upper case the symbol names could be
generated in upper case too?  So both you and me will be happy ;-)

IMO, a more general design would be to specify the open, close and
list symbols.  Thus the programmer will be free to choose what
terminal symbol names he wants, and more probably [important!] he will
be able to keep those that already exist in grammars.  This could
simplify things when porting existing grammars ;-)

Perhaps something like this:

(define-lex-block-analyzer semantic-lex-blah-blah-list
  '(BLOCK_BRACE
    ("{" . LBRACE)
    ("}" . RBRACE)
    )
  '(semantic-list
    ("(" . open-paren)
    (")" . close-paren)
    )
  ...
  )

 > This is because it is faster (I suspect) to do this:
 >
 > (cond ...
 >   ((looking-at "\\s(")
 >    (let ((text (...)))
 >       (cond ((string= "(" text) ...)
 >              (string= "{" text) ...))))))
 >   ...
 > )
 >
 > than several looking-at regexps.  (Only one regexp to compile).

This is a good point!

 > It might be nice to pre-name many of them, such as {, [, (, <.  Such
 > naming will help keep language files looking the same.  While
 > compiling the macro, it could search the syntax table for which ones
 > are available.  Hmm, the syntax table may not be available.  Perhaps
 > we can specify what the syntax table is somewhere since we already
 > offer syntax table deltaas.  Then only languages with really strange
 > characters would need to worry about the specific macro you have
 > proposed.

This could be good idea!  But, as I already said above, we should take
care of not making more difficult to port existing grammars, because
of hard-coded terminal symbol names.

 > Of course, this then begs the macro that detects "BEGIN", and finds
 > the matching "END" keyword for languages like Ada, pascal, and
 > OCTAVE. What fun.

;-)

David

>>> David Ponce <david@...> seems to think that:
>Eric,
>
>[...]
> >  To simplify list rule generation, perhaps something like this would
> >  be good:
> >
> > (define-lex-block-analyzer semantic-lex-blah-blah-list
> >    '( ( "{" "}" brace )
> >       ( "(" ")" paren ) )
> >   )
> >
> > It would auto-generate brace-open, brace-close, and brace-list.
> > (Sorry, I'm a fan of lower case, but am open to suggestions)
>
>In fact, I use upper case for terminal symbols to follow the
>recommendation from the Bison's manual:
>
>   "The Bison representation for a terminal symbol is also called a
>"token type".  Token types as well can be represented as C-like
>identifiers.  By convention, these identifiers should be upper case to
>distinguish them from nonterminals: for example, `INTEGER',
>`IDENTIFIER', `IF' or `RETURN'.  A terminal symbol that stands for a
>particular keyword in the language should be named after that keyword
>converted to upper case."

Ah, I remember this, which is how I wrote my grammars w/ the %token
keywords in the .bnf file.  Semantic introduced something different,
however, which was the concept of built-in lexical types which are
supertypes of what a typical lex returned.  Thus `semantic-list',
`open-paren', and `close-paren' for all classes of parenthitical.  In
the grammar, specifics were upper case.  Same for `punctuation' vs
`PERIOD'.

Since the analyzer you specified is for specifics, upper case does
make sense even in my world.

>Maybe if the prefix is given in upper case the symbol names could be
>generated in upper case too?  So both you and me will be happy ;-)
>
>IMO, a more general design would be to specify the open, close and
>list symbols.  Thus the programmer will be free to choose what
>terminal symbol names he wants, and more probably [important!] he will
>be able to keep those that already exist in grammars.  This could
>simplify things when porting existing grammars ;-)
>
>Perhaps something like this:
>
>(define-lex-block-analyzer semantic-lex-blah-blah-list
>  '(BLOCK_BRACE
>    ("{" . LBRACE)
>    ("}" . RBRACE)
>    )
>  '(semantic-list
>    ("(" . open-paren)
>    (")" . close-paren)
>    )
>  ...
>  )
  [ ... ]

Hi,

  This makes sense.  The base macro could be as you defined, and a
different macro would happen to pre-define the usual suspects for
authors starting new languages.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Eric,

>  This makes sense.  The base macro could be as you defined, and a
>different macro would happen to pre-define the usual suspects for
>authors starting new languages.
>
Attached you will find a first implementation of
`define-lex-block-analyzer', that takes into account your good remarks
;-)

I used it successfully for Wisent Java lexers!

As you will notice I simplified the way it is called, eliminating
unnecessary quote and parenthesis.  For example in the Java lexers I
use it like this:

(define-lex-block-analyzer wisent-java-lex-blocks
  "Detect and create a open, close or block token."
  (PAREN_BLOCK ("(" LPAREN) (")" RPAREN))
  (BRACE_BLOCK ("{" LBRACE) ("}" RBRACE))
  (BRACK_BLOCK ("[" LBRACK) ("]" RBRACK))
  )

Also, you were right, this new analyzer seems to be a little faster
than the previous different ones :-)

It would be nice if you could review my code.  And if you find it
good enough, I will add it into semantic-lex.el and check the change
in.

Thanks!
David

>>> David Ponce <david@...> seems to think that:
>Hi Eric,
>
>>  This makes sense.  The base macro could be as you defined, and a
>>different macro would happen to pre-define the usual suspects for
>>authors starting new languages.
>>
>Attached you will find a first implementation of
>`define-lex-block-analyzer', that takes into account your good remarks
>;-)
>
>I used it successfully for Wisent Java lexers!
>
>As you will notice I simplified the way it is called, eliminating
>unnecessary quote and parenthesis.  For example in the Java lexers I
>use it like this:
>
>(define-lex-block-analyzer wisent-java-lex-blocks
>  "Detect and create a open, close or block token."
>  (PAREN_BLOCK ("(" LPAREN) (")" RPAREN))
>  (BRACE_BLOCK ("{" LBRACE) ("}" RBRACE))
>  (BRACK_BLOCK ("[" LBRACK) ("]" RBRACK))
>  )

I like this better.

>Also, you were right, this new analyzer seems to be a little faster
>than the previous different ones :-)
>
>It would be nice if you could review my code.  And if you find it
>good enough, I will add it into semantic-lex.el and check the change
>in.
>
>Thanks!
>David
>
>
>
>;;; sem-lex-block.el --- Lexical analyzer for paired delimiters blocks
>
>;;; Commentary:
>;; 
>
>;;; History:
>;; 
>
>;;; Code:
>
>;; (define-lex-block-analyzer wisent-java-lex-blocks
>;;   "Detect and create a open, close or block token."
>;;   (PAREN_BLOCK ("(" LPAREN) (")" RPAREN))
>;;   (BRACE_BLOCK ("{" LBRACE) ("}" RBRACE))
>;;   (BRACK_BLOCK ("[" LBRACK) ("]" RBRACK))
>;;   )
>
>(defmacro define-lex-block-analyzer (name docstring spec1 &rest specs)
>  "Create a lexical analyzer NAME for paired delimiters blocks.
>It detects a paired delimiters block or the corresponding open or
>close delimiter depending on the value of the variable
>`current-depth'.  DOCSTRING is the documentation string of the lexical
>analyzer.  SPEC1 and SPECS specify the token symbols and open, close
>delimiters used.  Each SPEC has the form:
>
>\(BLOCK-SYM (OPEN-DELIM OPEN-SYM) (CLOSE-DELIM CLOSE-SYM))
> 
>where BLOCK-SYM is the symbol returned in a block token.  OPEN-DELIM
>and CLOSE-DELIM are respectively the open and close delimiters
>identifying a block.  OPEN-SYM and CLOSE-SYM are respectively the
>symbols returned in open and close tokens."
>  (let ((specs (cons spec1 specs))
>        spec ospec olist clist)
>    (while specs
>      (setq spec  (car specs)
>            specs (cdr specs)
>            open  (nth 1 spec)
>            ;; build alist ((OPEN-DELIM OPEN-SYM BLOCK-SYM) ...)
>            olist (cons (list (car open) (cadr open) (car spec)) olist)
>            ;; build alist ((CLOSE-DELIM . CLOSE-SYM) ...)
>            clist (cons (nth 2 spec) clist)))
>    `(eval-and-compile
>       (defvar ,name nil
>         ,docstring)
>       (setq ,name
>             '((and
>                (looking-at "\\(\\s(\\|\\s)\\)")
>                (let ((text (match-string 0)) match)
>                  (cond
>                   ((setq match (assoc text ',olist))
>                    (if (or (not depth) (< current-depth depth))
>                        (progn
>                          (setq current-depth (1+ current-depth))
>                          (semantic-lex-token
>                           (nth 1 match)
>                           (match-beginning 0) (match-end 0)))
>                      (semantic-lex-token
>                       (nth 2 match)
>                       (match-beginning 0)
>                       (save-excursion
>                         (condition-case nil
>                             (forward-list 1)
>                           ;; This case makes lex robust to broken lists.
>                           (error
>                            (goto-char
>                             (funcall
>                              semantic-lex-unterminated-syntax-end-function
>                              (nth 2 match) start end))))
>                         (setq end-point (point))))))
>                   ((setq match (assoc text ',clist))
>                    (setq current-depth (1- current-depth))
>                    (semantic-lex-token
>                     (nth 1 match)
>                     (match-beginning 0) (match-end 0)))))))
>             ))))
>
>(provide 'sem-lex-block)
>
>;;; sem-lex-block.el ends here
>

I tried very hard to find something wrong with the above, but was
unsuccessful.  It will make a great addition.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Eric,

>I tried very hard to find something wrong with the above, but was
>unsuccessful.  It will make a great addition.
>
Thanks!

I checked-in new semantic-lex.el and semantic-load.el with new
`define-lex-block-analyzer' macro.
I also checked-in new versions of wisent-java & wisent-wy stuff that use
the new lexical analysis tools ;-)

David

Hi Eric,

 > Hope you had fun.

Yes thanks!  I was very happy to see my friends in Germany :-)

 > I just got back from a trip running some of my catapults all day for
 > people I work with.  It was a good time.  We were throwing 2 liter
 > bottles 160ft with our human powered machine. Wee!

Nice!

[...]
 > > So, I just added a (require 'semantic-bovine) in such files to fix
 > > errors.
[...]

 > That makes sense.  I hadn't bumped into that problem yet.

I finally reverted back these changes (see below).

[...]
 > > - I wonder if it is really necessary to add new syntax in BNF or
 > >   WY files to define the lexer.  I found the various macros
 > >   provided in semantic-lex convenient enough to easily create a
 > >   new lexer that can be self contained in one ELisp file.  I can't
 > >   see any advantage of duplicating this code in a BNF or WY file,
 > >   in a different form.
 >
 > I'm not sure either.  I haven't been able to divine what a good new
 > syntax would be.  If the syntax table (which the lexer depends on)
 > also were to be defined, that could certainly benefit from a less
 > obscure mechanism than the existing syntax table code in Emacs.

Probably.  But as you I haven't been able to find a good new syntax
that don't simply duplicate the Elisp code!

 > > - I noticed that you moved the functions
 > >   `semantic-bovinate-from-nonterminal' and
 > >   `semantic-bovinate-from-nonterminal-full' in the new
 > >   semantic-bovine file.
[...]

 > That makes sense to do.  If you see more functions like that, feel
 > free to move them about.  I don't think they are a part of partial
 > reparsing only, which is what semantic-edit was pulled out for.
[...]

I finally moved back these runtime functions into semantic.el.  IMO
this is the best place where to put them for now.  Consequently there
is no more need to (require 'semantic-bovine) in files that use them
:)

 > > - IMO the variable `semantic-lex-analyzer' should be
 > >   `make-variable-buffer-local'.  I got strange bug when writing
 > >   the new Java lexer because I forgot to make it buffer local!
 >
 > I noticed that too but had not yet checked it in.

Done (see change log at end).

 > > I think I improved the compatibility between the old and new
 > > lexers. Now `semantic-lex-analyzer' is buffer local and its
 > > default value is `semantic-flex' for compatibility with Semantic
 > > 1.x API.  I also replaced all direct calls to `semantic-flex' by
 > > calls to `semantic-lex'.
 >
 > That's a good idea.  I'd like to remove semantic-flex someday, and
 > this is a good transition strategy.  My preference would be to have
 > it at the end of semantic-lex.el though to steer focus away from it.
[...]

Done (see change log at end).

 > > In the Wisent part, I improved the code in wisent-java-lex.el.  It
 > > now contains a new lexer for wisent-java.el ;-) plus some new test
 > > code.  I made some minor change to wisent-java-tags.el and
 > > wisent-java.el so they now work with the new lexers.
 >
 > That's great.  You had mentioned earlier about converting the LL
 > parser to use similar input as the LALR parser.  I am not yet sure
 > what that means, but perhaps it can be converged after these bigger
 > changes stabilize.

For now I added a new very simple function `wisent-lex' in
wisent-flex.el, that simply converts lexical tokens produced by
`semantic-lex' into ones suitable for wisent.

[...]
 > You occasionally generate performance numbers.  Would it make sense
 > to capture some of the code that creates them in the `test'
 > directory. We could periodically run them and capture the results in
 > a log to see how we're doing.

Good idea!  I will do that as soon as possible.

[...]
 > This is some great stuff and worth checking in.  I too have some
 > changes pending which would do well to go in after yours.
[...]

OK, I checked my changes in (see change log at end).

Sincerely,
David

------------------- Change Log --------------------------------------
* semantic.el:

(semantic-version): Updated to 2.0alpha2
(require 'semantic-lex): Added.
(semantic-flex-start, semantic-flex-end, semantic-flex-text)
(semantic-flex-keywords-obarray, semantic-flex-make-keyword-table)
(semantic-flex-keyword-p, semantic-flex-keyword-put)
(semantic-flex-keyword-get, semantic-flex-map-keywords)
(semantic-flex-keywords, semantic-flex-tokens)
(semantic-flex-unterminated-syntax-end-function)
(semantic-flex-extensions, semantic-flex-syntax-modifications)
(semantic-ignore-comments, semantic-flex-enable-newlines)
(semantic-flex-enable-whitespace, semantic-flex-enable-bol)
(semantic-number-expression, semantic-flex-depth, semantic-flex)
(semantic-flex-buffer, semantic-flex-list): Moved to semantic-lex.el.
(semantic-token-docstring, semantic-bovinate-toplevel): Call
`semantic-lex' instead of `semantic-flex'.
(semantic-lambda, semantic-bovinate-from-nonterminal)
(semantic-bovinate-from-nonterminal-full)
(semantic-bovinate-region-until-error): Moved back here from
semantic-bovine.el.  Use `semantic-lex' instead of `semantic-flex'.

* semantic-bovine.el:

(semantic-lambda, semantic-bovinate-from-nonterminal)
(semantic-bovinate-from-nonterminal-full)
(semantic-bovinate-region-until-error):  Moved these runtime
functions into semantic.el.

* semantic-edit.el:

(semantic-rebovinate-token): Call `semantic-lex' instead of
`semantic-flex'.

* semantic-util.el:

(semantic-find-doc-snarf-comment): Call `semantic-lex' instead of
`semantic-flex'.

* semantic-lex.el:

(with-syntax-table): Moved.
(semantic-lex-make-keyword-table, semantic-lex-keyword-p)
(semantic-lex-keyword-put, semantic-lex-keyword-get)
(semantic-lex-map-keywords, semantic-lex-keywords): New functions.
(semantic-lex-analyzer): Default to `semantic-flex'.  buffer-local
when set in any fashion.
(semantic-lex-depth): Fixed `make-variable-buffer-local'.
(define-lex): Use value of `semantic-lex-depth' as the default depth
if it is not specified.
(semantic-lex-token-class, semantic-lex-token-start)
(semantic-lex-token-end, semantic-lex-token-text): New functions.
(semantic-lex-symbol-or-keyword): Use `semantic-lex-keyword-p'
instead of `semantic-flex-keyword-p'.

Compatibility with Semantic 1.x API.

(semantic-flex-start, semantic-flex-end, semantic-flex-text)
(semantic-flex-make-keyword-table, semantic-flex-keyword-p)
(semantic-flex-keyword-put, semantic-flex-keyword-get)
(semantic-flex-map-keywords, semantic-flex-keywords): Aliased to
their counterparts in 2.x API.
(semantic-flex-keywords-obarray, semantic-flex-tokens)
(semantic-flex-unterminated-syntax-end-function)
(semantic-flex-extensions, semantic-flex-syntax-modifications)
(semantic-ignore-comments, semantic-flex-enable-newlines)
(semantic-flex-enable-whitespace, semantic-flex-enable-bol)
(semantic-number-expression, semantic-flex-depth, semantic-flex)
(semantic-flex-buffer, semantic-flex-list): Moved from semantic.el.
(semantic-flex-tokens): Default to value of `semantic-lex-tokens'.
(semantic-number-expression): Default to value of
`semantic-lex-number-expression'.

* wisent-flex:

(wisent-lex): New function.

Thanks for getting that stuff done.  It merged into my changes fine.

Eric

[...]
>------------------- Change Log --------------------------------------
>* semantic.el:
>
>(semantic-version): Updated to 2.0alpha2
>(require 'semantic-lex): Added.
>(semantic-flex-start, semantic-flex-end, semantic-flex-text)
>(semantic-flex-keywords-obarray, semantic-flex-make-keyword-table)
>(semantic-flex-keyword-p, semantic-flex-keyword-put)
>(semantic-flex-keyword-get, semantic-flex-map-keywords)
>(semantic-flex-keywords, semantic-flex-tokens)
>(semantic-flex-unterminated-syntax-end-function)
[...]

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org