CEDET

Hi Mike,

> I downloaded the beta from the website and followed the instructions. I'm
> using gnu emacs CVS. I got this error:
> 
>  /opt/emacs/bin/emacs -batch -l common-compile-script -f batch-byte-compile cedet-autogen.el cedet-compat.el ezimage.el inversion.el pprint.el sformat.el working.el
> 
>>>Error occurred processing cedet-autogen.el: error (("Local variables entry is missing the prefix"))

I think you need the fixed version of cedet-autogen.el which is not currently in the beta tarball.

You can download it from:

<http://cvs.sourceforge.net/viewcvs.py/*checkout*/cedet/cedet/common/cedet-autogen.el?rev=1.3.2.2>

David

>>> David PONCE <david.ponce@...> seems to think that:
>Eric,
>
>I updated the cogre wisent-dot stuff to work with the current version
>of the auto-generate analyzers feature.
>
>Attached is a tarball of the changed files: wisent-dot.el,
>wisent-dot.wy and wisent-dot-wy.el.
>
>Could you please check if parsing of .dot files works correctly?  I
>don't have support for `graphviz-dot-mode' in my installation.
  [ ... ]

Hi,

  I think I had written to the original author of graphviz mode a long
time ago, but he did not respond.  I was thinking of writing my own
version with just the basics for supporting it locally.

  Your tar files did not compile for me.  I may need to wait till CVS
is available and make sure I have all of your latest changes.

Thanks
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

>>> David PONCE <david.ponce@...> seems to think that:
  [ ... ]
>>>  * A keyword analyzer, defined by `define-lex-keyword-type-analyzer',
>>>    to handle only language keywords.  That permit to separate
>>>    analysis of true keywords from other symbols.
>> 
>> 
>> Does this mean a separate entry in the lexer construction?
>> ie. 1 call to find 'keywords' and a second to find 'symbol' %types?
>> That could slow things down some.
>
>Yes this means that.  For what I saw, using the new Java lexer,
>parsing hasn't slowed down.  Matching a keyword is very fast, as it
>uses a symbol table.  The only extra cost I see can be a redundant
>looking-at.  IMO this tiny extra cost is largely compensated by the
>flexibility given to the programmer.  Keywords and other symbols no
>more need to be handled in the same way, and can be matched by
>different syntax regexp.  For example we can imagine that keywords
>start with a percent (like in grammars) and ordinary symbols not.
>
>%type <keyword> syntax "%\\(\\sw\\|\\s_\\)+" matchdatatype keyword
>%type <symbol>  syntax "\\(\\sw\\|\\s_\\)+"  matchdatatype regexp
>
>In a such case using two different regexp to respectively match
>keywords and symbols could be faster than looking up in the keyword
>table for each symbol.

In addition for your grammar, a keyword regexp could be:

%type <keyword> syntax "%\\sw+" matchdatatype keyword

as there are no symbol constituents in any of the keywords (yet.)

This would be true for most languages where if, while, else, return,
etc are simple words.

>A more interesting case would be if, for some language, keywords were
>not based on Emacs symbol characters, as opposed to other ordinary
>symbols ;-)  Separating keyword from symbol analyzers permits to
>handle that.

AFAIK Emacs can use any characters in its symbols.  For example:

M-: (intern "moose pie") RET

The trick is when a reader sees it in your code it may not realize
what you mean. ;)

>>>- <type> in the generated analyzer name is now enclosed between < > to
>>>  improve readability.  For example:
>>>
>>>  %package my-foo
>>>  ...
>>>  %type <symbol> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype regexp
>> 
>> 
>> This entry is beginning to strike me as being a bit long.
>> Technically, the word "syntax" and "matchdatatype" are sugar, meaning
>> they add anchors for the developer to know what is coming.
>> 
>> Are sections of this optional?  eg:
>> 
>> %type <symbol> syntax "blahblah"
>> 
>> implies a matchdatatype of regexp
>> 
>> and 
>> 
>> %type <symbol> matchdatatype regexp
>> 
>> implies some sort of default syntax regular expression?
>> 
>> I guess just 
>> 
>> %type <symbol>
>> 
>> could mean something if there are defaults.
>
>It will be easy to preset useful defaults in the type table for well
>known types like <symbol>, <string>, <punctuation>, etc.., by adding
>things like the following in `semantic-lex-make-type-table':
>
>;; Set up some useful default properties
>(semantic-lex-type-put "punctuation"
>                       'syntax "\\(\\s.\\|\\s$\\|\\s'\\)"
>                       t)
>(semantic-lex-type-put "punctuation"
>                       'matchdatatype 'regexp
>                       t)
>(semantic-lex-type-put "symbol"
>                       'syntax "\\(\\sw\\|\\s_\\)+"
>                       t)
>(semantic-lex-type-put "symbol"
>                       'matchdatatype 'regexp
>                       t)

That's a nice idea.  For simple stuff, a language author could just
specify a few %type entries and that's it.

>For now generation of a <type> analyzer is controlled by the existence
>or a syntax property for that <type>.
>
>To let the developer control what analyzers to generate it could be
>possible to use a special flag like "%type <xxx> generate t", or
>simply check that a %type statement is present.

Yes, I can imagine just a few:

%type <symbol>
%type <punctuation>

to build a default analyzer.

I assume the order of the %type specifiers controls the order of
placement in the constructed lexer?  This will be an important
feature that must be documented.

>[...]
>> That's fine.  I'd probably adapt it to use all your new stuff anyway. ;)
>> I like the new representation as it says a lot more with the <> around
>> it.
>
>I committed the changes.
  [ ... ]

Thanks!  I will try them out.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Eric,

[...]
>>After converting wisent-java.wy, I will be able to completely remove
>>wisent-java-lex.el.  And probably some specific predefined analyzers
>>in semantic-lex.el could be removed too ;-)

I just converted the wisent-java.wy grammar to auto-generate the
lexical analyzers, and removed the dependency on wisent-java-lex from
wisent-java.el.

Parsing seems to work just fine using that grammar and the new
auto-generated lexer :-)

I committed these changes:

	* cedet/semantic/semantic-grammar.el 

	(semantic-grammar-insert-defanalyzer): Fix use of quote in
	declaration of regexp analyzer.

	* cedet/semantic/wisent/wisent-java.el

	(require 'wisent-java-lex): Remove.
	
	* cedet/semantic/wisent/wisent-java.wy

	Use %keyword to declare keywords.
	Automatically generate analyzers used to build the lexer.

	(<block>, <punctuation>, <symbol>, <string>)
	(<number>, <keyword>): Add %type declaration.
	(epilogue): Add lexer declaration.

David

[...]
> Yes, I can imagine just a few:
> 
> %type <symbol>
> %type <punctuation>
> 
> to build a default analyzer.

I will look at that as soon as possible ;-)

> I assume the order of the %type specifiers controls the order of
> placement in the constructed lexer?  This will be an important
> feature that must be documented.
[...]

For now only the basic analyzers are generated.  The developer is
responsible of putting them together in a lexer in the relevant order.

There are examples in the epilogue code in wisent-java.wy and
wisent-java-tags.wy.

Perhaps in the future we could imagine a new declarative statement
that defines the lexer itself.  Something like:

%lexer
 {
  ignore-whitespace
  ignore-newline
  ignore-comments
  <number>
  <string>
  ;; Must detect keywords before other symbols
  <keyword>
  <symbol>
  <punctuation>
  <block>
  default-action  
 }

<type> items would be replaced by corresponding generated analyzers
and other items like ignore-whitespace would be replaced by a
corresponding `semantic-lex-ignore-whitespace'.

However I am not sure that it will be worth implementing that
compared to the simplicity of doing the same thing in Lisp.

David

>>> David PONCE <david.ponce@...> seems to think that:
  [ ... ]
>Perhaps in the future we could imagine a new declarative statement
>that defines the lexer itself.  Something like:
>
>%lexer
> {
>  ignore-whitespace
>  ignore-newline
>  ignore-comments
>  <number>
>  <string>
>  ;; Must detect keywords before other symbols
>  <keyword>
>  <symbol>
>  <punctuation>
>  <block>
>  default-action  
> }
>
><type> items would be replaced by corresponding generated analyzers
>and other items like ignore-whitespace would be replaced by a
>corresponding `semantic-lex-ignore-whitespace'.
>
>However I am not sure that it will be worth implementing that
>compared to the simplicity of doing the same thing in Lisp.
  [ ... ]

I agree with your assessment.  There will always be analyzers that
cannot be adequately represented with a %type statement, such as the
#include file name in the C bovine grammar.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Eric,

[...]
 > Yes, I can imagine just a few:
 >
 > %type <symbol>
 > %type <punctuation>
 >
 > to build a default analyzer.

Following is a new patch that implements your suggestion above ;-)

I also modified the Java grammars to use the default values of the
well known token types <block>, <punctuation>, <symbol>, <string>,
<number> and <keyword>.

So now, for "standard" cases it is very easy to auto-generate the
basic type analyzers :-)

Finally, I added an optional argument `force' to the command
`semantic-grammar-create-package', to unconditionally re-generate
Lisp code from the grammar.  It is very useful to test generation
without modifying the grammar, by typing C-u C-c C-c.

If you don't have objections, I will commit the changes.

Here is the change log:

	* cedet/semantic/wisent/wisent-java.wy

	(<block>, <punctuation>, <symbol>, <string>)
	(<number>, <keyword>): Use default values in %type declarations.

	* cedet/semantic/wisent/wisent-java-tags.wy

	(<block>, <punctuation>, <symbol>, <string>)
	(<number>, <keyword>): Use default values in %type declarations.

	* cedet/semantic/semantic-grammar.el

	(semantic-grammar-token-%type-properties): Remember types
	explicitly declared with a %type statement.
	(semantic-grammar-insert-defanalyzer): Generate analyzer for
	explicitly declared types.
	(semantic-grammar-create-package): Add optional argument force to
	unconditionally re-generate Lisp code.

	* cedet/semantic/semantic-grammar-wy.el

	Re-generated.

	* cedet/semantic/semantic-grammar.wy

	(plist_opt): New rule.
	(type_decl): Use it.
	
	* cedet/semantic/semantic-lex.el
	
	(semantic-lex-preset-default-types): New function.
	(semantic-lex-make-type-table): Use it.

Enjoy!

David

Index: semantic/semantic-grammar-wy.el
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-grammar-wy.el,v
retrieving revision 1.9
diff -c -r1.9 semantic-grammar-wy.el
*** semantic/semantic-grammar-wy.el	14 Jan 2004 09:54:24 -0000	1.9
--- semantic/semantic-grammar-wy.el	18 Jan 2004 09:55:37 -0000
***************
*** 3,11 ****
   ;; Copyright (C) 2002, 2003, 2004 David Ponce

   ;; Author: David Ponce <david@...>
! ;; Created: 2004-01-14 10:33:53+0100
   ;; Keywords: syntax
! ;; X-RCS: $Id: semantic-grammar-wy.el,v 1.9 2004/01/14 09:54:24 ponced Exp $

   ;; This file is not part of GNU Emacs.
   ;;
--- 3,11 ----
   ;; Copyright (C) 2002, 2003, 2004 David Ponce

   ;; Author: David Ponce <david@...>
! ;; Created: 2004-01-18 10:52:15+0100
   ;; Keywords: syntax
! ;; X-RCS: $Id$

   ;; This file is not part of GNU Emacs.
   ;;
***************
*** 258,266 ****
           ((LT SYMBOL GT)
            (progn $2)))
          (type_decl
!         ((TYPE token_type plist)
            `(wisent-raw-tag
              (semantic-tag ',$2 'type :value ',$3))))
          (plist
           ((plist put_value)
            (append
--- 258,269 ----
           ((LT SYMBOL GT)
            (progn $2)))
          (type_decl
!         ((TYPE token_type plist_opt)
            `(wisent-raw-tag
              (semantic-tag ',$2 'type :value ',$3))))
+        (plist_opt
+         (nil)
+         ((plist)))
          (plist
           ((plist put_value)
            (append
Index: semantic/semantic-grammar.el
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-grammar.el,v
retrieving revision 1.52
diff -c -r1.52 semantic-grammar.el
*** semantic/semantic-grammar.el	16 Jan 2004 15:43:18 -0000	1.52
--- semantic/semantic-grammar.el	18 Jan 2004 09:55:38 -0000
***************
*** 438,443 ****
--- 438,445 ----
     (let (type)
       (dolist (tag (semantic-find-tags-by-class 'type (current-buffer)))
         (setq type (semantic-tag-name tag))
+       ;; Indicate to auto-generate the analyzer for this type
+       (push (list type :declared t) props)
         (dolist (e (semantic-tag-get-attribute tag :value))
           (push (list type (intern (car e)) (read (or (cdr e) "nil")))
                 props)))
***************
*** 796,805 ****
     (let* ((type-name  (symbol-name type))
            (type-value (symbol-value type))
            (syntax     (get type 'syntax))
            spec mtype prefix name doc)
!     ;; Generate an analyzer if at least syntactic/lexical matches are
!     ;; provided.
!     (when syntax
         (setq prefix (file-name-sans-extension
                       (semantic-grammar-buffer-file
                        semantic--grammar-output-buffer))
--- 798,809 ----
     (let* ((type-name  (symbol-name type))
            (type-value (symbol-value type))
            (syntax     (get type 'syntax))
+          (declared   (get type :declared))
            spec mtype prefix name doc)
!     ;; Generate an analyzer if the corresponding type has been
!     ;; explicitly declared in a %type statement, and if at least the
!     ;; syntax property has been provided.
!     (when (and declared syntax)
         (setq prefix (file-name-sans-extension
                       (semantic-grammar-buffer-file
                        semantic--grammar-output-buffer))
***************
*** 873,881 ****
         (noninteractive)
       noninteractive))

! (defun semantic-grammar-create-package ()
!   "Create package Lisp code from grammar in current buffer."
!   (interactive)
     (semantic-bovinate-toplevel t)
     (let* (
            ;; Values of the following local variables are obtained from
--- 877,889 ----
         (noninteractive)
       noninteractive))

! (defun semantic-grammar-create-package (&optional force)
!   "Create package Lisp code from grammar in current buffer.
! Does nothing if the Lisp code seems up to date.
! If optional argument FORCE is non-nil, unconditionally re-generate the
! Lisp code."
!   (interactive "P")
!   (setq force (or force current-prefix-arg))
     (semantic-bovinate-toplevel t)
     (let* (
            ;; Values of the following local variables are obtained from
***************
*** 890,896 ****
            (epilogue (semantic-grammar-epilogue))
            (footer   (semantic-grammar-footer))
            )
!     (if (and (not (buffer-modified-p))
                (file-newer-than-file-p
                 (buffer-file-name semantic--grammar-output-buffer)
                 (buffer-file-name semantic--grammar-input-buffer)))
--- 898,905 ----
            (epilogue (semantic-grammar-epilogue))
            (footer   (semantic-grammar-footer))
            )
!     (if (and (not force)
!              (not (buffer-modified-p))
                (file-newer-than-file-p
                 (buffer-file-name semantic--grammar-output-buffer)
                 (buffer-file-name semantic--grammar-input-buffer)))
Index: semantic/semantic-grammar.wy
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-grammar.wy,v
retrieving revision 1.12
diff -c -r1.12 semantic-grammar.wy
*** semantic/semantic-grammar.wy	14 Jan 2004 09:53:56 -0000	1.12
--- semantic/semantic-grammar.wy	18 Jan 2004 09:55:38 -0000
***************
*** 258,267 ****
     ;

   type_decl:
!     TYPE token_type plist
       `(TAG ',$2 'type :value ',$3)
     ;

   plist:
       plist put_value
       (append (list $2) $1)
--- 258,272 ----
     ;

   type_decl:
!     TYPE token_type plist_opt
       `(TAG ',$2 'type :value ',$3)
     ;

+ plist_opt:
+  ;;EMPTY
+   | plist
+   ;
+
   plist:
       plist put_value
       (append (list $2) $1)
Index: semantic/semantic-lex.el
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-lex.el,v
retrieving revision 1.30
diff -c -r1.30 semantic-lex.el
*** semantic/semantic-lex.el	16 Jan 2004 08:56:43 -0000	1.30
--- semantic/semantic-lex.el	18 Jan 2004 09:55:39 -0000
***************
*** 26,32 ****
   ;; In semantic 1.x, the lexical analyzer was an all purpose routine.
   ;; To boost efficiency, the analyzer is now a series of routines that
   ;; are constructed at build time into a single routine.  This will
! ;; eliminate uneeded if statements to speed the lexer.

   (require 'semantic-fw)
   ;;; Code:
--- 26,32 ----
   ;; In semantic 1.x, the lexical analyzer was an all purpose routine.
   ;; To boost efficiency, the analyzer is now a series of routines that
   ;; are constructed at build time into a single routine.  This will
! ;; eliminate unneeded if statements to speed the lexer.

   (require 'semantic-fw)
   ;;; Code:
***************
*** 217,222 ****
--- 217,238 ----
         (unless noerror
           (semantic-lex-type-invalid type)))))

+ (defun semantic-lex-preset-default-types ()
+   "Install useful default properties for well known types."
+   (semantic-lex-type-put "punctuation" 'matchdatatype 'string t)
+   (semantic-lex-type-put "punctuation" 'syntax "\\s.\\|\\s$\\|\\s'")
+   (semantic-lex-type-put "keyword" 'matchdatatype 'keyword t)
+   (semantic-lex-type-put "keyword" 'syntax "\\(\\sw\\|\\s_\\)+")
+   (semantic-lex-type-put "symbol"  'matchdatatype 'regexp t)
+   (semantic-lex-type-put "symbol"  'syntax "\\(\\sw\\|\\s_\\)+")
+   (semantic-lex-type-put "string"  'matchdatatype 'sexp t)
+   (semantic-lex-type-put "string"  'syntax "\\s\"")
+   (semantic-lex-type-put "number"  'matchdatatype 'regexp t)
+   (semantic-lex-type-put "number"  'syntax 'semantic-lex-number-expression)
+   (semantic-lex-type-put "block"   'matchdatatype 'block t)
+   (semantic-lex-type-put "block"   'syntax "\\s(\\|\\s)")
+   )
+
   (defun semantic-lex-make-type-table (specs &optional propspecs)
     "Convert type SPECS into an obarray and return it.
   SPECS must be a list of (TYPE . TOKENS) elements, where:
***************
*** 255,260 ****
--- 271,278 ----
             (setq default token)))
         ;; Ensure the default matching spec is the first one.
         (semantic-lex-type-set type (cons default (nreverse alist))))
+     ;; Install useful default types & properties
+     (semantic-lex-preset-default-types)
       ;; Apply all properties
       (while propspecs
         (setq spec (car propspecs)
Index: semantic/wisent/wisent-java-tags.wy
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-java-tags.wy,v
retrieving revision 1.12
diff -c -r1.12 wisent-java-tags.wy
*** semantic/wisent/wisent-java-tags.wy	16 Jan 2004 08:57:53 -0000	1.12
--- semantic/wisent/wisent-java-tags.wy	18 Jan 2004 09:55:41 -0000
***************
*** 49,55 ****
   ;; -----------------------------
   ;; Block & Parenthesis terminals
   ;; -----------------------------
! %type  <block>       syntax "\\s(\\|\\s)" matchdatatype block

   %token <block>       PAREN_BLOCK "(LPAREN RPAREN)"
   %token <block>       BRACE_BLOCK "(LBRACE RBRACE)"
--- 49,55 ----
   ;; -----------------------------
   ;; Block & Parenthesis terminals
   ;; -----------------------------
! %type  <block>       ;;syntax "\\s(\\|\\s)" matchdatatype block

   %token <block>       PAREN_BLOCK "(LPAREN RPAREN)"
   %token <block>       BRACE_BLOCK "(LBRACE RBRACE)"
***************
*** 65,71 ****
   ;; ------------------
   ;; Operator terminals
   ;; ------------------
! %type  <punctuation> syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string

   %token <punctuation> NOT         "!"
   %token <punctuation> NOTEQ       "!="
--- 65,71 ----
   ;; ------------------
   ;; Operator terminals
   ;; ------------------
! %type  <punctuation> ;;syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string

   %token <punctuation> NOT         "!"
   %token <punctuation> NOTEQ       "!="
***************
*** 111,123 ****
   ;; -----------------
   ;; Literal terminals
   ;; -----------------
! %type  <symbol>      syntax "\\(\\sw\\|\\s_\\)+"
   %token <symbol>      IDENTIFIER

! %type  <string>      syntax "\\s\"" matchdatatype sexp
   %token <string>      STRING_LITERAL

! %type  <number>      syntax semantic-lex-number-expression
   %token <number>      NUMBER_LITERAL

   ;; -----------------
--- 111,123 ----
   ;; -----------------
   ;; Literal terminals
   ;; -----------------
! %type  <symbol>      ;;syntax "\\(\\sw\\|\\s_\\)+"
   %token <symbol>      IDENTIFIER

! %type  <string>      ;;syntax "\\s\"" matchdatatype sexp
   %token <string>      STRING_LITERAL

! %type  <number>      ;;syntax semantic-lex-number-expression
   %token <number>      NUMBER_LITERAL

   ;; -----------------
***************
*** 125,131 ****
   ;; -----------------

   ;; Generate a keyword analyzer
! %type  <keyword> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword

   %keyword ABSTRACT     "abstract"
   %put     ABSTRACT summary
--- 125,131 ----
   ;; -----------------

   ;; Generate a keyword analyzer
! %type  <keyword> ;;syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword

   %keyword ABSTRACT     "abstract"
   %put     ABSTRACT summary
Index: semantic/wisent/wisent-java.wy
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-java.wy,v
retrieving revision 1.14
diff -c -r1.14 wisent-java.wy
*** semantic/wisent/wisent-java.wy	16 Jan 2004 13:14:11 -0000	1.14
--- semantic/wisent/wisent-java.wy	18 Jan 2004 09:55:41 -0000
***************
*** 46,52 ****
   ;; -----------------------------
   ;; Block & Parenthesis terminals
   ;; -----------------------------
! %type  <block>       syntax "\\s(\\|\\s)" matchdatatype block

   %token <block>       PAREN_BLOCK "(LPAREN RPAREN)"
   %token <block>       BRACE_BLOCK "(LBRACE RBRACE)"
--- 46,52 ----
   ;; -----------------------------
   ;; Block & Parenthesis terminals
   ;; -----------------------------
! %type  <block>       ;;syntax "\\s(\\|\\s)" matchdatatype block

   %token <block>       PAREN_BLOCK "(LPAREN RPAREN)"
   %token <block>       BRACE_BLOCK "(LBRACE RBRACE)"
***************
*** 62,68 ****
   ;; ------------------
   ;; Operator terminals
   ;; ------------------
! %type  <punctuation> syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string

   %token <punctuation> NOT         "!"
   %token <punctuation> NOTEQ       "!="
--- 62,68 ----
   ;; ------------------
   ;; Operator terminals
   ;; ------------------
! %type  <punctuation> ;;syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string

   %token <punctuation> NOT         "!"
   %token <punctuation> NOTEQ       "!="
***************
*** 108,130 ****
   ;; -----------------
   ;; Literal terminals
   ;; -----------------
! %type  <symbol>      syntax "\\(\\sw\\|\\s_\\)+"
   %token <symbol>      NULL_LITERAL    "\\`null\\'"
   %token <symbol>      BOOLEAN_LITERAL "\\`false\\'"
   %token <symbol>      BOOLEAN_LITERAL "\\`true\\'"
   %token <symbol>      IDENTIFIER

! %type  <string>      syntax "\\s\"" matchdatatype sexp
   %token <string>      STRING_LITERAL

! %type  <number>      syntax semantic-lex-number-expression
   %token <number>      NUMBER_LITERAL

   ;; -----------------
   ;; Keyword terminals
   ;; -----------------
   ;; Generate a keyword analyzer
! %type  <keyword> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword

   %keyword ABSTRACT     "abstract"
   %put     ABSTRACT summary
--- 108,130 ----
   ;; -----------------
   ;; Literal terminals
   ;; -----------------
! %type  <symbol>      ;;syntax "\\(\\sw\\|\\s_\\)+"
   %token <symbol>      NULL_LITERAL    "\\`null\\'"
   %token <symbol>      BOOLEAN_LITERAL "\\`false\\'"
   %token <symbol>      BOOLEAN_LITERAL "\\`true\\'"
   %token <symbol>      IDENTIFIER

! %type  <string>      ;;syntax "\\s\"" matchdatatype sexp
   %token <string>      STRING_LITERAL

! %type  <number>      ;;syntax semantic-lex-number-expression
   %token <number>      NUMBER_LITERAL

   ;; -----------------
   ;; Keyword terminals
   ;; -----------------
   ;; Generate a keyword analyzer
! %type  <keyword> ;;syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword

   %keyword ABSTRACT     "abstract"
   %put     ABSTRACT summary

>>> David Ponce <david.ponce@...> seems to think that:
>Hi Eric,
>
>[...]
> > Yes, I can imagine just a few:
> >
> > %type <symbol>
> > %type <punctuation>
> >
> > to build a default analyzer.
>
>Following is a new patch that implements your suggestion above ;-)
>
>I also modified the Java grammars to use the default values of the
>well known token types <block>, <punctuation>, <symbol>, <string>,
><number> and <keyword>.
>
>So now, for "standard" cases it is very easy to auto-generate the
>basic type analyzers :-)
>
>Finally, I added an optional argument `force' to the command
>`semantic-grammar-create-package', to unconditionally re-generate
>Lisp code from the grammar.  It is very useful to test generation
>without modifying the grammar, by typing C-u C-c C-c.
>
>If you don't have objections, I will commit the changes.

That sounds greawt!  Please do.

  [ ... ]
>===================================================================
>RCS file: /cvsroot/cedet/cedet/semantic/semantic-grammar.el,v
>retrieving revision 1.52
>diff -c -r1.52 semantic-grammar.el
>*** semantic/semantic-grammar.el	16 Jan 2004 15:43:18 -0000	1.52
>--- semantic/semantic-grammar.el	18 Jan 2004 09:55:38 -0000
>***************
>*** 438,443 ****
>--- 438,445 ----
>     (let (type)
>       (dolist (tag (semantic-find-tags-by-class 'type (current-buffer)))
>         (setq type (semantic-tag-name tag))
>+       ;; Indicate to auto-generate the analyzer for this type
>+       (push (list type :declared t) props)
  [ ... ]

I saw this curiosity that wasn't used elsewhere.  What thoughts do
you have on this setting?  I can imagine a sort of "I compiled your
language and here is a status report." type thing.

That, of course, got me thinking that any messages the parser may spit
out (like the wisent messages about having to guess what to do) could
be output as comments in the output file.  Perhaps in the commentary.
Then such messages would never be lost.

Have fun
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Eric,

[...]
 >>If you don't have objections, I will commit the changes.
 >
 >
 > That sounds greawt!  Please do.

Done.

[...]
 >>+       ;; Indicate to auto-generate the analyzer for this type
 >>+       (push (list type :declared t) props)
 >
 >   [ ... ]
 >
 > I saw this curiosity that wasn't used elsewhere.  What thoughts do
 > you have on this setting?  I can imagine a sort of "I compiled your
 > language and here is a status report." type thing.

:declared is an internal property used to remember what types have
been declared explicitly using a %type statement.  Some types could
have been declared implicitly using %token <type> statements.

Then `semantic-grammar-insert-defanalyzer' will use the :declared
property to know which types have been declared with a %type
statement, and, for those only, to try to generate a lexical analyzer.

That simple mechanism preserve the compatibility of the new generator
with grammars that don't use [yet] %type statements to declare types
and to generate analyzers ;-)

 > That, of course, got me thinking that any messages the parser may spit
 > out (like the wisent messages about having to guess what to do) could
 > be output as comments in the output file.  Perhaps in the commentary.
 > Then such messages would never be lost.

I am sorry, but I don't understand clearly your idea here.  Could you
please elaborate or give an example?

Thanks!

David

>>> David Ponce <david.ponce@...> seems to think that:
> > That, of course, got me thinking that any messages the parser may spit
> > out (like the wisent messages about having to guess what to do) could
> > be output as comments in the output file.  Perhaps in the commentary.
> > Then such messages would never be lost.
>
>I am sorry, but I don't understand clearly your idea here.  Could you
>please elaborate or give an example?
  [ ... ]

When wisent compiles a grammar, sometimes it outputs info like this
from wisent-awk.wy:

--------
semacs -batch -q -l grammar-make-script -f semantic-grammar-batch-build-packages wisent-java-tags.wy wisent-awk.wy wisent-calc.wy wisent-cim.wy wisent-java.wy wisent-python.wy
Package `wisent-java-tags-wy' is up to date.
Loading vc-cvs...
*Warning* default value of <<no-type>> tokens changed to LEX_LENGTH, was LEX_BUILTIN
*Warning* default value of <<no-type>> tokens changed to LEX_BUILTIN, was DECREMENT
*Warning* default value of <<no-type>> tokens changed to DECREMENT, was INCREMENT
*Warning* default value of <<no-type>> tokens changed to INCREMENT, was LEX_OR

[ ... ]
Wrote /home/zappo/cedet/semantic/wisent/wisent-awk-wy.el
Grammar in wisent-awk-wy.el contains 65 shift/reduce conflicts
-------

These could be added as comments into the -wy.el file, though now I
seem to remember that some of them are generated after the file is
written?

Now that I've had time to think about it, I'm not sure what I was
contemplating is possible.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Eric,

I reached a new step in auto-generation of lexical analyzers.

As a demonstration, I adapted the Java tags grammar so that now the
lexer is completely auto-generated from the grammar ;-) I did some
quick tests and Java parsing seems to work quite well with the new
lexer!

After converting wisent-java.wy, I will be able to completely remove
wisent-java-lex.el.  And probably some specific predefined analyzers
in semantic-lex.el could be removed too ;-)

Here is a summary of the important changes I made:

- Fixed use of quote inside backquote expressions.

- Updated font-lock keywords for new define-lex-... macros.

- The above highlighting now also works for Lisp code in grammars.

- Defined two new `matchdatatype' generated analyzers:

   * A sexp analyzer, defined by `define-lex-sexp-type-analyzer', to
     handle tokens which are s-expressions anchored by a `syntax'
     regexp.  A trivial example is given by strings, anchored by
     "\\s"" and whose end is given by a "guarded" `forward-sexp'.

   * A keyword analyzer, defined by `define-lex-keyword-type-analyzer',
     to handle only language keywords.  That permit to separate
     analysis of true keywords from other symbols.

- <type> in the generated analyzer name is now enclosed between < > to
   improve readability.  For example:

   %package my-foo
   ...
   %type <symbol> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype regexp

   now generate an analyzer named `my-foo--<symbol>-regexp-analyzer'.

   Unfortunately, wisent-dot.el will need a minor update to continue to
   work. :-(

The change log and the patch follows.

David

	* cedet/semantic/semantic-fw.el

	(semantic-fw-font-lock-keywords): Add new macro names.

	* cedet/semantic/semantic-grammar.el

	(semantic-grammar-token-%type-properties): %type statements
	unconditionally declare new types.
	(semantic-grammar-token-properties): Updated.
	(semantic-grammar-quoted-form): New function.
	(semantic-grammar-insert-defanalyzer): Use it to fix use of quote
	in analyzer declarations. Enclose type between < > in generated
	analyzer name.  Fix conditions that select the type of analyzer to
	declare.  Add declaration of sexp and keyword analyzers.
	(semantic-grammar-mode-keywords-1): Add highlighting of semantic
	keywords.
	
	* cedet/semantic/semantic-lex.el

	(semantic-lex-make-type-table): Implicitly create types with
	properties but without tokens associated.
	(define-lex-keyword-type-analyzer)
	(define-lex-sexp-type-analyzer): New macros.

	* cedet/semantic/wisent/wisent-java-tags.el

	(require 'wisent-java-lex): Remove.
	
	* cedet/semantic/wisent/wisent-java-tags.wy

	Automatically generate analyzers used to build the lexer.
	
	(<block>, <punctuation>, <symbol>, <string>)
	(<number>, <keyword>): Add %type declaration.
	(epilogue): Add lexer declaration.
	
Index: semantic/semantic-fw.el
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-fw.el,v
retrieving revision 1.25
diff -c -r1.25 semantic-fw.el
*** semantic/semantic-fw.el	23 Dec 2003 02:07:45 -0000	1.25
--- semantic/semantic-fw.el	15 Jan 2004 16:15:13 -0000
***************
*** 605,610 ****
--- 605,615 ----
                     "define-lex-block-analyzer"
                     "define-lex-regex-analyzer"
                     "define-lex-simple-regex-analyzer"
+                   "define-lex-keyword-type-analyzer"
+                   "define-lex-sexp-type-analyzer"
+                   "define-lex-regex-type-analyzer"
+                   "define-lex-string-type-analyzer"
+                   "define-lex-block-type-analyzer"
                     "define-mode-overload-implementation"
                     "define-semantic-child-mode"
                     "define-overload"
Index: semantic/semantic-grammar.el
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-grammar.el,v
retrieving revision 1.49
diff -c -r1.49 semantic-grammar.el
*** semantic/semantic-grammar.el	14 Jan 2004 09:59:33 -0000	1.49
--- semantic/semantic-grammar.el	15 Jan 2004 16:15:14 -0000
***************
*** 430,449 ****
           (setcdr assoc (cons (cons term value) (cdr assoc)))))
       alist))

! (defun semantic-grammar-token-%type-properties (tokens &optional props)
!   "For types found in TOKENS, return properties set by %type statements.
   If optional argument PROPS is non-nil, it is an existing list of
   properties where to add new properties."
!   (let (found type)
       (dolist (tag (semantic-find-tags-by-class 'type (current-buffer)))
!       (setq type  (semantic-tag-name tag)
!             found (assoc type tokens))
!       (if (null found)
!           nil ;; %type <type> ignored, no token defined
!         (setq type (car found))
!         (dolist (e (semantic-tag-get-attribute tag :value))
!           (push (list type (intern (car e)) (read (or (cdr e) "nil")))
!                 props))))
       props))

   (defun semantic-grammar-token-%put-properties (tokens)
--- 430,446 ----
           (setcdr assoc (cons (cons term value) (cdr assoc)))))
       alist))

! (defun semantic-grammar-token-%type-properties (&optional props)
!   "Return properties set by %type statements.
! This declare a new type if necessary.
   If optional argument PROPS is non-nil, it is an existing list of
   properties where to add new properties."
!   (let (type)
       (dolist (tag (semantic-find-tags-by-class 'type (current-buffer)))
!       (setq type (semantic-tag-name tag))
!       (dolist (e (semantic-tag-get-attribute tag :value))
!         (push (list type (intern (car e)) (read (or (cdr e) "nil")))
!               props)))
       props))

   (defun semantic-grammar-token-%put-properties (tokens)
***************
*** 462,473 ****
       props))

   (defsubst semantic-grammar-token-properties (tokens)
!   "Return properties of types found in TOKENS.
! That is properties set to any <type> by %put and %type statements.
   Properties set by %type statements take precedence over those set by
   %put statements."
     (let ((props (semantic-grammar-token-%put-properties tokens)))
!     (semantic-grammar-token-%type-properties tokens props)))

   (defun semantic-grammar-use-macros ()
     "Return macro definitions from %use-macros statements.
--- 459,472 ----
       props))

   (defsubst semantic-grammar-token-properties (tokens)
!   "Return properties of declared types.
! Types are explicitly declared by %type statements.  Types found in
! TOKENS are those declared implicitly by %token statements.
! Properties can be set by %put and %type statements.
   Properties set by %type statements take precedence over those set by
   %put statements."
     (let ((props (semantic-grammar-token-%put-properties tokens)))
!     (semantic-grammar-token-%type-properties props)))

   (defun semantic-grammar-use-macros ()
     "Return macro definitions from %use-macros statements.
***************
*** 785,790 ****
--- 784,796 ----
      (t
       semantic-grammar--lex-block-specs)))

+ (defsubst semantic-grammar-quoted-form (exp)
+   "Return a quoted form of EXP if it isn't a self evaluating form."
+   (if (and (not (null exp))
+            (or (listp exp) (symbolp exp)))
+       (list 'quote exp)
+     exp))
+
   (defun semantic-grammar-insert-defanalyzer (type)
     "Insert declaration of the lexical analyzer defined with TYPE."
     (let* ((type-name  (symbol-name type))
***************
*** 798,820 ****
                       (semantic-grammar-buffer-file
                        semantic--grammar-output-buffer))
               mtype (or (get type 'matchdatatype) 'regexp)
!             name (intern (format "%s--%s-%s-analyzer" prefix type mtype))
               doc (format "%s analyzer for <%s> tokens." mtype type))
         (cond
          ;; Regexp match analyzer
!        ((and (eq mtype 'regexp)
!              (setq spec (cdr type-value)))
           (semantic-grammar-insert-define
            `(define-lex-regex-type-analyzer ,name
!             ,doc ',syntax ',spec
               ',(or (car type-value) (intern type-name))))
           )
          ;; String compare analyzer
!        ((and (eq mtype 'string)
!              (setq spec (cdr type-value)))
           (semantic-grammar-insert-define
            `(define-lex-string-type-analyzer ,name
!             ,doc ',syntax ',spec
               ',(or (car type-value) (intern type-name))))
           )
          ;; Block analyzer
--- 804,825 ----
                       (semantic-grammar-buffer-file
                        semantic--grammar-output-buffer))
               mtype (or (get type 'matchdatatype) 'regexp)
!             name (intern (format "%s--<%s>-%s-analyzer" prefix type mtype))
               doc (format "%s analyzer for <%s> tokens." mtype type))
         (cond
          ;; Regexp match analyzer
!        ((eq mtype 'regexp)
           (semantic-grammar-insert-define
            `(define-lex-regex-type-analyzer ,name
!             ,doc ,syntax ,(cdr type-value)
               ',(or (car type-value) (intern type-name))))
           )
          ;; String compare analyzer
!        ((eq mtype 'string)
           (semantic-grammar-insert-define
            `(define-lex-string-type-analyzer ,name
!             ,doc ,syntax
!             ,(semantic-grammar-quoted-form (cdr type-value))
               ',(or (car type-value) (intern type-name))))
           )
          ;; Block analyzer
***************
*** 822,829 ****
                (setq spec (semantic-grammar--lex-block-specs)))
           (semantic-grammar-insert-define
            `(define-lex-block-type-analyzer ,name
!             ,doc ',syntax ',spec))
!         )))
       ))

   (defun semantic-grammar-insert-defanalyzers ()
--- 827,849 ----
                (setq spec (semantic-grammar--lex-block-specs)))
           (semantic-grammar-insert-define
            `(define-lex-block-type-analyzer ,name
!             ,doc ,syntax
!             ,(semantic-grammar-quoted-form spec)))
!         )
!        ;; Sexp analyzer
!        ((eq mtype 'sexp)
!         (semantic-grammar-insert-define
!          `(define-lex-sexp-type-analyzer ,name
!             ,doc ,syntax
!             ',(or (car type-value) (intern type-name))))
!         )
!        ;; keyword analyzer
!        ((eq mtype 'keyword)
!         (semantic-grammar-insert-define
!          `(define-lex-keyword-type-analyzer ,name
!             ,doc ,syntax))
!         )
!        ))
       ))

   (defun semantic-grammar-insert-defanalyzers ()
***************
*** 1152,1157 ****
--- 1172,1179 ----
       ;; grammar mode!
       ("[\r\n\t ]+:\\sw+\\>"
        0 font-lock-builtin-face)
+     ;; Append the Semantic keywords
+     ,@semantic-fw-font-lock-keywords
       )
     "Font Lock keywords used to highlight Semantic grammar buffers.")

Index: semantic/semantic-lex.el
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-lex.el,v
retrieving revision 1.29
diff -c -r1.29 semantic-lex.el
*** semantic/semantic-lex.el	15 Jan 2004 08:06:20 -0000	1.29
--- semantic/semantic-lex.el	15 Jan 2004 16:15:16 -0000
***************
*** 259,265 ****
       (while propspecs
         (setq spec (car propspecs)
               propspecs (cdr propspecs))
!       (semantic-lex-type-put (car spec) (nth 1 spec) (nth 2 spec)))
       semantic-lex-types-obarray))

   (defsubst semantic-lex-map-types (fun &optional property)
--- 259,266 ----
       (while propspecs
         (setq spec (car propspecs)
               propspecs (cdr propspecs))
!       ;; Create the type if necessary.
!       (semantic-lex-type-put (car spec) (nth 1 spec) (nth 2 spec) t))
       semantic-lex-types-obarray))

   (defsubst semantic-lex-map-types (fun &optional property)
***************
*** 1203,1208 ****
--- 1204,1239 ----
   
   ;;; Analyzers generated from grammar.
   ;;
+ (defmacro define-lex-keyword-type-analyzer (name doc syntax)
+   "Define a keyword type analyzer NAME with DOC string.
+ SYNTAX is the regexp that matches a keyword syntactic expression."
+   (let ((key (make-symbol "key")))
+     `(define-lex-analyzer ,name
+        ,doc
+        (and (looking-at ,syntax)
+             (let ((,key (semantic-lex-keyword-p (match-string 0))))
+               (when ,key
+                 (semantic-lex-push-token
+                  (semantic-lex-token
+                   ,key (match-beginning 0) (match-end 0)))))))
+     ))
+
+ (defmacro define-lex-sexp-type-analyzer (name doc syntax token)
+   "Define a sexp type analyzer NAME with DOC string.
+ SYNTAX is the regexp that matches the beginning of the s-expression.
+ TOKEN is the lexical token returned when SYNTAX matches."
+   `(define-lex-regex-analyzer ,name
+      ,doc
+      ,syntax
+      (semantic-lex-push-token
+       (semantic-lex-token
+        ,token (point)
+        (save-excursion
+          (semantic-lex-unterminated-syntax-protection ,token
+            (forward-sexp 1)
+            (point))))))
+   )
+
   (defmacro define-lex-regex-type-analyzer (name doc syntax matches default)
     "Define a regexp type analyzer NAME with DOC string.
   SYNTAX is the regexp that matches a syntactic expression.
Index: semantic/wisent/wisent-java-tags.el
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-java-tags.el,v
retrieving revision 1.27
diff -c -r1.27 wisent-java-tags.el
*** semantic/wisent/wisent-java-tags.el	5 Sep 2003 10:33:15 -0000	1.27
--- semantic/wisent/wisent-java-tags.el	15 Jan 2004 16:15:18 -0000
***************
*** 35,41 ****

   (require 'wisent-bovine)
   (require 'wisent-java-tags-wy)
! (require 'wisent-java-lex)
   (require 'semantic-java)
   (eval-when-compile
     (require 'semantic-util)
--- 35,41 ----

   (require 'wisent-bovine)
   (require 'wisent-java-tags-wy)
! ;;(require 'wisent-java-lex)
   (require 'semantic-java)
   (eval-when-compile
     (require 'semantic-util)
Index: semantic/wisent/wisent-java-tags.wy
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-java-tags.wy,v
retrieving revision 1.11
diff -c -r1.11 wisent-java-tags.wy
*** semantic/wisent/wisent-java-tags.wy	5 Dec 2003 09:18:58 -0000	1.11
--- semantic/wisent/wisent-java-tags.wy	15 Jan 2004 16:15:18 -0000
***************
*** 46,305 ****
   %start interface_member_declaration
   %start formal_parameters

! ;; ---------------------
! ;; Parenthesis terminals
! ;; ---------------------
! %token <open-paren>   LPAREN           "("
! %token <close-paren>  RPAREN           ")"
! %token <open-paren>   LBRACE           "{"
! %token <close-paren>  RBRACE           "}"
! %token <open-paren>   LBRACK           "["
! %token <close-paren>  RBRACK           "]"
!
! ;; ---------------
! ;; Block terminals
! ;; ---------------
! %token <block>        PAREN_BLOCK      "(LPAREN RPAREN)"
! %token <block>        BRACE_BLOCK      "(LBRACE RBRACE)"
! %token <block>        BRACK_BLOCK      "(LBRACK RBRACK)"

   ;; ------------------
   ;; Operator terminals
   ;; ------------------
! %token <punctuation>   NOT             "!"
! %token <punctuation>   NOTEQ           "!="
! %token <punctuation>   MOD             "%"
! %token <punctuation>   MODEQ           "%="
! %token <punctuation>   AND             "&"
! %token <punctuation>   ANDAND          "&&"
! %token <punctuation>   ANDEQ           "&="
! %token <punctuation>   MULT            "*"
! %token <punctuation>   MULTEQ          "*="
! %token <punctuation>   PLUS            "+"
! %token <punctuation>   PLUSPLUS        "++"
! %token <punctuation>   PLUSEQ          "+="
! %token <punctuation>   COMMA           ","
! %token <punctuation>   MINUS           "-"
! %token <punctuation>   MINUSMINUS      "--"
! %token <punctuation>   MINUSEQ         "-="
! %token <punctuation>   DOT             "."
! %token <punctuation>   DIV             "/"
! %token <punctuation>   DIVEQ           "/="
! %token <punctuation>   COLON           ":"
! %token <punctuation>   SEMICOLON       ";"
! %token <punctuation>   LT              "<"
! %token <punctuation>   LSHIFT          "<<"
! %token <punctuation>   LSHIFTEQ        "<<="
! %token <punctuation>   LTEQ            "<="
! %token <punctuation>   EQ              "="
! %token <punctuation>   EQEQ            "=="
! %token <punctuation>   GT              ">"
! %token <punctuation>   GTEQ            ">="
! %token <punctuation>   RSHIFT          ">>"
! %token <punctuation>   RSHIFTEQ        ">>="
! %token <punctuation>   URSHIFT         ">>>"
! %token <punctuation>   URSHIFTEQ       ">>>="
! %token <punctuation>   QUESTION        "?"
! %token <punctuation>   XOR             "^"
! %token <punctuation>   XOREQ           "^="
! %token <punctuation>   OR              "|"
! %token <punctuation>   OREQ            "|="
! %token <punctuation>   OROR            "||"
! %token <punctuation>   COMP            "~"

   ;; -----------------
   ;; Literal terminals
   ;; -----------------
   %token <symbol>      IDENTIFIER
   %token <string>      STRING_LITERAL
   %token <number>      NUMBER_LITERAL

   ;; -----------------
   ;; Keyword terminals
   ;; -----------------
! %token ABSTRACT     "abstract"
! %put ABSTRACT summary
   "Class|Method declaration modifier: abstract {class|<type>} <name> ..."

! %token BOOLEAN      "boolean"
! %put BOOLEAN summary
   "Primitive logical quantity type (true or false)"

! %token BREAK        "break"
! %put BREAK summary
   "break [<label>] ;"

! %token BYTE         "byte"
! %put BYTE summary
   "Integral primitive type (-128 to 127)"

! %token CASE         "case"
! %put CASE summary
   "switch(<expr>) {case <const-expr>: <stmts> ... }"

! %token CATCH        "catch"
! %put CATCH summary
   "try {<stmts>} catch(<parm>) {<stmts>} ... "

! %token CHAR         "char"
! %put CHAR summary
   "Integral primitive type ('\u0000' to '\uffff') (0 to 65535)"

! %token CLASS        "class"
! %put CLASS summary
   "Class declaration: class <name>"

! %token CONST        "const"
! %put CONST summary
   "Unused reserved word"

! %token CONTINUE     "continue"
! %put CONTINUE summary
   "continue [<label>] ;"

! %token DEFAULT      "default"
! %put DEFAULT summary
   "switch(<expr>) { ... default: <stmts>}"

! %token DO           "do"
! %put DO summary
   "do <stmt> while (<expr>);"

! %token DOUBLE       "double"
! %put DOUBLE summary
   "Primitive floating-point type (double-precision 64-bit IEEE 754)"

! %token ELSE         "else"
! %put ELSE summary
   "if (<expr>) <stmt> else <stmt>"

! %token EXTENDS      "extends"
! %put EXTENDS summary
   "SuperClass|SuperInterfaces declaration: extends <name> [, ...]"

! %token FINAL        "final"
! %put FINAL summary
   "Class|Member declaration modifier: final {class|<type>} <name> ..."

! %token FINALLY      "finally"
! %put FINALLY summary
   "try {<stmts>} ... finally {<stmts>}"

! %token FLOAT        "float"
! %put FLOAT summary
   "Primitive floating-point type (single-precision 32-bit IEEE 754)"

! %token FOR          "for"
! %put FOR summary
   "for ([<init-expr>]; [<expr>]; [<update-expr>]) <stmt>"

! %token GOTO         "goto"
! %put GOTO summary
   "Unused reserved word"

! %token IF           "if"
! %put IF summary
   "if (<expr>) <stmt> [else <stmt>]"

! %token IMPLEMENTS   "implements"
! %put IMPLEMENTS summary
   "Class SuperInterfaces declaration: implements <name> [, ...]"

! %token IMPORT       "import"
! %put IMPORT summary
   "Import package declarations: import <package>"

! %token INSTANCEOF   "instanceof"

! %token INT          "int"
! %put INT summary
   "Integral primitive type (-2147483648 to 2147483647)"

! %token INTERFACE    "interface"
! %put INTERFACE summary
   "Interface declaration: interface <name>"

! %token LONG         "long"
! %put LONG summary
   "Integral primitive type (-9223372036854775808 to 9223372036854775807)"

! %token NATIVE       "native"
! %put NATIVE summary
   "Method declaration modifier: native <type> <name> ..."

! %token NEW          "new"

! %token PACKAGE      "package"
! %put PACKAGE summary
   "Package declaration: package <name>"

! %token PRIVATE      "private"
! %put PRIVATE summary
   "Access level modifier: private {class|interface|<type>} <name> ..."

! %token PROTECTED    "protected"
! %put PROTECTED summary
   "Access level modifier: protected {class|interface|<type>} <name> ..."

! %token PUBLIC       "public"
! %put PUBLIC summary
   "Access level modifier: public {class|interface|<type>} <name> ..."

! %token RETURN       "return"
! %put RETURN summary
   "return [<expr>] ;"

! %token SHORT        "short"
! %put SHORT summary
   "Integral primitive type (-32768 to 32767)"

! %token STATIC       "static"
! %put STATIC summary
   "Declaration modifier: static {class|interface|<type>} <name> ..."

! %token STRICTFP     "strictfp"
! %put STRICTFP summary
   "Declaration modifier: strictfp {class|interface|<type>} <name> ..."

! %token SUPER        "super"

! %token SWITCH       "switch"
! %put SWITCH summary
   "switch(<expr>) {[case <const-expr>: <stmts> ...] [default: <stmts>]}"


! %token SYNCHRONIZED "synchronized"
! %put SYNCHRONIZED summary
   "synchronized (<expr>) ... | Method decl. modifier: synchronized <type> <name> ..."

! %token THIS         "this"

! %token THROW        "throw"
! %put THROW summary
   "throw <expr> ;"

! %token THROWS       "throws"
! %put THROWS summary
   "Method|Constructor declaration: throws <classType>, ..."

! %token TRANSIENT    "transient"
! %put TRANSIENT summary
   "Field declaration modifier: transient <type> <name> ..."

! %token TRY          "try"
! %put TRY summary
   "try {<stmts>} [catch(<parm>) {<stmts>} ...] [finally {<stmts>}]"

! %token VOID         "void"
! %put VOID summary
   "Method return type: void <name> ..."

! %token VOLATILE     "volatile"
! %put VOLATILE summary
   "Field declaration modifier: volatile <type> <name> ..."

! %token WHILE        "while"
! %put WHILE summary
   "while (<expr>) <stmt> | do <stmt> while (<expr>);"

   ;; --------------------------
--- 46,315 ----
   %start interface_member_declaration
   %start formal_parameters

! ;; -----------------------------
! ;; Block & Parenthesis terminals
! ;; -----------------------------
! %type  <block>       syntax "\\s(\\|\\s)" matchdatatype block
!
! %token <block>       PAREN_BLOCK "(LPAREN RPAREN)"
! %token <block>       BRACE_BLOCK "(LBRACE RBRACE)"
! %token <block>       BRACK_BLOCK "(LBRACK RBRACK)"
!
! %token <open-paren>  LPAREN      "("
! %token <close-paren> RPAREN      ")"
! %token <open-paren>  LBRACE      "{"
! %token <close-paren> RBRACE      "}"
! %token <open-paren>  LBRACK      "["
! %token <close-paren> RBRACK      "]"

   ;; ------------------
   ;; Operator terminals
   ;; ------------------
! %type  <punctuation> syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string
!
! %token <punctuation> NOT         "!"
! %token <punctuation> NOTEQ       "!="
! %token <punctuation> MOD         "%"
! %token <punctuation> MODEQ       "%="
! %token <punctuation> AND         "&"
! %token <punctuation> ANDAND      "&&"
! %token <punctuation> ANDEQ       "&="
! %token <punctuation> MULT        "*"
! %token <punctuation> MULTEQ      "*="
! %token <punctuation> PLUS        "+"
! %token <punctuation> PLUSPLUS    "++"
! %token <punctuation> PLUSEQ      "+="
! %token <punctuation> COMMA       ","
! %token <punctuation> MINUS       "-"
! %token <punctuation> MINUSMINUS  "--"
! %token <punctuation> MINUSEQ     "-="
! %token <punctuation> DOT         "."
! %token <punctuation> DIV         "/"
! %token <punctuation> DIVEQ       "/="
! %token <punctuation> COLON       ":"
! %token <punctuation> SEMICOLON   ";"
! %token <punctuation> LT          "<"
! %token <punctuation> LSHIFT      "<<"
! %token <punctuation> LSHIFTEQ    "<<="
! %token <punctuation> LTEQ        "<="
! %token <punctuation> EQ          "="
! %token <punctuation> EQEQ        "=="
! %token <punctuation> GT          ">"
! %token <punctuation> GTEQ        ">="
! %token <punctuation> RSHIFT      ">>"
! %token <punctuation> RSHIFTEQ    ">>="
! %token <punctuation> URSHIFT     ">>>"
! %token <punctuation> URSHIFTEQ   ">>>="
! %token <punctuation> QUESTION    "?"
! %token <punctuation> XOR         "^"
! %token <punctuation> XOREQ       "^="
! %token <punctuation> OR          "|"
! %token <punctuation> OREQ        "|="
! %token <punctuation> OROR        "||"
! %token <punctuation> COMP        "~"

   ;; -----------------
   ;; Literal terminals
   ;; -----------------
+ %type  <symbol>      syntax "\\(\\sw\\|\\s_\\)+"
   %token <symbol>      IDENTIFIER
+
+ %type  <string>      syntax "\\s\"" matchdatatype sexp
   %token <string>      STRING_LITERAL
+
+ %type  <number>      syntax semantic-lex-number-expression
   %token <number>      NUMBER_LITERAL

   ;; -----------------
   ;; Keyword terminals
   ;; -----------------
!
! ;; Generate a keyword analyzer
! %type  <keyword> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword
!
! %keyword ABSTRACT     "abstract"
! %put     ABSTRACT summary
   "Class|Method declaration modifier: abstract {class|<type>} <name> ..."

! %keyword BOOLEAN      "boolean"
! %put     BOOLEAN summary
   "Primitive logical quantity type (true or false)"

! %keyword BREAK        "break"
! %put     BREAK summary
   "break [<label>] ;"

! %keyword BYTE         "byte"
! %put     BYTE summary
   "Integral primitive type (-128 to 127)"

! %keyword CASE         "case"
! %put     CASE summary
   "switch(<expr>) {case <const-expr>: <stmts> ... }"

! %keyword CATCH        "catch"
! %put     CATCH summary
   "try {<stmts>} catch(<parm>) {<stmts>} ... "

! %keyword CHAR         "char"
! %put     CHAR summary
   "Integral primitive type ('\u0000' to '\uffff') (0 to 65535)"

! %keyword CLASS        "class"
! %put     CLASS summary
   "Class declaration: class <name>"

! %keyword CONST        "const"
! %put     CONST summary
   "Unused reserved word"

! %keyword CONTINUE     "continue"
! %put     CONTINUE summary
   "continue [<label>] ;"

! %keyword DEFAULT      "default"
! %put     DEFAULT summary
   "switch(<expr>) { ... default: <stmts>}"

! %keyword DO           "do"
! %put     DO summary
   "do <stmt> while (<expr>);"

! %keyword DOUBLE       "double"
! %put     DOUBLE summary
   "Primitive floating-point type (double-precision 64-bit IEEE 754)"

! %keyword ELSE         "else"
! %put     ELSE summary
   "if (<expr>) <stmt> else <stmt>"

! %keyword EXTENDS      "extends"
! %put     EXTENDS summary
   "SuperClass|SuperInterfaces declaration: extends <name> [, ...]"

! %keyword FINAL        "final"
! %put     FINAL summary
   "Class|Member declaration modifier: final {class|<type>} <name> ..."

! %keyword FINALLY      "finally"
! %put     FINALLY summary
   "try {<stmts>} ... finally {<stmts>}"

! %keyword FLOAT        "float"
! %put     FLOAT summary
   "Primitive floating-point type (single-precision 32-bit IEEE 754)"

! %keyword FOR          "for"
! %put     FOR summary
   "for ([<init-expr>]; [<expr>]; [<update-expr>]) <stmt>"

! %keyword GOTO         "goto"
! %put     GOTO summary
   "Unused reserved word"

! %keyword IF           "if"
! %put     IF summary
   "if (<expr>) <stmt> [else <stmt>]"

! %keyword IMPLEMENTS   "implements"
! %put     IMPLEMENTS summary
   "Class SuperInterfaces declaration: implements <name> [, ...]"

! %keyword IMPORT       "import"
! %put     IMPORT summary
   "Import package declarations: import <package>"

! %keyword INSTANCEOF   "instanceof"

! %keyword INT          "int"
! %put     INT summary
   "Integral primitive type (-2147483648 to 2147483647)"

! %keyword INTERFACE    "interface"
! %put     INTERFACE summary
   "Interface declaration: interface <name>"

! %keyword LONG         "long"
! %put     LONG summary
   "Integral primitive type (-9223372036854775808 to 9223372036854775807)"

! %keyword NATIVE       "native"
! %put     NATIVE summary
   "Method declaration modifier: native <type> <name> ..."

! %keyword NEW          "new"

! %keyword PACKAGE      "package"
! %put     PACKAGE summary
   "Package declaration: package <name>"

! %keyword PRIVATE      "private"
! %put     PRIVATE summary
   "Access level modifier: private {class|interface|<type>} <name> ..."

! %keyword PROTECTED    "protected"
! %put     PROTECTED summary
   "Access level modifier: protected {class|interface|<type>} <name> ..."

! %keyword PUBLIC       "public"
! %put     PUBLIC summary
   "Access level modifier: public {class|interface|<type>} <name> ..."

! %keyword RETURN       "return"
! %put     RETURN summary
   "return [<expr>] ;"

! %keyword SHORT        "short"
! %put     SHORT summary
   "Integral primitive type (-32768 to 32767)"

! %keyword STATIC       "static"
! %put     STATIC summary
   "Declaration modifier: static {class|interface|<type>} <name> ..."

! %keyword STRICTFP     "strictfp"
! %put     STRICTFP summary
   "Declaration modifier: strictfp {class|interface|<type>} <name> ..."

! %keyword SUPER        "super"

! %keyword SWITCH       "switch"
! %put     SWITCH summary
   "switch(<expr>) {[case <const-expr>: <stmts> ...] [default: <stmts>]}"


! %keyword SYNCHRONIZED "synchronized"
! %put     SYNCHRONIZED summary
   "synchronized (<expr>) ... | Method decl. modifier: synchronized <type> <name> ..."

! %keyword THIS         "this"

! %keyword THROW        "throw"
! %put     THROW summary
   "throw <expr> ;"

! %keyword THROWS       "throws"
! %put     THROWS summary
   "Method|Constructor declaration: throws <classType>, ..."

! %keyword TRANSIENT    "transient"
! %put     TRANSIENT summary
   "Field declaration modifier: transient <type> <name> ..."

! %keyword TRY          "try"
! %put     TRY summary
   "try {<stmts>} [catch(<parm>) {<stmts>} ...] [finally {<stmts>}]"

! %keyword VOID         "void"
! %put     VOID summary
   "Method return type: void <name> ..."

! %keyword VOLATILE     "volatile"
! %put     VOLATILE summary
   "Field declaration modifier: volatile <type> <name> ..."

! %keyword WHILE        "while"
! %put     WHILE summary
   "while (<expr>) <stmt> | do <stmt> while (<expr>);"

   ;; --------------------------
***************
*** 326,355 ****
   ;; - `with-ref' (optional) if non-nil indicates that the tag is
   ;;   followed by a reference like in "@see <reference>".

! %token _AUTHOR      "@author"
! %put   _AUTHOR      javadoc (seq 1 usage (type))
! %token _VERSION     "@version"
! %put   _VERSION     javadoc (seq 2 usage (type))
! %token _PARAM       "@param"
! %put   _PARAM       javadoc (seq 3 usage (function) with-name t)
! %token _RETURN      "@return"
! %put   _RETURN      javadoc (seq 4 usage (function))
! %token _EXCEPTION   "@exception"
! %put   _EXCEPTION   javadoc (seq 5 usage (function) with-name t)
! %token _THROWS      "@throws"
! %put   _THROWS      javadoc (seq 6 usage (function) with-name t)
! %token _SEE         "@see"
! %put   _SEE         javadoc (seq 7 usage (type function variable) opt t with-ref t)
! %token _SINCE       "@since"
! %put   _SINCE       javadoc (seq 8 usage (type function variable) opt t)
! %token _SERIAL      "@serial"
! %put   _SERIAL      javadoc (seq 9 usage (variable) opt t)
! %token _SERIALDATA  "@serialData"
! %put   _SERIALDATA  javadoc (seq 10 usage (function) opt t)
! %token _SERIALFIELD "@serialField"
! %put   _SERIALFIELD javadoc (seq 11 usage (variable) opt t)
! %token _DEPRECATED  "@deprecated"
! %put   _DEPRECATED  javadoc (seq 12 usage (type function variable) opt t)

   %%

--- 336,365 ----
   ;; - `with-ref' (optional) if non-nil indicates that the tag is
   ;;   followed by a reference like in "@see <reference>".

! %keyword _AUTHOR      "@author"
! %put     _AUTHOR      javadoc (seq 1 usage (type))
! %keyword _VERSION     "@version"
! %put     _VERSION     javadoc (seq 2 usage (type))
! %keyword _PARAM       "@param"
! %put     _PARAM       javadoc (seq 3 usage (function) with-name t)
! %keyword _RETURN      "@return"
! %put     _RETURN      javadoc (seq 4 usage (function))
! %keyword _EXCEPTION   "@exception"
! %put     _EXCEPTION   javadoc (seq 5 usage (function) with-name t)
! %keyword _THROWS      "@throws"
! %put     _THROWS      javadoc (seq 6 usage (function) with-name t)
! %keyword _SEE         "@see"
! %put     _SEE         javadoc (seq 7 usage (type function variable) opt t with-ref t)
! %keyword _SINCE       "@since"
! %put     _SINCE       javadoc (seq 8 usage (type function variable) opt t)
! %keyword _SERIAL      "@serial"
! %put     _SERIAL      javadoc (seq 9 usage (variable) opt t)
! %keyword _SERIALDATA  "@serialData"
! %put     _SERIALDATA  javadoc (seq 10 usage (function) opt t)
! %keyword _SERIALFIELD "@serialField"
! %put     _SERIALFIELD javadoc (seq 11 usage (variable) opt t)
! %keyword _DEPRECATED  "@deprecated"
! %put     _DEPRECATED  javadoc (seq 12 usage (type function variable) opt t)

   %%

***************
*** 707,710 ****
--- 717,740 ----
       (identity "[]")
     ;

+ %%
+
+ ;; Define the lexer for this grammar
+ (define-lex wisent-java-tags-lexer
+   "Lexical analyzer that handles Java buffers.
+ It ignores whitespaces, newlines and comments."
+   semantic-lex-ignore-whitespace
+   semantic-lex-ignore-newline
+   semantic-lex-ignore-comments
+   ;;;; Auto-generated analyzers.
+   wisent-java-tags-wy--<number>-regexp-analyzer
+   wisent-java-tags-wy--<string>-sexp-analyzer
+   ;; Must detect keywords before other symbols
+   wisent-java-tags-wy--<keyword>-keyword-analyzer
+   wisent-java-tags-wy--<symbol>-regexp-analyzer
+   wisent-java-tags-wy--<punctuation>-string-analyzer
+   wisent-java-tags-wy--<block>-block-analyzer
+   ;;;;
+   semantic-lex-default-action)
+
   ;;; wisent-java-tags.wy ends here

>>> David Ponce <david.ponce@...> seems to think that:
>Hi Eric,
>
>I reached a new step in auto-generation of lexical analyzers.
>
>As a demonstration, I adapted the Java tags grammar so that now the
>lexer is completely auto-generated from the grammar ;-) I did some
>quick tests and Java parsing seems to work quite well with the new
>lexer!
>
>After converting wisent-java.wy, I will be able to completely remove
>wisent-java-lex.el.  And probably some specific predefined analyzers
>in semantic-lex.el could be removed too ;-)

Wow, that is pretty impressive.  I like the idea of having everything
together in one file that explains how to parse a file.

Don't you still need wisent-java-lex.el for the other java (non tag)
parser?

>Here is a summary of the important changes I made:
>
>- Fixed use of quote inside backquote expressions.
>
>- Updated font-lock keywords for new define-lex-... macros.
>
>- The above highlighting now also works for Lisp code in grammars.
>
>- Defined two new `matchdatatype' generated analyzers:
>
>   * A sexp analyzer, defined by `define-lex-sexp-type-analyzer', to
>     handle tokens which are s-expressions anchored by a `syntax'
>     regexp.  A trivial example is given by strings, anchored by
>     "\\s"" and whose end is given by a "guarded" `forward-sexp'.

Very nice.

It might not be much of a stretch for this to imply any kind of
matching, not just those available in the syntax table?  Like BEGIN ->
END in Ada, for example.

That's a project for another day... far in the future. ;)

>   * A keyword analyzer, defined by `define-lex-keyword-type-analyzer',
>     to handle only language keywords.  That permit to separate
>     analysis of true keywords from other symbols.

Does this mean a separate entry in the lexer construction?
ie. 1 call to find 'keywords' and a second to find 'symbol' %types?
That could slow things down some.

>- <type> in the generated analyzer name is now enclosed between < > to
>   improve readability.  For example:
>
>   %package my-foo
>   ...
>   %type <symbol> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype regexp

This entry is beginning to strike me as being a bit long.
Technically, the word "syntax" and "matchdatatype" are sugar, meaning
they add anchors for the developer to know what is coming.

Are sections of this optional?  eg:

%type <symbol> syntax "blahblah"

implies a matchdatatype of regexp

and 

%type <symbol> matchdatatype regexp

implies some sort of default syntax regular expression?

I guess just 

%type <symbol>

could mean something if there are defaults.

>   now generate an analyzer named `my-foo--<symbol>-regexp-analyzer'.
>
>   Unfortunately, wisent-dot.el will need a minor update to continue to
>   work. :-(

That's fine.  I'd probably adapt it to use all your new stuff anyway. ;)
I like the new representation as it says a lot more with the <> around
it.

Thanks!
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

>>> David PONCE <david.ponce@...> seems to think that:
>Hi Eric,
>
>
>>   This is great.  If as you say, the change don't effect existing
>> grammars, I recommend checking it in.  I may not get a chance to play
>> with it much until the weekend.
>
>OK.  I committed the changes.

Thanks.

>[...]
>>>I attached a sample grammar gr-test.wy that you can use to experiment
>>>lexical analyzer generation ;-)
>> 
>> I did not find this file, but it may be wise to check it into the
>> 'test' directory so it can be regularly tested along with the other
>> items in there.
>
>Oops!  I forgot to attach it.  Sorry.  I checked it into the test
>directory.

I see it now.  I will look at it too.

>[...]
>>>! (defun semantic-grammar-types-properties (tokens &optional props)
>>>!   "Return the list of type properties for lexical tokens TOKENS.
>>>! If optional argument PROPS is non-nil, it is an existing list of
>>>! properties where to add type properties."
>> 
>> 
>> I'm having trouble following some of your more advanced code in here.
>> In this function, and several others, I am confused by the names
>> again.
>> 
>> Since % is a valid character for symbols in Emacs Lisp, perhaps
>> functions and variable that refer to a "token" or "tokens" that
>> represent output from %token or %type, or %start could use "%token"
>> to distinguish from what the lexer produces which are also tokens.
>> 
>> This is particularly important since you are referencing special
>> tokens which are really tags (in our parser speak) that describe
>> tokens from a lexer.
>> 
>> Unfortunately, that implies renaming vast amounts in this file since
>> most commands that reference any %keyword type thing would need a new
>> name to be consistent. :(
>[...]
>
>I created two new functions: `semantic-grammar-token-%put-properties'
>and `semantic-grammar-token-%type-properties' which, I hope, are
>better named.  I also wrote clearer doc strings.  For compatibility
>reason, I kept `semantic-grammar-token-properties' which now just
>call the two above functions.  I also rewrote its doc string to be
>clearer.

Thanks. When you feel you have reached a final stage on the code, it
would be good to add it to the doc.  While I have poked through many
sections recently, I have not been through the lex portion.

>> Unfortunately I don't really have time for a detailed look today.
>> If you check it in, I will use it to port my dot grammar to see how
>> it works.  That would be a good excuse to test my new idle completion
>> tool in a .wy file too. ;)
>
>The new idle scheduler seems to work fine.  It is a great idea and
>addition to Semantic ;-)  Maybe in the future, it could become a CEDET
>common tool.

That is a good idea I hadn't thought of.  The most important feature
of the idle scheduler is that it synchronizes the tag table in the
current buffer in a special way.  It is really geared toward semantic
in that way.

If you write a tool that does not use semantic tags, using and idle
scheduler yourself is probably just as easy.

>The only minor (but a little annoying problem) I encountered with the
>idle scheduler re-parse feature, is that sometimes re-parse seems to
>have been disabled, and is no more scheduled.  The only way I found to
>re-enable it, is to disable then enable the minor mode.  Unfortunately
>I didn't had time enough to track that problem down, and I don't have
>more information to give you for now.  Perhaps have you some idea on
>what happens?
  [ ... ]

Yes, I noticed that too, but thought it was because I was changing
the idle code while it was running. ;)  I will examine it more
closely.  Since the scheduler has to reschedule itself, there was
probably and error before that action.

Thanks!
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

>>> David PONCE <david.ponce@...> seems to think that:
  [ ... ]
>> I did not find this file, but it may be wise to check it into the
>> 'test' directory so it can be regularly tested along with the other
>> items in there.
>
>Oops!  I forgot to attach it.  Sorry.  I checked it into the test
>directory.
  [ ... ]

Hi David,

In gr_test.wy, this line:

;; Generate a block analyzer for paren-like blocks
%type <block> syntax "\\(\\s(\\|\\s)\\)" matchdatatype block

Is it necessary to wrap the contents in \\( type delimiters?
An earlier match looked like this:

;; Generate a string compare analyzer for punctuations
%type <punctuation> syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string

where (string-match 1) may not be what you expect.


I'm converting my dot language (in cedet/cogre/wisent-dot.wy) and got
this compilation warning:

Compiling file /home/zappo/cedet/cogre/wisent-dot-wy.el at Wed Jan 14 20:18:04 2004

In toplevel form:
wisent-dot-wy.el:253:34:Warning: reference to free variable elt

which looks like it might be here on line 1265 in semantic-lex.el:

                       ,val (substring ,val 0 ,len)))
               (when elt ;; Adjust token end position.
                 (setq ,elt (car ,elt)

Note other refs are to ,elt.

Once I fixed that, I could byte compile, load, and run my wisent
parser against my test dot file with success!

I tried out %type with both punctuation and blocks.  I also used the
new %keyword keyword.  I didn't need anything fancy for other aspects of
the dot language.

It worked well.  The new syntax is very nice.  The block analyzers in
particular are much clearer in the .wy format than in the Emacs Lisp
format.  This is good stuff!

Thanks
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Eric,

[...]
> In gr_test.wy, this line:
> 
> ;; Generate a block analyzer for paren-like blocks
> %type <block> syntax "\\(\\s(\\|\\s)\\)" matchdatatype block
> 
> Is it necessary to wrap the contents in \\( type delimiters?

You're right it isn't necessary.  I will remove the \\( and \\)
delimiters.

> An earlier match looked like this:
> 
> ;; Generate a string compare analyzer for punctuations
> %type <punctuation> syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string
> 
> where (string-match 1) may not be what you expect.

I am afraid, is there any problem with that regexp?  The buffer
substring that matches a `syntax' regexp is always obtained using
(match-string 0).

> I'm converting my dot language (in cedet/cogre/wisent-dot.wy) and got
> this compilation warning:
> 
> Compiling file /home/zappo/cedet/cogre/wisent-dot-wy.el at Wed Jan 14 20:18:04 2004
> 
> In toplevel form:
> wisent-dot-wy.el:253:34:Warning: reference to free variable elt
> 
> which looks like it might be here on line 1265 in semantic-lex.el:
> 
>                        ,val (substring ,val 0 ,len)))
>                (when elt ;; Adjust token end position.
>                  (setq ,elt (car ,elt)
> 
> Note other refs are to ,elt.

Thanks!  I will fix the missing comma.

> Once I fixed that, I could byte compile, load, and run my wisent
> parser against my test dot file with success!

Yeah!

> I tried out %type with both punctuation and blocks.  I also used the
> new %keyword keyword.  I didn't need anything fancy for other aspects of
> the dot language.
> 
> It worked well.  The new syntax is very nice.  The block analyzers in
> particular are much clearer in the .wy format than in the Emacs Lisp
> format.  This is good stuff!

I am very happy that it works and you like it ;-)

I also plan to add a generic compare analyzer that can use a user's
defined predicate to match tokens.  Something like this:

%{
  (defun my-predicate (elt1 el2)
    ;; my code to see if elt1 match elt2
    )
  }

%type <my-type> syntax "some-syntax-regexp" 
                matchdatatype (predicate my-predicate)

The analyzer could be defined like this:

(defun semantic-lex-predicate-rassoc (key list &optional predicate)
  "Find the first item whose cdr matches KEY in LIST.
Optional argument PREDICATE is a function used to compare each LIST
element to KEY.  It default to `equal'."
  (cond
   ((or (null predicate) (eq predicate 'equal))
    (rassoc key list))
   ((eq predicate 'eq)
    (rassq key list))
   ((let (found)
      (while (and (not found) list)
        (if (funcall predicate (cdar list) key)
            (setq found (car list))
          (setq list (cdr list))))
      found))))

(defmacro define-lex-predicate-type-analyzer (name doc syntax matches default
                                                   &optional predicate)
  "Define a predicate based type analyzer NAME with DOC string.
SYNTAX is the regexp that matches a syntactic expression.
MATCHES is an alist of lexical elements used to refine the syntactic
expression.
DEFAULT is the default lexical token returned when no MATCHES.
Optional PREDICATE is the function used to compare the string matched
by SYNTAX with values in MATCHES.  It defaults to `equal'."
  (if matches
      (let* ((elt (make-symbol "elt"))
             (pos (make-symbol "pos"))
             (end (make-symbol "end")))
        `(define-lex-analyzer ,name
           ,doc
           (and (looking-at ,syntax)
                (let* ((,pos (match-beginning 0))
                       (,end (match-end 0))
                       (,elt (semantic-lex-predicate-rassoc
                              (match-string 0)
                              ,matches
                              ,predicate)))
                  (semantic-lex-push-token
                   (semantic-lex-token
                    (if ,elt (car ,elt) ,default) ,pos ,end)))))
        )
    `(define-lex-simple-regex-analyzer ,name
       ,doc
       ,syntax ,default)
    ))

What do you think?

Thanks! 
David

>>> David PONCE <david.ponce@...> seems to think that:
>Hi Eric,
  [ ... ]
>> An earlier match looked like this:
>> 
>> ;; Generate a string compare analyzer for punctuations
>> %type <punctuation> syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string
>> 
>> where (string-match 1) may not be what you expect.
>
>I am afraid, is there any problem with that regexp?  The buffer
>substring that matches a `syntax' regexp is always obtained using
>(match-string 0).

I was using the above as an example of why I thought the earlier
quoted regexp had unnecessary \\( grouping characters.  I was
expecting (match-string 0) was the means for extracting.

  [ ... ]
>> It worked well.  The new syntax is very nice.  The block analyzers in
>> particular are much clearer in the .wy format than in the Emacs Lisp
>> format.  This is good stuff!
>
>I am very happy that it works and you like it ;-)
>
>I also plan to add a generic compare analyzer that can use a user's
>defined predicate to match tokens.  Something like this:
>
>%{
>  (defun my-predicate (elt1 el2)
>    ;; my code to see if elt1 match elt2
>    )
>  }
>
>%type <my-type> syntax "some-syntax-regexp" 
>                matchdatatype (predicate my-predicate)
>
  [ ... ]

I'm not sure I understand the need for a predicate.  For example,
what if I did this:

%type <cmacro> syntax "^#\\w+" matchdatatype ( ... )

Hmm, I answer my own question, the predicate might test if
the matched string was a #include, or a #define.

I will wait to see what examples you have.  It sounds pretty nifty
though!  It may be that we don't need the plethora of default lexical
analyzers for more than creating a default lexer.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org