Re[2]: [CEDET-devel] Auto-generation of lexical analyzer (next step)

[Eric M. Ludlam <eric@si...>]

>>> David PONCE <david.ponce@...> seems to think that:
  [ ... ]
>>>  * A keyword analyzer, defined by `define-lex-keyword-type-analyzer',
>>>    to handle only language keywords.  That permit to separate
>>>    analysis of true keywords from other symbols.
>> 
>> 
>> Does this mean a separate entry in the lexer construction?
>> ie. 1 call to find 'keywords' and a second to find 'symbol' %types?
>> That could slow things down some.
>
>Yes this means that.  For what I saw, using the new Java lexer,
>parsing hasn't slowed down.  Matching a keyword is very fast, as it
>uses a symbol table.  The only extra cost I see can be a redundant
>looking-at.  IMO this tiny extra cost is largely compensated by the
>flexibility given to the programmer.  Keywords and other symbols no
>more need to be handled in the same way, and can be matched by
>different syntax regexp.  For example we can imagine that keywords
>start with a percent (like in grammars) and ordinary symbols not.
>
>%type <keyword> syntax "%\\(\\sw\\|\\s_\\)+" matchdatatype keyword
>%type <symbol>  syntax "\\(\\sw\\|\\s_\\)+"  matchdatatype regexp
>
>In a such case using two different regexp to respectively match
>keywords and symbols could be faster than looking up in the keyword
>table for each symbol.

In addition for your grammar, a keyword regexp could be:

%type <keyword> syntax "%\\sw+" matchdatatype keyword

as there are no symbol constituents in any of the keywords (yet.)

This would be true for most languages where if, while, else, return,
etc are simple words.

>A more interesting case would be if, for some language, keywords were
>not based on Emacs symbol characters, as opposed to other ordinary
>symbols ;-)  Separating keyword from symbol analyzers permits to
>handle that.

AFAIK Emacs can use any characters in its symbols.  For example:

M-: (intern "moose pie") RET

The trick is when a reader sees it in your code it may not realize
what you mean. ;)

>>>- <type> in the generated analyzer name is now enclosed between < > to
>>>  improve readability.  For example:
>>>
>>>  %package my-foo
>>>  ...
>>>  %type <symbol> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype regexp
>> 
>> 
>> This entry is beginning to strike me as being a bit long.
>> Technically, the word "syntax" and "matchdatatype" are sugar, meaning
>> they add anchors for the developer to know what is coming.
>> 
>> Are sections of this optional?  eg:
>> 
>> %type <symbol> syntax "blahblah"
>> 
>> implies a matchdatatype of regexp
>> 
>> and 
>> 
>> %type <symbol> matchdatatype regexp
>> 
>> implies some sort of default syntax regular expression?
>> 
>> I guess just 
>> 
>> %type <symbol>
>> 
>> could mean something if there are defaults.
>
>It will be easy to preset useful defaults in the type table for well
>known types like <symbol>, <string>, <punctuation>, etc.., by adding
>things like the following in `semantic-lex-make-type-table':
>
>;; Set up some useful default properties
>(semantic-lex-type-put "punctuation"
>                       'syntax "\\(\\s.\\|\\s$\\|\\s'\\)"
>                       t)
>(semantic-lex-type-put "punctuation"
>                       'matchdatatype 'regexp
>                       t)
>(semantic-lex-type-put "symbol"
>                       'syntax "\\(\\sw\\|\\s_\\)+"
>                       t)
>(semantic-lex-type-put "symbol"
>                       'matchdatatype 'regexp
>                       t)

That's a nice idea.  For simple stuff, a language author could just
specify a few %type entries and that's it.

>For now generation of a <type> analyzer is controlled by the existence
>or a syntax property for that <type>.
>
>To let the developer control what analyzers to generate it could be
>possible to use a special flag like "%type <xxx> generate t", or
>simply check that a %type statement is present.

Yes, I can imagine just a few:

%type <symbol>
%type <punctuation>

to build a default analyzer.

I assume the order of the %type specifiers controls the order of
placement in the constructed lexer?  This will be an important
feature that must be documented.

>[...]
>> That's fine.  I'd probably adapt it to use all your new stuff anyway. ;)
>> I like the new representation as it says a lot more with the <> around
>> it.
>
>I committed the changes.
  [ ... ]

Thanks!  I will try them out.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Re: [CEDET-devel] Auto-generation of lexical analyzer (next step)

[David PONCE <david.ponce@wa...>]

Hi Eric,

[...]
>>After converting wisent-java.wy, I will be able to completely remove
>>wisent-java-lex.el.  And probably some specific predefined analyzers
>>in semantic-lex.el could be removed too ;-)
> 
> 
> Wow, that is pretty impressive.  I like the idea of having everything
> together in one file that explains how to parse a file.
> 
> Don't you still need wisent-java-lex.el for the other java (non tag)
> parser?

No, wisent-java-lex is only used in wisent-java-tags.el and
wisent-java.el to provide the lexers.  When both lexers will be
auto-generated that file wont be necessary anymore.

>>- Defined two new `matchdatatype' generated analyzers:
>>
>>  * A sexp analyzer, defined by `define-lex-sexp-type-analyzer', to
>>    handle tokens which are s-expressions anchored by a `syntax'
>>    regexp.  A trivial example is given by strings, anchored by
>>    "\\s"" and whose end is given by a "guarded" `forward-sexp'.
> 
> 
> Very nice.
> 
> It might not be much of a stretch for this to imply any kind of
> matching, not just those available in the syntax table?  Like BEGIN ->
> END in Ada, for example.

That's a good idea.  Perhaps a solution could be to allow overriding
of the hard-coded `forward-sexp' by a user's defined function that can
handle specific language expression like BEGIN -> END in Ada.  The
sexp and block type analyzers could both benefit of that feature.

[...]
>>  * A keyword analyzer, defined by `define-lex-keyword-type-analyzer',
>>    to handle only language keywords.  That permit to separate
>>    analysis of true keywords from other symbols.
> 
> 
> Does this mean a separate entry in the lexer construction?
> ie. 1 call to find 'keywords' and a second to find 'symbol' %types?
> That could slow things down some.

Yes this means that.  For what I saw, using the new Java lexer,
parsing hasn't slowed down.  Matching a keyword is very fast, as it
uses a symbol table.  The only extra cost I see can be a redundant
looking-at.  IMO this tiny extra cost is largely compensated by the
flexibility given to the programmer.  Keywords and other symbols no
more need to be handled in the same way, and can be matched by
different syntax regexp.  For example we can imagine that keywords
start with a percent (like in grammars) and ordinary symbols not.

%type <keyword> syntax "%\\(\\sw\\|\\s_\\)+" matchdatatype keyword
%type <symbol>  syntax "\\(\\sw\\|\\s_\\)+"  matchdatatype regexp

In a such case using two different regexp to respectively match
keywords and symbols could be faster than looking up in the keyword
table for each symbol.

A more interesting case would be if, for some language, keywords were
not based on Emacs symbol characters, as opposed to other ordinary
symbols ;-)  Separating keyword from symbol analyzers permits to
handle that.

>>- <type> in the generated analyzer name is now enclosed between < > to
>>  improve readability.  For example:
>>
>>  %package my-foo
>>  ...
>>  %type <symbol> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype regexp
> 
> 
> This entry is beginning to strike me as being a bit long.
> Technically, the word "syntax" and "matchdatatype" are sugar, meaning
> they add anchors for the developer to know what is coming.
> 
> Are sections of this optional?  eg:
> 
> %type <symbol> syntax "blahblah"
> 
> implies a matchdatatype of regexp
> 
> and 
> 
> %type <symbol> matchdatatype regexp
> 
> implies some sort of default syntax regular expression?
> 
> I guess just 
> 
> %type <symbol>
> 
> could mean something if there are defaults.

It will be easy to preset useful defaults in the type table for well
known types like <symbol>, <string>, <punctuation>, etc.., by adding
things like the following in `semantic-lex-make-type-table':

;; Set up some useful default properties
(semantic-lex-type-put "punctuation"
                       'syntax "\\(\\s.\\|\\s$\\|\\s'\\)"
                       t)
(semantic-lex-type-put "punctuation"
                       'matchdatatype 'regexp
                       t)
(semantic-lex-type-put "symbol"
                       'syntax "\\(\\sw\\|\\s_\\)+"
                       t)
(semantic-lex-type-put "symbol"
                       'matchdatatype 'regexp
                       t)

For now generation of a <type> analyzer is controlled by the existence
or a syntax property for that <type>.

To let the developer control what analyzers to generate it could be
possible to use a special flag like "%type <xxx> generate t", or
simply check that a %type statement is present.

[...]
> That's fine.  I'd probably adapt it to use all your new stuff anyway. ;)
> I like the new representation as it says a lot more with the <> around
> it.

I committed the changes.

Thanks!
David

Re: [CEDET-devel] Auto-generation of lexical analyzer (next step)

[David PONCE <david.ponce@wa...>]

Eric,

[...]
>>After converting wisent-java.wy, I will be able to completely remove
>>wisent-java-lex.el.  And probably some specific predefined analyzers
>>in semantic-lex.el could be removed too ;-)

I just converted the wisent-java.wy grammar to auto-generate the
lexical analyzers, and removed the dependency on wisent-java-lex from
wisent-java.el.

Parsing seems to work just fine using that grammar and the new
auto-generated lexer :-)

I committed these changes:

	* cedet/semantic/semantic-grammar.el 

	(semantic-grammar-insert-defanalyzer): Fix use of quote in
	declaration of regexp analyzer.

	* cedet/semantic/wisent/wisent-java.el

	(require 'wisent-java-lex): Remove.
	
	* cedet/semantic/wisent/wisent-java.wy

	Use %keyword to declare keywords.
	Automatically generate analyzers used to build the lexer.

	(<block>, <punctuation>, <symbol>, <string>)
	(<number>, <keyword>): Add %type declaration.
	(epilogue): Add lexer declaration.

David

Re: [CEDET-devel] Auto-generation of lexical analyzer (next step)

[David PONCE <david.ponce@wa...>]

[...]
> Yes, I can imagine just a few:
> 
> %type <symbol>
> %type <punctuation>
> 
> to build a default analyzer.

I will look at that as soon as possible ;-)

> I assume the order of the %type specifiers controls the order of
> placement in the constructed lexer?  This will be an important
> feature that must be documented.
[...]

For now only the basic analyzers are generated.  The developer is
responsible of putting them together in a lexer in the relevant order.

There are examples in the epilogue code in wisent-java.wy and
wisent-java-tags.wy.

Perhaps in the future we could imagine a new declarative statement
that defines the lexer itself.  Something like:

%lexer
 {
  ignore-whitespace
  ignore-newline
  ignore-comments
  <number>
  <string>
  ;; Must detect keywords before other symbols
  <keyword>
  <symbol>
  <punctuation>
  <block>
  default-action  
 }

<type> items would be replaced by corresponding generated analyzers
and other items like ignore-whitespace would be replaced by a
corresponding `semantic-lex-ignore-whitespace'.

However I am not sure that it will be worth implementing that
compared to the simplicity of doing the same thing in Lisp.

David

Re[2]: [CEDET-devel] Auto-generation of lexical analyzer (next step)

[Eric M. Ludlam <eric@si...>]

>>> David PONCE <david.ponce@...> seems to think that:
  [ ... ]
>Perhaps in the future we could imagine a new declarative statement
>that defines the lexer itself.  Something like:
>
>%lexer
> {
>  ignore-whitespace
>  ignore-newline
>  ignore-comments
>  <number>
>  <string>
>  ;; Must detect keywords before other symbols
>  <keyword>
>  <symbol>
>  <punctuation>
>  <block>
>  default-action  
> }
>
><type> items would be replaced by corresponding generated analyzers
>and other items like ignore-whitespace would be replaced by a
>corresponding `semantic-lex-ignore-whitespace'.
>
>However I am not sure that it will be worth implementing that
>compared to the simplicity of doing the same thing in Lisp.
  [ ... ]

I agree with your assessment.  There will always be analyzers that
cannot be adequately represented with a %type statement, such as the
#include file name in the C bovine grammar.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Re[2]: [CEDET-devel] Auto-generation of lexical analyzer (next step)

[Eric M. Ludlam <eric@si...>]

>>> David PONCE <david.ponce@...> seems to think that:
  [ ... ]
>>>  * A keyword analyzer, defined by `define-lex-keyword-type-analyzer',
>>>    to handle only language keywords.  That permit to separate
>>>    analysis of true keywords from other symbols.
>> 
>> 
>> Does this mean a separate entry in the lexer construction?
>> ie. 1 call to find 'keywords' and a second to find 'symbol' %types?
>> That could slow things down some.
>
>Yes this means that.  For what I saw, using the new Java lexer,
>parsing hasn't slowed down.  Matching a keyword is very fast, as it
>uses a symbol table.  The only extra cost I see can be a redundant
>looking-at.  IMO this tiny extra cost is largely compensated by the
>flexibility given to the programmer.  Keywords and other symbols no
>more need to be handled in the same way, and can be matched by
>different syntax regexp.  For example we can imagine that keywords
>start with a percent (like in grammars) and ordinary symbols not.
>
>%type <keyword> syntax "%\\(\\sw\\|\\s_\\)+" matchdatatype keyword
>%type <symbol>  syntax "\\(\\sw\\|\\s_\\)+"  matchdatatype regexp
>
>In a such case using two different regexp to respectively match
>keywords and symbols could be faster than looking up in the keyword
>table for each symbol.

In addition for your grammar, a keyword regexp could be:

%type <keyword> syntax "%\\sw+" matchdatatype keyword

as there are no symbol constituents in any of the keywords (yet.)

This would be true for most languages where if, while, else, return,
etc are simple words.

>A more interesting case would be if, for some language, keywords were
>not based on Emacs symbol characters, as opposed to other ordinary
>symbols ;-)  Separating keyword from symbol analyzers permits to
>handle that.

AFAIK Emacs can use any characters in its symbols.  For example:

M-: (intern "moose pie") RET

The trick is when a reader sees it in your code it may not realize
what you mean. ;)

>>>- <type> in the generated analyzer name is now enclosed between < > to
>>>  improve readability.  For example:
>>>
>>>  %package my-foo
>>>  ...
>>>  %type <symbol> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype regexp
>> 
>> 
>> This entry is beginning to strike me as being a bit long.
>> Technically, the word "syntax" and "matchdatatype" are sugar, meaning
>> they add anchors for the developer to know what is coming.
>> 
>> Are sections of this optional?  eg:
>> 
>> %type <symbol> syntax "blahblah"
>> 
>> implies a matchdatatype of regexp
>> 
>> and 
>> 
>> %type <symbol> matchdatatype regexp
>> 
>> implies some sort of default syntax regular expression?
>> 
>> I guess just 
>> 
>> %type <symbol>
>> 
>> could mean something if there are defaults.
>
>It will be easy to preset useful defaults in the type table for well
>known types like <symbol>, <string>, <punctuation>, etc.., by adding
>things like the following in `semantic-lex-make-type-table':
>
>;; Set up some useful default properties
>(semantic-lex-type-put "punctuation"
>                       'syntax "\\(\\s.\\|\\s$\\|\\s'\\)"
>                       t)
>(semantic-lex-type-put "punctuation"
>                       'matchdatatype 'regexp
>                       t)
>(semantic-lex-type-put "symbol"
>                       'syntax "\\(\\sw\\|\\s_\\)+"
>                       t)
>(semantic-lex-type-put "symbol"
>                       'matchdatatype 'regexp
>                       t)

That's a nice idea.  For simple stuff, a language author could just
specify a few %type entries and that's it.

>For now generation of a <type> analyzer is controlled by the existence
>or a syntax property for that <type>.
>
>To let the developer control what analyzers to generate it could be
>possible to use a special flag like "%type <xxx> generate t", or
>simply check that a %type statement is present.

Yes, I can imagine just a few:

%type <symbol>
%type <punctuation>

to build a default analyzer.

I assume the order of the %type specifiers controls the order of
placement in the constructed lexer?  This will be an important
feature that must be documented.

>[...]
>> That's fine.  I'd probably adapt it to use all your new stuff anyway. ;)
>> I like the new representation as it says a lot more with the <> around
>> it.
>
>I committed the changes.
  [ ... ]

Thanks!  I will try them out.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Re: [CEDET-devel] Auto-generation of lexical analyzer (next step)

[David Ponce <david.ponce@wa...>]

Hi Eric,

[...]
 > Yes, I can imagine just a few:
 >
 > %type <symbol>
 > %type <punctuation>
 >
 > to build a default analyzer.

Following is a new patch that implements your suggestion above ;-)

I also modified the Java grammars to use the default values of the
well known token types <block>, <punctuation>, <symbol>, <string>,
<number> and <keyword>.

So now, for "standard" cases it is very easy to auto-generate the
basic type analyzers :-)

Finally, I added an optional argument `force' to the command
`semantic-grammar-create-package', to unconditionally re-generate
Lisp code from the grammar.  It is very useful to test generation
without modifying the grammar, by typing C-u C-c C-c.

If you don't have objections, I will commit the changes.

Here is the change log:

	* cedet/semantic/wisent/wisent-java.wy

	(<block>, <punctuation>, <symbol>, <string>)
	(<number>, <keyword>): Use default values in %type declarations.

	* cedet/semantic/wisent/wisent-java-tags.wy

	(<block>, <punctuation>, <symbol>, <string>)
	(<number>, <keyword>): Use default values in %type declarations.

	* cedet/semantic/semantic-grammar.el

	(semantic-grammar-token-%type-properties): Remember types
	explicitly declared with a %type statement.
	(semantic-grammar-insert-defanalyzer): Generate analyzer for
	explicitly declared types.
	(semantic-grammar-create-package): Add optional argument force to
	unconditionally re-generate Lisp code.

	* cedet/semantic/semantic-grammar-wy.el

	Re-generated.

	* cedet/semantic/semantic-grammar.wy

	(plist_opt): New rule.
	(type_decl): Use it.
	
	* cedet/semantic/semantic-lex.el
	
	(semantic-lex-preset-default-types): New function.
	(semantic-lex-make-type-table): Use it.

Enjoy!

David

Index: semantic/semantic-grammar-wy.el
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-grammar-wy.el,v
retrieving revision 1.9
diff -c -r1.9 semantic-grammar-wy.el
*** semantic/semantic-grammar-wy.el	14 Jan 2004 09:54:24 -0000	1.9
--- semantic/semantic-grammar-wy.el	18 Jan 2004 09:55:37 -0000
***************
*** 3,11 ****
   ;; Copyright (C) 2002, 2003, 2004 David Ponce

   ;; Author: David Ponce <david@...>
! ;; Created: 2004-01-14 10:33:53+0100
   ;; Keywords: syntax
! ;; X-RCS: $Id: semantic-grammar-wy.el,v 1.9 2004/01/14 09:54:24 ponced Exp $

   ;; This file is not part of GNU Emacs.
   ;;
--- 3,11 ----
   ;; Copyright (C) 2002, 2003, 2004 David Ponce

   ;; Author: David Ponce <david@...>
! ;; Created: 2004-01-18 10:52:15+0100
   ;; Keywords: syntax
! ;; X-RCS: $Id$

   ;; This file is not part of GNU Emacs.
   ;;
***************
*** 258,266 ****
           ((LT SYMBOL GT)
            (progn $2)))
          (type_decl
!         ((TYPE token_type plist)
            `(wisent-raw-tag
              (semantic-tag ',$2 'type :value ',$3))))
          (plist
           ((plist put_value)
            (append
--- 258,269 ----
           ((LT SYMBOL GT)
            (progn $2)))
          (type_decl
!         ((TYPE token_type plist_opt)
            `(wisent-raw-tag
              (semantic-tag ',$2 'type :value ',$3))))
+        (plist_opt
+         (nil)
+         ((plist)))
          (plist
           ((plist put_value)
            (append
Index: semantic/semantic-grammar.el
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-grammar.el,v
retrieving revision 1.52
diff -c -r1.52 semantic-grammar.el
*** semantic/semantic-grammar.el	16 Jan 2004 15:43:18 -0000	1.52
--- semantic/semantic-grammar.el	18 Jan 2004 09:55:38 -0000
***************
*** 438,443 ****
--- 438,445 ----
     (let (type)
       (dolist (tag (semantic-find-tags-by-class 'type (current-buffer)))
         (setq type (semantic-tag-name tag))
+       ;; Indicate to auto-generate the analyzer for this type
+       (push (list type :declared t) props)
         (dolist (e (semantic-tag-get-attribute tag :value))
           (push (list type (intern (car e)) (read (or (cdr e) "nil")))
                 props)))
***************
*** 796,805 ****
     (let* ((type-name  (symbol-name type))
            (type-value (symbol-value type))
            (syntax     (get type 'syntax))
            spec mtype prefix name doc)
!     ;; Generate an analyzer if at least syntactic/lexical matches are
!     ;; provided.
!     (when syntax
         (setq prefix (file-name-sans-extension
                       (semantic-grammar-buffer-file
                        semantic--grammar-output-buffer))
--- 798,809 ----
     (let* ((type-name  (symbol-name type))
            (type-value (symbol-value type))
            (syntax     (get type 'syntax))
+          (declared   (get type :declared))
            spec mtype prefix name doc)
!     ;; Generate an analyzer if the corresponding type has been
!     ;; explicitly declared in a %type statement, and if at least the
!     ;; syntax property has been provided.
!     (when (and declared syntax)
         (setq prefix (file-name-sans-extension
                       (semantic-grammar-buffer-file
                        semantic--grammar-output-buffer))
***************
*** 873,881 ****
         (noninteractive)
       noninteractive))

! (defun semantic-grammar-create-package ()
!   "Create package Lisp code from grammar in current buffer."
!   (interactive)
     (semantic-bovinate-toplevel t)
     (let* (
            ;; Values of the following local variables are obtained from
--- 877,889 ----
         (noninteractive)
       noninteractive))

! (defun semantic-grammar-create-package (&optional force)
!   "Create package Lisp code from grammar in current buffer.
! Does nothing if the Lisp code seems up to date.
! If optional argument FORCE is non-nil, unconditionally re-generate the
! Lisp code."
!   (interactive "P")
!   (setq force (or force current-prefix-arg))
     (semantic-bovinate-toplevel t)
     (let* (
            ;; Values of the following local variables are obtained from
***************
*** 890,896 ****
            (epilogue (semantic-grammar-epilogue))
            (footer   (semantic-grammar-footer))
            )
!     (if (and (not (buffer-modified-p))
                (file-newer-than-file-p
                 (buffer-file-name semantic--grammar-output-buffer)
                 (buffer-file-name semantic--grammar-input-buffer)))
--- 898,905 ----
            (epilogue (semantic-grammar-epilogue))
            (footer   (semantic-grammar-footer))
            )
!     (if (and (not force)
!              (not (buffer-modified-p))
                (file-newer-than-file-p
                 (buffer-file-name semantic--grammar-output-buffer)
                 (buffer-file-name semantic--grammar-input-buffer)))
Index: semantic/semantic-grammar.wy
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-grammar.wy,v
retrieving revision 1.12
diff -c -r1.12 semantic-grammar.wy
*** semantic/semantic-grammar.wy	14 Jan 2004 09:53:56 -0000	1.12
--- semantic/semantic-grammar.wy	18 Jan 2004 09:55:38 -0000
***************
*** 258,267 ****
     ;

   type_decl:
!     TYPE token_type plist
       `(TAG ',$2 'type :value ',$3)
     ;

   plist:
       plist put_value
       (append (list $2) $1)
--- 258,272 ----
     ;

   type_decl:
!     TYPE token_type plist_opt
       `(TAG ',$2 'type :value ',$3)
     ;

+ plist_opt:
+  ;;EMPTY
+   | plist
+   ;
+
   plist:
       plist put_value
       (append (list $2) $1)
Index: semantic/semantic-lex.el
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/semantic-lex.el,v
retrieving revision 1.30
diff -c -r1.30 semantic-lex.el
*** semantic/semantic-lex.el	16 Jan 2004 08:56:43 -0000	1.30
--- semantic/semantic-lex.el	18 Jan 2004 09:55:39 -0000
***************
*** 26,32 ****
   ;; In semantic 1.x, the lexical analyzer was an all purpose routine.
   ;; To boost efficiency, the analyzer is now a series of routines that
   ;; are constructed at build time into a single routine.  This will
! ;; eliminate uneeded if statements to speed the lexer.

   (require 'semantic-fw)
   ;;; Code:
--- 26,32 ----
   ;; In semantic 1.x, the lexical analyzer was an all purpose routine.
   ;; To boost efficiency, the analyzer is now a series of routines that
   ;; are constructed at build time into a single routine.  This will
! ;; eliminate unneeded if statements to speed the lexer.

   (require 'semantic-fw)
   ;;; Code:
***************
*** 217,222 ****
--- 217,238 ----
         (unless noerror
           (semantic-lex-type-invalid type)))))

+ (defun semantic-lex-preset-default-types ()
+   "Install useful default properties for well known types."
+   (semantic-lex-type-put "punctuation" 'matchdatatype 'string t)
+   (semantic-lex-type-put "punctuation" 'syntax "\\s.\\|\\s$\\|\\s'")
+   (semantic-lex-type-put "keyword" 'matchdatatype 'keyword t)
+   (semantic-lex-type-put "keyword" 'syntax "\\(\\sw\\|\\s_\\)+")
+   (semantic-lex-type-put "symbol"  'matchdatatype 'regexp t)
+   (semantic-lex-type-put "symbol"  'syntax "\\(\\sw\\|\\s_\\)+")
+   (semantic-lex-type-put "string"  'matchdatatype 'sexp t)
+   (semantic-lex-type-put "string"  'syntax "\\s\"")
+   (semantic-lex-type-put "number"  'matchdatatype 'regexp t)
+   (semantic-lex-type-put "number"  'syntax 'semantic-lex-number-expression)
+   (semantic-lex-type-put "block"   'matchdatatype 'block t)
+   (semantic-lex-type-put "block"   'syntax "\\s(\\|\\s)")
+   )
+
   (defun semantic-lex-make-type-table (specs &optional propspecs)
     "Convert type SPECS into an obarray and return it.
   SPECS must be a list of (TYPE . TOKENS) elements, where:
***************
*** 255,260 ****
--- 271,278 ----
             (setq default token)))
         ;; Ensure the default matching spec is the first one.
         (semantic-lex-type-set type (cons default (nreverse alist))))
+     ;; Install useful default types & properties
+     (semantic-lex-preset-default-types)
       ;; Apply all properties
       (while propspecs
         (setq spec (car propspecs)
Index: semantic/wisent/wisent-java-tags.wy
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-java-tags.wy,v
retrieving revision 1.12
diff -c -r1.12 wisent-java-tags.wy
*** semantic/wisent/wisent-java-tags.wy	16 Jan 2004 08:57:53 -0000	1.12
--- semantic/wisent/wisent-java-tags.wy	18 Jan 2004 09:55:41 -0000
***************
*** 49,55 ****
   ;; -----------------------------
   ;; Block & Parenthesis terminals
   ;; -----------------------------
! %type  <block>       syntax "\\s(\\|\\s)" matchdatatype block

   %token <block>       PAREN_BLOCK "(LPAREN RPAREN)"
   %token <block>       BRACE_BLOCK "(LBRACE RBRACE)"
--- 49,55 ----
   ;; -----------------------------
   ;; Block & Parenthesis terminals
   ;; -----------------------------
! %type  <block>       ;;syntax "\\s(\\|\\s)" matchdatatype block

   %token <block>       PAREN_BLOCK "(LPAREN RPAREN)"
   %token <block>       BRACE_BLOCK "(LBRACE RBRACE)"
***************
*** 65,71 ****
   ;; ------------------
   ;; Operator terminals
   ;; ------------------
! %type  <punctuation> syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string

   %token <punctuation> NOT         "!"
   %token <punctuation> NOTEQ       "!="
--- 65,71 ----
   ;; ------------------
   ;; Operator terminals
   ;; ------------------
! %type  <punctuation> ;;syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string

   %token <punctuation> NOT         "!"
   %token <punctuation> NOTEQ       "!="
***************
*** 111,123 ****
   ;; -----------------
   ;; Literal terminals
   ;; -----------------
! %type  <symbol>      syntax "\\(\\sw\\|\\s_\\)+"
   %token <symbol>      IDENTIFIER

! %type  <string>      syntax "\\s\"" matchdatatype sexp
   %token <string>      STRING_LITERAL

! %type  <number>      syntax semantic-lex-number-expression
   %token <number>      NUMBER_LITERAL

   ;; -----------------
--- 111,123 ----
   ;; -----------------
   ;; Literal terminals
   ;; -----------------
! %type  <symbol>      ;;syntax "\\(\\sw\\|\\s_\\)+"
   %token <symbol>      IDENTIFIER

! %type  <string>      ;;syntax "\\s\"" matchdatatype sexp
   %token <string>      STRING_LITERAL

! %type  <number>      ;;syntax semantic-lex-number-expression
   %token <number>      NUMBER_LITERAL

   ;; -----------------
***************
*** 125,131 ****
   ;; -----------------

   ;; Generate a keyword analyzer
! %type  <keyword> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword

   %keyword ABSTRACT     "abstract"
   %put     ABSTRACT summary
--- 125,131 ----
   ;; -----------------

   ;; Generate a keyword analyzer
! %type  <keyword> ;;syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword

   %keyword ABSTRACT     "abstract"
   %put     ABSTRACT summary
Index: semantic/wisent/wisent-java.wy
===================================================================
RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-java.wy,v
retrieving revision 1.14
diff -c -r1.14 wisent-java.wy
*** semantic/wisent/wisent-java.wy	16 Jan 2004 13:14:11 -0000	1.14
--- semantic/wisent/wisent-java.wy	18 Jan 2004 09:55:41 -0000
***************
*** 46,52 ****
   ;; -----------------------------
   ;; Block & Parenthesis terminals
   ;; -----------------------------
! %type  <block>       syntax "\\s(\\|\\s)" matchdatatype block

   %token <block>       PAREN_BLOCK "(LPAREN RPAREN)"
   %token <block>       BRACE_BLOCK "(LBRACE RBRACE)"
--- 46,52 ----
   ;; -----------------------------
   ;; Block & Parenthesis terminals
   ;; -----------------------------
! %type  <block>       ;;syntax "\\s(\\|\\s)" matchdatatype block

   %token <block>       PAREN_BLOCK "(LPAREN RPAREN)"
   %token <block>       BRACE_BLOCK "(LBRACE RBRACE)"
***************
*** 62,68 ****
   ;; ------------------
   ;; Operator terminals
   ;; ------------------
! %type  <punctuation> syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string

   %token <punctuation> NOT         "!"
   %token <punctuation> NOTEQ       "!="
--- 62,68 ----
   ;; ------------------
   ;; Operator terminals
   ;; ------------------
! %type  <punctuation> ;;syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string

   %token <punctuation> NOT         "!"
   %token <punctuation> NOTEQ       "!="
***************
*** 108,130 ****
   ;; -----------------
   ;; Literal terminals
   ;; -----------------
! %type  <symbol>      syntax "\\(\\sw\\|\\s_\\)+"
   %token <symbol>      NULL_LITERAL    "\\`null\\'"
   %token <symbol>      BOOLEAN_LITERAL "\\`false\\'"
   %token <symbol>      BOOLEAN_LITERAL "\\`true\\'"
   %token <symbol>      IDENTIFIER

! %type  <string>      syntax "\\s\"" matchdatatype sexp
   %token <string>      STRING_LITERAL

! %type  <number>      syntax semantic-lex-number-expression
   %token <number>      NUMBER_LITERAL

   ;; -----------------
   ;; Keyword terminals
   ;; -----------------
   ;; Generate a keyword analyzer
! %type  <keyword> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword

   %keyword ABSTRACT     "abstract"
   %put     ABSTRACT summary
--- 108,130 ----
   ;; -----------------
   ;; Literal terminals
   ;; -----------------
! %type  <symbol>      ;;syntax "\\(\\sw\\|\\s_\\)+"
   %token <symbol>      NULL_LITERAL    "\\`null\\'"
   %token <symbol>      BOOLEAN_LITERAL "\\`false\\'"
   %token <symbol>      BOOLEAN_LITERAL "\\`true\\'"
   %token <symbol>      IDENTIFIER

! %type  <string>      ;;syntax "\\s\"" matchdatatype sexp
   %token <string>      STRING_LITERAL

! %type  <number>      ;;syntax semantic-lex-number-expression
   %token <number>      NUMBER_LITERAL

   ;; -----------------
   ;; Keyword terminals
   ;; -----------------
   ;; Generate a keyword analyzer
! %type  <keyword> ;;syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword

   %keyword ABSTRACT     "abstract"
   %put     ABSTRACT summary

Re[2]: [CEDET-devel] Auto-generation of lexical analyzer (next step)

[Eric M. Ludlam <eric@si...>]

>>> David Ponce <david.ponce@...> seems to think that:
>Hi Eric,
>
>[...]
> > Yes, I can imagine just a few:
> >
> > %type <symbol>
> > %type <punctuation>
> >
> > to build a default analyzer.
>
>Following is a new patch that implements your suggestion above ;-)
>
>I also modified the Java grammars to use the default values of the
>well known token types <block>, <punctuation>, <symbol>, <string>,
><number> and <keyword>.
>
>So now, for "standard" cases it is very easy to auto-generate the
>basic type analyzers :-)
>
>Finally, I added an optional argument `force' to the command
>`semantic-grammar-create-package', to unconditionally re-generate
>Lisp code from the grammar.  It is very useful to test generation
>without modifying the grammar, by typing C-u C-c C-c.
>
>If you don't have objections, I will commit the changes.

That sounds greawt!  Please do.

  [ ... ]
>===================================================================
>RCS file: /cvsroot/cedet/cedet/semantic/semantic-grammar.el,v
>retrieving revision 1.52
>diff -c -r1.52 semantic-grammar.el
>*** semantic/semantic-grammar.el	16 Jan 2004 15:43:18 -0000	1.52
>--- semantic/semantic-grammar.el	18 Jan 2004 09:55:38 -0000
>***************
>*** 438,443 ****
>--- 438,445 ----
>     (let (type)
>       (dolist (tag (semantic-find-tags-by-class 'type (current-buffer)))
>         (setq type (semantic-tag-name tag))
>+       ;; Indicate to auto-generate the analyzer for this type
>+       (push (list type :declared t) props)
  [ ... ]

I saw this curiosity that wasn't used elsewhere.  What thoughts do
you have on this setting?  I can imagine a sort of "I compiled your
language and here is a status report." type thing.

That, of course, got me thinking that any messages the parser may spit
out (like the wisent messages about having to guess what to do) could
be output as comments in the output file.  Perhaps in the commentary.
Then such messages would never be lost.

Have fun
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Re: [CEDET-devel] Auto-generation of lexical analyzer (next step)

[David Ponce <david.ponce@wa...>]

Eric,

[...]
 >>If you don't have objections, I will commit the changes.
 >
 >
 > That sounds greawt!  Please do.

Done.

[...]
 >>+       ;; Indicate to auto-generate the analyzer for this type
 >>+       (push (list type :declared t) props)
 >
 >   [ ... ]
 >
 > I saw this curiosity that wasn't used elsewhere.  What thoughts do
 > you have on this setting?  I can imagine a sort of "I compiled your
 > language and here is a status report." type thing.

:declared is an internal property used to remember what types have
been declared explicitly using a %type statement.  Some types could
have been declared implicitly using %token <type> statements.

Then `semantic-grammar-insert-defanalyzer' will use the :declared
property to know which types have been declared with a %type
statement, and, for those only, to try to generate a lexical analyzer.

That simple mechanism preserve the compatibility of the new generator
with grammars that don't use [yet] %type statements to declare types
and to generate analyzers ;-)

 > That, of course, got me thinking that any messages the parser may spit
 > out (like the wisent messages about having to guess what to do) could
 > be output as comments in the output file.  Perhaps in the commentary.
 > Then such messages would never be lost.

I am sorry, but I don't understand clearly your idea here.  Could you
please elaborate or give an example?

Thanks!

David

Re[2]: [CEDET-devel] Auto-generation of lexical analyzer (next step)

[Eric M. Ludlam <eric@si...>]

>>> David Ponce <david.ponce@...> seems to think that:
> > That, of course, got me thinking that any messages the parser may spit
> > out (like the wisent messages about having to guess what to do) could
> > be output as comments in the output file.  Perhaps in the commentary.
> > Then such messages would never be lost.
>
>I am sorry, but I don't understand clearly your idea here.  Could you
>please elaborate or give an example?
  [ ... ]

When wisent compiles a grammar, sometimes it outputs info like this
from wisent-awk.wy:

--------
semacs -batch -q -l grammar-make-script -f semantic-grammar-batch-build-packages wisent-java-tags.wy wisent-awk.wy wisent-calc.wy wisent-cim.wy wisent-java.wy wisent-python.wy
Package `wisent-java-tags-wy' is up to date.
Loading vc-cvs...
*Warning* default value of <<no-type>> tokens changed to LEX_LENGTH, was LEX_BUILTIN
*Warning* default value of <<no-type>> tokens changed to LEX_BUILTIN, was DECREMENT
*Warning* default value of <<no-type>> tokens changed to DECREMENT, was INCREMENT
*Warning* default value of <<no-type>> tokens changed to INCREMENT, was LEX_OR

[ ... ]
Wrote /home/zappo/cedet/semantic/wisent/wisent-awk-wy.el
Grammar in wisent-awk-wy.el contains 65 shift/reduce conflicts
-------

These could be added as comments into the -wy.el file, though now I
seem to remember that some of them are generated after the file is
written?

Now that I've had time to think about it, I'm not sure what I was
contemplating is possible.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org