[CEDET-devel] Auto-generation of lexical analyzer (next step)
Brought to you by:
zappo
From: David P. <dav...@wa...> - 2004-01-15 20:04:17
|
Hi Eric, I reached a new step in auto-generation of lexical analyzers. As a demonstration, I adapted the Java tags grammar so that now the lexer is completely auto-generated from the grammar ;-) I did some quick tests and Java parsing seems to work quite well with the new lexer! After converting wisent-java.wy, I will be able to completely remove wisent-java-lex.el. And probably some specific predefined analyzers in semantic-lex.el could be removed too ;-) Here is a summary of the important changes I made: - Fixed use of quote inside backquote expressions. - Updated font-lock keywords for new define-lex-... macros. - The above highlighting now also works for Lisp code in grammars. - Defined two new `matchdatatype' generated analyzers: * A sexp analyzer, defined by `define-lex-sexp-type-analyzer', to handle tokens which are s-expressions anchored by a `syntax' regexp. A trivial example is given by strings, anchored by "\\s"" and whose end is given by a "guarded" `forward-sexp'. * A keyword analyzer, defined by `define-lex-keyword-type-analyzer', to handle only language keywords. That permit to separate analysis of true keywords from other symbols. - <type> in the generated analyzer name is now enclosed between < > to improve readability. For example: %package my-foo ... %type <symbol> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype regexp now generate an analyzer named `my-foo--<symbol>-regexp-analyzer'. Unfortunately, wisent-dot.el will need a minor update to continue to work. :-( The change log and the patch follows. David * cedet/semantic/semantic-fw.el (semantic-fw-font-lock-keywords): Add new macro names. * cedet/semantic/semantic-grammar.el (semantic-grammar-token-%type-properties): %type statements unconditionally declare new types. (semantic-grammar-token-properties): Updated. (semantic-grammar-quoted-form): New function. (semantic-grammar-insert-defanalyzer): Use it to fix use of quote in analyzer declarations. Enclose type between < > in generated analyzer name. Fix conditions that select the type of analyzer to declare. Add declaration of sexp and keyword analyzers. (semantic-grammar-mode-keywords-1): Add highlighting of semantic keywords. * cedet/semantic/semantic-lex.el (semantic-lex-make-type-table): Implicitly create types with properties but without tokens associated. (define-lex-keyword-type-analyzer) (define-lex-sexp-type-analyzer): New macros. * cedet/semantic/wisent/wisent-java-tags.el (require 'wisent-java-lex): Remove. * cedet/semantic/wisent/wisent-java-tags.wy Automatically generate analyzers used to build the lexer. (<block>, <punctuation>, <symbol>, <string>) (<number>, <keyword>): Add %type declaration. (epilogue): Add lexer declaration. Index: semantic/semantic-fw.el =================================================================== RCS file: /cvsroot/cedet/cedet/semantic/semantic-fw.el,v retrieving revision 1.25 diff -c -r1.25 semantic-fw.el *** semantic/semantic-fw.el 23 Dec 2003 02:07:45 -0000 1.25 --- semantic/semantic-fw.el 15 Jan 2004 16:15:13 -0000 *************** *** 605,610 **** --- 605,615 ---- "define-lex-block-analyzer" "define-lex-regex-analyzer" "define-lex-simple-regex-analyzer" + "define-lex-keyword-type-analyzer" + "define-lex-sexp-type-analyzer" + "define-lex-regex-type-analyzer" + "define-lex-string-type-analyzer" + "define-lex-block-type-analyzer" "define-mode-overload-implementation" "define-semantic-child-mode" "define-overload" Index: semantic/semantic-grammar.el =================================================================== RCS file: /cvsroot/cedet/cedet/semantic/semantic-grammar.el,v retrieving revision 1.49 diff -c -r1.49 semantic-grammar.el *** semantic/semantic-grammar.el 14 Jan 2004 09:59:33 -0000 1.49 --- semantic/semantic-grammar.el 15 Jan 2004 16:15:14 -0000 *************** *** 430,449 **** (setcdr assoc (cons (cons term value) (cdr assoc))))) alist)) ! (defun semantic-grammar-token-%type-properties (tokens &optional props) ! "For types found in TOKENS, return properties set by %type statements. If optional argument PROPS is non-nil, it is an existing list of properties where to add new properties." ! (let (found type) (dolist (tag (semantic-find-tags-by-class 'type (current-buffer))) ! (setq type (semantic-tag-name tag) ! found (assoc type tokens)) ! (if (null found) ! nil ;; %type <type> ignored, no token defined ! (setq type (car found)) ! (dolist (e (semantic-tag-get-attribute tag :value)) ! (push (list type (intern (car e)) (read (or (cdr e) "nil"))) ! props)))) props)) (defun semantic-grammar-token-%put-properties (tokens) --- 430,446 ---- (setcdr assoc (cons (cons term value) (cdr assoc))))) alist)) ! (defun semantic-grammar-token-%type-properties (&optional props) ! "Return properties set by %type statements. ! This declare a new type if necessary. If optional argument PROPS is non-nil, it is an existing list of properties where to add new properties." ! (let (type) (dolist (tag (semantic-find-tags-by-class 'type (current-buffer))) ! (setq type (semantic-tag-name tag)) ! (dolist (e (semantic-tag-get-attribute tag :value)) ! (push (list type (intern (car e)) (read (or (cdr e) "nil"))) ! props))) props)) (defun semantic-grammar-token-%put-properties (tokens) *************** *** 462,473 **** props)) (defsubst semantic-grammar-token-properties (tokens) ! "Return properties of types found in TOKENS. ! That is properties set to any <type> by %put and %type statements. Properties set by %type statements take precedence over those set by %put statements." (let ((props (semantic-grammar-token-%put-properties tokens))) ! (semantic-grammar-token-%type-properties tokens props))) (defun semantic-grammar-use-macros () "Return macro definitions from %use-macros statements. --- 459,472 ---- props)) (defsubst semantic-grammar-token-properties (tokens) ! "Return properties of declared types. ! Types are explicitly declared by %type statements. Types found in ! TOKENS are those declared implicitly by %token statements. ! Properties can be set by %put and %type statements. Properties set by %type statements take precedence over those set by %put statements." (let ((props (semantic-grammar-token-%put-properties tokens))) ! (semantic-grammar-token-%type-properties props))) (defun semantic-grammar-use-macros () "Return macro definitions from %use-macros statements. *************** *** 785,790 **** --- 784,796 ---- (t semantic-grammar--lex-block-specs))) + (defsubst semantic-grammar-quoted-form (exp) + "Return a quoted form of EXP if it isn't a self evaluating form." + (if (and (not (null exp)) + (or (listp exp) (symbolp exp))) + (list 'quote exp) + exp)) + (defun semantic-grammar-insert-defanalyzer (type) "Insert declaration of the lexical analyzer defined with TYPE." (let* ((type-name (symbol-name type)) *************** *** 798,820 **** (semantic-grammar-buffer-file semantic--grammar-output-buffer)) mtype (or (get type 'matchdatatype) 'regexp) ! name (intern (format "%s--%s-%s-analyzer" prefix type mtype)) doc (format "%s analyzer for <%s> tokens." mtype type)) (cond ;; Regexp match analyzer ! ((and (eq mtype 'regexp) ! (setq spec (cdr type-value))) (semantic-grammar-insert-define `(define-lex-regex-type-analyzer ,name ! ,doc ',syntax ',spec ',(or (car type-value) (intern type-name)))) ) ;; String compare analyzer ! ((and (eq mtype 'string) ! (setq spec (cdr type-value))) (semantic-grammar-insert-define `(define-lex-string-type-analyzer ,name ! ,doc ',syntax ',spec ',(or (car type-value) (intern type-name)))) ) ;; Block analyzer --- 804,825 ---- (semantic-grammar-buffer-file semantic--grammar-output-buffer)) mtype (or (get type 'matchdatatype) 'regexp) ! name (intern (format "%s--<%s>-%s-analyzer" prefix type mtype)) doc (format "%s analyzer for <%s> tokens." mtype type)) (cond ;; Regexp match analyzer ! ((eq mtype 'regexp) (semantic-grammar-insert-define `(define-lex-regex-type-analyzer ,name ! ,doc ,syntax ,(cdr type-value) ',(or (car type-value) (intern type-name)))) ) ;; String compare analyzer ! ((eq mtype 'string) (semantic-grammar-insert-define `(define-lex-string-type-analyzer ,name ! ,doc ,syntax ! ,(semantic-grammar-quoted-form (cdr type-value)) ',(or (car type-value) (intern type-name)))) ) ;; Block analyzer *************** *** 822,829 **** (setq spec (semantic-grammar--lex-block-specs))) (semantic-grammar-insert-define `(define-lex-block-type-analyzer ,name ! ,doc ',syntax ',spec)) ! ))) )) (defun semantic-grammar-insert-defanalyzers () --- 827,849 ---- (setq spec (semantic-grammar--lex-block-specs))) (semantic-grammar-insert-define `(define-lex-block-type-analyzer ,name ! ,doc ,syntax ! ,(semantic-grammar-quoted-form spec))) ! ) ! ;; Sexp analyzer ! ((eq mtype 'sexp) ! (semantic-grammar-insert-define ! `(define-lex-sexp-type-analyzer ,name ! ,doc ,syntax ! ',(or (car type-value) (intern type-name)))) ! ) ! ;; keyword analyzer ! ((eq mtype 'keyword) ! (semantic-grammar-insert-define ! `(define-lex-keyword-type-analyzer ,name ! ,doc ,syntax)) ! ) ! )) )) (defun semantic-grammar-insert-defanalyzers () *************** *** 1152,1157 **** --- 1172,1179 ---- ;; grammar mode! ("[\r\n\t ]+:\\sw+\\>" 0 font-lock-builtin-face) + ;; Append the Semantic keywords + ,@semantic-fw-font-lock-keywords ) "Font Lock keywords used to highlight Semantic grammar buffers.") Index: semantic/semantic-lex.el =================================================================== RCS file: /cvsroot/cedet/cedet/semantic/semantic-lex.el,v retrieving revision 1.29 diff -c -r1.29 semantic-lex.el *** semantic/semantic-lex.el 15 Jan 2004 08:06:20 -0000 1.29 --- semantic/semantic-lex.el 15 Jan 2004 16:15:16 -0000 *************** *** 259,265 **** (while propspecs (setq spec (car propspecs) propspecs (cdr propspecs)) ! (semantic-lex-type-put (car spec) (nth 1 spec) (nth 2 spec))) semantic-lex-types-obarray)) (defsubst semantic-lex-map-types (fun &optional property) --- 259,266 ---- (while propspecs (setq spec (car propspecs) propspecs (cdr propspecs)) ! ;; Create the type if necessary. ! (semantic-lex-type-put (car spec) (nth 1 spec) (nth 2 spec) t)) semantic-lex-types-obarray)) (defsubst semantic-lex-map-types (fun &optional property) *************** *** 1203,1208 **** --- 1204,1239 ---- ;;; Analyzers generated from grammar. ;; + (defmacro define-lex-keyword-type-analyzer (name doc syntax) + "Define a keyword type analyzer NAME with DOC string. + SYNTAX is the regexp that matches a keyword syntactic expression." + (let ((key (make-symbol "key"))) + `(define-lex-analyzer ,name + ,doc + (and (looking-at ,syntax) + (let ((,key (semantic-lex-keyword-p (match-string 0)))) + (when ,key + (semantic-lex-push-token + (semantic-lex-token + ,key (match-beginning 0) (match-end 0))))))) + )) + + (defmacro define-lex-sexp-type-analyzer (name doc syntax token) + "Define a sexp type analyzer NAME with DOC string. + SYNTAX is the regexp that matches the beginning of the s-expression. + TOKEN is the lexical token returned when SYNTAX matches." + `(define-lex-regex-analyzer ,name + ,doc + ,syntax + (semantic-lex-push-token + (semantic-lex-token + ,token (point) + (save-excursion + (semantic-lex-unterminated-syntax-protection ,token + (forward-sexp 1) + (point)))))) + ) + (defmacro define-lex-regex-type-analyzer (name doc syntax matches default) "Define a regexp type analyzer NAME with DOC string. SYNTAX is the regexp that matches a syntactic expression. Index: semantic/wisent/wisent-java-tags.el =================================================================== RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-java-tags.el,v retrieving revision 1.27 diff -c -r1.27 wisent-java-tags.el *** semantic/wisent/wisent-java-tags.el 5 Sep 2003 10:33:15 -0000 1.27 --- semantic/wisent/wisent-java-tags.el 15 Jan 2004 16:15:18 -0000 *************** *** 35,41 **** (require 'wisent-bovine) (require 'wisent-java-tags-wy) ! (require 'wisent-java-lex) (require 'semantic-java) (eval-when-compile (require 'semantic-util) --- 35,41 ---- (require 'wisent-bovine) (require 'wisent-java-tags-wy) ! ;;(require 'wisent-java-lex) (require 'semantic-java) (eval-when-compile (require 'semantic-util) Index: semantic/wisent/wisent-java-tags.wy =================================================================== RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-java-tags.wy,v retrieving revision 1.11 diff -c -r1.11 wisent-java-tags.wy *** semantic/wisent/wisent-java-tags.wy 5 Dec 2003 09:18:58 -0000 1.11 --- semantic/wisent/wisent-java-tags.wy 15 Jan 2004 16:15:18 -0000 *************** *** 46,305 **** %start interface_member_declaration %start formal_parameters ! ;; --------------------- ! ;; Parenthesis terminals ! ;; --------------------- ! %token <open-paren> LPAREN "(" ! %token <close-paren> RPAREN ")" ! %token <open-paren> LBRACE "{" ! %token <close-paren> RBRACE "}" ! %token <open-paren> LBRACK "[" ! %token <close-paren> RBRACK "]" ! ! ;; --------------- ! ;; Block terminals ! ;; --------------- ! %token <block> PAREN_BLOCK "(LPAREN RPAREN)" ! %token <block> BRACE_BLOCK "(LBRACE RBRACE)" ! %token <block> BRACK_BLOCK "(LBRACK RBRACK)" ;; ------------------ ;; Operator terminals ;; ------------------ ! %token <punctuation> NOT "!" ! %token <punctuation> NOTEQ "!=" ! %token <punctuation> MOD "%" ! %token <punctuation> MODEQ "%=" ! %token <punctuation> AND "&" ! %token <punctuation> ANDAND "&&" ! %token <punctuation> ANDEQ "&=" ! %token <punctuation> MULT "*" ! %token <punctuation> MULTEQ "*=" ! %token <punctuation> PLUS "+" ! %token <punctuation> PLUSPLUS "++" ! %token <punctuation> PLUSEQ "+=" ! %token <punctuation> COMMA "," ! %token <punctuation> MINUS "-" ! %token <punctuation> MINUSMINUS "--" ! %token <punctuation> MINUSEQ "-=" ! %token <punctuation> DOT "." ! %token <punctuation> DIV "/" ! %token <punctuation> DIVEQ "/=" ! %token <punctuation> COLON ":" ! %token <punctuation> SEMICOLON ";" ! %token <punctuation> LT "<" ! %token <punctuation> LSHIFT "<<" ! %token <punctuation> LSHIFTEQ "<<=" ! %token <punctuation> LTEQ "<=" ! %token <punctuation> EQ "=" ! %token <punctuation> EQEQ "==" ! %token <punctuation> GT ">" ! %token <punctuation> GTEQ ">=" ! %token <punctuation> RSHIFT ">>" ! %token <punctuation> RSHIFTEQ ">>=" ! %token <punctuation> URSHIFT ">>>" ! %token <punctuation> URSHIFTEQ ">>>=" ! %token <punctuation> QUESTION "?" ! %token <punctuation> XOR "^" ! %token <punctuation> XOREQ "^=" ! %token <punctuation> OR "|" ! %token <punctuation> OREQ "|=" ! %token <punctuation> OROR "||" ! %token <punctuation> COMP "~" ;; ----------------- ;; Literal terminals ;; ----------------- %token <symbol> IDENTIFIER %token <string> STRING_LITERAL %token <number> NUMBER_LITERAL ;; ----------------- ;; Keyword terminals ;; ----------------- ! %token ABSTRACT "abstract" ! %put ABSTRACT summary "Class|Method declaration modifier: abstract {class|<type>} <name> ..." ! %token BOOLEAN "boolean" ! %put BOOLEAN summary "Primitive logical quantity type (true or false)" ! %token BREAK "break" ! %put BREAK summary "break [<label>] ;" ! %token BYTE "byte" ! %put BYTE summary "Integral primitive type (-128 to 127)" ! %token CASE "case" ! %put CASE summary "switch(<expr>) {case <const-expr>: <stmts> ... }" ! %token CATCH "catch" ! %put CATCH summary "try {<stmts>} catch(<parm>) {<stmts>} ... " ! %token CHAR "char" ! %put CHAR summary "Integral primitive type ('\u0000' to '\uffff') (0 to 65535)" ! %token CLASS "class" ! %put CLASS summary "Class declaration: class <name>" ! %token CONST "const" ! %put CONST summary "Unused reserved word" ! %token CONTINUE "continue" ! %put CONTINUE summary "continue [<label>] ;" ! %token DEFAULT "default" ! %put DEFAULT summary "switch(<expr>) { ... default: <stmts>}" ! %token DO "do" ! %put DO summary "do <stmt> while (<expr>);" ! %token DOUBLE "double" ! %put DOUBLE summary "Primitive floating-point type (double-precision 64-bit IEEE 754)" ! %token ELSE "else" ! %put ELSE summary "if (<expr>) <stmt> else <stmt>" ! %token EXTENDS "extends" ! %put EXTENDS summary "SuperClass|SuperInterfaces declaration: extends <name> [, ...]" ! %token FINAL "final" ! %put FINAL summary "Class|Member declaration modifier: final {class|<type>} <name> ..." ! %token FINALLY "finally" ! %put FINALLY summary "try {<stmts>} ... finally {<stmts>}" ! %token FLOAT "float" ! %put FLOAT summary "Primitive floating-point type (single-precision 32-bit IEEE 754)" ! %token FOR "for" ! %put FOR summary "for ([<init-expr>]; [<expr>]; [<update-expr>]) <stmt>" ! %token GOTO "goto" ! %put GOTO summary "Unused reserved word" ! %token IF "if" ! %put IF summary "if (<expr>) <stmt> [else <stmt>]" ! %token IMPLEMENTS "implements" ! %put IMPLEMENTS summary "Class SuperInterfaces declaration: implements <name> [, ...]" ! %token IMPORT "import" ! %put IMPORT summary "Import package declarations: import <package>" ! %token INSTANCEOF "instanceof" ! %token INT "int" ! %put INT summary "Integral primitive type (-2147483648 to 2147483647)" ! %token INTERFACE "interface" ! %put INTERFACE summary "Interface declaration: interface <name>" ! %token LONG "long" ! %put LONG summary "Integral primitive type (-9223372036854775808 to 9223372036854775807)" ! %token NATIVE "native" ! %put NATIVE summary "Method declaration modifier: native <type> <name> ..." ! %token NEW "new" ! %token PACKAGE "package" ! %put PACKAGE summary "Package declaration: package <name>" ! %token PRIVATE "private" ! %put PRIVATE summary "Access level modifier: private {class|interface|<type>} <name> ..." ! %token PROTECTED "protected" ! %put PROTECTED summary "Access level modifier: protected {class|interface|<type>} <name> ..." ! %token PUBLIC "public" ! %put PUBLIC summary "Access level modifier: public {class|interface|<type>} <name> ..." ! %token RETURN "return" ! %put RETURN summary "return [<expr>] ;" ! %token SHORT "short" ! %put SHORT summary "Integral primitive type (-32768 to 32767)" ! %token STATIC "static" ! %put STATIC summary "Declaration modifier: static {class|interface|<type>} <name> ..." ! %token STRICTFP "strictfp" ! %put STRICTFP summary "Declaration modifier: strictfp {class|interface|<type>} <name> ..." ! %token SUPER "super" ! %token SWITCH "switch" ! %put SWITCH summary "switch(<expr>) {[case <const-expr>: <stmts> ...] [default: <stmts>]}" ! %token SYNCHRONIZED "synchronized" ! %put SYNCHRONIZED summary "synchronized (<expr>) ... | Method decl. modifier: synchronized <type> <name> ..." ! %token THIS "this" ! %token THROW "throw" ! %put THROW summary "throw <expr> ;" ! %token THROWS "throws" ! %put THROWS summary "Method|Constructor declaration: throws <classType>, ..." ! %token TRANSIENT "transient" ! %put TRANSIENT summary "Field declaration modifier: transient <type> <name> ..." ! %token TRY "try" ! %put TRY summary "try {<stmts>} [catch(<parm>) {<stmts>} ...] [finally {<stmts>}]" ! %token VOID "void" ! %put VOID summary "Method return type: void <name> ..." ! %token VOLATILE "volatile" ! %put VOLATILE summary "Field declaration modifier: volatile <type> <name> ..." ! %token WHILE "while" ! %put WHILE summary "while (<expr>) <stmt> | do <stmt> while (<expr>);" ;; -------------------------- --- 46,315 ---- %start interface_member_declaration %start formal_parameters ! ;; ----------------------------- ! ;; Block & Parenthesis terminals ! ;; ----------------------------- ! %type <block> syntax "\\s(\\|\\s)" matchdatatype block ! ! %token <block> PAREN_BLOCK "(LPAREN RPAREN)" ! %token <block> BRACE_BLOCK "(LBRACE RBRACE)" ! %token <block> BRACK_BLOCK "(LBRACK RBRACK)" ! ! %token <open-paren> LPAREN "(" ! %token <close-paren> RPAREN ")" ! %token <open-paren> LBRACE "{" ! %token <close-paren> RBRACE "}" ! %token <open-paren> LBRACK "[" ! %token <close-paren> RBRACK "]" ;; ------------------ ;; Operator terminals ;; ------------------ ! %type <punctuation> syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype string ! ! %token <punctuation> NOT "!" ! %token <punctuation> NOTEQ "!=" ! %token <punctuation> MOD "%" ! %token <punctuation> MODEQ "%=" ! %token <punctuation> AND "&" ! %token <punctuation> ANDAND "&&" ! %token <punctuation> ANDEQ "&=" ! %token <punctuation> MULT "*" ! %token <punctuation> MULTEQ "*=" ! %token <punctuation> PLUS "+" ! %token <punctuation> PLUSPLUS "++" ! %token <punctuation> PLUSEQ "+=" ! %token <punctuation> COMMA "," ! %token <punctuation> MINUS "-" ! %token <punctuation> MINUSMINUS "--" ! %token <punctuation> MINUSEQ "-=" ! %token <punctuation> DOT "." ! %token <punctuation> DIV "/" ! %token <punctuation> DIVEQ "/=" ! %token <punctuation> COLON ":" ! %token <punctuation> SEMICOLON ";" ! %token <punctuation> LT "<" ! %token <punctuation> LSHIFT "<<" ! %token <punctuation> LSHIFTEQ "<<=" ! %token <punctuation> LTEQ "<=" ! %token <punctuation> EQ "=" ! %token <punctuation> EQEQ "==" ! %token <punctuation> GT ">" ! %token <punctuation> GTEQ ">=" ! %token <punctuation> RSHIFT ">>" ! %token <punctuation> RSHIFTEQ ">>=" ! %token <punctuation> URSHIFT ">>>" ! %token <punctuation> URSHIFTEQ ">>>=" ! %token <punctuation> QUESTION "?" ! %token <punctuation> XOR "^" ! %token <punctuation> XOREQ "^=" ! %token <punctuation> OR "|" ! %token <punctuation> OREQ "|=" ! %token <punctuation> OROR "||" ! %token <punctuation> COMP "~" ;; ----------------- ;; Literal terminals ;; ----------------- + %type <symbol> syntax "\\(\\sw\\|\\s_\\)+" %token <symbol> IDENTIFIER + + %type <string> syntax "\\s\"" matchdatatype sexp %token <string> STRING_LITERAL + + %type <number> syntax semantic-lex-number-expression %token <number> NUMBER_LITERAL ;; ----------------- ;; Keyword terminals ;; ----------------- ! ! ;; Generate a keyword analyzer ! %type <keyword> syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword ! ! %keyword ABSTRACT "abstract" ! %put ABSTRACT summary "Class|Method declaration modifier: abstract {class|<type>} <name> ..." ! %keyword BOOLEAN "boolean" ! %put BOOLEAN summary "Primitive logical quantity type (true or false)" ! %keyword BREAK "break" ! %put BREAK summary "break [<label>] ;" ! %keyword BYTE "byte" ! %put BYTE summary "Integral primitive type (-128 to 127)" ! %keyword CASE "case" ! %put CASE summary "switch(<expr>) {case <const-expr>: <stmts> ... }" ! %keyword CATCH "catch" ! %put CATCH summary "try {<stmts>} catch(<parm>) {<stmts>} ... " ! %keyword CHAR "char" ! %put CHAR summary "Integral primitive type ('\u0000' to '\uffff') (0 to 65535)" ! %keyword CLASS "class" ! %put CLASS summary "Class declaration: class <name>" ! %keyword CONST "const" ! %put CONST summary "Unused reserved word" ! %keyword CONTINUE "continue" ! %put CONTINUE summary "continue [<label>] ;" ! %keyword DEFAULT "default" ! %put DEFAULT summary "switch(<expr>) { ... default: <stmts>}" ! %keyword DO "do" ! %put DO summary "do <stmt> while (<expr>);" ! %keyword DOUBLE "double" ! %put DOUBLE summary "Primitive floating-point type (double-precision 64-bit IEEE 754)" ! %keyword ELSE "else" ! %put ELSE summary "if (<expr>) <stmt> else <stmt>" ! %keyword EXTENDS "extends" ! %put EXTENDS summary "SuperClass|SuperInterfaces declaration: extends <name> [, ...]" ! %keyword FINAL "final" ! %put FINAL summary "Class|Member declaration modifier: final {class|<type>} <name> ..." ! %keyword FINALLY "finally" ! %put FINALLY summary "try {<stmts>} ... finally {<stmts>}" ! %keyword FLOAT "float" ! %put FLOAT summary "Primitive floating-point type (single-precision 32-bit IEEE 754)" ! %keyword FOR "for" ! %put FOR summary "for ([<init-expr>]; [<expr>]; [<update-expr>]) <stmt>" ! %keyword GOTO "goto" ! %put GOTO summary "Unused reserved word" ! %keyword IF "if" ! %put IF summary "if (<expr>) <stmt> [else <stmt>]" ! %keyword IMPLEMENTS "implements" ! %put IMPLEMENTS summary "Class SuperInterfaces declaration: implements <name> [, ...]" ! %keyword IMPORT "import" ! %put IMPORT summary "Import package declarations: import <package>" ! %keyword INSTANCEOF "instanceof" ! %keyword INT "int" ! %put INT summary "Integral primitive type (-2147483648 to 2147483647)" ! %keyword INTERFACE "interface" ! %put INTERFACE summary "Interface declaration: interface <name>" ! %keyword LONG "long" ! %put LONG summary "Integral primitive type (-9223372036854775808 to 9223372036854775807)" ! %keyword NATIVE "native" ! %put NATIVE summary "Method declaration modifier: native <type> <name> ..." ! %keyword NEW "new" ! %keyword PACKAGE "package" ! %put PACKAGE summary "Package declaration: package <name>" ! %keyword PRIVATE "private" ! %put PRIVATE summary "Access level modifier: private {class|interface|<type>} <name> ..." ! %keyword PROTECTED "protected" ! %put PROTECTED summary "Access level modifier: protected {class|interface|<type>} <name> ..." ! %keyword PUBLIC "public" ! %put PUBLIC summary "Access level modifier: public {class|interface|<type>} <name> ..." ! %keyword RETURN "return" ! %put RETURN summary "return [<expr>] ;" ! %keyword SHORT "short" ! %put SHORT summary "Integral primitive type (-32768 to 32767)" ! %keyword STATIC "static" ! %put STATIC summary "Declaration modifier: static {class|interface|<type>} <name> ..." ! %keyword STRICTFP "strictfp" ! %put STRICTFP summary "Declaration modifier: strictfp {class|interface|<type>} <name> ..." ! %keyword SUPER "super" ! %keyword SWITCH "switch" ! %put SWITCH summary "switch(<expr>) {[case <const-expr>: <stmts> ...] [default: <stmts>]}" ! %keyword SYNCHRONIZED "synchronized" ! %put SYNCHRONIZED summary "synchronized (<expr>) ... | Method decl. modifier: synchronized <type> <name> ..." ! %keyword THIS "this" ! %keyword THROW "throw" ! %put THROW summary "throw <expr> ;" ! %keyword THROWS "throws" ! %put THROWS summary "Method|Constructor declaration: throws <classType>, ..." ! %keyword TRANSIENT "transient" ! %put TRANSIENT summary "Field declaration modifier: transient <type> <name> ..." ! %keyword TRY "try" ! %put TRY summary "try {<stmts>} [catch(<parm>) {<stmts>} ...] [finally {<stmts>}]" ! %keyword VOID "void" ! %put VOID summary "Method return type: void <name> ..." ! %keyword VOLATILE "volatile" ! %put VOLATILE summary "Field declaration modifier: volatile <type> <name> ..." ! %keyword WHILE "while" ! %put WHILE summary "while (<expr>) <stmt> | do <stmt> while (<expr>);" ;; -------------------------- *************** *** 326,355 **** ;; - `with-ref' (optional) if non-nil indicates that the tag is ;; followed by a reference like in "@see <reference>". ! %token _AUTHOR "@author" ! %put _AUTHOR javadoc (seq 1 usage (type)) ! %token _VERSION "@version" ! %put _VERSION javadoc (seq 2 usage (type)) ! %token _PARAM "@param" ! %put _PARAM javadoc (seq 3 usage (function) with-name t) ! %token _RETURN "@return" ! %put _RETURN javadoc (seq 4 usage (function)) ! %token _EXCEPTION "@exception" ! %put _EXCEPTION javadoc (seq 5 usage (function) with-name t) ! %token _THROWS "@throws" ! %put _THROWS javadoc (seq 6 usage (function) with-name t) ! %token _SEE "@see" ! %put _SEE javadoc (seq 7 usage (type function variable) opt t with-ref t) ! %token _SINCE "@since" ! %put _SINCE javadoc (seq 8 usage (type function variable) opt t) ! %token _SERIAL "@serial" ! %put _SERIAL javadoc (seq 9 usage (variable) opt t) ! %token _SERIALDATA "@serialData" ! %put _SERIALDATA javadoc (seq 10 usage (function) opt t) ! %token _SERIALFIELD "@serialField" ! %put _SERIALFIELD javadoc (seq 11 usage (variable) opt t) ! %token _DEPRECATED "@deprecated" ! %put _DEPRECATED javadoc (seq 12 usage (type function variable) opt t) %% --- 336,365 ---- ;; - `with-ref' (optional) if non-nil indicates that the tag is ;; followed by a reference like in "@see <reference>". ! %keyword _AUTHOR "@author" ! %put _AUTHOR javadoc (seq 1 usage (type)) ! %keyword _VERSION "@version" ! %put _VERSION javadoc (seq 2 usage (type)) ! %keyword _PARAM "@param" ! %put _PARAM javadoc (seq 3 usage (function) with-name t) ! %keyword _RETURN "@return" ! %put _RETURN javadoc (seq 4 usage (function)) ! %keyword _EXCEPTION "@exception" ! %put _EXCEPTION javadoc (seq 5 usage (function) with-name t) ! %keyword _THROWS "@throws" ! %put _THROWS javadoc (seq 6 usage (function) with-name t) ! %keyword _SEE "@see" ! %put _SEE javadoc (seq 7 usage (type function variable) opt t with-ref t) ! %keyword _SINCE "@since" ! %put _SINCE javadoc (seq 8 usage (type function variable) opt t) ! %keyword _SERIAL "@serial" ! %put _SERIAL javadoc (seq 9 usage (variable) opt t) ! %keyword _SERIALDATA "@serialData" ! %put _SERIALDATA javadoc (seq 10 usage (function) opt t) ! %keyword _SERIALFIELD "@serialField" ! %put _SERIALFIELD javadoc (seq 11 usage (variable) opt t) ! %keyword _DEPRECATED "@deprecated" ! %put _DEPRECATED javadoc (seq 12 usage (type function variable) opt t) %% *************** *** 707,710 **** --- 717,740 ---- (identity "[]") ; + %% + + ;; Define the lexer for this grammar + (define-lex wisent-java-tags-lexer + "Lexical analyzer that handles Java buffers. + It ignores whitespaces, newlines and comments." + semantic-lex-ignore-whitespace + semantic-lex-ignore-newline + semantic-lex-ignore-comments + ;;;; Auto-generated analyzers. + wisent-java-tags-wy--<number>-regexp-analyzer + wisent-java-tags-wy--<string>-sexp-analyzer + ;; Must detect keywords before other symbols + wisent-java-tags-wy--<keyword>-keyword-analyzer + wisent-java-tags-wy--<symbol>-regexp-analyzer + wisent-java-tags-wy--<punctuation>-string-analyzer + wisent-java-tags-wy--<block>-block-analyzer + ;;;; + semantic-lex-default-action) + ;;; wisent-java-tags.wy ends here |