Thread: Re: [CEDET-devel] macros in .[bw]y files
Brought to you by:
zappo
From: David P. <dav...@wa...> - 2003-07-29 10:34:56
|
Hi Eric, [...] >>It looks that the %macro statement could be easily converted into an >>expander function. >> >>%macro FUNCTION-TAG(&rest args) >> (wisent-raw-tag (semantic-tag-new-function ,@args)) >> >>would become something like: >> >>(defun FUNCTION-TAG (&rest args) >> (wisent-raw-tag (semantic-tag-new-function ,@args))) > > > The current macros are defmacro macros, but yes, that was my thought > as well. I thought a little more on that, and finally I wonder if we are not reinventing the wheel? I am not sure that it is really worth complicating the grammar with %macro statements not really different from true lisp macro/function definitions in a .el support file. Maybe a better approach could be to adopt the grammar outline already used by bison, that is: %{ PROLOGUE %} DECLARATIONS %% GRAMMAR RULES %% EPILOGUE The bison manual says: - "The PROLOGUE section contains macro definitions and declarations of functions and variables that are used in the actions in the grammar rules. These are copied to the beginning of the parser file [...]" - "The EPILOGUE is copied verbatim to the end of the parser file, just as the PROLOGUE is copied to the beginning." Maybe we could also eliminate the %setupfunction and use a placeholder for the generated code, and automatically generate the %keywordtable, %parsetable, and %tokentable names from the %outputfile name? In my idea, the wisent-java.wy example grammar could resemble to this: ----- <file starts here> ----- %{ ;; PROLOGUE (require 'some-feature) (defun FUNCTION-TAG (&rest args) (wisent-raw-tag (semantic-tag-new-function ,@args))) ... }% ;; DECLARATIONS %outputfile "wisent-java" ... %% ;; GRAMMAR RULES ... %% ;; EPILOGUE (defun wisent-java-default-setup () "Setup buffer for Semantic." ;; My setup code (setq semantic-lex-number-expression semantic-java-number-regexp semantic-lex-analyzer 'wisent-java-tags-lexer semantic-tag-expand-function 'wisent-java-expand-tag) ... ;; Generated setup code that setup the parser, that is the ;; `parse-stream' override, the `semantic-parser-name', ;; `semantic-toplevel-bovine-table', `semantic-debug-parser-source', ;; `semantic-flex-keywords-obarray', and ;; `semantic-lex-types-obarray' buffer local variables, etc.. $PARSER-SETUP ) ;;;###autoload (add-hook 'java-mode-hook 'wisent-java-default-setup) ----- <file ends here> ------- And the generated wisent-java.el (or wisent-java.wyc ?) would look like this: ----- <file starts here> ----- ;;; wisent-java.el --- GENERATED HEADER ;; PROLOGUE (require 'some-feature) (defun FUNCTION-TAG (&rest args) (wisent-raw-tag (semantic-tag-new-function ,@args))) ... ;; THESES DECLARATIONS ARE GENERATED (defvar wisent-java-parse-table ... "The `wisent-java' parser table.") (defvar wisent-java-keyword-table ... "The `wisent-java' keyword table.") (defvar wisent-java-token-table ... "The `wisent-java' token table.") ;; EPILOGUE (defun wisent-java-default-setup () "Setup buffer for Semantic." ;; My setup code (setq semantic-lex-number-expression semantic-java-number-regexp semantic-lex-analyzer 'wisent-java-tags-lexer semantic-tag-expand-function 'wisent-java-expand-tag) ... ;; Generated setup code that setup the parser, that is the ;; `parse-stream' override, the `semantic-parser-name', ;; `semantic-toplevel-bovine-table', `semantic-debug-parser-source', ;; `semantic-flex-keywords-obarray', and ;; `semantic-lex-types-obarray' buffer local variables, etc.. ;; GENERATED $PARSER-SETUP CODE: (progn (semantic-install-function-overrides '((parse-stream . wisent-parse-stream))) (setq semantic-parser-name "LALR" semantic-toplevel-bovine-table wisent-java-parser-tables semantic-debug-parser-source "wisent-java-tags.wy" semantic-flex-keywords-obarray wisent-java-keywords semantic-lex-types-obarray wisent-java-tokens) ;; Collect unmatched syntax lexical tokens (semantic-make-local-hook 'wisent-discarding-token-functions) (add-hook 'wisent-discarding-token-functions 'wisent-collect-unmatched-syntax nil t)) ) ;;;###autoload (add-hook 'java-mode-hook #'wisent-java-default-setup) ;; GENERATED FOOTER (provide 'wisent-java) ;;; wisent-java.el ends here ----- <file end here> ------- What do you think? David |
From: David P. <dav...@wa...> - 2003-07-31 11:07:38
Attachments:
semantic-grammar.tar.gz
|
Hi Eric, > I like your idea. The macros for TAGS could live in one specific > .elparser install file which could be (require'd) in the Prologue. > That satisfies all requirements I think, and will make wisent more > familiar and easier to use. I implemented the grammar outline of bison for semantic grammars: %{ PROLOGUE %} DECLARATIONS %% GRAMMAR RULES %% EPILOGUE The keywords %outputfile, %keywordtable, %tokentable, %parsetable, and %setupfunction are now obsolete and ignored (issue a warning when parsed). I changed the %outputfile clause into a more generic "%package SYMBOL" clause, that just identify the library where lisp code is generated. For example: %package foo Will generate the lisp code into the file foo.el. It will be easy to generate into a .wyc or .byc file later. The lisp code generation works like this: 1. The <package>.el file is opened and erased. 2. A standard header + the prologue code is copied at the beginning of the buffer. 3. These declarations follow: - keyword table (defconst <package>-keyword-table ...) - token table (defconst <package>-token-table ...) - parse table (defconst <package>-parse-table ...) - parser install function (defun <package>-install-parser () ...) 4. The epilogue code is copied at the end of the buffer, followed by a standard footer (<package> is provided). The `<package>-install-parser' function setup the parser internals. That is, the parser function override, the parser name, debugging informations, keywords and tokens obarrays, bovinate table, and other parser specific stuff. That function must be called from the user's mode hook function (no more generated) where other user's semantic variables and hooks are setup. Unfortunately that introduce an incompatibility with the previous behavior :-( However I think the new design is clearer and conform to bison's one. For now I updated the `wisent-grammar-mode', as it is needed to process input grammars! It should not be difficult to update the `bovine-grammar-mode' accordingly, and I will do that shortly. I also updated the sample wisent-calc grammar and support file. To resume, with the new outline, for each grammar we now have at least three files: - the grammar - the generated library - a support file (lexer, overrides, etc.) that requires the above. > The trick being the creation of wisent MACROS which > also require a modification to the wisent code generator via a > cond statement. Perhaps `define-wisent-macro' or some such could > be used to do both tasks in some clever way. > > You could write only one .wy file, compiled to .wyc, and have a > working tag generator. All the overload functions would probably > still want to live in a second Emacs Lisp file which is require'd in > the prologue, however. I must confess that I am stuck on that subject :-( In my idea, the current "hard coded" implementation could be simply removed and replaced by true Elisp macros that could be put in a separate file required in the grammar prologue. The macro substitution would not be done at code generation, but in the normal lisp manner, at byte-compilation or execution time. Consequently, font-locking wouldn't be possible anymore, unless we adopt and recommend a strict naming convention for grammar macros. I guess the difficulty is that Emacs don't support packages, so it is necessary to prefix macro names with the library name to avoid collisions. Maybe have you better ideas? Anyway, attached you will find a tarball of the new grammar outline implementation. The files are semantic-grammar-2.wy, semantic-grammar-2.el, wisent-grammar-2.el and a new semantic-grammar-support.el file generated from the grammar. The wisent-calc sample is also included. To try it, I suggest you put these files in a separate directory in your load-path, and (require 'wisent-grammar-2) before opening the new grammars. The other grammar/support files must be converted before making the new implementation the default :-( I am looking forward for your feedback. Enjoy! David |
From: Eric M. L. <er...@si...> - 2003-07-31 14:04:10
|
>>> David PONCE <dav...@wa...> seems to think that: >Hi Eric, > >> I like your idea. The macros for TAGS could live in one specific >> .elparser install file which could be (require'd) in the Prologue. >> That satisfies all requirements I think, and will make wisent more >> familiar and easier to use. > >I implemented the grammar outline of bison for semantic grammars: > > %{ > PROLOGUE > %} > > DECLARATIONS > > %% > GRAMMAR RULES > %% > > EPILOGUE > >The keywords %outputfile, %keywordtable, %tokentable, %parsetable, and >%setupfunction are now obsolete and ignored (issue a warning when >parsed). This is great. Reducing the work needed to write a new grammar is a good idea. >I changed the %outputfile clause into a more generic "%package SYMBOL" >clause, that just identify the library where lisp code is generated. > >For example: > >%package foo > >Will generate the lisp code into the file foo.el. It will be easy to >generate into a .wyc or .byc file later. Clever idea. This abstraction already shows additional uses as you indirectly indicate in the next paragraph. >The lisp code generation works like this: > >1. The <package>.el file is opened and erased. > >2. A standard header + the prologue code is copied at the beginning of > the buffer. > >3. These declarations follow: > > - keyword table > > (defconst <package>-keyword-table > ...) > > - token table > > (defconst <package>-token-table > ...) > > - parse table > (defconst <package>-parse-table > ...) > > - parser install function > > (defun <package>-install-parser () > ...) Previously these developer named items were `semantic-<package>-something-table'. A developer will never see these names, perhaps there should be an additional prefix to make sure cc-mode.el doesn't accidentally use a `token-table' for some other use. >4. The epilogue code is copied at the end of the buffer, followed by a > standard footer (<package> is provided). semantic-grammar-2.wy does not have an epilogue. wisent-calc.wy has a setup function in its epilogue. It appears wisent-calc.el initializes the parser twice, an the lexical analyzer twice. >The `<package>-install-parser' function setup the parser internals. >That is, the parser function override, the parser name, debugging >informations, keywords and tokens obarrays, bovinate table, and other >parser specific stuff. > >That function must be called from the user's mode hook function (no >more generated) where other user's semantic variables and hooks are >setup. Unfortunately that introduce an incompatibility with the >previous behavior :-( However I think the new design is clearer and >conform to bison's one. This seems like a good idea. It keeps the generated code separate from the user code in a cleaner way. >For now I updated the `wisent-grammar-mode', as it is needed to >process input grammars! > >It should not be difficult to update the `bovine-grammar-mode' >accordingly, and I will do that shortly. > >I also updated the sample wisent-calc grammar and support file. > >To resume, with the new outline, for each grammar we now have at >least three files: > >- the grammar >- the generated library >- a support file (lexer, overrides, etc.) that requires the above. We had discussed in the past if each language got it's own directory in the hierarchy, or just lived with other grammars. In the distant future, I could imagine additional files for font lock, and indentation as well. >> The trick being the creation of wisent MACROS which >> also require a modification to the wisent code generator via a >> cond statement. Perhaps `define-wisent-macro' or some such could >> be used to do both tasks in some clever way. >> >> You could write only one .wy file, compiled to .wyc, and have a >> working tag generator. All the overload functions would probably >> still want to live in a second Emacs Lisp file which is require'd in >> the prologue, however. > >I must confess that I am stuck on that subject :-( In my idea, the >current "hard coded" implementation could be simply removed and >replaced by true Elisp macros that could be put in a separate file >required in the grammar prologue. The macro substitution would not be >done at code generation, but in the normal lisp manner, at >byte-compilation or execution time. Consequently, font-locking >wouldn't be possible anymore, unless we adopt and recommend a strict >naming convention for grammar macros. I guess the difficulty is that >Emacs don't support packages, so it is necessary to prefix macro names >with the library name to avoid collisions. > >Maybe have you better ideas? By macros, I mean that if you write a tagging parser, FUNCTION-TAG is good. If you wrote a font lock parser (in yet another file) you would want a FUNCTION-COLOR macro instead. In an indenting parser, you would want an INCREMENT-INDENT function instead. There would be no naming conflict. Unfortunately, such a future world would require three different parsers! That, in itself, is kind of icky. It would be much nicer to have one grammar, and use it to do all three. Having three sets of actions in one grammar would also be icky. The grammar could create an intermediate representation (full parse tree) and then Lisp code could walk the tree to do work. The dissociation of actions from the grammar is icky too. I do not know what the right goal is. >Anyway, attached you will find a tarball of the new grammar outline >implementation. The files are semantic-grammar-2.wy, >semantic-grammar-2.el, wisent-grammar-2.el and a new >semantic-grammar-support.el file generated from the grammar. The >wisent-calc sample is also included. Thanks! I will endeavor to try it. >To try it, I suggest you put these files in a separate directory in >your load-path, and (require 'wisent-grammar-2) before opening the >new grammars. The other grammar/support files must be converted >before making the new implementation the default :-( The other grammars are already compiled to lisp, so you could make this the default as long as you don't try to recompile the older .wy files, or did internal stuff change beyond just the format of the grammar files? Have fun! Eric -- Eric Ludlam: za...@gn..., er...@si... Home: http://www.ludlam.net Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: David P. <dav...@wa...> - 2003-07-31 16:55:12
|
Eric, [...] >>3. These declarations follow: >> >> - keyword table >> >> (defconst <package>-keyword-table >> ...) >> >> - token table >> >> (defconst <package>-token-table >> ...) >> >> - parse table >> (defconst <package>-parse-table >> ...) >> >> - parser install function >> >> (defun <package>-install-parser () >> ...) > > > Previously these developer named items were > `semantic-<package>-something-table'. A developer will never see > these names, perhaps there should be an additional prefix to make > sure cc-mode.el doesn't accidentally use a `token-table' for some > other use. That is a good point. Perhaps we could use something like `<package>--' as prefix? The only name that the developer must see is the name of the install-parser function that must be called from a mode hook. >>4. The epilogue code is copied at the end of the buffer, followed by a >> standard footer (<package> is provided). > > > semantic-grammar-2.wy does not have an epilogue. There is no need for one. The code to setup semantic is part of the `semantic-grammar-mode' setup. > wisent-calc.wy has a setup function in its epilogue. It appears > wisent-calc.el initializes the parser twice, an the lexical analyzer > twice. I don't see your point here. For me all seems OK. The epilogue defines the user function `wisent-calc-setup-parser' that calls the generated internal setup function `wisent-calc-support-install-parser'. Where are the parser and lexer initialized twice? [...] >>To resume, with the new outline, for each grammar we now have at >>least three files: >> >>- the grammar >>- the generated library >>- a support file (lexer, overrides, etc.) that requires the above. > > > We had discussed in the past if each language got it's own directory > in the hierarchy, or just lived with other grammars. To avoid multiplication of small directories (and avoid cluttering up the `load-path' with them), maybe should I remove the %package clause and make it implicit, like bison does? That is, a grammar foo.[wb]y could produce a library foo-[wb]y.el (an implicit foo-[wb]y package). Thus for each language "foo" we will have: foo.[wb]y, foo-[wb]y.el and a main foo.el file. [...] > By macros, I mean that if you write a tagging parser, FUNCTION-TAG is > good. If you wrote a font lock parser (in yet another file) you > would want a FUNCTION-COLOR macro instead. In an indenting parser, > you would want an INCREMENT-INDENT function instead. There would be > no naming conflict. I am afraid but I don't clearly see your point here. Could you please give a short example on how these different parsers would work? [...] > The other grammars are already compiled to lisp, so you could make > this the default as long as you don't try to recompile the older .wy > files, or did internal stuff change beyond just the format of the > grammar files? There is no other change but the format. So, I can commit my work and start to convert grammars and support files right now :) More probably, after we have closed this thread ;-) Thanks! David |
From: Eric M. L. <er...@si...> - 2003-07-31 18:18:56
|
>>> David PONCE <dav...@wa...> seems to think that: >Eric, > >[...] >>>3. These declarations follow: >>> >>> - keyword table >>> >>> (defconst <package>-keyword-table >>> ...) >>> >>> - token table >>> >>> (defconst <package>-token-table >>> ...) >>> >>> - parse table >>> (defconst <package>-parse-table >>> ...) >>> >>> - parser install function >>> >>> (defun <package>-install-parser () >>> ...) >> >> >> Previously these developer named items were >> `semantic-<package>-something-table'. A developer will never see >> these names, perhaps there should be an additional prefix to make >> sure cc-mode.el doesn't accidentally use a `token-table' for some >> other use. > >That is a good point. Perhaps we could use something like >`<package>--' as prefix? That might be a good idea. >The only name that the developer must see is the name of the >install-parser function that must be called from a mode hook. Perhaps we could define an EIEIO object for a parser, and hide all that stuff as slots in an instance of that object. We already have a strong dependency on EIEIO so it might not be so bad. >>>4. The epilogue code is copied at the end of the buffer, followed by a >>> standard footer (<package> is provided). >> >> >> semantic-grammar-2.wy does not have an epilogue. > >There is no need for one. The code to setup semantic is part of >the `semantic-grammar-mode' setup. > >> wisent-calc.wy has a setup function in its epilogue. It appears >> wisent-calc.el initializes the parser twice, an the lexical analyzer >> twice. > >I don't see your point here. For me all seems OK. The epilogue >defines the user function `wisent-calc-setup-parser' that calls the >generated internal setup function `wisent-calc-support-install-parser'. > >Where are the parser and lexer initialized twice? I think I had my symbols crossed. I thought wisent-calc called both wisent-calc-setup-parser and wisent-calc-support-install-parser, but I was wrong. Strangely wisent-calc-setup-parser turns on lexical features, and confused semantic-lex-init with something specific to calc. >[...] >>>To resume, with the new outline, for each grammar we now have at >>>least three files: >>> >>>- the grammar >>>- the generated library >>>- a support file (lexer, overrides, etc.) that requires the above. >> >> >> We had discussed in the past if each language got it's own directory >> in the hierarchy, or just lived with other grammars. > >To avoid multiplication of small directories (and avoid cluttering up >the `load-path' with them), maybe should I remove the %package clause >and make it implicit, like bison does? That is, a grammar foo.[wb]y >could produce a library foo-[wb]y.el (an implicit foo-[wb]y package). >Thus for each language "foo" we will have: foo.[wb]y, foo-[wb]y.el and >a main foo.el file. It seems to me that the effect would be the same. Bison does offer command line arguments to change where the output goes, however. Perhaps package could just be optional. >[...] >> By macros, I mean that if you write a tagging parser, FUNCTION-TAG is >> good. If you wrote a font lock parser (in yet another file) you >> would want a FUNCTION-COLOR macro instead. In an indenting parser, >> you would want an INCREMENT-INDENT function instead. There would be >> no naming conflict. > >I am afraid but I don't clearly see your point here. Could you please >give a short example on how these different parsers would work? Here is an example from semantic-grammar-2.wy: token_decl: TOKEN token_type_opt SYMBOL string_value `(TAG ',$3 ',(if $2 'token 'keyword) :type ',$2 :value ',$4) | TOKEN token_type_opt symbols `(TAG ',(car $3) 'token :type ',$2 :rest ',(cdr $3)) ; This create s tags. Font lock might look like this: token_decl: TOKEN token_type_opt SYMBOL string_value `(progn (FACE keyword $1) (FACE type $2) (FACE variable $3) (FACE string $4)) | TOKEN token_type_opt symbols ... ; As macros, they do macro things just like TAG. You probably don't want a FACE macro available while tagging, nor a TAG macro available while font locking. >[...] >> The other grammars are already compiled to lisp, so you could make >> this the default as long as you don't try to recompile the older .wy >> files, or did internal stuff change beyond just the format of the >> grammar files? > >There is no other change but the format. So, I can commit my work and >start to convert grammars and support files right now :) >More probably, after we have closed this thread ;-) [ ... ] Sounds good to me. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: http://www.ludlam.net Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: David P. <dav...@wa...> - 2003-08-01 10:54:37
|
Hi Eric, [...] >>That is a good point. Perhaps we could use something like >>`<package>--' as prefix? > > > That might be a good idea. I did that. >>The only name that the developer must see is the name of the >>install-parser function that must be called from a mode hook. > > > Perhaps we could define an EIEIO object for a parser, and hide all > that stuff as slots in an instance of that object. We already have a > strong dependency on EIEIO so it might not be so bad. Sounds like a good idea. We could replace all the buffer local variables used for parsing by one buffer local variable that would point to the current parser instance, and use methods to retrieve the keyword, token, and parser tables, the parser name, and the grammar source file name for debugging. Also we could have abstract methods to install hooks and overrides. The root abstract parser class could look like this: My code is probably wrong. I miss competence in the use of EIEIO ;-) (defclass semantic-parser () ((name :accessor semantic-parser-name :documentation "Parser name") (keyword-table :accessor semantic-parser-keyword-table :documentation "Table of language keywords.") (token-table :accessor semantic-parser-token-table :documentation "Table of language tokens.") (parse-table :accessor semantic-parser-parse-table :documentation "Table of parser rules.") (grammar-file :accessor semantic-parser-grammar-file :documentation "Source of the grammar used to build this parser.") ) "A Semantic parser." :abstract t) (defmethod semantic-parser-install-overrides ((parser semantic-parser)) "Install parser function overrides." nil) (defmethod semantic-parser-install-hooks ((parser semantic-parser)) "Install hooks used by this parser." nil) Using a such paradigm would lead to make EIEIO mandatory for Semantic. [...] >>To avoid multiplication of small directories (and avoid cluttering up >>the `load-path' with them), maybe should I remove the %package clause >>and make it implicit, like bison does? That is, a grammar foo.[wb]y >>could produce a library foo-[wb]y.el (an implicit foo-[wb]y package). >>Thus for each language "foo" we will have: foo.[wb]y, foo-[wb]y.el and >>a main foo.el file. > > > It seems to me that the effect would be the same. Bison does offer > command line arguments to change where the output goes, however. > Perhaps package could just be optional. Good idea. I made %package optional. By default it is defined as proposed above. [...] >>I am afraid but I don't clearly see your point here. Could you please >>give a short example on how these different parsers would work? > > > Here is an example from semantic-grammar-2.wy: > > token_decl: > TOKEN token_type_opt SYMBOL string_value > `(TAG ',$3 ',(if $2 'token 'keyword) :type ',$2 :value ',$4) > | TOKEN token_type_opt symbols > `(TAG ',(car $3) 'token :type ',$2 :rest ',(cdr $3)) > ; > > This create s tags. Font lock might look like this: > > token_decl: > TOKEN token_type_opt SYMBOL string_value > `(progn (FACE keyword $1) > (FACE type $2) > (FACE variable $3) > (FACE string $4)) > | TOKEN token_type_opt symbols > ... > ; > > As macros, they do macro things just like TAG. You probably don't > want a FACE macro available while tagging, nor a TAG macro available > while font locking. In fact, what I don't clearly see is how to associate a set of grammar macros to a specific grammar. For example: foo-macros.el: (defmacro TAG (&rest args) `(foo-something ,@args)) foo.wy: %{ (require 'foo-macros) %} rule: component (TAG $1) ; foo-wy.el (require 'foo-macros) (defconst foo-wy--parse-table ... (rule ((component) (TAG $1))) ...) At byte-compilation/runtime TAG will be expanded to `foo-something'. However if there is a "bar" grammar defined like this: bar-macros.el: (defmacro TAG (&rest args) `(bar-something ,@args)) bar.wy: %{ (require bar-macros) %} rule: component (TAG $1) ; bar-wy.el (require 'bar-macros) (defconst bar-wy--parse-table ... (rule ((component) (TAG $1))) ...) TAG will be expanded to `bar-something' as expected, but the new TAG macro will override the previous definition provided by foo-macros (there is a name conflict). How do you think we can solve that? >>There is no other change but the format. So, I can commit my work and >>start to convert grammars and support files right now :) >>More probably, after we have closed this thread ;-) > > [ ... ] > > Sounds good to me. I already converted the whole wisent directory + the bovine-grammar mode. It remains to convert .by and support files. A big change in view ;-) David |
From: Eric M. L. <er...@si...> - 2003-08-01 12:20:09
|
>>> David PONCE <dav...@wa...> seems to think that: >Hi Eric, > >[...] >>>That is a good point. Perhaps we could use something like >>>`<package>--' as prefix? >> >> >> That might be a good idea. > >I did that. Great! >>>The only name that the developer must see is the name of the >>>install-parser function that must be called from a mode hook. >> >> >> Perhaps we could define an EIEIO object for a parser, and hide all >> that stuff as slots in an instance of that object. We already have a >> strong dependency on EIEIO so it might not be so bad. > >Sounds like a good idea. We could replace all the buffer local >variables used for parsing by one buffer local variable that would >point to the current parser instance, and use methods to retrieve the >keyword, token, and parser tables, the parser name, and the grammar >source file name for debugging. Also we could have abstract methods >to install hooks and overrides. The root abstract parser class could >look like this: > >My code is probably wrong. I miss competence in the use of EIEIO ;-) > >(defclass semantic-parser () > ((name > :accessor semantic-parser-name > :documentation "Parser name") > (keyword-table > :accessor semantic-parser-keyword-table > :documentation "Table of language keywords.") > (token-table > :accessor semantic-parser-token-table > :documentation "Table of language tokens.") > (parse-table > :accessor semantic-parser-parse-table > :documentation "Table of parser rules.") > (grammar-file > :accessor semantic-parser-grammar-file > :documentation "Source of the grammar used to build this parser.") > ) > "A Semantic parser." > :abstract t) A great start. I rarely use the :accessor slot attribute. CLOS needs it as a short hand because the eieio `oset' equivalent is (fset (slot-value obj 'field) value), and an accessor called `foo' makes this more readable like this: (fset (foo obj) value) I do use accessors to make code more readable. You can also create read-only slots with them, where there is an :initarg, and :accessor, and a :protection of 'private. I tend not to think quite that far ahead though I probably should. Anyway, here is a start using more slot attributes: (defclass semantic-parser () ((name :type string :accessor semantic-parser-name :protection private :allocation class :documentation "Parser name") (keyword-table :type list :protection private :accessor semantic-parser-keyword-table :allocation class :documentation "Table of language keywords.") (token-table :type list :protection private :accessor semantic-parser-token-table :allocation class :documentation "Table of language tokens.") (parse-table :type list :protection private :accessor semantic-parser-parse-table :allocation class :documentation "Table of parser rules.") (grammar-file :type string :accessor semantic-parser-grammar-file :allocation class :documentation "Source of the grammar used to build this parser.") ) "A Semantic parser." :abstract t) You could then subclass for a language: (defclass java-parser (semantic-parser) (name :initform "java") (keyword-table :initform ( big list o stuff)) (token-table :initform ( bigger list o stuff)) ... (java-thingy :documentation "Useful storage while parsing java.") "Parser for Java.") which would basically share all root data with a core class. A run-time modification of the language grammar would effect all buffers without any additional work. >(defmethod semantic-parser-install-overrides ((parser semantic-parser)) > "Install parser function overrides." > nil) > >(defmethod semantic-parser-install-hooks ((parser semantic-parser)) > "Install hooks used by this parser." > nil) Good ideas here too. >Using a such paradigm would lead to make EIEIO mandatory for Semantic. It is already mandatory if you want to use the Makefile. As time moves on I worry about this less because EIEIO makes life so much easier. >[...] >>>To avoid multiplication of small directories (and avoid cluttering up >>>the `load-path' with them), maybe should I remove the %package clause >>>and make it implicit, like bison does? That is, a grammar foo.[wb]y >>>could produce a library foo-[wb]y.el (an implicit foo-[wb]y package). >>>Thus for each language "foo" we will have: foo.[wb]y, foo-[wb]y.el and >>>a main foo.el file. >> >> >> It seems to me that the effect would be the same. Bison does offer >> command line arguments to change where the output goes, however. >> Perhaps package could just be optional. > >Good idea. I made %package optional. By default it is defined as >proposed above. Thanks! >[...] [ ... ] > >In fact, what I don't clearly see is how to associate a set of >grammar macros to a specific grammar. > >For example: > >foo-macros.el: > >(defmacro TAG (&rest args) > `(foo-something ,@args)) > >foo.wy: > >%{ > (require 'foo-macros) > %} > >rule: component > (TAG $1) > ; > >foo-wy.el > >(require 'foo-macros) > >(defconst foo-wy--parse-table > ... > (rule > ((component) > (TAG $1))) > ...) > >At byte-compilation/runtime TAG will be expanded to `foo-something'. > >However if there is a "bar" grammar defined like this: > >bar-macros.el: > >(defmacro TAG (&rest args) > `(bar-something ,@args)) > >bar.wy: > >%{ > (require bar-macros) > %} > >rule: component > (TAG $1) > ; > >bar-wy.el > >(require 'bar-macros) > >(defconst bar-wy--parse-table > ... > (rule > ((component) > (TAG $1))) > ...) > >TAG will be expanded to `bar-something' as expected, but the new TAG >macro will override the previous definition provided by foo-macros >(there is a name conflict). > >How do you think we can solve that? I understand your question now! The parser generator needs to associate a string with a macro, so the macro defined in a parser macros file may look like this: (defmacro foo-TAG (&rest args) `(bar-something ,@args)) (define-macro-for-this-run-of-the-parser-generator "TAG" 'foo-TAG) This seems problematic though. Perhaps the association of the string "TAG" to 'foo-tag needs to somehow be in the grammar since it is the parser-generator which uses that string, and nothing else. >>>There is no other change but the format. So, I can commit my work and >>>start to convert grammars and support files right now :) >>>More probably, after we have closed this thread ;-) >> >> [ ... ] >> >> Sounds good to me. > >I already converted the whole wisent directory + the bovine-grammar >mode. It remains to convert .by and support files. A big change in >view ;-) [ ... ] That sounds great. I will be leaving for a short holiday this evening, so hopefully I can investigate these tasks when I return next week. Thanks for your efforts! Eric -- Eric Ludlam: za...@gn..., er...@si... Home: http://www.ludlam.net Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: David P. <dav...@wa...> - 2003-08-02 08:43:03
|
Hi Eric, >>Sounds like a good idea. We could replace all the buffer local >>variables used for parsing by one buffer local variable that would >>point to the current parser instance, and use methods to retrieve the >>keyword, token, and parser tables, the parser name, and the grammar >>source file name for debugging. Also we could have abstract methods >>to install hooks and overrides. The root abstract parser class could >>look like this: [...] > > A great start. I rarely use the :accessor slot attribute. CLOS needs > it as a short hand because the eieio `oset' equivalent is [...] > > Anyway, here is a start using more slot attributes: > Thanks for your explanation! However, I would prefer to let you implement those classes ;-) >>Using a such paradigm would lead to make EIEIO mandatory for Semantic. > > It is already mandatory if you want to use the Makefile. As time > moves on I worry about this less because EIEIO makes life so much > easier. Agreed. [...] > I understand your question now! The parser generator needs to > associate a string with a macro, so the macro defined in a parser > macros file may look like this: > > (defmacro foo-TAG (&rest args) > `(bar-something ,@args)) > > (define-macro-for-this-run-of-the-parser-generator "TAG" 'foo-TAG) > > This seems problematic though. Perhaps the association of the string > "TAG" to 'foo-tag needs to somehow be in the grammar since it is the > parser-generator which uses that string, and nothing else. I thought to that too. Perhaps could we use a new %expandmacro keyword like this: %expandmacro TAG foo-TAG To generate association of a grammar macro with an expander? [...] >>I already converted the whole wisent directory + the bovine-grammar >>mode. It remains to convert .by and support files. A big change in >>view ;-) > > [ ... ] > > That sounds great. I will be leaving for a short holiday this > evening, so hopefully I can investigate these tasks when I return > next week. Hope you will have a fun holiday ;-) I terminated conversion to the new grammar layout, and checked-in a bunch of changes! Following is the change log. Hope these changes didn't break things ;-) David 2003-08-01 David Ponce <da...@dp...> New outline of semantic input grammars, conform to bison's one. Lisp code generated from a grammar is now in its own library. * semantic/semantic-grammar.el (semantic-grammar-wy, sformat): Require. (semantic-grammar-lex-prologue) (semantic-grammar-lex-epilogue): New lex analyzers. (semantic-grammar-lexer): Use them. (semantic-grammar-lex-epilogue-start): New function. (semantic-grammar-automaton) (semantic-grammar-keywords) (semantic-grammar-tokens): Remove. (semantic-grammar-prologue) (semantic-grammar-epilogue) (semantic-grammar-package): New functions. (semantic-grammar-setupcode-text) (semantic-grammar-setupcode-forms) (semantic-grammar-outputfile): Remove. (semantic-grammar-setupfunction) (semantic-grammar-tokentable) (semantic-grammar-parsetable) (semantic-grammar-keywordtable): New implementation. (semantic-grammar-autogen-cookie) (semantic-grammar-autogen-cookie-re) (semantic-grammar-buffer) (semantic-grammar-beginning-of-code) (semantic-grammar-beginning-of-body) (semantic-grammar-with-outputfile) (semantic-grammar-with-grammar-buffer) (semantic-grammar-update-def) (semantic-grammar-update-parsetable) (semantic-grammar-update-keywordtable) (semantic-grammar-update-tokentable) (semantic-grammar-update-setupfunction) (semantic-grammar-update-outputfile) (semantic-grammar-setupcode-value) (semantic-grammar-keywordtable-value) (semantic-grammar-tokentable-value) (semantic-grammar-parsetable-value): Remove. (semantic--grammar-input-buffer) (semantic--grammar-output-buffer) (semantic-grammar-insert-defconst) (semantic-grammar-insert-defun) (semantic-grammar-header-template) (semantic-grammar-footer-template) (semantic-grammar-header) (semantic-grammar-footer) (semantic-grammar-token-data) (semantic-grammar-keyword-data) (semantic-grammar-parser-data) (semantic-grammar-setup-data) (semantic-grammar-create-package): New. (semantic-grammar-syntax-table): Give word syntax class to '%'. (semantic-grammar-mode-keywords-1): Update for new syntax. (semantic-grammar-map): Map C-c C-c to `semantic-grammar-create-package'. (semantic-grammar-in-lisp-p): Take into account the epilogue code. (emantic-grammar-do-lisp-indent): Likewise. (semantic-grammar-mode): Doc fix. Update semantic setup. * semantic/semantic-grammar.wy Parse new grammar outline. (setupcode, %outputfile, %parsetable) (%tokentable, %setupfunction): Remove. (%start) Add prologue epilogue. (LANGUAGEMODE, LEFT, NONASSOC, PREC, PUT, QUOTEMODE) (RIGHT, SCOPESTART, START, TOKEN) (KEYWORDTABLE, OUTPUTFILE, PARSETABLE) (SETUPFUNCTION, TOKENTABLE): Keywords begin with `%'. (PACKAGE): New keyword. (PERCENT_PERCENT, PROLOGUE, EPILOGUE): New tokens. (PERCENT): Remove. (grammar): Add prologue and epilogue rules. (prologue, epilogue): New rules. (decl): Add package_decl, precedence_decl rules. Remove left_decl, right_decl and nonassoc decl. (package_decl, precedence_decl, associativity): New rules. (any_symbol, level): Remove. All callers changed. * semantic/semantic-grammar-wy.el New generated file. * semantic/bovine/bovine-grammar.el (bovine-grammar-setupcode-builder): Only setup parser internals. * semantic/bovine/c.by (setupcode, %outputfile, %parsetable) (%keywordtable, %setupfunction): Remove. (%package): New. * semantic/bovine/semantic-c.el (semantic-c-by): Require. (semantic-toplevel-c-bovine-table) (semantic-c-keyword-table): Remove. (semantic-default-c-setup): Call generated function to setup parser internals. Do user's setup here. Use new variable `senator-step-at-tag-classes'. * semantic/bovine/semantic-c-by.el New generated file. * semantic/bovine/make.by (setupcode, %outputfile, %parsetable) (%keywordtable, %setupfunction): Remove. (%package): New. * semantic/bovine/semantic-make.el (semantic-make-by): Require. (semantic-toplevel-make-bovine-table) (semantic-make-keyword-table): Remove. (semantic-default-make-setup): Call generated function to setup parser internals. Do user's setup here. * semantic/bovine/semantic-make-by.el New generated file. * semantic/bovine/scheme.by (setupcode, %outputfile, %parsetable) (%keywordtable, %setupfunction): Remove. (%package): New. * semantic/bovine/semantic-scm.el (semantic-scm-by): Require. (semantic-toplevel-scheme-bovine-table) (semantic-scheme-keyword-table): Remove. (semantic-default-scheme-setup): Call generated function to setup parser internals. Do user's setup here. * semantic/bovine/semantic-scm-by.el New generated file. * semantic/bovine/skeleton.by (setupcode, %outputfile, %parsetable) (%keywordtable, %setupfunction): Remove. (%package): New. * semantic/bovine/semantic-skel.el (semantic-skeleton-by): Require. (semantic-toplevel-skeleton-bovine-table) (semantic-scheme-skeleton-table): Remove. (semantic-default-skel-setup): Call generated function to setup parser internals. Do user's setup here. * semantic/bovine/semantic-skeleton-by.el New generated file. * semantic/wisent/wisent-awk.wy (%outputfile, %parsetable, %tokentable) (%keywordtable, %setupfunction): Remove. * semantic/wisent/wisent-cim.wy (%outputfile, %parsetable, %tokentable) (%keywordtable, %setupfunction): Remove. * semantic/wisent/wisent-awk.el Obsolete file removed. * semantic/wisent/wisent-cim.el Obsolete file removed. * semantic/wisent/wisent-c.wy (%outputfile, %parsetable, %tokentable) (%keywordtable, %setupfunction): Remove. (prologue): New. * semantic/wisent/wisent-c.el (wisent-c-wy): Require. (semantic-ast): Remove. It is required in above library. (wisent-c-parser-automaton, wisent-c-keyword-table) (wisent-c-token-table): Remove. (wisent-c-default-setup): Call generated function to setup parser internals. Do user's setup here. Use new variable `senator-step-at-tag-classes'. * semantic/wisent/wisent-c-wy.el New generated file. * semantic/wisent/wisent-calc.wy (%outputfile, %parsetable, %tokentable) (%setupfunction, setupcode): Remove. (epilogue) <wisent-calc-setup-parser>: New. * semantic/wisent/wisent-calc.el (wisent-calc-wy): Require. (wisent-calc-automaton, wisent-c-keyword-table) (wisent-calc-tokens): Remove. (wisent-calc-setup-parser): Moved in grammar epilogue. (wisent-calc): No more depend on `wisent-calc-automaton'. * semantic/wisent/wisent-calc-wy.el New generated file. * semantic/wisent/wisent-grammar.el (require 'cl): Remove. (wisent-grammar-EXPAND) (wisent-grammar-EXPANDFULL): Fix error message. (wisent-grammar-CODE-TAG): Add missing comma. (wisent-grammar-expand-builtins): No more expand macros. (wisent-grammar-expand-sexpr): Remove. All callers changed. (wisent-grammar-setupcode-builder): Return only code to setup parser internals. * semantic/wisent/wisent-java.wy (%outputfile, %parsetable, %tokentable) (%keywordtable, %setupfunction, setupcode): Remove. * semantic/wisent/wisent-java-tags.wy (%outputfile, %parsetable, %tokentable) (%keywordtable, %setupfunction, setupcode): Remove. * semantic/wisent/wisent-java.el (wisent-java-wy): Require. (wisent-java-parser-tables, wisent-java-keywords) (wisent-java-tokens): Remove. (wisent-java-default-setup): Call generated function to setup parser internals. Do user's setup here. Use new variable `senator-step-at-tag-classes'. (semantic-default-java-setup): Remove obsolete `remove-hook'. * semantic/wisent/wisent-java-tags.el (wisent-java-tags-wy): Require. (wisent-java-parser-tables, wisent-java-keywords) (wisent-java-tokens): Remove. (wisent-java-default-setup): Call generated function to setup parser internals. Do user's setup here. Use new variable `senator-step-at-tag-classes'. * semantic/wisent/wisent-java-wy.el New generated file. * semantic/wisent/wisent-java-tags-wy.el New generated file. * semantic/wisent/wisent-python.wy (%outputfile, %parsetable, %tokentable) (%keywordtable, %setupfunction, setupcode): Remove. * semantic/wisent/wisent-python.el (wisent-python-wy): Require. (wisent-python-parser-tables, wisent-python-keywords) (wisent-python-tokens): Remove. (wisent-python-default-setup): Moved. Call generated function to setup parser internals. Do user's setup here. * semantic/wisent/wisent-python-wy.el New generated file. |
From: Eric M. L. <er...@si...> - 2003-08-04 16:58:15
|
>>> David Ponce <dav...@wa...> seems to think that: >Hi Eric, > > >>Sounds like a good idea. We could replace all the buffer local > >>variables used for parsing by one buffer local variable that would > >>point to the current parser instance, and use methods to retrieve the > >>keyword, token, and parser tables, the parser name, and the grammar > >>source file name for debugging. Also we could have abstract methods > >>to install hooks and overrides. The root abstract parser class could > >>look like this: >[...] > > > > A great start. I rarely use the :accessor slot attribute. CLOS needs > > it as a short hand because the eieio `oset' equivalent is >[...] > > > > Anyway, here is a start using more slot attributes: > > > >Thanks for your explanation! However, I would prefer to let you >implement those classes ;-) I think I've finally finished eradicating all old poor uses of "token", so I'll do that next. [ ... ] > > I understand your question now! The parser generator needs to > > associate a string with a macro, so the macro defined in a parser > > macros file may look like this: > > > > (defmacro foo-TAG (&rest args) > > `(bar-something ,@args)) > > > > (define-macro-for-this-run-of-the-parser-generator "TAG" 'foo-TAG) > > > > This seems problematic though. Perhaps the association of the string > > "TAG" to 'foo-tag needs to somehow be in the grammar since it is the > > parser-generator which uses that string, and nothing else. > >I thought to that too. Perhaps could we use a new %expandmacro >keyword like this: > >%expandmacro TAG foo-TAG > >To generate association of a grammar macro with an expander? That sounds like a good idea. It implies the need for a "%include", or some other cleverness indicating what set of macro expand keys should be loaded. Perhaps it would instead be declarative, such as "%gererate tags" or "%macroset font-lock" or "%import indent" [ ... ] >I terminated conversion to the new grammar layout, and checked-in a >bunch of changes! Following is the change log. Hope these changes >didn't break things ;-) I need to updated the EDE project to build a Makefile to create the intermediate files, such as: %.el: %.wy $(EMACS) -f wisent-something $@ so that wisent-awk.el is created from wisent-awk.wy. Do you have somethingalready? If not, I can create something to get the Makefiles working again. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: http://www.ludlam.net Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: David P. <dav...@wa...> - 2003-08-04 20:27:25
|
Hi Eric, [...] > I think I've finally finished eradicating all old poor uses of > "token", so I'll do that next. Thanks! I saw you did a great work! [...] >>>This seems problematic though. Perhaps the association of the string >>>"TAG" to 'foo-tag needs to somehow be in the grammar since it is the >>>parser-generator which uses that string, and nothing else. >> >>I thought to that too. Perhaps could we use a new %expandmacro >>keyword like this: >> >>%expandmacro TAG foo-TAG >> >>To generate association of a grammar macro with an expander? > > > That sounds like a good idea. It implies the need for a "%include", > or some other cleverness indicating what set of macro expand keys > should be loaded. > > Perhaps it would instead be declarative, such as "%gererate tags" or > "%macroset font-lock" or "%import indent" I worked a little more on that and implemented a solution that seems to work :-) For now I created a new %expanders statement that looks like this: %expanders ( (ASSOC . semantic-grammar-ASSOC) (EXPAND . wisent-grammar-EXPAND) (EXPANDFULL . wisent-grammar-EXPANDFULL) (TAG . wisent-grammar-TAG) (VARIABLE-TAG . wisent-grammar-VARIABLE-TAG) (FUNCTION-TAG . wisent-grammar-FUNCTION-TAG) (TYPE-TAG . wisent-grammar-TYPE-TAG) (INCLUDE-TAG . wisent-grammar-INCLUDE-TAG) (PACKAGE-TAG . wisent-grammar-PACKAGE-TAG) (EXPANDTAG . wisent-grammar-EXPANDTAG) (CODE-TAG . wisent-grammar-CODE-TAG) (AST-ADD . wisent-grammar-AST-ADD) (AST-PUT . wisent-grammar-AST-PUT) (AST-GET . wisent-grammar-AST-GET) (AST-GET1 . wisent-grammar-AST-GET1) (AST-GET-STRING . wisent-grammar-AST-GET-STRING) (AST-MERGE . wisent-grammar-AST-MERGE) ) It associates grammar macro names with function expanders. At generation these macros are expanded using the given functions. I also hacked the generation process to evaluate the prologue before expansion, so it is possible to simply use a `require' statement to include macro expanders: %{ (require 'wisent-grammar-macros) %} A little inconvenient with that scheme, is that macro definitions are also required at run time (copied from prologue in the generated lisp file), even if not necessary. I also provided a new `semantic-grammar-macros' variable that can be used to provide default definitions for commonly used macros. For example I defined the following default for WY grammars: (defvar-mode-local wisent-grammar-mode semantic-grammar-macros '( (ASSOC . semantic-grammar-ASSOC) (EXPAND . wisent-grammar-EXPAND) (EXPANDFULL . wisent-grammar-EXPANDFULL) (TAG . wisent-grammar-TAG) (VARIABLE-TAG . wisent-grammar-VARIABLE-TAG) (FUNCTION-TAG . wisent-grammar-FUNCTION-TAG) (TYPE-TAG . wisent-grammar-TYPE-TAG) (INCLUDE-TAG . wisent-grammar-INCLUDE-TAG) (PACKAGE-TAG . wisent-grammar-PACKAGE-TAG) (EXPANDTAG . wisent-grammar-EXPANDTAG) (CODE-TAG . wisent-grammar-CODE-TAG) (AST-ADD . wisent-grammar-AST-ADD) (AST-PUT . wisent-grammar-AST-PUT) (AST-GET . wisent-grammar-AST-GET) (AST-GET1 . wisent-grammar-AST-GET1) (AST-GET-STRING . wisent-grammar-AST-GET-STRING) (AST-MERGE . wisent-grammar-AST-MERGE) ) "Definitions of macros used in WY grammars.") I enhanced grammar font-locking to automagically highlight macro names defined in both `semantic-grammar-macros' and in the %expanders clause in the current buffer! Now that macros are no more hard coded, I would like to replace the expanders statement with something like this: %with-macros (TAG EXPAND ...) wisent-grammar-macro ... %with-macros (ASSOC) semantic-grammar The %with-macros statement would indicate to expand the given list of macros using definitions loaded from the indicated library. The names of expander functions would be the name of the macro prefixed with the library name. In the above examples that would give these associations: (TAG . wisent-grammar-macro-TAG) (EXPAND . wisent-grammar-macro-EXPAND) ... (ASSOC . semantic-grammar-ASSOC) The advantage would be simplicity, standardized naming, and possible load of macro libraries at generation time only (should be easy to implement). It could be worth adopting a similar consistent form to setup default macro expanders by customizing `semantic-grammar-macros': (defvar-mode-local wisent-grammar-mode semantic-grammar-macros '( ((TAG EXPAND ...) wisent-grammar-macro) ... ((ASSOC) semantic-grammar) ) ) Finally, I must look at the bovine grammar generator to see how it could benefit of a such improved macro architecture ;-) What do you think? [...] >>I terminated conversion to the new grammar layout, and checked-in a >>bunch of changes! Following is the change log. Hope these changes >>didn't break things ;-) > > > I need to updated the EDE project to build a Makefile to create the > intermediate files, such as: > > %.el: %.wy > $(EMACS) -f wisent-something $@ > > so that wisent-awk.el is created from wisent-awk.wy. > > Do you have somethingalready? If not, I can create something to get > the Makefiles working again. Oops! I forgot the EDE stuff. Sorry! I see a problem here with the %.el: %.wy rule because the file names are not necessarily identical. The current rule is that the generated lisp file name is the one indicated in the grammar %package statement if provided, or a default generated name otherwise. The default name is generated like this: foo.[wb]y -> foo-[wb]y.el More generally: foo.<ext> -> foo-<ext>.el The problem is with an explicit %package name, like used in bovine grammars. For example c.by uses a "%package bovine-c" and generates a bovine-c-by.el file. How it is possible to handle that with make? What I have to do too, is to allow `semantic-grammar-create-package' to work in batch mode, to allow: $(EMACS) -f semantic-grammar-create-package $@ Unfortunately, I will be very busy during the rest of August and probably September too. So I probably will not be able to do things quickly :-( David |
From: Eric M. L. <er...@si...> - 2003-08-05 04:53:50
|
>>> David Ponce <dav...@wa...> seems to think that: [ ... ] > >>>This seems problematic though. Perhaps the association of the string > >>>"TAG" to 'foo-tag needs to somehow be in the grammar since it is the > >>>parser-generator which uses that string, and nothing else. > >> > >>I thought to that too. Perhaps could we use a new %expandmacro > >>keyword like this: > >> > >>%expandmacro TAG foo-TAG > >> > >>To generate association of a grammar macro with an expander? > > > > > > That sounds like a good idea. It implies the need for a "%include", > > or some other cleverness indicating what set of macro expand keys > > should be loaded. > > > > Perhaps it would instead be declarative, such as "%gererate tags" or > > "%macroset font-lock" or "%import indent" > >I worked a little more on that and implemented a solution that seems >to work :-) > >For now I created a new %expanders statement that looks like this: > >%expanders ( > (ASSOC . semantic-grammar-ASSOC) > (EXPAND . wisent-grammar-EXPAND) > (EXPANDFULL . wisent-grammar-EXPANDFULL) > (TAG . wisent-grammar-TAG) > (VARIABLE-TAG . wisent-grammar-VARIABLE-TAG) > (FUNCTION-TAG . wisent-grammar-FUNCTION-TAG) > (TYPE-TAG . wisent-grammar-TYPE-TAG) > (INCLUDE-TAG . wisent-grammar-INCLUDE-TAG) > (PACKAGE-TAG . wisent-grammar-PACKAGE-TAG) > (EXPANDTAG . wisent-grammar-EXPANDTAG) > (CODE-TAG . wisent-grammar-CODE-TAG) > (AST-ADD . wisent-grammar-AST-ADD) > (AST-PUT . wisent-grammar-AST-PUT) > (AST-GET . wisent-grammar-AST-GET) > (AST-GET1 . wisent-grammar-AST-GET1) > (AST-GET-STRING . wisent-grammar-AST-GET-STRING) > (AST-MERGE . wisent-grammar-AST-MERGE) > ) > >It associates grammar macro names with function expanders. At >generation these macros are expanded using the given functions. I >also hacked the generation process to evaluate the prologue before >expansion, so it is possible to simply use a `require' statement to >include macro expanders: > >%{ > (require 'wisent-grammar-macros) > %} > >A little inconvenient with that scheme, is that macro definitions are >also required at run time (copied from prologue in the generated lisp >file), even if not necessary. > >I also provided a new `semantic-grammar-macros' variable that can be >used to provide default definitions for commonly used macros. For >example I defined the following default for WY grammars: > >(defvar-mode-local wisent-grammar-mode semantic-grammar-macros > '( > (ASSOC . semantic-grammar-ASSOC) > (EXPAND . wisent-grammar-EXPAND) > (EXPANDFULL . wisent-grammar-EXPANDFULL) > (TAG . wisent-grammar-TAG) > (VARIABLE-TAG . wisent-grammar-VARIABLE-TAG) > (FUNCTION-TAG . wisent-grammar-FUNCTION-TAG) > (TYPE-TAG . wisent-grammar-TYPE-TAG) > (INCLUDE-TAG . wisent-grammar-INCLUDE-TAG) > (PACKAGE-TAG . wisent-grammar-PACKAGE-TAG) > (EXPANDTAG . wisent-grammar-EXPANDTAG) > (CODE-TAG . wisent-grammar-CODE-TAG) > (AST-ADD . wisent-grammar-AST-ADD) > (AST-PUT . wisent-grammar-AST-PUT) > (AST-GET . wisent-grammar-AST-GET) > (AST-GET1 . wisent-grammar-AST-GET1) > (AST-GET-STRING . wisent-grammar-AST-GET-STRING) > (AST-MERGE . wisent-grammar-AST-MERGE) > ) > "Definitions of macros used in WY grammars.") > >I enhanced grammar font-locking to automagically highlight macro names >defined in both `semantic-grammar-macros' and in the %expanders clause >in the current buffer! That's great. Breaking the hard link between code and macro symbols is an important step. >Now that macros are no more hard coded, I would like to replace the >expanders statement with something like this: > >%with-macros (TAG EXPAND ...) wisent-grammar-macro >... >%with-macros (ASSOC) semantic-grammar > >The %with-macros statement would indicate to expand the given list of >macros using definitions loaded from the indicated library. The >names of expander functions would be the name of the macro prefixed >with the library name. In the above examples that would give these >associations: > >(TAG . wisent-grammar-macro-TAG) >(EXPAND . wisent-grammar-macro-EXPAND) >... >(ASSOC . semantic-grammar-ASSOC) > >The advantage would be simplicity, standardized naming, and possible >load of macro libraries at generation time only (should be easy to >implement). > >It could be worth adopting a similar consistent form to setup default >macro expanders by customizing `semantic-grammar-macros': > >(defvar-mode-local wisent-grammar-mode semantic-grammar-macros > '( > ((TAG EXPAND ...) wisent-grammar-macro) > ... > ((ASSOC) semantic-grammar) > ) > ) > >Finally, I must look at the bovine grammar generator to see how it >could benefit of a such improved macro architecture ;-) > >What do you think? I think I got a little lost in the above description. In an ideal situation, someone writing a grammar would put this into their grammar file: %with-macros TAG and that would both find the macro association TAG with foo-TAG, but also add (require 'foo-tag) (or whatever) as part of the prologue. I would also make the AST macros separate from the TAG macros. Then the C grammar may write: %with TAGS %with AST or (if you feel daring) %with TAGS, AST >[...] > >>I terminated conversion to the new grammar layout, and checked-in a > >>bunch of changes! Following is the change log. Hope these changes > >>didn't break things ;-) > > > > > > I need to updated the EDE project to build a Makefile to create the > > intermediate files, such as: > > > > %.el: %.wy > > $(EMACS) -f wisent-something $@ > > > > so that wisent-awk.el is created from wisent-awk.wy. > > > > Do you have somethingalready? If not, I can create something to get > > the Makefiles working again. > >Oops! I forgot the EDE stuff. Sorry! > >I see a problem here with the %.el: %.wy rule because the file names >are not necessarily identical. The current rule is that the generated >lisp file name is the one indicated in the grammar %package statement >if provided, or a default generated name otherwise. The default name >is generated like this: > >foo.[wb]y -> foo-[wb]y.el > >More generally: > >foo.<ext> -> foo-<ext>.el Ah yes, I had forgotten. The make syntax would then be: %.wy: %-wy.el >The problem is with an explicit %package name, like used in bovine >grammars. For example c.by uses a "%package bovine-c" and generates a >bovine-c-by.el file. How it is possible to handle that with make? Well, an external script can read all .wy files, determine the correct association based on %package statements, and build a dependency file. OR, we can claim that %package is useful, but we won't use it, thus simplifying our Makefile. ;) >What I have to do too, is to allow `semantic-grammar-create-package' >to work in batch mode, to allow: > >$(EMACS) -f semantic-grammar-create-package $@ > >Unfortunately, I will be very busy during the rest of August and probably >September too. So I probably will not be able to do things quickly :-( [ ... ] I will attempt to tackle this problem. My brain is still reeling from my first pass at a set of parser classes and needs a break. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: http://www.ludlam.net Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: David P. <dav...@wa...> - 2003-08-05 07:28:01
|
Hi Eric, [...] >>Now that macros are no more hard coded, I would like to replace the >>expanders statement with something like this: >> >>%with-macros (TAG EXPAND ...) wisent-grammar-macro >>... >>%with-macros (ASSOC) semantic-grammar >> [...] >>What do you think? > > > I think I got a little lost in the above description. I am sorry. Look like my explanations wasn't clear enough :-( > In an ideal situation, someone writing a grammar would put this into > their grammar file: > > %with-macros TAG > > and that would both find the macro association TAG with foo-TAG, but > also add (require 'foo-tag) (or whatever) as part of the prologue. That would certainly be very simple, and easy to use! However, in your above example, I don't see the magic that tells to expand TAG with foo-TAG and to load the foo-tag library? Also I don't think it is necessary to include the (require 'foo-tag) in the prologue, as macros are only needed when doing expansion, that is when running `semantic-grammar-create-package'. My %with[-macros] declarative syntax is a tentative to keep things simple and relatively explicit. When you write: %with foo (TAG ...) You tell `semantic-grammar-create-package' to: 1. require the `foo' library. 2. Associate TAG with an expander function named `foo-TAG' (that should have been previously loaded from the `foo' library). 3. Repeat 1. & 2. above for each %with statement found. 4. Convert grammar rules into Lisp code. At this point macros declared above will be expanded by calling their associated expander functions. After the Lisp code has been generated, the macro libraries loaded from step 1. above are no more necessary. There is no need to require them in the prologue of the generated Lisp file. Thus, in WY grammar you could have: %with wisent-grammar-macro (TAG ...) to expand (TAG something) into: (wisent-raw-tag (semantic-tag something)) And in BY grammar: %with bovine-grammar-macro (TAG ...) to expand (TAG something) into: (semantic-tag something) Is my explanation clear? > I would also make the AST macros separate from the TAG macros. Then > the C grammar may write: > > %with TAGS > %with AST > > or (if you feel daring) > > %with TAGS, AST Good idea. Should be easy to do ;-) [...] >>I see a problem here with the %.el: %.wy rule because the file names >>are not necessarily identical. The current rule is that the generated >>lisp file name is the one indicated in the grammar %package statement >>if provided, or a default generated name otherwise. The default name >>is generated like this: >> >>foo.[wb]y -> foo-[wb]y.el >> >>More generally: >> >>foo.<ext> -> foo-<ext>.el > > > Ah yes, I had forgotten. The make syntax would then be: > > %.wy: %-wy.el > > >>The problem is with an explicit %package name, like used in bovine >>grammars. For example c.by uses a "%package bovine-c" and generates a >>bovine-c-by.el file. How it is possible to handle that with make? > > > Well, an external script can read all .wy files, determine the > correct association based on %package statements, and build a > dependency file. That would be great! > > OR, we can claim that %package is useful, but we won't use it, thus > simplifying our Makefile. ;) Probably the best thing to do with our grammars. Maybe it would be better to rename <lang>.by files to bovine-<lang>.by, so default package would be bovine-<lang>-by.el instead of simply <lang>-by.el? [...] > I will attempt to tackle this problem. My brain is still reeling > from my first pass at a set of parser classes and needs a break. Thanks! David |
From: Eric M. L. <er...@si...> - 2003-08-05 12:24:38
|
>>> David PONCE <dav...@wa...> seems to think that: >Hi Eric, > >[...] >>>Now that macros are no more hard coded, I would like to replace the >>>expanders statement with something like this: >>> >>>%with-macros (TAG EXPAND ...) wisent-grammar-macro >>>... >>>%with-macros (ASSOC) semantic-grammar >>> >[...] >>>What do you think? >> >> >> I think I got a little lost in the above description. > >I am sorry. Look like my explanations wasn't clear enough :-( > >> In an ideal situation, someone writing a grammar would put this into >> their grammar file: >> >> %with-macros TAG >> >> and that would both find the macro association TAG with foo-TAG, but >> also add (require 'foo-tag) (or whatever) as part of the prologue. > >That would certainly be very simple, and easy to use! However, in >your above example, I don't see the magic that tells to expand TAG >with foo-TAG and to load the foo-tag library? %with-macros TAG would have to identify "TAG.wy" or "TAG.wyh", and load in extensions from there. That would contain %expandmacros, or other clever thing that makes all the associations. Wisent would need to somehow merge those macros in with the current macro set. Basically, you would have to create: TAG.wy (or .wyh, or maybe .why?) TAG-macros.el or have some other well known naming convention. Macros would then live in these file pairs. >Also I don't think it is necessary to include the (require 'foo-tag) >in the prologue, as macros are only needed when doing expansion, that >is when running `semantic-grammar-create-package'. Ok. >My %with[-macros] declarative syntax is a tentative to keep things >simple and relatively explicit. When you write: > >%with foo (TAG ...) > >You tell `semantic-grammar-create-package' to: > >1. require the `foo' library. > >2. Associate TAG with an expander function named `foo-TAG' (that > should have been previously loaded from the `foo' library). > >3. Repeat 1. & 2. above for each %with statement found. > >4. Convert grammar rules into Lisp code. At this point macros > declared above will be expanded by calling their associated > expander functions. The only dilemma I see above is every grammar must define every TAG generating macro. >After the Lisp code has been generated, the macro libraries loaded >from step 1. above are no more necessary. There is no need to >require them in the prologue of the generated Lisp file. Clever. >Thus, in WY grammar you could have: > >%with wisent-grammar-macro (TAG ...) > >to expand (TAG something) into: > >(wisent-raw-tag (semantic-tag something)) > >And in BY grammar: > >%with bovine-grammar-macro (TAG ...) > >to expand (TAG something) into: > >(semantic-tag something) > >Is my explanation clear? It is more clear now. Thanks. >> I would also make the AST macros separate from the TAG macros. Then >> the C grammar may write: >> >> %with TAGS >> %with AST >> >> or (if you feel daring) >> >> %with TAGS, AST > >Good idea. Should be easy to do ;-) Now I'm confused again. I thought you didn't like my single command that pulls in multiple macros. >[...] >>>I see a problem here with the %.el: %.wy rule because the file names >>>are not necessarily identical. The current rule is that the generated >>>lisp file name is the one indicated in the grammar %package statement >>>if provided, or a default generated name otherwise. The default name >>>is generated like this: >>> >>>foo.[wb]y -> foo-[wb]y.el >>> >>>More generally: >>> >>>foo.<ext> -> foo-<ext>.el >> >> >> Ah yes, I had forgotten. The make syntax would then be: >> >> %.wy: %-wy.el >> >> >>>The problem is with an explicit %package name, like used in bovine >>>grammars. For example c.by uses a "%package bovine-c" and generates a >>>bovine-c-by.el file. How it is possible to handle that with make? >> >> >> Well, an external script can read all .wy files, determine the >> correct association based on %package statements, and build a >> dependency file. > >That would be great! > >> >> OR, we can claim that %package is useful, but we won't use it, thus >> simplifying our Makefile. ;) > >Probably the best thing to do with our grammars. Maybe it would be >better to rename <lang>.by files to bovine-<lang>.by, so default >package would be bovine-<lang>-by.el instead of simply <lang>-by.el? [ ... ] Heh, I was thinking it would be nice to rename wisent-<lang>.wy files to just <lang>.wy. :) It would certainly be nice to be consistent between the two. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: http://www.ludlam.net Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: David P. <dav...@wa...> - 2003-08-05 14:03:05
|
[...] >>>In an ideal situation, someone writing a grammar would put this into >>>their grammar file: >>> >>>%with-macros TAG >>> >>>and that would both find the macro association TAG with foo-TAG, but >>>also add (require 'foo-tag) (or whatever) as part of the prologue. >> >>That would certainly be very simple, and easy to use! However, in >>your above example, I don't see the magic that tells to expand TAG >>with foo-TAG and to load the foo-tag library? > > > %with-macros TAG > > would have to identify "TAG.wy" or "TAG.wyh", and load in extensions > from there. That would contain %expandmacros, or other clever thing > that makes all the associations. Wisent would need to somehow merge > those macros in with the current macro set. > > Basically, you would have to create: > > TAG.wy (or .wyh, or maybe .why?) > TAG-macros.el > > or have some other well known naming convention. Macros would then > live in these file pairs. If I understand well, in your idea the %with-macros statement looks like a sort of grammar include statement. In my proposal bellow all is handled at Lisp level, which IMO is more simple to implement. Also there is no need to create extra TAG.wy files. [...] >>My %with[-macros] declarative syntax is a tentative to keep things >>simple and relatively explicit. When you write: >> >>%with foo (TAG ...) >> >>You tell `semantic-grammar-create-package' to: >> >>1. require the `foo' library. >> >>2. Associate TAG with an expander function named `foo-TAG' (that >> should have been previously loaded from the `foo' library). >> >>3. Repeat 1. & 2. above for each %with statement found. >> >>4. Convert grammar rules into Lisp code. At this point macros >> declared above will be expanded by calling their associated >> expander functions. > > > The only dilemma I see above is every grammar must define every TAG > generating macro. I don't see why. Macros are defined in Lisp libraries and you choose which ones will be used in your grammar using a %with declaration. Also, some macros can be defined implicitly via customization of the `semantic-grammar-macros' variable that is an equivalent of the %with statement at Lisp level. For example instead of including: %with wisent-grammar-macro (TAG ...) in every WY grammars. It is possible to: (defvar-mode-local wisent-grammar-mode semantic-grammar-macros <lisp form of the above %with statement> ) So each time you open a WY grammar buffer, you get the default set of macro definitions. The %with statement would be useful to define macros specific to certain grammars, or to override a default definition. The generation process will look first at %with declarations in the current grammar, then in declarations provided in the variable `semantic-grammar-macros'. [...] >>>I would also make the AST macros separate from the TAG macros. Then >>>the C grammar may write: >>> >>>%with TAGS >>>%with AST >>> >>>or (if you feel daring) >>> >>>%with TAGS, AST >> >>Good idea. Should be easy to do ;-) > > > Now I'm confused again. I thought you didn't like my single command > that pulls in multiple macros. Sorry, I just agreed on the principle. The definitive solution probably needs more discussion ;-) [...] >>Probably the best thing to do with our grammars. Maybe it would be >>better to rename <lang>.by files to bovine-<lang>.by, so default >>package would be bovine-<lang>-by.el instead of simply <lang>-by.el? > > [ ... ] > > Heh, I was thinking it would be nice to rename wisent-<lang>.wy files > to just <lang>.wy. :) It would certainly be nice to be consistent > between the two. I would prefer to rename <lang>.by files to bovine-<lang>.by to be consistent with Lisp naming convention. If we rename wisent-<lang>.wy file to <lang>.wy, the auto generated Lisp library will be <lang>-wy and probably the main support file should be <lang>.el. To avoid possible conflicts, IMO it is better to keep the `wisent' or `bovine' prefix in library names: wisent-<lang>.el -> main support library that requires: wisent-<lang>-wy.el -> generated file. wisent-<lang>-lex.el -> lexer etc. Same thing for bovine stuff: bovine-<lang>.el -> main support library that requires: bovine-<lang>-by.el -> generated file. bovine-<lang>-lex.el -> lexer etc. However, I will follow your decision ;-) David |
From: Eric M. L. <er...@si...> - 2003-07-29 13:56:28
|
David, I like your idea. The macros for TAGS could live in one specific .el file which could be (require'd) in the Prologue. That satisfies all requirements I think, and will make wisent more familiar and easier to use. The trick being the creation of wisent MACROS which also require a modification to the wisent code generator via a cond statement. Perhaps `define-wisent-macro' or some such could be used to do both tasks in some clever way. You could write only one .wy file, compiled to .wyc, and have a working tag generator. All the overload functions would probably still want to live in a second Emacs Lisp file which is require'd in the prologue, however. Eric >>> David PONCE <dav...@wa...> seems to think that: >Hi Eric, > >[...] >>>It looks that the %macro statement could be easily converted into an >>>expander function. >>> >>>%macro FUNCTION-TAG(&rest args) >>> (wisent-raw-tag (semantic-tag-new-function ,@args)) >>> >>>would become something like: >>> >>>(defun FUNCTION-TAG (&rest args) >>> (wisent-raw-tag (semantic-tag-new-function ,@args))) >> >> >> The current macros are defmacro macros, but yes, that was my thought >> as well. > >I thought a little more on that, and finally I wonder if we are not >reinventing the wheel? I am not sure that it is really worth >complicating the grammar with %macro statements not really different >from true lisp macro/function definitions in a .el support file. > >Maybe a better approach could be to adopt the grammar outline already >used by bison, that is: > > %{ > PROLOGUE > %} > > DECLARATIONS > > %% > GRAMMAR RULES > %% > > EPILOGUE > >The bison manual says: > >- "The PROLOGUE section contains macro definitions and declarations of > functions and variables that are used in the actions in the grammar > rules. These are copied to the beginning of the parser file [...]" > >- "The EPILOGUE is copied verbatim to the end of the parser file, just > as the PROLOGUE is copied to the beginning." > >Maybe we could also eliminate the %setupfunction and use a placeholder >for the generated code, and automatically generate the %keywordtable, >%parsetable, and %tokentable names from the %outputfile name? > >In my idea, the wisent-java.wy example grammar could resemble to this: > >----- <file starts here> ----- >%{ >;; PROLOGUE > >(require 'some-feature) > >(defun FUNCTION-TAG (&rest args) > (wisent-raw-tag (semantic-tag-new-function ,@args))) > >... > >}% > >;; DECLARATIONS >%outputfile "wisent-java" >... > >%% >;; GRAMMAR RULES >... > >%% >;; EPILOGUE > >(defun wisent-java-default-setup () > "Setup buffer for Semantic." > > ;; My setup code > (setq semantic-lex-number-expression semantic-java-number-regexp > semantic-lex-analyzer 'wisent-java-tags-lexer > semantic-tag-expand-function 'wisent-java-expand-tag) > ... > > ;; Generated setup code that setup the parser, that is the > ;; `parse-stream' override, the `semantic-parser-name', > ;; `semantic-toplevel-bovine-table', `semantic-debug-parser-source', > ;; `semantic-flex-keywords-obarray', and > ;; `semantic-lex-types-obarray' buffer local variables, etc.. > $PARSER-SETUP > ) > >;;;###autoload >(add-hook 'java-mode-hook 'wisent-java-default-setup) >----- <file ends here> ------- > >And the generated wisent-java.el (or wisent-java.wyc ?) would look >like this: > >----- <file starts here> ----- >;;; wisent-java.el --- GENERATED HEADER > >;; PROLOGUE > >(require 'some-feature) > >(defun FUNCTION-TAG (&rest args) > (wisent-raw-tag (semantic-tag-new-function ,@args))) > >... > >;; THESES DECLARATIONS ARE GENERATED >(defvar wisent-java-parse-table > ... > "The `wisent-java' parser table.") > >(defvar wisent-java-keyword-table > ... > "The `wisent-java' keyword table.") > >(defvar wisent-java-token-table > ... > "The `wisent-java' token table.") > >;; EPILOGUE > >(defun wisent-java-default-setup () > "Setup buffer for Semantic." > > ;; My setup code > (setq semantic-lex-number-expression semantic-java-number-regexp > semantic-lex-analyzer 'wisent-java-tags-lexer > semantic-tag-expand-function 'wisent-java-expand-tag) > ... > > ;; Generated setup code that setup the parser, that is the > ;; `parse-stream' override, the `semantic-parser-name', > ;; `semantic-toplevel-bovine-table', `semantic-debug-parser-source', > ;; `semantic-flex-keywords-obarray', and > ;; `semantic-lex-types-obarray' buffer local variables, etc.. > > ;; GENERATED $PARSER-SETUP CODE: > (progn > (semantic-install-function-overrides > '((parse-stream . wisent-parse-stream))) > (setq semantic-parser-name "LALR" > semantic-toplevel-bovine-table wisent-java-parser-tables > semantic-debug-parser-source "wisent-java-tags.wy" > semantic-flex-keywords-obarray wisent-java-keywords > semantic-lex-types-obarray wisent-java-tokens) > ;; Collect unmatched syntax lexical tokens > (semantic-make-local-hook 'wisent-discarding-token-functions) > (add-hook 'wisent-discarding-token-functions > 'wisent-collect-unmatched-syntax nil t)) > ) > >;;;###autoload >(add-hook 'java-mode-hook #'wisent-java-default-setup) > >;; GENERATED FOOTER >(provide 'wisent-java) > >;;; wisent-java.el ends here >----- <file end here> ------- > >What do you think? > >David > -- Eric Ludlam: za...@gn..., er...@si... Home: http://www.ludlam.net Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |