Thread: [CEDET-devel] Semantic framework: semantic-fw.el
Brought to you by:
zappo
From: Eric M. L. <er...@si...> - 2002-07-21 13:57:32
|
Hi, I added a new `framework' file into semantic: semantic-fw.el. This file contains the overlay compatibility code, and other bits that are used in several files that isn't really specific to parsing. Eventually, I hope to remove some circular dependencies in the byte compilation phase. Anyway, one of the things that moved into the framework file is the code for override methods. I'd like to see the lex and parser variables replaced by override methods instead. The only bit that worries me is that if we turn the lexer into such a method, it will add a hash operation which might slow things down. On the other hand, it will specify what is really wanted which is a method for lexing or parsing. In preparing for this (eventual) change, there is a new `semantic-defoverload'. I'm not sure I like the name though. Anyway, I used this macro for all the token->text functions in semantic-util. Creating a new override method looks like this: (semantic-defoverload semantic-name-nonterminal (token &optional parent color) "Return the name string describing TOKEN. The name is the shortest possible representation. Optional argument PARENT is the parent type if TOKEN is a detail. Optional argument COLOR means highlight the prototype with font-lock colors.") where all the old code is now auto-generated. The macro not only fills in the boiler plate, but will leave a tag behind. It will then be possible to write a function or two that lets you list all functions that can be overridden. This in combination with some code to browse those functions that are overriding may help new language authors figure out what to do. I'd like to have a second macro for overriding a function. I'm not really sure how that would work, in that I'd like the language setup functions to say "set-up-overrides" with no arguments, and have it figure it out on it's own based on something from the macro. Lastly, I'd like these override tables set up once per mode, instead of once per file. I thought it would be needed to override some function in one buffer only, but I haven't come up with a use for this. This is important for semanticdb. Right now, it is only reliable to run override methods for tokens in buffers that are active. It would be nice to use token->text on a token from Emacs Lisp while a C buffer is active. This is pretty much required for speedbar and ECB which are currently just lucky that things work via the defaults ok. Wacky new languages will start exposing this deficiency. Well, I'm off for a week and won't be checking email, so I'll catch up with everyone later. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: David P. <da...@dp...> - 2002-07-21 18:47:17
|
Hi Eric, > I added a new `framework' file into semantic: semantic-fw.el. > This file contains the overlay compatibility code, and other bits > that are used in several files that isn't really specific to > parsing. Eventually, I hope to remove some circular dependencies in > the byte compilation phase. This looks very good! > Anyway, one of the things that moved into the framework file is > the code for override methods. I'd like to see the lex and parser > variables replaced by override methods instead. The only bit that > worries me is that if we turn the lexer into such a method, it will > add a hash operation which might slow things down. On the other > hand, it will specify what is really wanted which is a method for > lexing or parsing. I agree. Maybe we could try a first implementation and if it really slows things down too much, keep the current mechanism? > In preparing for this (eventual) change, there is a new > `semantic-defoverload'. I'm not sure I like the name though. > Anyway, I used this macro for all the token->text functions in > semantic-util. Creating a new override method looks like this: [...] I definitively like this idea to have macros to define things! For now we have `define-lex', `define-lex-analyzer' and so on, and `define-wisent-lexer'. To be consistent, why not just `define-overload'? > The macro not only fills in the boiler plate, but will leave a tag > behind. It will then be possible to write a function or two that > lets you list all functions that can be overridden. This in > combination with some code to browse those functions that are > overriding may help new language authors figure out what to do. Nice! > I'd like to have a second macro for overriding a function. I'm > not really sure how that would work, in that I'd like the language > setup functions to say "set-up-overrides" with no arguments, and > have it figure it out on it's own based on something from the macro. I don't understand well your idea here, could you give an example? > Lastly, I'd like these override tables set up once per mode, > instead of once per file. I thought it would be needed to override > some function in one buffer only, but I haven't come up with a use > for this. [...] Yes this would be definitively better to setup overrides per mode. Maybe we could use a global override table and prefix symbol names with mode symbols, both when installing (explicit) and fetching (implicit). Something like this: (defun semantic-install-function-overrides (modes overrides &optional transient) ...) where modes will be a list of `major-mode' symbols (to take care of derived modes ;) Of course installation will be done once instead of for each new buffer! Calling `semantic-fetch-overload' will not change, but it will prefix the symbol argument with current `major-mode'. > Well, I'm off for a week and won't be checking email, so I'll > catch up with everyone later. Vacation? If so, enjoy! David |
From: Eric M. L. <er...@si...> - 2002-07-21 19:31:40
|
>>> David Ponce <da...@dp...> seems to think that: [ ... ] > > In preparing for this (eventual) change, there is a new > > `semantic-defoverload'. I'm not sure I like the name though. > > Anyway, I used this macro for all the token->text functions in > > semantic-util. Creating a new override method looks like this: >[...] > >I definitively like this idea to have macros to define things! For >now we have `define-lex', `define-lex-analyzer' and so on, and >`define-wisent-lexer'. To be consistent, why not just >`define-overload'? That makes sense. > > The macro not only fills in the boiler plate, but will leave a tag > > behind. It will then be possible to write a function or two that > > lets you list all functions that can be overridden. This in > > combination with some code to browse those functions that are > > overriding may help new language authors figure out what to do. > >Nice! > > > I'd like to have a second macro for overriding a function. I'm > > not really sure how that would work, in that I'd like the language > > setup functions to say "set-up-overrides" with no arguments, and > > have it figure it out on it's own based on something from the macro. > >I don't understand well your idea here, could you give an example? For example, someone might write: (define-overload short-name-of-override (args) "Doc about why specific" spiffy code) and that's it. It has overridden `short-name-of-override' and installed it in the right override table. As you say, perhaps an extra argument of the major mode is needed. > > Lastly, I'd like these override tables set up once per mode, > > instead of once per file. I thought it would be needed to override > > some function in one buffer only, but I haven't come up with a use > > for this. >[...] > >Yes this would be definitively better to setup overrides per mode. >Maybe we could use a global override table and prefix symbol names >with mode symbols, both when installing (explicit) and fetching >(implicit). Something like this: > >(defun semantic-install-function-overrides (modes overrides > &optional transient) > ...) > >where modes will be a list of `major-mode' symbols (to take care of >derived modes ;) Of course installation will be done once instead >of for each new buffer! > >Calling `semantic-fetch-overload' will not change, but it will prefix >the symbol argument with current `major-mode'. This is similar to what I was thinking. I was contemplating hanging mode-specific override tables of the major mode symbol's plist. I'm not sure which is faster to use. What would be nice is not to have to pass the major mode down for every call to an override method. Perhaps there would be an active override table that could be locally bound when operating on buffer-less token lists. > > Well, I'm off for a week and won't be checking email, so I'll > > catch up with everyone later. > >Vacation? If so, enjoy! Yes, a week away from everything. It should be fun. Packing is taking a long time though. Later. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: David P. <da...@dp...> - 2002-07-26 13:33:36
Attachments:
semantic.patch
|
Hi Eric, Attached you will find a patch to semantic-fw.el, where I re-implemented `semantic-install-function-overrides' and `semantic-fetch-overload', and the macro `semantic-defoverload' as `define-overload'. There is also a patch to semantic-util.el to use `define-overload', plus small fixes. And another minor patch to semantic.el to inline some functions (see change log at end). As you will see `semantic-install-function-overrides' and `semantic-fetch-overload' are fully compatible with previous versions. But they can also work on mode basis if requested :-) `semantic-install-function-overrides' accepts a new optional symbol argument specifying a major mode where to statically install function overrides. By default it dynamically (via mode hook) installs the overrides in `semantic-override-table'. It is even possible to mix mode and local installation! When doing a buffer local installation, the function checks if a table exists for the current `major-mode' (or its parents) and, if so, installs the local overrides into a local copy of that table (a sort of poor man inheritance mechanism ;-) I provided two new functions: `semantic-current-overrides' and `semantic-new-overrides', to respectively find and return the current active overrides table, and to create a new one, maybe by doing a deep copy of an existing table. To improve performance when used on mode basis, `semantic-fetch-overload' set `semantic-override-table' with the mode override table it found when first called. So next calls just fetch values from the buffer local table. The nice thing is that when `semantic-override-table' was already setup before the first call, it will be used like in previous version ;-) Finally, I followed your idea and used a 'semantic-override-table property of the major mode symbol to store the mode override table. I found it easy to implement, taking care of derived modes. I did some tests and the new code seems to works very well with existing language stuff. The performance is not so different of what I measured with the old version. I also hacked semantic-java.el to install function overrides in java-mode. And had no problem with the JDEE which locally installs its own version of 'prototype-nonterminal! What's your opinion? Can I check my changes in? Thanks! Hope you had a nice week ;-) David ----------------------------- Change Log ----------------------------- * semantic-fw.el: (semantic-current-overrides, semantic-new-overrides): New functions. (semantic-install-function-overrides): Use them. New optional argument MODE to install into. (semantic-fetch-overload): Use `semantic-current-overrides'. (semantic-defoverload): Renamed to... (define-overload): New macro. * semantic-util.el: (semantic-name-nonterminal, semantic-abbreviate-nonterminal) (semantic-summarize-nonterminal, semantic-prototype-nonterminal) (semantic-concise-prototype-nonterminal) (semantic-uml-abbreviate-nonterminal) (semantic-uml-prototype-nonterminal) (semantic-uml-concise-prototype-nonterminal) (semantic-nonterminal-static): Use `define-overload' instead of `semantic-defoverload'. (semantic-find-dependency, semantic-find-nonterminal) (semantic-find-documentation): Put the right property 'semantic-overload. * semantic.el: (semantic-bovinate-from-nonterminal) (semantic-bovinate-from-nonterminal-full) (semantic-bovinate-region-until-error): Inlined to avoid an unnecessary extra call. |
From: Eric M. L. <er...@si...> - 2002-07-29 03:07:52
|
>>> David Ponce <da...@dp...> seems to think that: >Hi Eric, > >Attached you will find a patch to semantic-fw.el, where I >re-implemented `semantic-install-function-overrides' and >`semantic-fetch-overload', and the macro `semantic-defoverload' as >`define-overload'. There is also a patch to semantic-util.el to use >`define-overload', plus small fixes. And another minor patch to >semantic.el to inline some functions (see change log at end). [ ... ] > >What's your opinion? Can I check my changes in? This seems like a great idea. Please check them in. >Thanks! Hope you had a nice week ;-) [ ... ] The weather was pleasant, and my wife and I got some free babysitting. Can't go wrong there. I also re-wrote semantic-edit.el on my laptop. The new incremental parser has some nice new features, and helps prevent full-reparses more often, including insertion and deletion of tokens. Yay! On the bad side, I had to also build a couple new minor modes to replace the old show-dirty minor mode, and the auto-parse mode needs some more adjusting I think. My next task is getting that code off my laptop as the CVS install on it doesn't seem to work very well. :( If the wisent parser was working with the old style rebovinate token, then this new incremental parser should also work well with it. Have fun Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: David P. <da...@dp...> - 2002-07-29 17:52:31
|
Hi Eric, [...] > > What's your opinion? Can I check my changes in? > > This seems like a great idea. Please check them in. Thanks! Done. [...] > I also re-wrote semantic-edit.el on my laptop. The new incremental > parser has some nice new features, and helps prevent full-reparses > more often, including insertion and deletion of tokens. Yay! On > the bad side, I had to also build a couple new minor modes to > replace the old show-dirty minor mode, and the auto-parse mode needs > some more adjusting I think. [...] You really did a great work! I tried to use the new incremental parser and found some bugs I fixed (checked-in). I also added the new minor modes to Senator menu. Finally I used different hooks in semantic-imenu that seems to work far better that the previous one (no more messages "Imenu is out of date, try again. (internal bug)"). It remains a problem when `semantic-edits-bovinate-region' just returns nil when the parser fails. In that case `semantic-edits-splice-remove' fails with the following error because cache-list is nil: setcdr(nil nil) (if (eq first (car chil)) (progn (setq cacheend chil) (while ... ...) (setcar chil ...) (setcdr chil ...)) (while (and cachestart ...) (setq cachestart ...)) (setq cacheend cachestart) (while (and cacheend ...) (setq cacheend ...)) (setcdr cachestart (cdr cacheend))) (let* ((first ...) (last ...) (chil ...) (cachestart cachelist) (cacheend nil) (tmp oldtokens)) (if (eq first ...) (progn ... ... ... ...) (while ... ...) (setq cacheend cachestart) (while ... ...) (setcdr cachestart ...)) (while oldtokens (semantic-deoverlay-token ...) (setq oldtokens ...))) semantic-edits-splice-remove((("v[]" variable "int" nil (...) nil (...) #<overlay from 2489 to 2547 in Essai.java>)) nil nil) In fact I got this problem on a statement like this (Java): private int v[ ] = {1, 2, 3, 4}, w = 0; that results in two tokens, one for `v' from `private' to the comma, and a second one for `w' from `w' to after the semicolon. In such case the region covered by each token can't be parsed. Only the whole statement is valid. So when you change something in that statement the incremental parser always fails. Unfortunately I don't found yet how to get both incremental parse and token differentiation (useful for navigation, completion, etc.) work together. Maybe we need a new property telling the true bounds of such tokens? Have you any other idea? Sincerely, David P.S.: Here is the change log of what I checked-in: ------------------------------------------------------------------- * semantic-edit.el: (semantic-edits-bovinate-region): Added missing parameter reparse-symbol. (semantic-edits-incremental-parser): Set reparse-symbol. (semantic-edits-splice-remove): Fixed typo. * semantic-fw.el: (semantic-current-overrides, semantic-new-overrides): New functions. (semantic-install-function-overrides): Use them. New optional argument MODE to install into. (semantic-fetch-overload): Use `semantic-current-overrides'. (semantic-defoverload): Renamed to... (define-overload): New macro. * semantic-imenu.el: (semantic-imenu-flush-fcn, semantic-create-imenu-index): Use `semantic-after-toplevel-cache-change-hook' and `semantic-after-partial-cache-change-hook' instead of `semantic--before-toplevel-cache-flush-hook' and `semantic-clean-token-hooks'. * semantic-util.el: (semantic-name-nonterminal, semantic-abbreviate-nonterminal) (semantic-summarize-nonterminal, semantic-prototype-nonterminal) (semantic-concise-prototype-nonterminal) (semantic-uml-abbreviate-nonterminal) (semantic-uml-prototype-nonterminal) (semantic-uml-concise-prototype-nonterminal) (semantic-nonterminal-static): Use `define-overload' instead of `semantic-defoverload'. (semantic-find-dependency, semantic-find-nonterminal) (semantic-find-documentation): Put the right property 'semantic-overload. * semantic.el: (semantic-bovinate-toplevel): Added missing `working-status-form' around incremental parse. (semantic-bovinate-from-nonterminal) (semantic-bovinate-from-nonterminal-full) (semantic-bovinate-region-until-error): Inlined to avoid an unnecessary extra call. * senator.el: (senator-register-mode-menu-entry ["Highlight Dirty Tokens"]): Commented out. (senator-register-mode-menu-entry ["Highlight changes"]) (senator-register-mode-menu-entry ["Show parser state"]): Added. |
From: Eric M. L. <er...@si...> - 2002-07-30 01:39:51
|
>>> David Ponce <da...@dp...> seems to think that: >Hi Eric, > >[...] > > > What's your opinion? Can I check my changes in? > > > > This seems like a great idea. Please check them in. > >Thanks! Done. > >[...] > > I also re-wrote semantic-edit.el on my laptop. The new incremental > > parser has some nice new features, and helps prevent full-reparses > > more often, including insertion and deletion of tokens. Yay! On > > the bad side, I had to also build a couple new minor modes to > > replace the old show-dirty minor mode, and the auto-parse mode needs > > some more adjusting I think. >[...] > >You really did a great work! I tried to use the new incremental >parser and found some bugs I fixed (checked-in). I also added the new >minor modes to Senator menu. Finally I used different hooks in >semantic-imenu that seems to work far better that the previous one >(no more messages "Imenu is out of date, try again. (internal bug)"). Thanks. I did a lot of testing in situations without the parent/child relationship. There are many other issues and missing features from the incremental parser at the moment, and it is nice to have some verification in Java. I have a change for senator also. When adding senator-completion-cache-flush-fcn to some parser hooks, the hook semantic-after-partial-cache-change-hook should be used instead of `semantic-clean-token-hooks'. >It remains a problem when `semantic-edits-bovinate-region' just >returns nil when the parser fails. In that case >`semantic-edits-splice-remove' fails with the following error because >cache-list is nil: > > setcdr(nil nil) > (if (eq first (car chil)) (progn (setq cacheend chil) (while ... > ...) (setcar chil ...) (setcdr chil ...)) (while (and cachestart > ...) (setq cachestart ...)) (setq cacheend cachestart) (while (and > cacheend ...) (setq cacheend ...)) (setcdr cachestart (cdr > cacheend))) (let* ((first ...) (last ...) (chil ...) (cachestart > cachelist) (cacheend nil) (tmp oldtokens)) (if (eq first ...) (progn > ... ... ... ...) (while ... ...) (setq cacheend cachestart) (while > ... ...) (setcdr cachestart ...)) (while oldtokens > (semantic-deoverlay-token ...) (setq oldtokens ...))) > semantic-edits-splice-remove((("v[]" variable "int" nil (...) nil > (...) #<overlay from 2489 to 2547 in Essai.java>)) nil nil) > >In fact I got this problem on a statement like this (Java): > > private int v[ > ] = {1, > 2, > 3, > 4}, > w = 0; > >that results in two tokens, one for `v' from `private' to the comma, >and a second one for `w' from `w' to after the semicolon. In such >case the region covered by each token can't be parsed. Only the >whole statement is valid. So when you change something in that >statement the incremental parser always fails. Unfortunately I don't >found yet how to get both incremental parse and token differentiation >(useful for navigation, completion, etc.) work together. > >Maybe we need a new property telling the true bounds of such tokens? > >Have you any other idea? I would be loath to change how overlays are used for explicit token boundaries. This is how a user wants to think about their text. As this is only for the incremental parser, perhaps the notion of a parsable unit, or a tag segment, or other useful name is needed. Something to represent the minimum text needed for the rule in question to generate the tag list. Functions for -start and -end for this unit could mime `semantic-token-start' and `semantic-token-end' in most cases, but if a token had an extra-specifier or a property indicating which user-visible tag started or ended the parsable unit the tag was generated from. Unfortunately, this information would require special care when saving and restoring a tags database on disk unless it was not a reference, but a key of some sort (such as a name, or index forward and backward through the current cache.) I would like to add support for more types of overlays in semantic set into the extra-specifier or property list. In particular, I'd like tokens to be able to specify a segment within themselves that can be edited w/out concern for the cache. (ei, if you edit within that space, don't bother parsing.) This too could speed things up. The assumption is that even if parens go out of sync, the user will try to fix it up later. With some generic support for such overlay based properties, a solution for the above problem would be solved. Of course, it could be as simple as creating overloadable functions and you could use some clever regexp based hack to get the correct bounds. Of course, short term, the fail should automatically revert to requiring a full reparse. If not, a condition-case or lex configuration may be needed. Well, enough of that for tonight. Have fun Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |