Thread: [CEDET-devel] Regression Test Generic tag format
Brought to you by:
zappo
From: Eric M. L. <er...@si...> - 2003-04-12 17:32:06
|
I've added two functions to the end of Klaus' new file `semantic-regtest.el'. These routines convert a semantic 2.0 tag into a "generic format", meaning the tags are good for comparing two parser outputs. The new tags format is: ("name" 'class :sym1 val1 :sym2 val2 ...) All "properties" are stripped since a grammar may change names of it's parser nodes. The overlay is stripped as well. In addition, the :symN symbols are all sorted in ascending order in case the order of those elements change from one parse to the next. This will ensure maximum flexibility. Have fun Eric -- Eric Ludlam: za...@gn..., er...@si... Home: http://www.ludlam.net Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: Klaus B. <kla...@sd...> - 2003-04-14 10:04:42
|
On Sat, 12 Apr 2003, Eric M. Ludlam wrote: > I've added two functions to the end of Klaus' new file > `semantic-regtest.el'. These routines convert a semantic 2.0 tag > into a "generic format", meaning the tags are good for comparing two > parser outputs. The new tags format is: > > ("name" 'class :sym1 val1 :sym2 val2 ...) > > All "properties" are stripped since a grammar may change names of it's > parser nodes. The overlay is stripped as well. > > In addition, the :symN symbols are all sorted in ascending order in > case the order of those elements change from one parse to the next. > This will ensure maximum flexibility. Thanks a lot, Eric! Nevertheless i have some questions: 1. Does these function also print out the parents of a token, namely if the token is a type - are the parents contained in the 2.0-attributes? 2. Does these functions print out the children of a token? I ask because i have made a first version for semantic 1.4: (defun semantic-regtest-convert-tag (tag) (let ((name (semantic-token-name tag)) (class (semantic-token-token tag)) (attr (semantic-token-extra-specs tag)) (generic nil)) (while attr (let ((sym (car attr)) (val (car (cdr attr)))) (cond ((semantic-token-p val) ;; This attribute is a tag (ie, a type perhaps?) (setq val (semantic-regtest-convert-tag val))) ((and (listp val) (semantic-token-p (car val))) ;; List of more tags in this property. Children/members (setq val (semantic-regtest-convert-tag-table val))) (t nil)) (setq generic (cons (list sym val) generic)) (setq attr (cdr (cdr attr))))) ;; At this point, generic is an ALIST, not a PROPERTY LIST. ;; We need to sort it so that order changes do not effect the ;; test. (setq generic (sort generic (lambda (a b) (string< (symbol-name (car a)) (symbol-name (car b)))))) (append (list name class) (apply 'append generic)) )) This function does neither 1. nor 2. (see above). IMHO both of them is important, in every case the parents of a token should be reported, about the children I'm not sure if this is necessary (because every token (sorry, i mean tag ;-) is processed in a file, so maybe reporting children is not necessary?!) In 1.4 i can find out the parents (semantic-token-type-parent and semantic-token-function-parent). IMHO the independed tag-format should contain a slot for the parents of a token. Are there other slots beside tag-name, tag-type, tag-parents which should be handled specially? Thanks, Klaus > > Have fun > Eric -- Klaus Berndl mailto: kla...@sd... sd&m AG http://www.sdm.de software design & management Thomas-Dehler-Str. 27, 81737 München, Germany Tel +49 89 63812-392, Fax -220 |
From: Klaus B. <kla...@sd...> - 2003-04-14 11:08:17
|
On 14 Apr 2003, Klaus Berndl wrote: > On Sat, 12 Apr 2003, Eric M. Ludlam wrote: > > > I've added two functions to the end of Klaus' new file > > `semantic-regtest.el'. These routines convert a semantic 2.0 tag > > into a "generic format", meaning the tags are good for comparing two > > parser outputs. The new tags format is: > > > > ("name" 'class :sym1 val1 :sym2 val2 ...) > > > > All "properties" are stripped since a grammar may change names of it's > > parser nodes. The overlay is stripped as well. > > > > In addition, the :symN symbols are all sorted in ascending order in > > case the order of those elements change from one parse to the next. > > This will ensure maximum flexibility. > > Thanks a lot, Eric! > > Nevertheless i have some questions: > 1. Does these function also print out the parents of a token, namely if the > token is a type - are the parents contained in the 2.0-attributes? > > 2. Does these functions print out the children of a token? > > I ask because i have made a first version for semantic 1.4: > [...] OK, here is the code for semantic 1.4 which returns a generic-tag-fomat like: (<tag-name> <tag-type> (<tag-parent-names>) (<tag-children-names>) sym1 val1 sym2 val2 ....) Thoughts? (defun semantic-regtest-tag-parent-names (parents) (if parents (let* ((parent (car parents)) (name (cond ((semantic-token-p parent) (semantic-token-name parent)) ((stringp parent) parent)))) (if name (cons name (semantic-regtest-tag-parent-names (cdr parents))) (if (listp parent) (append (semantic-regtest-tag-parent-names parent) (semantic-regtest-tag-parent-names (cdr parents)))))) (list nil))) (defun semantic-regtest-tag-children-names (children) (if (null children) (list nil) (mapcar (function (lambda (child) (semantic-token-name child))) children))) (defun semantic-regtest-convert-tag-table (table) "Convert the tag table TABLE to a generic format." (mapcar #'semantic-regtest-convert-tag table)) (defun semantic-regtest-convert-tag (tag) (let ((name (semantic-token-name tag)) (class (semantic-token-token tag)) (parents (cond ((equal (semantic-token-token tag) 'type) (semantic-regtest-tag-parent-names (semantic-token-type-parent tag))) ((equal (semantic-token-token tag) 'function) (list (semantic-token-function-parent tag))) (t (list nil)))) (children (semantic-regtest-tag-children-names (semantic-nonterminal-children tag))) (attr (semantic-token-extra-specs tag)) (generic nil)) (while attr ;; In semantic 1.4 the extra-specs is a list of cons-cells, where the ;; car is the symbol and the cdr is the value. (let ((sym (car (car attr))) (val (cdr (car attr)))) (cond ((semantic-token-p val) ;; This attribute is a tag (ie, a type perhaps?) (setq val (semantic-regtest-convert-tag val))) ((and (listp val) (semantic-token-p (car val))) ;; List of more tags in this property. Children/members (setq val (semantic-regtest-convert-tag-table val))) (t nil)) (setq generic (cons (list sym val) generic)) (setq attr (cdr attr)))) ;; At this point, generic is an ALIST, not a PROPERTY LIST. ;; We need to sort it so that order changes do not effect the ;; test. (setq generic (sort generic (lambda (a b) (string< (symbol-name (car a)) (symbol-name (car b)))))) (append (list name class parents children) (apply 'append generic)) )) Maybe we should sort the parents and children too?! Eric, if you find this senseful, could you please adjust your convert code for 2.0 so it return the same generic format?! Ciao, Klaus -- Klaus Berndl mailto: kla...@sd... sd&m AG http://www.sdm.de software design & management Thomas-Dehler-Str. 27, 81737 München, Germany Tel +49 89 63812-392, Fax -220 |
From: Eric M. L. <er...@si...> - 2003-04-14 12:23:37
|
>>> Klaus Berndl <kla...@sd...> seems to think that: >On Sat, 12 Apr 2003, Eric M. Ludlam wrote: > >> I've added two functions to the end of Klaus' new file >> `semantic-regtest.el'. These routines convert a semantic 2.0 tag >> into a "generic format", meaning the tags are good for comparing two >> parser outputs. The new tags format is: >> >> ("name" 'class :sym1 val1 :sym2 val2 ...) >> >> All "properties" are stripped since a grammar may change names of it's >> parser nodes. The overlay is stripped as well. >> >> In addition, the :symN symbols are all sorted in ascending order in >> case the order of those elements change from one parse to the next. >> This will ensure maximum flexibility. > >Thanks a lot, Eric! > >Nevertheless i have some questions: >1. Does these function also print out the parents of a token, namely if the > token is a type - are the parents contained in the 2.0-attributes? Yes. >2. Does these functions print out the children of a token? Yes. Well, sort of yes. It doesn't actually print anything. It just changes the internal representation which can then use `prin1' to output it to a text stream. >I ask because i have made a first version for semantic 1.4: > >(defun semantic-regtest-convert-tag (tag) > (let ((name (semantic-token-name tag)) > (class (semantic-token-token tag)) > (attr (semantic-token-extra-specs tag)) > (generic nil)) > (while attr > (let ((sym (car attr)) > (val (car (cdr attr)))) > (cond ((semantic-token-p val) > ;; This attribute is a tag (ie, a type perhaps?) > (setq val (semantic-regtest-convert-tag val))) > ((and (listp val) > (semantic-token-p (car val))) > ;; List of more tags in this property. Children/members > (setq val (semantic-regtest-convert-tag-table val))) > (t nil)) > (setq generic (cons (list sym val) generic)) > (setq attr (cdr (cdr attr))))) > ;; At this point, generic is an ALIST, not a PROPERTY LIST. > ;; We need to sort it so that order changes do not effect the > ;; test. > (setq generic (sort generic (lambda (a b) > (string< (symbol-name (car a)) > (symbol-name (car b)))))) > (append (list name class) > (apply 'append generic)) > )) > >This function does neither 1. nor 2. (see above). IMHO both of them is >important, in every case the parents of a token should be reported, about the >children I'm not sure if this is necessary (because every token (sorry, i mean >tag ;-) is processed in a file, so maybe reporting children is not >necessary?!) > >In 1.4 i can find out the parents (semantic-token-type-parent and >semantic-token-function-parent). IMHO the independed tag-format should contain >a slot for the parents of a token. > >Are there other slots beside tag-name, tag-type, tag-parents which should be >handled specially? [ ... ] If you look at semantic-tag.el in semantic 2.0, look for the functions like this: (defsubst semantic-tag-new-variable (name type default-value &rest attributes) "Create a semantic tag of class variable. NAME is the name of this variable. TYPE is a string or semantic tag representing the type of this variable. DEFAULT-VALUE is a string representing the default value of this variable. ATTRIBUTES is a list of additional attributes belonging to this tag." (apply 'semantic-tag name 'variable :type type :default-value default-value attributes)) The arguments for each of these functions mirror the internal tag format of semantic 1.4 (which simplified conversion.) You can use the argument list of these functions to find out the old 1.4 tag format, AND what the individual slots should be called to maximize compatibility. It the case of a type: (defsubst semantic-tag-new-type (name type members parents &rest attributes) "Create a semantic tag of class type. NAME is the name of this type. TYPE is a string or semantic tag representing the type of this type. MEMBERS is a list of strings or semantic tags representing the elements that make up this type if it is a composite type. PARENTS is a cons cell. (EXPLICIT-PARENTS . INTERFACE-PARENTS) EXPLICIT-PARENTS can be a single string (Just one parent) or a list of parents (in a multiple inheritance situation). It can also be nil. INTERFACE-PARENTS is a list of strings representing the names of all INTERFACES, or abstract classes inherited from. It can also be nil. This slot can be interesting because the form: ( nil \"string\") is a valid parent where there is no explicit parent, and only an interface. ATTRIBUTES is a list of additional attributes belonging to this tag." (apply 'semantic-tag name 'type :type type :members members :superclasses (car parents) :interfaces (cdr parents) attributes)) you see members (elements of the type) and parents are two special slots. For semantic 1.4, you could copy these functions w/ new names into regtest, and then for each 1.4 token: (apply 'semantic-tag-new-type TAG) to convert it into 2.0 format, then use the 2.0 converter to finish the job. Good Luck Eric -- Eric Ludlam: za...@gn..., er...@si... Home: http://www.ludlam.net Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: Klaus B. <kla...@sd...> - 2003-04-14 14:23:20
|
On Mon, 14 Apr 2003, Eric M. Ludlam wrote: > >>> Klaus Berndl <kla...@sd...> seems to think that: > >On Sat, 12 Apr 2003, Eric M. Ludlam wrote: > > > >> I've added two functions to the end of Klaus' new file > >> `semantic-regtest.el'. These routines convert a semantic 2.0 tag > >> into a "generic format", meaning the tags are good for comparing two > >> parser outputs. The new tags format is: > >> > >> ("name" 'class :sym1 val1 :sym2 val2 ...) > >> > >> All "properties" are stripped since a grammar may change names of it's > >> parser nodes. The overlay is stripped as well. > >> > >> In addition, the :symN symbols are all sorted in ascending order in > >> case the order of those elements change from one parse to the next. > >> This will ensure maximum flexibility. > > > >Thanks a lot, Eric! > > > >Nevertheless i have some questions: > >1. Does these function also print out the parents of a token, namely if the > > token is a type - are the parents contained in the 2.0-attributes? > > Yes. > > >2. Does these functions print out the children of a token? > > Yes. > > Well, sort of yes. It doesn't actually print anything. It just > changes the internal representation which can then use `prin1' to > output it to a text stream. Of course, this was already clear to me ;-) [...] > > (defsubst semantic-tag-new-type (name type members parents &rest > attributes) > "Create a semantic tag of class type. > NAME is the name of this type. > TYPE is a string or semantic tag representing the type of this type. > MEMBERS is a list of strings or semantic tags representing the > elements that make up this type if it is a composite type. > PARENTS is a cons cell. (EXPLICIT-PARENTS . INTERFACE-PARENTS) > EXPLICIT-PARENTS can be a single string (Just one parent) or a > list of parents (in a multiple inheritance situation). It can also > be nil. > INTERFACE-PARENTS is a list of strings representing the names of > all INTERFACES, or abstract classes inherited from. It can also be > nil. > This slot can be interesting because the form: > ( nil \"string\") > is a valid parent where there is no explicit parent, and only an > interface. > ATTRIBUTES is a list of additional attributes belonging to this tag." > (apply 'semantic-tag name 'type > :type type > :members members > :superclasses (car parents) > :interfaces (cdr parents) > attributes)) > > > you see members (elements of the type) and parents are two special > slots. > > For semantic 1.4, you could copy these functions w/ new names into > regtest, and then for each 1.4 token: > > (apply 'semantic-tag-new-type TAG) > > to convert it into 2.0 format, then use the 2.0 converter to finish > the job. generally a good idea, but in details there are problems at least if i want it to do so elegant like you described with (apply 'semantic-tag-new-type TAG). The function semantic-teg-new-* are not able to get a pure semantic-1.4-token/tag as input because the argument-list of these functions expects already the tag-class filtered out. Here is what i hacked together to convert a semantic 1.4 token into a 2.0 tag: (defconst semantic-regtest-new-tag-alist '((type . semantic-tag-new-type) (function . semantic-tag-new-function) (variable . semantic-tag-new-variable) (include . semantic-tag-new-include) (package . semantic-tag-new-package) (code . semantic-tag-new-code))) (defun semantic-regtest-convert-tag-1.4-to-2.0 (tag) (let ((token-name (semantic-token-name tag)) (tag-creation-fcn (cdr (assoc (semantic-token-token tag) semantic-regtest-new-tag-alist)))) (if tag-creation-fcn (apply tag-creation-fcn token-name (cdr (cdr tag))) ;; there is no predefined tag-creation function, so we use the generic ;; one. (semantic-tag token-name (semantic-token-token tag) (cdr (cdr tag)))))) `semantic-regtest-convert-tag-1.4-to-2.0' can be called for a token returned for example by `semantic-current-nonterminal' (1.4 version) and returns a tag in 2.0 format. Right? This 2.0-tag can i pump through `semantic-regtest-convert-tag' (your 2.0 version) to get a generic tag-format for regtesting. This one i can print out with prin1-to-string for example. Right or false? ;-) Thanks, Klaus -- Klaus Berndl mailto: kla...@sd... sd&m AG http://www.sdm.de software design & management Thomas-Dehler-Str. 27, 81737 München, Germany Tel +49 89 63812-392, Fax -220 |
From: Klaus B. <kla...@sd...> - 2003-04-14 16:39:22
|
On Mon, 14 Apr 2003, Eric M. Ludlam wrote: > >>> Klaus Berndl <kla...@sd...> seems to think that: > >On Sat, 12 Apr 2003, Eric M. Ludlam wrote: > > [...] > >Are there other slots beside tag-name, tag-type, tag-parents which should > >be handled specially? [...] > > For semantic 1.4, you could copy these functions w/ new names into > regtest, and then for each 1.4 token: > > (apply 'semantic-tag-new-type TAG) > > to convert it into 2.0 format, then use the 2.0 converter to finish > the job. I have checked in a new semantic-regtest.el which contains now a first try of compatibility code which should work with semantic 2.0 and semantic 1.4. Now the library should be able to run regression-tests also between semantic 1.4 and semantic 2.0 grammars/parsers. A first test with semantic 1.4 looks good. But in general i think i could need some feedback about my changes if they are senseful or which parts of my code could/must be written better or different. Could you please take a look into the latest CVS-version of semantic-regtest.el? Thanks a lot for your patience and help! Klaus -- Klaus Berndl mailto: kla...@sd... sd&m AG http://www.sdm.de software design & management Thomas-Dehler-Str. 27, 81737 München, Germany Tel +49 89 63812-392, Fax -220 |
From: Eric M. L. <er...@si...> - 2003-04-15 15:38:26
|
>>> Klaus Berndl <kla...@sd...> seems to think that: >On Mon, 14 Apr 2003, Eric M. Ludlam wrote: > >> >>> Klaus Berndl <kla...@sd...> seems to think that: >> >On Sat, 12 Apr 2003, Eric M. Ludlam wrote: >> > > >[...] > >> >Are there other slots beside tag-name, tag-type, tag-parents which should >> >be handled specially? > >[...] > >> >> For semantic 1.4, you could copy these functions w/ new names into >> regtest, and then for each 1.4 token: >> >> (apply 'semantic-tag-new-type TAG) >> >> to convert it into 2.0 format, then use the 2.0 converter to finish >> the job. > >I have checked in a new semantic-regtest.el which contains now a first try of >compatibility code which should work with semantic 2.0 and semantic 1.4. Now >the library should be able to run regression-tests also between semantic 1.4 >and semantic 2.0 grammars/parsers. A first test with semantic 1.4 looks good. > >But in general i think i could need some feedback about my changes if they are >senseful or which parts of my code could/must be written better or different. [ ... ] Hi, sorry for the delay. I think in a previous email you said that a 1.4 tag could not be passed into your convert function because of the tags format including properties, documentation, and an overlay. I think this assessment is correct and would cause problems so I am surprised (and glad) that you say it is working now. The code you have checked in like this: (or (fboundp 'semantic-tag-new-type) (defun semantic-tag-new-type (name type members parents &rest attributes) "See semantic 2.X for a description of this function." (apply 'semantic-tag name 'type :type type :members members :superclasses (car parents) :interfaces (cdr parents) attributes))) may need to be augmented to pre-strip the tag for the semantic 2.0 creator. I'm guessing something like this is needed (for the type class of token) (let ((tmp (copy-sequence inputtag))) (setcdr (nthcdr 4 tmp (semantic-token-type-extra-specs inputtag))) (funcall 'semantic-tag-new-type tmp)) which would rebuild the input tag as something compatible with the arguments to the new tag type. Ug, this is all very messy. It might be easier to just know you need to do this: (apply 'semantic-tag-new-type (semantic-token-name tag) (semantic-token-token tag) (semantic-token-type tag) (semantic-token-type-parts tag) (semantic-token-type-parent tag) (semantic-token-type-extra-specs tag)) Indeed, this is the type of mess that inspired us to change the tag format. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: http://www.ludlam.net Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |