cedet-devel — Developers of CEDET

Re[3]: [CEDET-devel] problem with python parser

[<ry...@ds...>]

Eric,

Thanks for pointing out the manual section.
I vaguely recall this now that you mention it.
I'll read it carefully and make sure python tokens comply
with the guide.

Python parser does not currently comply with the guide with
regard to the PARENTS part of the type token in that the
PARENTS is a list of tokens rather than a list of base
class names.  This is because python base classes are
specified within parenthesis which are parsed via the
EXPANDFULL mechanism as shown below:

    ;; classdef: 'class' NAME ['(' testlist ')'] ':' suite
    classdef
      : CLASS NAME paren_class_list_opt COLON suite
        (wisent-token $2 'type $1 $5 $3)
      ;

    ;; ['(' testlist ')']
    paren_class_list_opt
      : ;;EMPTY
      | paren_class_list
      ;

    paren_class_list
      : PAREN_BLOCK
        (EXPANDFULL $1 paren_classes)
      ;

    ;; parameters: '(' [varargslist] ')'
    paren_classes
      : LPAREN
        ()
      | RPAREN
        ()
      | paren_class COMMA
        (wisent-token $1 'variable nil nil)
      | paren_class RPAREN
        (wisent-token $1 'variable nil nil)
      ;

    ;; In general, the base class can be specified by a general expression
    ;; which evalue to a class object, i.e., base classes are not just names!
    ;; However base classes are names in most cases.  Thus the
    ;; non-terminal below work only with simple names.  Even if the
    ;; parser can parse general expressions, I don't see much benefit in
    ;; generating a string of expression as base class "name".
    paren_class
      : NAME
      ;

For example:

    class Foo(One,Two):
        x = 1
        y = 2

produces

     ("Foo" type "class"
      (("x" variable nil nil 241 246)
       ("y" variable nil nil 252 257))
      (("One" variable nil nil
        ((reparse-symbol . paren_classes))
        [226 230])
       ("Two" variable nil nil
        ((reparse-symbol . paren_classes))
        [230 234]))
      nil #<overlay from 216 to 299 in test.py>)

"One" and "Two" above should be simple names rather than tokens.

In order to convert these tokens to names,
I cooked up the code below:

    (define-mode-overload-implementation
      semantic-parse-region python-mode
      (start end &optional nonterminal depth returnonerror)
      "Over-ride so that 'paren_classes' non-terminal tokens can be intercepted
    then converted to simple names to comply with the semantic token style guide."
      (let ((tokens (semantic-parse-region-default
    		 start end nonterminal depth returnonerror)))
        (if (eq nonterminal 'paren_classes)
    	(mapcar #'semantic-token-name tokens)
          tokens)))

With this addition, the following token is generated:

     ("Foo" type "class"
      (("x" variable nil nil 241 246)
       ("y" variable nil nil 252 257))
      ("One" "Two")
      nil #<overlay from 216 to 299 in test.py>)

This seems to do the job.
What do you think this of solution?
Am I abusing this overload mechanism?


>>>>> "EL" == Eric M Ludlam <er...@si...> writes:
    EL> 
    >>>> "Eric M. Ludlam" <er...@si...> seems to think that:
    >> The token format is described in the `semantic-token-style-guide'.
    >> As new parts were added, I've tried keeping it up to date.
    >> 
    EL>   [ ... ]
    EL> 
    EL> Ooops,  I think in Lisp too much.
    EL> 
    EL> It is described in the info manual in the chapter "Semantic Token
    EL> Style Guide".
    EL> 
    EL> Eric
    EL> 
    EL> -- 
    EL>           Eric Ludlam:                 za...@gn..., er...@si...
    EL>    Home: http://www.ludlam.net            Siege: www.siege-engine.com
    EL> Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org
    EL> 
    EL> 
    EL> -------------------------------------------------------
    EL> This SF.NET email is sponsored by:
    EL> SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
    EL> http://www.vasoftware.com
    EL> _______________________________________________
    EL> Cedet-devel mailing list
    EL> Ced...@li...
    EL> https://lists.sourceforge.net/lists/listinfo/cedet-devel

Re[2]: [CEDET-devel] Lexers and parsers and unmatched syntax, oh my!

[Eric M. L. <er...@si...>]

>>> "David PONCE" <Dav...@wa...> seems to think that:
  [ ... ]
>I personally use the following function to calculate elapsed time:
>
>(if (fboundp 'float-time)
>
>    ;; GNU Emacs 21
>    (defalias 'cpu-time 'float-time)
>  
>  ;; Other Emacs/XEmacs flavors
>  (defun cpu-time (&optional time)
>    "Convert TIME to a floating point number.
>If TIME is nil use `current-time' instead."
>    (or time (setq time (current-time)))
>    (+ (* (car time) 65536.0)
>       (nth 1 time)
>       (/ (or (nth 2 time) 0) 1000000.0)))
>
>  )
>
>Maybe it could help ;-)
  [ ... ]

I was unable to figure out how this calculated elapsed time.  I did
find an elapsed time calculator in elp.el  It now looks like this in
semantic.el.

(defun semantic-elapsed-time (start end)
  "Copied from elp.el.  Was elp-elapsed-time.
Argument START and END bound the time being calculated."
  (+ (* (- (car end) (car start)) 65536.0)
     (- (car (cdr end)) (car (cdr start)))
     (/ (- (car (cdr (cdr end))) (car (cdr (cdr start)))) 1000000.0)))


Eric

-- 
          Eric Ludlam:                 za...@gn..., er...@si...
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org

Re[1]: [CEDET-devel] About `semanticdb-project-predicates'

[Eric M. L. <er...@si...>]

This looks like a good update.  Thanks.

Eric

>>> "David PONCE" <Dav...@wa...> seems to think that:
>Eric,
>
>Just some [minor] points about `semanticdb-project-predicates' ;-)
>
>Its documentation says that:
>
>"Project Management software (such as EDE and JDE) should add their
>own predicates with `add-hook' to this variable, and semanticdb will
>save token caches in directories controlled by them."
>
>So clearly `semanticdb-project-predicates' seems to be what the Elisp
>reference manual calls an "abnormal hook".
>
>My first point is that probably it would be better to rename it
>`semanticdb-project-predicate-functions' to follow the naming
>convention recommended in the Elisp reference manual.
>
>Secondly, I think the implementation `semanticdb-write-directory-p' in
>semanticdb-file.el could benefit of using the standard built-in
>function `run-hook-with-args-until-success', like this:
>
>*** semanticdb-file.el.ori	Fri Jan 31 11:18:43 2003
>--- semanticdb-file.el	Fri Jan 31 11:17:32 2003
>***************
>*** 188,207 ****
>  	       (if (string= (oref obj reference-directory) (car path))
>  		   (throw 'found t)))
>  	      ((eq (car path) 'project)
>! 	       (let ((predicates semanticdb-project-predicates))
>! 		 (if predicates
>! 		     (while predicates
>! 		       (if (funcall (car predicates)
>! 				    (oref obj reference-directory))
>! 			   (throw 'found t))
>! 		       (setq predicates (cdr predicates)))
>! 		   ;; If the mode is 'project, and there are no project
>! 		   ;; modes, then just always save the file.  If users
>! 		   ;; wish to restrict the search, modify
>! 		   ;; `semanticdb-persistent-path' to include desired paths.
>! 		   (if (= (length semanticdb-persistent-path) 1)
>  		       (throw 'found t))
>! 		   )))
>  	      ((eq (car path) 'never)
>  	       (throw 'found nil))
>  	      ((eq (car path) 'always)
>--- 188,205 ----
>  	       (if (string= (oref obj reference-directory) (car path))
>  		   (throw 'found t)))
>  	      ((eq (car path) 'project)
>! 	       (if semanticdb-project-predicates
>! 		   (if (run-hook-with-args-until-success
>! 			'semanticdb-project-predicates
>! 			(oref obj reference-directory))
>  		       (throw 'found t))
>! 		 ;; If the mode is 'project, and there are no project
>! 		 ;; modes, then just always save the file.  If users
>! 		 ;; wish to restrict the search, modify
>! 		 ;; `semanticdb-persistent-path' to include desired paths.
>! 		 (if (= (length semanticdb-persistent-path) 1)
>! 		     (throw 'found t))
>! 		 ))
>  	      ((eq (car path) 'never)
>  	       (throw 'found nil))
>  	      ((eq (car path) 'always)
  [ ... ]

-- 
          Eric Ludlam:                 za...@gn..., er...@si...
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org

Re: [CEDET-devel] Lexers and parsers and unmatched syntax, oh my!

[David P. <Dav...@wa...>]

> One new thing I've noticed is that when I use C-c C-c to update the
> code, it doesn't activate the changes.  In a bnf file, C-c C-c also
> made sure all created code was evaluated into emacs.  When it was
> done, it also went through all buffers, and any buffer that was of
> the mode specified by the language would get the semantic install
> function run.
> 
> Is there an objection to this old behavior=3F  If not I will add those
> features to semantic-grammar.el

Of course not!  Feel free to do it.  I probably put such things
somewhere in my TODO list, and forgot them ;-)

>>I apologize, I didn't take into account compatibility with BNF
>>debugging when I wrote the new grammar framework :-(
>>
>>So it is highly probable that you'll encounter all sort of bugs when
>>debugging BY grammar.  Sorry.
> 
>   [ ... ]
> 
> In that case I will probably try to fix the debugger so I can debug
> my grammar.  Does the debugger work with wisent=3F

Unfortunately no.  Wisent implementation is very different from
"Bovinator" one and source debugging probably must be re-implemented
for it.

> Or maybe I'll implement that new debugger API I've been thinking
> about for the past year.  Yea, that's the ticket.
> 
> (I finally finished some web site work I've been doing for my wife so
> I should have time to play here again.  Yay!)

Hurray!

David

Re[2]: [CEDET-devel] Lexers and parsers and unmatched syntax, oh my!

[Eric M. L. <er...@si...>]

>>> "David PONCE" <Dav...@wa...> seems to think that:
>Hi Eric,
>
>> I was bovinating some stuff today and discovered this in my messages
>> buffer:
>> 
>> Retrieving tokens took 1.-64 seconds.
>> Making completion list...
>> Retrieving tokens took 1.36 seconds.
>> Making completion list...
>> Retrieving tokens took 1.-71 seconds.
>
>I personally use the following function to calculate elapsed time:
>
>(if (fboundp 'float-time)
>
>    ;; GNU Emacs 21
>    (defalias 'cpu-time 'float-time)
>  
>  ;; Other Emacs/XEmacs flavors
>  (defun cpu-time (&optional time)
>    "Convert TIME to a floating point number.
>If TIME is nil use `current-time' instead."
>    (or time (setq time (current-time)))
>    (+ (* (car time) 65536.0)
>       (nth 1 time)
>       (/ (or (nth 2 time) 0) 1000000.0)))
>
>  )
>
>Maybe it could help ;-)

Thanks.  I'll try to fix up the bovinate command.

  [ ... ]
>> which basically provides a `token-stream' for tokens to be placed.
>> It then returns that stream.  It would be nil if empty.  The forms
>> still move the cursor so you can see what was analyzed.
>
>Good point!  I think you should check it in.

Done.

>> Anyway, I'm having trouble doing the conversion to the .by format.
>> I've converted it, built a new lexer that looks ok.  So I tried
>> `bovinate-debug'.  To my dismay, it switched to the .by file (as
>> appropriate), parsed some file (the Makefile I presume) and then put
>> the unmatched syntax for the Makefile into make.by.  Hmmm.
>[...]
>
>I had a look at make.by and noticed that there are rules like this:
>
>opt-whitespace : whitespace ( nil )
>	       | EMPTY
>	       ;
>
>Like in Bison, the EMPTY token does not exist as a "special" token.
>There are actually "empty" rules that must be written like this:
>
>opt-whitespace : whitespace ( nil )
>	       | ;;EMPTY
>	       ;
>
>Probably fixing that should help.

Ah yes, I had forgotten this step,  I did this but it didn't help my
current dilemma.  Still more to do for me.

One new thing I've noticed is that when I use C-c C-c to update the
code, it doesn't activate the changes.  In a bnf file, C-c C-c also
made sure all created code was evaluated into emacs.  When it was
done, it also went through all buffers, and any buffer that was of
the mode specified by the language would get the semantic install
function run.

Is there an objection to this old behavior?  If not I will add those
features to semantic-grammar.el


>I apologize, I didn't take into account compatibility with BNF
>debugging when I wrote the new grammar framework :-(
>
>So it is highly probable that you'll encounter all sort of bugs when
>debugging BY grammar.  Sorry.
  [ ... ]

In that case I will probably try to fix the debugger so I can debug
my grammar.  Does the debugger work with wisent?

Or maybe I'll implement that new debugger API I've been thinking
about for the past year.  Yea, that's the ticket.

(I finally finished some web site work I've been doing for my wife so
I should have time to play here again.  Yay!)

Eric

-- 
          Eric Ludlam:                 za...@gn..., er...@si...
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org

Re[3]: [CEDET-devel] problem with python parser

[Eric M. L. <er...@si...>]

>>> "Eric M. Ludlam" <er...@si...> seems to think that:
>The token format is described in the `semantic-token-style-guide'.
>As new parts were added, I've tried keeping it up to date.
>
  [ ... ]

Ooops,  I think in Lisp too much.

It is described in the info manual in the chapter "Semantic Token
Style Guide".

Eric

-- 
          Eric Ludlam:                 za...@gn..., er...@si...
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org

Re[2]: [CEDET-devel] problem with python parser

[Eric M. L. <er...@si...>]

The token format is described in the `semantic-token-style-guide'.
As new parts were added, I've tried keeping it up to date.

Of course, we had plans for a while to switch to a different token
format.  Don't know when we will get to that.

Eric

>>> ry...@ds... seems to think that:
>Hi David,
>
>Thanks for your help.  I checked in your changes pretty much
>as is.  So simple formal parameters are now parsed and
>included within the 'function token.   Other changes checked
>in include clean up of many semantic actions, as well as
>parsing of base class list within 'class tokens.
>
>I also checked couple of sentences in NEWS files.
>
>Next I'll be studying the tokens generated by the java
>parser and I'll try to have the python parser generate
>similar tokens.  In particular, I'll try to include
>auxiliary informations such as formal parameters and base
>class lists in the same position within the token list.
>
>After that, I will go through as many files as I can in
>/usr/lib/python2.2/ directory and make sure that the parser
>can handle all the files.
>
>>>>>> "DP" == David PONCE <Dav...@wa...> writes:
>    DP> 
>    DP> Hi Richard,
>    >> I would like to ask your help in figuring out why python grammar does
>    >> not seem to work as I expected.  I include the full copy of
>    >> wisent-python.wy at the end, but here are the three NT's of interest.
>    DP> [...]
>    >> 
>    >> This is my attempt at ading function arguments to 'function tokens.
>    >> As you can see the code above is very similar to what you have in
>    >> wisent-java-tag.wy.  The last NT, function_parameters, is very
>    >> simplified version since it only matches NAME terminal.
>    DP> [...]
>    >> The problem I can't solve now is that if I add
>    >> 
>    >> %start function_parameters
>    >> 
>    >> then rebuild the tables and re-bovinate, then no token is generated
>    >> from the same input lines!  It seems like adding this one line
>    >> seems to break the parser.  What is going on?
>    DP> 
>    DP> If fact the result of an EXPANDFULL clause nonterminal must be a
>    DP> valid semantic token otherwise the iterative parser just return nil.
>    DP> 
>    DP> As your NAME rule just return a string (value of NAME) it is not
>    DP> considered as a valid token and ignored.
>    DP> 
>    DP> After some minor changes in your grammar I managed to successfully
>    DP> parse your small example below:
>    DP> 
>    DP> def ttt(x):
>    DP>     i = 1
>    DP> 
>    DP> That produced this parse tree:
>    DP> 
>    DP> (("ttt" function nil
>    DP>   (("x" variable nil nil nil nil
>    DP>     ((reparse-symbol . function_parameters))
>    DP>     #<overlay from 9 to 10 in test.py>))
>    DP>   nil #<overlay from 1 to 23 in test.py>))
>    DP> 
>    DP> I attached the new grammar.  Using diff you will easily see what has
>    DP> changed.  Particularly I had to remove the NEWLINE following
>    DP> compound_stmt in the goal nonterminal, as it appeared to me that the
>    DP> lexical stream never terminates with a NEWLINE lexical token!
>    DP> 
>    DP> Hope it helps.
>    DP> Thanks for working on that!
>    DP> 
>    DP> David
>
>
>
>-------------------------------------------------------
>This SF.NET email is sponsored by:
>SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
>http://www.vasoftware.com
>_______________________________________________
>Cedet-devel mailing list
>Ced...@li...
>https://lists.sourceforge.net/lists/listinfo/cedet-devel
>

-- 
          Eric Ludlam:                 za...@gn..., er...@si...
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org

[CEDET-devel] About `semanticdb-project-predicates'

[David P. <Dav...@wa...>]

Eric,

Just some [minor] points about `semanticdb-project-predicates' ;-)

Its documentation says that:

"Project Management software (such as EDE and JDE) should add their
own predicates with `add-hook' to this variable, and semanticdb will
save token caches in directories controlled by them."

So clearly `semanticdb-project-predicates' seems to be what the Elisp
reference manual calls an "abnormal hook".

My first point is that probably it would be better to rename it
`semanticdb-project-predicate-functions' to follow the naming
convention recommended in the Elisp reference manual.

Secondly, I think the implementation `semanticdb-write-directory-p' in
semanticdb-file.el could benefit of using the standard built-in
function `run-hook-with-args-until-success', like this:

*** semanticdb-file.el.ori=09Fri Jan 31 11:18:43 2003
--- semanticdb-file.el=09Fri Jan 31 11:17:32 2003
***************
*** 188,207 ****
  =09       (if (string=3D (oref obj reference-directory) (car path))
  =09=09   (throw 'found t)))
  =09      ((eq (car path) 'project)
! =09       (let ((predicates semanticdb-project-predicates))
! =09=09 (if predicates
! =09=09     (while predicates
! =09=09       (if (funcall (car predicates)
! =09=09=09=09    (oref obj reference-directory))
! =09=09=09   (throw 'found t))
! =09=09       (setq predicates (cdr predicates)))
! =09=09   ;; If the mode is 'project, and there are no project
! =09=09   ;; modes, then just always save the file.  If users
! =09=09   ;; wish to restrict the search, modify
! =09=09   ;; `semanticdb-persistent-path' to include desired paths.
! =09=09   (if (=3D (length semanticdb-persistent-path) 1)
  =09=09       (throw 'found t))
! =09=09   )))
  =09      ((eq (car path) 'never)
  =09       (throw 'found nil))
  =09      ((eq (car path) 'always)
--- 188,205 ----
  =09       (if (string=3D (oref obj reference-directory) (car path))
  =09=09   (throw 'found t)))
  =09      ((eq (car path) 'project)
! =09       (if semanticdb-project-predicates
! =09=09   (if (run-hook-with-args-until-success
! =09=09=09'semanticdb-project-predicates
! =09=09=09(oref obj reference-directory))
  =09=09       (throw 'found t))
! =09=09 ;; If the mode is 'project, and there are no project
! =09=09 ;; modes, then just always save the file.  If users
! =09=09 ;; wish to restrict the search, modify
! =09=09 ;; `semanticdb-persistent-path' to include desired paths.
! =09=09 (if (=3D (length semanticdb-persistent-path) 1)
! =09=09     (throw 'found t))
! =09=09 ))
  =09      ((eq (car path) 'never)
  =09       (throw 'found nil))
  =09      ((eq (car path) 'always)

What do you think=3F

David

Re: [CEDET-devel] Lexers and parsers and unmatched syntax, oh my!

[David P. <Dav...@wa...>]

Hi Eric,

> I was bovinating some stuff today and discovered this in my messages
> buffer:
> 
> Retrieving tokens took 1.-64 seconds.
> Making completion list...
> Retrieving tokens took 1.36 seconds.
> Making completion list...
> Retrieving tokens took 1.-71 seconds.

I personally use the following function to calculate elapsed time:

(if (fboundp 'float-time)

    ;; GNU Emacs 21
    (defalias 'cpu-time 'float-time)
  
  ;; Other Emacs/XEmacs flavors
  (defun cpu-time (&optional time)
    "Convert TIME to a floating point number.
If TIME is nil use `current-time' instead."
    (or time (setq time (current-time)))
    (+ (* (car time) 65536.0)
       (nth 1 time)
       (/ (or (nth 2 time) 0) 1000000.0)))

  )

Maybe it could help ;-)

> Anyway, I was attempting to convert make.bnf into make.by.  I was
> fiddling with analyzers, and the new feature that defines a function
> for each individual analyzer failed to work in stand alone mode when
> the new analyzer had a call to semantic-lex-token.  I solved that
> with this patch.
[...]
> which basically provides a `token-stream' for tokens to be placed.
> It then returns that stream.  It would be nil if empty.  The forms
> still move the cursor so you can see what was analyzed.

Good point!  I think you should check it in.

> Anyway, I'm having trouble doing the conversion to the .by format.
> I've converted it, built a new lexer that looks ok.  So I tried
> `bovinate-debug'.  To my dismay, it switched to the .by file (as
> appropriate), parsed some file (the Makefile I presume) and then put
> the unmatched syntax for the Makefile into make.by.  Hmmm.
[...]

I had a look at make.by and noticed that there are rules like this:

opt-whitespace : whitespace ( nil )
=09       | EMPTY
=09       ;

Like in Bison, the EMPTY token does not exist as a "special" token.
There are actually "empty" rules that must be written like this:

opt-whitespace : whitespace ( nil )
=09       | ;;EMPTY
=09       ;

Probably fixing that should help.

I apologize, I didn't take into account compatibility with BNF
debugging when I wrote the new grammar framework :-(

So it is highly probable that you'll encounter all sort of bugs when
debugging BY grammar.  Sorry.

Anyway it is a good thing to convert the make grammar to use new
lexical and grammar tools.  Thanks ;-)

David

Re: [CEDET-devel] problem with python parser

[<ry...@ds...>]

Hi David,

Thanks for your help.  I checked in your changes pretty much
as is.  So simple formal parameters are now parsed and
included within the 'function token.   Other changes checked
in include clean up of many semantic actions, as well as
parsing of base class list within 'class tokens.

I also checked couple of sentences in NEWS files.

Next I'll be studying the tokens generated by the java
parser and I'll try to have the python parser generate
similar tokens.  In particular, I'll try to include
auxiliary informations such as formal parameters and base
class lists in the same position within the token list.

After that, I will go through as many files as I can in
/usr/lib/python2.2/ directory and make sure that the parser
can handle all the files.

>>>>> "DP" == David PONCE <Dav...@wa...> writes:
    DP> 
    DP> Hi Richard,
    >> I would like to ask your help in figuring out why python grammar does
    >> not seem to work as I expected.  I include the full copy of
    >> wisent-python.wy at the end, but here are the three NT's of interest.
    DP> [...]
    >> 
    >> This is my attempt at ading function arguments to 'function tokens.
    >> As you can see the code above is very similar to what you have in
    >> wisent-java-tag.wy.  The last NT, function_parameters, is very
    >> simplified version since it only matches NAME terminal.
    DP> [...]
    >> The problem I can't solve now is that if I add
    >> 
    >> %start function_parameters
    >> 
    >> then rebuild the tables and re-bovinate, then no token is generated
    >> from the same input lines!  It seems like adding this one line
    >> seems to break the parser.  What is going on?
    DP> 
    DP> If fact the result of an EXPANDFULL clause nonterminal must be a
    DP> valid semantic token otherwise the iterative parser just return nil.
    DP> 
    DP> As your NAME rule just return a string (value of NAME) it is not
    DP> considered as a valid token and ignored.
    DP> 
    DP> After some minor changes in your grammar I managed to successfully
    DP> parse your small example below:
    DP> 
    DP> def ttt(x):
    DP>     i = 1
    DP> 
    DP> That produced this parse tree:
    DP> 
    DP> (("ttt" function nil
    DP>   (("x" variable nil nil nil nil
    DP>     ((reparse-symbol . function_parameters))
    DP>     #<overlay from 9 to 10 in test.py>))
    DP>   nil #<overlay from 1 to 23 in test.py>))
    DP> 
    DP> I attached the new grammar.  Using diff you will easily see what has
    DP> changed.  Particularly I had to remove the NEWLINE following
    DP> compound_stmt in the goal nonterminal, as it appeared to me that the
    DP> lexical stream never terminates with a NEWLINE lexical token!
    DP> 
    DP> Hope it helps.
    DP> Thanks for working on that!
    DP> 
    DP> David

[CEDET-devel] Lexers and parsers and unmatched syntax, oh my!

[Eric M. L. <er...@si...>]

I was bovinating some stuff today and discovered this in my messages
buffer:

Retrieving tokens took 1.-64 seconds.
Making completion list...
Retrieving tokens took 1.36 seconds.
Making completion list...
Retrieving tokens took 1.-71 seconds.

Anyway, I was attempting to convert make.bnf into make.by.  I was
fiddling with analyzers, and the new feature that defines a function
for each individual analyzer failed to work in stand alone mode when
the new analyzer had a call to semantic-lex-token.  I solved that
with this patch.

*** semantic-lex.el.~1.15.~	Tue Jan 28 07:30:49 2003
--- semantic-lex.el	Thu Jan 30 20:58:56 2003
***************
*** 616,622 ****
       ;; function help is automatically provided, and perhaps the
       ;; function could be useful for testing and debugging one
       ;; analyzer.
!      (fset ',name (lambda () ,doc (when ,condition ,@forms)))
       ))
  
  ;;;###autoload
--- 617,626 ----
       ;; function help is automatically provided, and perhaps the
       ;; function could be useful for testing and debugging one
       ;; analyzer.
!      (fset ',name (lambda () ,doc
! 		    (let ((token-stream nil))
! 		      (when ,condition ,@forms)
! 		      token-stream)))
       ))
  
  ;;;###autoload


which basically provides a `token-stream' for tokens to be placed.
It then returns that stream.  It would be nil if empty.  The forms
still move the cursor so you can see what was analyzed.

Anyway, I'm having trouble doing the conversion to the .by format.
I've converted it, built a new lexer that looks ok.  So I tried
`bovinate-debug'.  To my dismay, it switched to the .by file (as
appropriate), parsed some file (the Makefile I presume) and then put
the unmatched syntax for the Makefile into make.by.  Hmmm.

Anyway, one problem at a time.  I'll keep working on the .by
conversion.

Have fun
Eric

-- 
          Eric Ludlam:                 za...@gn..., er...@si...
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org

[CEDET-devel] Re: [cedet-semantic] New C++-parser available

[David P. <Dav...@wa...>]

Klaus,

[...]
> Just fixed this and checked in a new c.bnf, semantic-c.el, NEWS and
> tests/template.cpp

Many thanks!  I will sync. the trunk.

> For next 1.4.3 release of semantic c/c++-parsing is now "closed"!
> 
> David, this were my last changes for the next release. This is my GO.
> Hopefully its not to late...

There's no hurry Klaus.  We can delay 1.4.3 as long as needed to
stabilize the C/C++ support ;-)

David

[CEDET-devel] RE: [cedet-semantic] New C++-parser available

[Berndl, K. <kla...@sd...>]

>One more thing I just discovered:
>
>> **** Template-specifier can now contain normal args like "int i" which is very
>>      important.

>template <int i> struct foo { }; // ok

>foo<0> bar; // not ok

Just fixed this and checked in a new c.bnf, semantic-c.el, NEWS and tests/template.cpp

For next 1.4.3 release of semantic c/c++-parsing is now "closed"!

David, this were my last changes for the next release. This is my GO.
Hopefully its not to late...

Ciao,
Klaus

Re: [CEDET-devel] Authors

[David P. <Dav...@wa...>]

Hi Eric,

>   To help semantic be a better GNU project, I added an AUTHORS file.
> Please make sure I picked out the right email address for you, or
> that I didn't erroneously forget about you.

Good idea!   My email address is right.

>   I also updated the Project.ede file.

So the Makefile probably need to be updated too ;-)
I will do that.

>   Also, I still have erlang.tar sitting on my PC which was sent to me
> a while back.  It uses the semantic 1.4 .bnf file notation.  I
> checked them in mostly unchanged into the bovine directory.  David,
> if you want to put them into semantic 1.4.3, that is ok by me.

Maybe Vladimir should be in the AUTHORS.

I would prefer to release 1.4.3 as it is now with all the precious C/C++
enhancements.

An 1.4.4 could be released shortly after with new erlang support.
Probably semantic-load has to be updated to "autoload" erlang support=3F

Thanks!

David

[CEDET-devel] Authors

[Eric M. L. <er...@si...>]

Hi all,

  To help semantic be a better GNU project, I added an AUTHORS file.
Please make sure I picked out the right email address for you, or
that I didn't erroneously forget about you.

  I also updated the Project.ede file.

  Also, I still have erlang.tar sitting on my PC which was sent to me
a while back.  It uses the semantic 1.4 .bnf file notation.  I checked
them in mostly unchanged into the bovine directory.  David, if you
want to put them into semantic 1.4.3, that is ok by me.

  For semantic 2.0, I left them out of the project file pending
translation to new APIs, such as the lexer and others.  If anyone on
the list knows erlang, a test file would be helpful.

Thanks
Eric

-- 
          Eric Ludlam:                 za...@gn..., er...@si...
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org

Re: [CEDET-devel] FW: Parsing of C string constants - L"..=?iso-8859-1?Q?=09." sol=09ved?=

[David P. <Dav...@wa...>]

Hi Klaus,

[...]
> Ok, i have checked in (v1p4) new semantic-c.el, new NEWS (for David
> ;-) and also new tests/test.cpp....Now all wide-char stuff should
> work. IMHO 1.4.3 is ready for release... :-)

I updated semantic-c.el and test.cpp in the trunk too.
I will prepare a new 1.4.3 release as soon as possible.

Thank you so much for doing all that!

David

P.S.: I noticed that in CVS some lines in your check-in comments begin
with tabs.  Probably you forgot to remove them when copy/pasting from
your local Changelog ;-) It is not catastrophic, just that these tabs
will be duplicated by rcs2log or cvs2cl.pl, that add tabs too, when
building a Changelog.

RE: [CEDET-devel] FW: Parsing of C string constants - L".. ." sol ved

[Berndl, K. <kla...@sd...>]

> > Eric, what do you say? Is this a good and save solution?

>IMO your solution is a good one.

Ok, i have checked in (v1p4) new semantic-c.el, new NEWS (for David ;-)
and also new tests/test.cpp....Now all wide-char stuff should work. IMHO
1.4.3 is ready for release... :-)

>I think the L prefix must be part of the token value.  So it will be
>possible (in future use?) to distinguish wchar_t and char strings.

Yes, i agree. Done.

>I synch. the trunk with your previous changes ;-)

Fine!

Ciao,
Klaus

Re[3]: [CEDET-devel] FW: Parsing of C string constants - L".. ." solved

[Eric M. L. <er...@si...>]

>>> "Berndl, Klaus" <kla...@sd...> seems to think that:
>Ok, i think i have found the answer for myself - but first after taking a look into
>semantic-flex and seeing that the semantic-flex-extensions are used by just doing
>a looking-at at the cars...
>
>Ok, what i have done is:
>
>
>(defvar semantic-flex-c-extensions
>  '(("^\\s-*#if\\s-*0$" . semantic-flex-c-if-0)
>    ("^#\\(if\\(def\\)?\\|el\\(if\\|se\\)\\|endif\\)" . semantic-flex-c-if)
>    ("<[^\n>]+>" . semantic-flex-c-include-system)
>    ("\\(\\\\\n\\)" . semantic-flex-backslash-newline)
>    ("L\\s\"" . semantic-flex-wide-char-literal)
>    )
>  "Extensions to the flexer for C.")
>
>(defun semantic-flex-wide-char-literal ()
>  (let ((start (point)))
>    (forward-char 1)
>    ;; now we should be onto the string-delimiter direct after the L
>    (forward-sexp 1)
>    (cons 'string (cons start (point)))))
>
>
>This parses now correct stuff like:
>
>char const *p = "string1";
>char const *q = "string1" "str\"ing2" "string3";
>wchar_t testc = L'a';
>
>wchar_t const *wp = L"string with a \" in it";
>wchar_t const *wq = L"string \n\t\"test" L"string2";
>wchar_t const *wr = L"string L";
>
>Currently the L is now contained in the string. I'm  not sure if
>this is good or if it would be better to parse the L"..." but
>then ignoring the L and just returning the plain string "..." as
>'string. Thoughts?
  [ ... ]

Right now, strings only show up in the "value" part of a variable.
As such, you can have it do whatever you like.  If the L is an important
part of what the string is about, then stick it on.  If you want, you
could just treat the L as whitespace and ignore it.

Eric

-- 
          Eric Ludlam:                 za...@gn..., er...@si...
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org

Re[3]: [CEDET-devel] FW: Parsing of C string constants

[Eric M. L. <er...@si...>]

>>> "Berndl, Klaus" <kla...@sd...> seems to think that:
  [ ... ]
>>This could be an easy piece in semantic 2, but in 1.4,
>>you need to modify semantic-c-flex-extensions with something like
>>this:
>
>>  "L\\s\""
>
>>which would then need the existing string logic hidden in the lexer
>>to finish the task.
>
>Hmm, IMHO this regexp is not enough because this would also match:
>
>"bla bla bla L"
  [ ... ]

it would not match "blah L" because the string "blah L" would be
matched first by the string analyzer, and the L analyzer would not
have the opportunity to see the L inside the string.

The extensions work like this:

  1) Does the regexp "L\\s\"" match
  2) If yes, run some code,

Thus, in your code, you can possibly use the lexical analyzer
recursively on (+1 (point)) to slurp the string.  If that works, then
you are all set, if it fails, return no token, and the L is turned
into a symbol, and the string a string, as opposed to a single wide
string.

Eric

-- 
          Eric Ludlam:                 za...@gn..., er...@si...
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org

Re: [CEDET-devel] FW: Parsing of C string constants - L".. ." solved

[David P. <dav...@wa...>]

Hi Klaus,

 > Ok, i think i have found the answer for myself - but first after
 > taking a look into semantic-flex and seeing that the
 > semantic-flex-extensions are used by just doing a looking-at at the
 > cars...
 >
 > Ok, what i have done is:

[...]

 > This parses now correct stuff like:
 >
 > char const *p = "string1";
 > char const *q = "string1" "str\"ing2" "string3";
 > wchar_t testc = L'a';
 >
 > wchar_t const *wp = L"string with a \" in it";
 > wchar_t const *wq = L"string \n\t\"test" L"string2";
 > wchar_t const *wr = L"string L";
 >
 > Currently the L is now contained in the string. I'm  not sure if
 > this is good or if it would be better to parse the L"..." but
 > then ignoring the L and just returning the plain string "..." as
 > 'string. Thoughts?
 >
 > Eric, what do you say? Is this a good and save solution?

IMO your solution is a good one.

I think the L prefix must be part of the token value.  So it will be
possible (in future use?) to distinguish wchar_t and char strings.

I synch. the trunk with your previous changes ;-)

Great work. Thanks!
David

RE: Re[1]: [CEDET-devel] Question to full/partial reparses

[Berndl, K. <kla...@sd...>]

Hi Eric.

>  You certainly have a dilemma.  It is exactly this problem that made
>  me attempt to make the new incremental parser more robust.  I would
>  also someday like to make the full-reparse go back and synchronize
>  against old overlays and cons cells also.

Yes, this would be a real milestone in reparsing...

> Anyway, what you need to do here is, whenever you "cache" a token,
> make a marker object, and position it at token start.

>(let ((m (make-marker)))
>  (move-marker m (semantic-token-start T))
     ...)

>  When you need to get back to the old token, you can do this:

>  (save-excursion
>     (goto-char m)
>     (semantic-current-nonterminal))

>  and this will work perhaps 90% of the time.  It will not work if T
>  was deleted, or cut from one location to another.  To protect
>  against that, you need to do name compares or something similar.

Thanks for this hint - i had already tried this yesterday with copy-marker
instead of move-marker but it seems not to be fully satisfying me...
Now i have implemented it like you suggested with move-marker...and it seems
to be the best possible solution for now.

Many thanks,
Klaus


>>> "Berndl, Klaus" <kla...@sd...> seems to think that:
>Hi guys,
>
>suppose semantic 1.4.2.
>
>Further suppose that i have a semantic-token T (e.g. a function-token of an
>elisp-file F). T contains all informations of a function-token and therefore
>among others also a valid overlay.
>
>Ok, now suppose that i store this token in a certain own cache. Then a full reparse
>is done (either autom. or manually triggered) because i have changes something other
>in file F - the elisp-code to which token T belongs (i.e. the function) is unchanged.
>But after the full reparse the overlay of the token T stored in my own cache (i need it
>for certain purposes - at least the overlay-informations of the tokens) is invalid.
>
>(Eric, you remember we have discussed a related topic a long time ago?)
>
>Ok, it sounds understandable that after a reparse the old overlays are invalid because
>replaced by the new one (generated by the bovination/parsing run).
>
>But now i have a problem: I want to use the overlay-informations of my cached token T
>(need overlay-buffer overlay-start/end) but the overlay of T is now invalid cause of the
>reparse and there is a new token T' with a new valid overlay.
>
>Now i'm wondering how to resync my cache with the new tokens (ok, i could throw away
>the whole cache after a reparse but this would be bad because this cache stores a 
>"navigation"-history similary back- and next-button of a webbrowser :-( or at least with
>the new overlays of the tokens.
>
>But for this i must recognize the the new generated token T' is equlivalent to the old token
>T (equivalent in the sense it is the token of the same unchanged function, has same argument-
>list etc. etc.) but i can not compare T and T' foe example with semantic-equivalent-tokens-p
>because this function needs for comparison the overlays and - remember - token T doesn't
>contain a valid overlay!
>
>To make a long story short: Any senseful or prakticable ways to check for token-equivalence if
>one of the token contains an INVALID overlay. AFAICS name, and type are not enough because for
>example in C++ files there can be method-declaration in the class and also the method-
>implementation outside of the class in the same file...hmm, this could be checked with adopted-
>attribute....
>
>Ok, anyway, thoughs? Or was my problem not described understandable??
>
>Thanks,
>Klaus
>
>
>-------------------------------------------------------
>This SF.NET email is sponsored by:
>SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
>http://www.vasoftware.com
>_______________________________________________
>Cedet-devel mailing list
>Ced...@li...
>https://lists.sourceforge.net/lists/listinfo/cedet-devel
>

-- 
          Eric Ludlam:                 za...@gn..., er...@si...
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org

RE: Re[1]: [CEDET-devel] FW: Parsing of C string constants - L".. ." solved

[Berndl, K. <kla...@sd...>]

Ok, i think i have found the answer for myself - but first after taking a look into
semantic-flex and seeing that the semantic-flex-extensions are used by just doing
a looking-at at the cars...

Ok, what i have done is:


(defvar semantic-flex-c-extensions
  '(("^\\s-*#if\\s-*0$" . semantic-flex-c-if-0)
    ("^#\\(if\\(def\\)?\\|el\\(if\\|se\\)\\|endif\\)" . semantic-flex-c-if)
    ("<[^\n>]+>" . semantic-flex-c-include-system)
    ("\\(\\\\\n\\)" . semantic-flex-backslash-newline)
    ("L\\s\"" . semantic-flex-wide-char-literal)
    )
  "Extensions to the flexer for C.")

(defun semantic-flex-wide-char-literal ()
  (let ((start (point)))
    (forward-char 1)
    ;; now we should be onto the string-delimiter direct after the L
    (forward-sexp 1)
    (cons 'string (cons start (point)))))


This parses now correct stuff like:

char const *p = "string1";
char const *q = "string1" "str\"ing2" "string3";
wchar_t testc = L'a';

wchar_t const *wp = L"string with a \" in it";
wchar_t const *wq = L"string \n\t\"test" L"string2";
wchar_t const *wr = L"string L";

Currently the L is now contained in the string. I'm  not sure if
this is good or if it would be better to parse the L"..." but
then ignoring the L and just returning the plain string "..." as
'string. Thoughts?

Eric, what do you say? Is this a good and save solution?

Ciao,
Klaus

-----Original Message-----
From: Berndl, Klaus 
Sent: Wednesday, January 29, 2003 6:59 PM
To: 'Eric M. Ludlam'
Cc: ced...@li...
Subject: RE: Re[1]: [CEDET-devel] FW: Parsing of C string constants


>The L"wide string" notation should probably be handled in the Lexical
>analyzer.

I agree

>This could be an easy piece in semantic 2, but in 1.4,
>you need to modify semantic-c-flex-extensions with something like
>this:

>  "L\\s\""

>which would then need the existing string logic hidden in the lexer
>to finish the task.

Hmm, IMHO this regexp is not enough because this would also match:

"bla bla bla L"
             ^^

IMHO this must be a regexp which matches exactly L and direct behind
a complete string. Is this right? If yes, what is the correct regexp
to match all possible strings, even such ones contained escaped " etc??


>As a rule, you could define a token L, and make a string rule that
>had an optional L in front, but I'm not sure if it would mess up if a
>user tried to define a function or variable L.

Have already tried this, but it seems not to work and IMHO this
is a job for the lexer not for the parser.

Many thanks in advance,
Klaus


-------------------------------------------------------
This SF.NET email is sponsored by:
SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
http://www.vasoftware.com
_______________________________________________
Cedet-devel mailing list
Ced...@li...
https://lists.sourceforge.net/lists/listinfo/cedet-devel

RE: Re[1]: [CEDET-devel] FW: Parsing of C string constants

[Berndl, K. <kla...@sd...>]

>The L"wide string" notation should probably be handled in the Lexical
>analyzer.

I agree

>This could be an easy piece in semantic 2, but in 1.4,
>you need to modify semantic-c-flex-extensions with something like
>this:

>  "L\\s\""

>which would then need the existing string logic hidden in the lexer
>to finish the task.

Hmm, IMHO this regexp is not enough because this would also match:

"bla bla bla L"
             ^^

IMHO this must be a regexp which matches exactly L and direct behind
a complete string. Is this right? If yes, what is the correct regexp
to match all possible strings, even such ones contained escaped " etc??


>As a rule, you could define a token L, and make a string rule that
>had an optional L in front, but I'm not sure if it would mess up if a
>user tried to define a function or variable L.

Have already tried this, but it seems not to work and IMHO this
is a job for the lexer not for the parser.

Many thanks in advance,
Klaus

[CEDET-devel] Re: [cedet-semantic] New C++-parser available - was: Lates semantic f rom CVS?

[<mar...@gi...>]

Berndl, Klaus wrote:

> Hi,
> 
> I have checked in new c.bnf with some new parsing features and some
>  fixes (see below), a newly generated semantic-c.el and also a new
> tests/test.cpp which tests latest fixes.
> 
> wchar_t and string-sequences like "str" "str" now works.

Just tested this part and I can comfirm it works now.

>> You have added many additional capabilities to the parser now.
>> Are you getting the correct types of output in ECB?  Occasionally
>> in the past I've added support for a stray & or const in type
>> definitions for variables.  Later, when one of the token->text
>> functions are called, it shows up in the wrong place.  For C this
>> has been one of the more challenging aspects of translating C
>> into a token, then back out again.  When last I ran tests, it was
>> pretty good, but missing the finer points of * and & placement.
>> With all your new template and const work it would be good to
>> asses the situation.
> 
> Done! You were right, Eric, in the function-arguments correct
> pointer counting and const recognizing was disrupted after my
> latest changes. This means parsing was ok, but the output was
> wrong.
> 
> Now i have fixed all this and now it works again! Now all the
> output from the new parsing capabilities is correct - at least for
> this i have tested ;-)

Hmm, are references important too? Just asking, because references are
not recognised but simply ignored, I think.

Markus

FW: [CEDET-devel] RE: Re[1]: [cedet-semantic] New C++-parser avai lable - was: Lates semantic f rom CVS?

[Berndl, K. <kla...@sd...>]

Forgot to say: All this is in branch v1p4!


-----Original Message-----
From: Berndl, Klaus 
Sent: Wednesday, January 29, 2003 5:30 PM
To: 'Eric M. Ludlam'; 'ced...@li...'
Cc: mar...@gi...; ced...@li...
Subject: [CEDET-devel] RE: Re[1]: [cedet-semantic] New C++-parser available - was: Lates semantic f rom CVS?


Hi,

I have checked in new c.bnf with some new parsing features and some
fixes (see below), a newly generated semantic-c.el and also a new tests/test.cpp
which tests latest fixes.

wchar_t and string-sequences like "str" "str" now works.

But the L"string" or L'c' stuff still fails....

Hmm, maybe Eric could fix this best?


>  You have added many additional capabilities to the parser now.  Are
>you getting the correct types of output in ECB?  Occasionally in the
>past I've added support for a stray & or const in type definitions for
>variables.  Later, when one of the token->text functions are called,
>it shows up in the wrong place.  For C this has been one of the more
>challenging aspects of translating C into a token, then back out
>again.  When last I ran tests, it was pretty good, but missing the
>finer points of * and & placement.  With all your new template and
>const work it would be good to asses the situation.

Done! You were right, Eric, in the function-arguments correct pointer
counting and const recognizing was disrupted after my latest changes.
This means parsing was ok, but the output was wrong.

Now i have fixed all this and now it works again! Now all the output
from the new parsing capabilities is correct - at least for this i have
tested ;-)

>  You can easily check what the outputs are with the command
>`semantic-test-all-token->text-functions' while the cursor is on a
>definition in your C code.

Yes, this is a very helpful command for this stuff...

Ciao,
Klaus


-------------------------------------------------------
This SF.NET email is sponsored by:
SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
http://www.vasoftware.com
_______________________________________________
Cedet-devel mailing list
Ced...@li...
https://lists.sourceforge.net/lists/listinfo/cedet-devel