Re: [cedet-semantic] Error parsing Python

[Eric M. L. <er...@si...>]

Hi Dale,

   An interesting find, and thanks for the detailed info about the 
reproduction and location of possible problems.

   The core lex function needs to turn off the syntax placed in text 
properties because some modes, like cc-mode makes some kinds of text 
completely invisible, or to appear unexpectedly to some Emacs commands. 
  Since the lexical analyzer typically depends on features like \\s* for 
searching for certain classes of syntax, having that change randomly is 
bad.

   As far as font lock is concerned, does python depend on font-lock to 
put text properties down before features like forward-sexp to work? 
This would be bad for the wisent parser to depend on, as font-lock is 
turned off during background parsing.

   The best bet is for wisent-python-forward-line to have some sort of 
custom 'forward-sexp', but I can imagine this being rather poor.  MATLAB 
has similar syntax around quotes which prevents it from using partial 
sexp matching.

   If python does not depend on font-lock to add the feature you need, 
then having that single function just re-enable 
parse-sexp-lookup-properties seems like a fine thing to me.

Does that help?  Sorry for not having a straight answer.
Eric

On 05/02/2010 12:12 PM, Dale Sedivec wrote:
> Greetings,
>
> 	I've found a bug in Semantic's parsing of Python code, and I'm
> not sure of the best way to fix it.  This two line Python exaqmple
> will trip the bug:
>
> def f():
>      return (''' ' ''')
>
> 	(This defines a function f that returns the string " ' ".  The
> "return" is actually unnecessary to trip this bug but makes the
> function a little easier for a non-Python programmer to understand.
> The parentheses on the second line are necessary to trip the bug,
> though superfluous as far as Python is concerned.)
>
> 	When I senator-force-refresh that code with debug-on-error and
> semantic-lex-debug-analyzers turned on I get an error calling
> scan-sexps with point on the left parenthesis on the second line.
>
> 	The problem seems to be that:
>
> 1. define-lex turns off parse-sexp-lookup-properties (semantic-lex.el
>     from HEAD, line 843).
> 2. wisent-python-forward-line calls (forward-sexp 1) when it's looking
>     at a parenthesis (wisent-python.el line 102).
> 3. python-mode relies on font-lock-syntactic-properties and syntax
>     properties (and thus parse-sexp-lookup-properties) to parse
>     triple-quoted strings.  (I believe this is the case at least as far
>     back as Emacs 22's python.el.  Search for python-quote-syntax.)
>
> 	I noticed this bug during normal use because I had a (much
> more complicated) function like this and I noticed which-func-mode
> wasn't working for any code following f().
>
> 	I would submit a patch but I'm not sure how this should be
> fixed:
>
> * I'm not sure if it's possible to keep python-mode from relying
>    on font-lock-syntactic-keywords because I don't know much about that
>    corner of Emacs (and I also suspect that parsing Python string
>    literals with, for example, a regular expression is impossible).
>
> * Should define-lex be allowed to turn off syntactic keywords?
>
> * The easiest fix is probably to have wisent-python-forward-line turn
>    parse-sexp-lookup-properties back on when it wants to use Emacs'
>    balanced expression parsing/movement, but that might be considered
>    unclean: how do you know why parse-sexp-lookup-properties was turned
>    off in the first place?
>
> 	I'd be most grateful if someone else could fix this bug or
> at least point me in the right direction to produce a fix myself.
>
> 	FYI I'm using Emacs 23.1.96 on OS X with CEDET from CVS.
>
> 	Thanks, and thanks for CEDET and Semantic!
>
> Dale
>
> ------------------------------------------------------------------------------
> _______________________________________________
> cedet-semantic mailing list
> ced...@li...
> https://lists.sourceforge.net/lists/listinfo/cedet-semantic
>