#83 Python mode parsing causes 100% cpu

closed-fixed
semantic (53)
5
2007-03-08
2006-12-15
Gary Oberbrunner
No

Maybe you folks already know this, but the current CVS version of cedet with Emacs 22 has a problem parsing python files -- it takes 100% of CPU for a very long time. I'm not sure it's really an infinite loop, but it's really long (minutes for a 2k line python file). Several web pages suggest removing the python-mode-hook from wisent-python.el, such as this one:

http://www.emacswiki.org/cgi-bin/wiki?action=browse;id=CollectionOfEmacsDevelopmentEnvironmentTools

This makes semantic not parse python files (ecb still uses imenu or other basic parsers), which works around the bug.

Thought you should know about it so either it can be fixed or that line removed from the cvs version so it doesn't bite others.

-- Gary Oberbrunner

Discussion

1 2 > >> (Page 1 of 2)
  • Eric M. Ludlam
    Eric M. Ludlam
    2007-02-08

    Logged In: YES
    user_id=88537
    Originator: NO

    I fixed a similar sounding problem last year which is in CEDET CVS.

    http://sourceforge.net/mailarchive/forum.php?thread_id=9678864&forum_id=4869

    I tried the test python file that is in cedet/semantic/tset.py but it did not replicate.
    Do you have an example I can try?

     
  • Eric M. Ludlam
    Eric M. Ludlam
    2007-02-08

    • assigned_to: nobody --> zappo
     
  • ahjf
    ahjf
    2007-02-09

    Logged In: YES
    user_id=1581128
    Originator: NO

    the same situation happens to my emacs 23 checked out from cvs when I open a c file!

     
  • Logged In: YES
    user_id=417980
    Originator: YES

    Still happens for me with a large python file (which I can't share, sorry):

    Debugger entered--Lisp error: (quit)
    scan-lists(83347 -1 1)
    byte-code("惀䈀#�� [scan-lists -1 1 t] 4)
    wisent-python-lexer(1 89088 nil nil)
    semantic-lex-python-mode(1 89088 nil nil)
    semantic-lex(1 89088 nil)
    semantic-parse-region-default(1 89088 nil nil nil)
    semantic-parse-region(1 89088)
    semantic-fetch-tags()
    byte-code("
    ⃀휀섀� [semantic-fetch-tags nil] 1)
    semantic-idle-scheduler-refresh-tags()
    byte-code("ᣆ...)
    semantic-idle-scheduler-refresh-tags()
    semantic-idle-core-handler()
    semantic-idle-scheduler-function()
    apply(semantic-idle-scheduler-function nil)
    timer-event-handler([t 0 2 0 t semantic-idle-scheduler-function nil t])

    The python file is valid and syntactically correct; python loads and runs it fine. But interestingly, if I try to continue from the above quit, I get another debugger trap:
    Debugger entered--Lisp error: (scan-error "Unbalanced parentheses" 88714 1)
    and if I manually run the function above:
    M-: (scan-lists 83347 -1 1)
    I get the same error:
    (scan-error "Unbalanced parentheses 83347 1)

    I'll try to chop it down and see if I can get a shorter snippet to fail.

    (btw, the comment below from ahjf *might* just be the long first-time stealth subversion check that ebrowse does rather than this bug. Is your source dir really big, ahjf?)

     
  • Logged In: YES
    user_id=417980
    Originator: YES

    Update: I let it continue, and after about 5 minutes, the CPU usage went back to zero, so it finished parsing my file. Apparently it cached it (in semantic.cache?) because now it's fine. I could delete the cache file and try it again to see why it's slow, but I won't unless you'd like me to. It's definitely not an infinite loop anyway. Still, over 5 min for a 2k line python file seems excessive. (Just for the record, my "SConscript" file is the one I used for testing.)

     
  • Logged In: YES
    user_id=417980
    Originator: YES

    OK, one more update on the speed of parsing. I think almost all the time is taken in the scan-lists call in wisent-python-implicit-line-joining-p. The problem is it tries to scan backward for an open list, and often ends up going all the way back to the beginning of the buffer. Since it calls this function basically on every line, the behavior is quadratic: once per line, it scans all lines up to the current one. Maybe a regex search there would speed it up (at the cost of accuracy?), or somehow limit the search back a few k chars (but does scan-lists obey buffer restrictions, I don't know.)

     
  • patch for wisent-python.el

     
  • Logged In: YES
    user_id=417980
    Originator: YES

    OK, here's a patch. Not brilliant; just keeps scan-lists from looking back more than 1000 chars.
    File Added: wisent-python.el.patch

     
  • ahjf
    ahjf
    2007-02-09

    Logged In: YES
    user_id=1581128
    Originator: NO

    No,not the first time subversion check,but every time;
    but fortunately after i checked out the latest cedet source code, re-timestamped those makefiles and make again,those troules just go away.
    it seems to be a bug already solved but not integrated into the 1.02 tarball!

     
  • ahjf
    ahjf
    2007-02-09

    Logged In: YES
    user_id=1581128
    Originator: NO

    a few mistakes:
    troubles not troules
    1.03pre not 1.02
    sorry!

     
1 2 > >> (Page 1 of 2)