#933 External lexer lag with large number of keywords

Bug
closed-accepted
Neil Hodgson
Scintilla (791)
2
2010-10-10
2010-03-28
Garthex
No

This problem arose from my lexer plugin for Notepad++, but part of the problem lies in Scintilla code from what I can tell. My lexer has an immense number of keywords and is loaded into Scintilla as an external lexer. On some machines, a noticeable lag is observed.

From studying the code, I've noticed that each time ExternalLexerModule::Lex is called, WordListsToStrings is also called, and once inside my plugin's lex function, the WordList must be recreated. I can cache the WordList object, but the WordListsToStrings is still called every time a character is typed as far as I can tell.

I'm certainly open to suggestions on how to improve the situation.

For reference, my plugin is called GmodLua (http://code.google.com/p/npp-gmod-lua/)

Discussion

  • Neil Hodgson
    Neil Hodgson
    2010-03-28

    I don't really like the external lexer mechanism - you may be better off discussing this on the scintilla-interest mailing list with those who use it.

    Probably the cheapest change would be to optimize WordListsToStrings. Its sure to be performing a lot of allocations which could be reduced to one by performing a measurement pass, allocating the memory, then copying all the words into that memory.

     
  • Neil Hodgson
    Neil Hodgson
    2010-03-28

    • priority: 5 --> 2
    • assigned_to: nobody --> nyamatongwe
     
  • Garthex
    Garthex
    2010-05-12

    Patch files v1

     
  • Garthex
    Garthex
    2010-05-12

    I think I've fixed the problem for the most part. What I did is save a variable in the WordList object, so that once it creates the string version of the keywords, it'll keep using that same variable instead of recreating it each call to Lex.

    The only drawback is one of security, since the same reference will be sent to the external lexers on each call, instead of creating a copy for each call. However, since internal lexers get a reference to the actual WordList object and could potential alter it from there, I don't think this is much of an issue.

     
  • Neil Hodgson
    Neil Hodgson
    2010-05-13

    There is soon going to be a major change to the way lexers work that will not have this problem. A document about the changes should be available in a day or so.

     
  • Garthex
    Garthex
    2010-10-10

    • status: open --> closed-accepted