#1930 Enormous delay when calling styler.SetLevel after document is folded.

[Stefanos]

Hi,

while implementing a plugin, I've noticed some huge delays using the styler.setLevel function when implementing folding.

Attached you will find the Lexer.cpp and the document I am using as testbench.

Bare in mind the following:

Upon opening the document, all the folding markers are instantly present and they are correctly displaying the folding points.

When removing/adding curly brackets at the very top of the document it is also instantly updating. The problem comes when I do the following:

1) Open document
2) Fold the entire document ( click on the very top folding header )
3) Unfold the document
4) Remove the very top opening curly bracket.

The set.Level function takes about 43 seconds to return...

There has to be some kind of bug, since this only happens if I previously fold the document and not before.. or on opening the document.

Using Notepad++ 7.3.3 32bit, Windows 10 64bit.

This version of Notepad++ uses scintilla version: 3.56

Attached u will find, the file I used, my lexer and two videos showing the issue.

[Neil Hodgson]

I am not seeing a long delay on Windows 10 64-bit with this lexer added to current SciTE 3.7.4. The file is a little janky when running on my (quite fast) machine but this isn't apparent in release mode.

Calling SetLevel may (depending on options) be notifying the application with an SC_MOD_CHANGEFOLD message. This may be performing arbitrary actions. You should step into the SetLevel call to find where the delay is occurring. It will be recurrently calling into your lexer to lex portions of the file before folding them and its possible the lengthy call is trying to lex the entire file.

Running the profiler can help understand where the time is going.

[Neil Hodgson]

Also tried Scintilla 3.5.6 and performance was the same as 3.7.4.

[Stefanos]

Any clues as to why this happens only after I fold the document once? Did u try it like this? I will check the notepad++ implementation to see if they react on this SC_MOD_CHANGEFOLD message. I think that it has something to do with caching the document first or similar. I can open any document , even bigger than the one I attached here and it will open instantly, only after fodling the document once, does this delay occur.

[Neil Hodgson]

Folding the document once is similar to scrolling to the end and back: the whole file is lexed and fold levels set.

With this file, removing the top bracket means that every line is one fold level lower. Every line is a subordinate of the top line and also of the second line so the 'last subordinate' that needs to be processed will be at the end of the file. Hence every line will be relexed and refolded with every line changing its fold level.

[Stefanos]

I checked the source code of Notepad++. This happens when SC_MOD_CHANGE_FOLD occurs:

void ScintillaEditView::foldChanged(int line, int levelNow, int levelPrev)
{
    if (levelNow & SC_FOLDLEVELHEADERFLAG)      //line can be folded
    {
        if (!(levelPrev & SC_FOLDLEVELHEADERFLAG))  //but previously couldnt
        {
            // Adding a fold point.
            execute(SCI_SETFOLDEXPANDED, line, 1);
            expand(line, true, false, 0, levelPrev);
        }
    }
    else if (levelPrev & SC_FOLDLEVELHEADERFLAG)
    {
        if (isFolded(line))
        {
            // Removing the fold from one that has been contracted so should expand
            // otherwise lines are left invisible with no way to make them visible
            execute(SCI_SETFOLDEXPANDED, line, 1);
            expand(line, true, false, 0, levelPrev);
        }
    }
    else if (!(levelNow & SC_FOLDLEVELWHITEFLAG) &&
            ((levelPrev & SC_FOLDLEVELNUMBERMASK) > (levelNow & SC_FOLDLEVELNUMBERMASK)))
    {
        // See if should still be hidden
        int parentLine = static_cast<int32_t>(execute(SCI_GETFOLDPARENT, line));
        if ((parentLine < 0) || !isFolded(parentLine && execute(SCI_GETLINEVISIBLE, parentLine)))
            execute(SCI_SHOWLINES, line, line);
    }
}

Where execute is simply a call to scintilla, BUT expand is this:

void ScintillaEditView::expand(int &line, bool doExpand, bool force, int visLevels, int level)
{
    int lineMaxSubord = int(execute(SCI_GETLASTCHILD, line, level & SC_FOLDLEVELNUMBERMASK));
    ++line;
    while (line <= lineMaxSubord)
    {
        if (force)
        {
            execute(((visLevels > 0) ? SCI_SHOWLINES : SCI_HIDELINES), line, line);
        }
        else
        {
            if (doExpand)
                execute(SCI_SHOWLINES, line, line);
        }

        int levelLine = level;
        if (levelLine == -1)
            levelLine = int(execute(SCI_GETFOLDLEVEL, line, 0));

        if (levelLine & SC_FOLDLEVELHEADERFLAG)
        {
            if (force)
            {
                if (visLevels > 1)
                    execute(SCI_SETFOLDEXPANDED, line, 1);
                else
                    execute(SCI_SETFOLDEXPANDED, line, 0);
                expand(line, doExpand, force, visLevels - 1);
            }
            else
            {
                if (doExpand)
                {
                    if (!isFolded(line))
                        execute(SCI_SETFOLDEXPANDED, line, 1);

                    expand(line, true, force, visLevels - 1);
                }
                else
                    expand(line, false, force, visLevels - 1);
            }
        }
        else
            ++line;
    }

    runMarkers(true, 0, true, false);
}

Theoretically there is recursion happening in here.. In any case, I might try to integrate my lexer with scite to see if I have improved responses.

[Neil Hodgson]

You could also try turning on Scintilla's built-in fold handling with http://www.scintilla.org/ScintillaDoc.html#SCI_SETAUTOMATICFOLD . This avoids calling back to the application and so, on Windows, avoids SendMessage calls which could be slow if there are any tools monitoring this system call.

[Stefanos]

First thing I'll do is just remove all handling of SC_MOD_CHANGE_FOLD from Notepad++. If this alleviates the issue then we know where the issue comes from. I am unaware of such option in Notepad++ to handle fodling by itself and I am not even sure it can work for external lexers if there is such an option.

[Stefanos]

I run the profiler without modifying any code from my plugin or from notepad++ this are the preliminary results. I'll try to analyze the hot paths now.

[Stefanos]

After removing handling of SC_MOD_CHANGEFOLD from NPP the editor responds instantly while adding/removing brackets. So this is definitely a Notepad++ issue.

[Stefanos]

U may close this issue. Thank u very much for your assistance.

[Neil Hodgson]

One difference between SciTE and Notepad++ is the runMarkers(true, 0, true, false); which could be taking the time. Try removing that.

SciTE moved away from recursion in the expansion handling and the equivalent of expand is now:

void SciTEBase::ExpandFolds(int line, bool expand, int level) {
    // Expand or contract line and all subordinates
    // level is the fold level of line
    const int lineMaxSubord = wEditor.Call(SCI_GETLASTCHILD, line, LevelNumber(level));
    line++;
    wEditor.Call(expand ? SCI_SHOWLINES : SCI_HIDELINES, line, lineMaxSubord);
    while (line <= lineMaxSubord) {
        const int levelLine = wEditor.Call(SCI_GETFOLDLEVEL, line);
        if (levelLine & SC_FOLDLEVELHEADERFLAG) {
            wEditor.Call(SCI_SETFOLDEXPANDED, line, expand ? 1 : 0);
        }
        line++;
    }
}

[Neil Hodgson]

I believe this is a problem with Notepad++ so closing.