#1567 Remove unnecessary `InvalidateStyleRedraw()` for `Message::SetUseTabs`

[YX Hao]

Hi Neil and maintainers,

I think Message::SetUseTabs only toggles wheather or not new tabs will be replaced with spaces, and the layout doesn't change. So, InvalidateStyleRedraw() is not necessary.

This eliminates a bug of Notepad4: doesn't work correctly on rectangular selections. If 0 column is selected, press 1+ times. The Ctrl + Tab feature can date back to the original Notepad2.

Thanks in advance!

[Zufu Liu]

useTabs field is only used inside Document::SetLineIndentation() and Editor::Indent() methods, call to InvalidateStyleRedraw() can be eliminated.

[Neil Hodgson]

InvalidateStyleRedraw() should be safe to call. If Ctrl+Tab is not working correctly then Message::Tab that it calls may need to perform more setup. If just the change proposed above is done then there may be other ways to reach Message::Tab in a state that fails.

The reason for the failure should be examined more closely and, perhaps fixed inside Editor::Indent that is called for Message::Tab.

While the mentioned InvalidateStyleRedraw() can be removed for performance, the potential failure is more important.

[Zufu Liu]

The mentioned code block inserts raw tab (\t) character at caret position:

case CMD_CTRLTAB:
    SciCall_SetTabIndents(false);
    SciCall_SetUseTabs(true);
    SciCall_Tab();
    SciCall_SetUseTabs(!fvCurFile.bTabsAsSpaces);
    SciCall_SetTabIndents(fvCurFile.bTabIndents);
    break;

1567before.gif is screenshot with InvalidateStyleRedraw(),
1567after.gif is screenshot without InvalidateStyleRedraw().

[Zufu Liu]

Wonder how YX Hao identified the bug and fix.
Apply 1567-Notepad4-debug-1013.diff onto today's Noetpad4 code, build (e.g. click batch file inside build\VisualStudio\ folder) and run Notepad4.exe from terminal, open a 2 bytes file (\n\n, just two new-lines), hold Alt key to make rectangular selection for the three lines, press Ctrl + Tab twice.

log without InvalidateStyleRedraw() for Message::SetUseTabs:

SetRectangularRange | Editor::ButtonUpWithModifiers(class Scintilla::Internal::Point,unsigned int,enum Scintilla::KeyMod):5267
SetRectangularRange range: 0
SetRectangularRange range: 1
SetRectangularRange range: 2
Tab before: ToString: R#2,0-2 | Editor::KeyCommand(enum Scintilla::Message):4088
ToString: R#2,0-2 | Editor::Indent(bool,bool):4190
Indent DeleteChars(0, 0)
ToString: R#2,0-2 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-3 | Editor::Indent(bool,bool):4260
ToString: R#2,1-3 | Editor::Indent(bool,bool):4190
Indent DeleteChars(2, 0)
ToString: R#2,1-3 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-4 | Editor::Indent(bool,bool):4260
ToString: R#2,1-4 | Editor::Indent(bool,bool):4190
Indent DeleteChars(4, 0)
ToString: R#2,1-4 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-4 | Editor::Indent(bool,bool):4260
ToString: R#2,1-4 | ModelState::RememberSelectionForRedoOntoStack(int,const class Scintilla::Internal::Selection &,__int64):79
Tab  after: ToString: R#2,1-4 | Editor::KeyCommand(enum Scintilla::Message):4096
Tab before: ToString: R#2,1-4 | Editor::KeyCommand(enum Scintilla::Message):4088
ToString: R#2,1-4 | Editor::Indent(bool,bool):4190
Indent DeleteChars(1, 0)
ToString: R#2,1-4 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,2-5 | Editor::Indent(bool,bool):4260
ToString: R#2,2-5 | Editor::Indent(bool,bool):4190
Indent DeleteChars(4, 0)
ToString: R#2,2-5 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,2-6 | Editor::Indent(bool,bool):4260
ToString: R#2,2-6 | Editor::Indent(bool,bool):4190
Indent DeleteChars(7, 0)
ToString: R#2,2-6 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,2-6 | Editor::Indent(bool,bool):4260
ToString: R#2,2-6 | ModelState::RememberSelectionForRedoOntoStack(int,const class Scintilla::Internal::Selection &,__int64):79
Tab  after: ToString: R#2,2-6 | Editor::KeyCommand(enum Scintilla::Message):4096

log with InvalidateStyleRedraw():

SetRectangularRange | Editor::ButtonUpWithModifiers(class Scintilla::Internal::Point,unsigned int,enum Scintilla::KeyMod):5267
SetRectangularRange range: 0
SetRectangularRange range: 1
SetRectangularRange range: 2
Tab before: ToString: R#2,0-2 | Editor::KeyCommand(enum Scintilla::Message):4088
ToString: R#2,0-2 | Editor::Indent(bool,bool):4190
Indent DeleteChars(0, 0)
ToString: R#2,0-2 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
RefreshStyleData | Editor::NotifyModified(class Scintilla::Internal::Document *,class Scintilla::Internal::DocModification,void *):2917
SetRectangularRange | Editor::RefreshStyleData(const struct std::source_location):260
SetRectangularRange range: 1-0
SetRectangularRange range: 2v8-2
SetRectangularRange range: 3v8-3
ToString: R#2,1-3 | Editor::Indent(bool,bool):4260
ToString: R#2,1-3 | Editor::Indent(bool,bool):4190
Indent DeleteChars(2, 0)
ToString: R#2,1-3 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-4 | Editor::Indent(bool,bool):4260
ToString: R#2,1-4 | Editor::Indent(bool,bool):4190
Indent DeleteChars(4, 0)
ToString: R#2,1-4 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-4 | Editor::Indent(bool,bool):4260
ToString: R#2,1-4 | ModelState::RememberSelectionForRedoOntoStack(int,const class Scintilla::Internal::Selection &,__int64):79
Tab  after: ToString: R#2,1-4 | Editor::KeyCommand(enum Scintilla::Message):4096
RefreshStyleData | ScintillaWin::PaintDC(struct HDC__ *):1406
SetRectangularRange | Editor::RefreshStyleData(const struct std::source_location):260
SetRectangularRange range: 1-0
SetRectangularRange range: 3-2
SetRectangularRange range: 5-4
Tab before: ToString: R#2,1-4 | Editor::KeyCommand(enum Scintilla::Message):4088
ToString: R#2,1-4 | Editor::Indent(bool,bool):4190
Indent DeleteChars(0, 1)
ToString: R#2,1-4 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
NotifyModified DeleteText 0, 1
ToString: R#2,1-4 | Editor::NotifyModified(class Scintilla::Internal::Document *,class Scintilla::Internal::DocModification,void *):2866
ToString: R#2,0-3 | Editor::NotifyModified(class Scintilla::Internal::Document *,class Scintilla::Internal::DocModification,void *):2870
RefreshStyleData | Editor::NotifyModified(class Scintilla::Internal::Document *,class Scintilla::Internal::DocModification,void *):2917
SetRectangularRange | Editor::RefreshStyleData(const struct std::source_location):260
SetRectangularRange range: 0
SetRectangularRange range: 1
SetRectangularRange range: 3
ToString: R#2,0-3 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-4 | Editor::Indent(bool,bool):4260
ToString: R#2,1-4 | Editor::Indent(bool,bool):4190
Indent DeleteChars(2, 0)
ToString: R#2,1-4 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-5 | Editor::Indent(bool,bool):4260
ToString: R#2,1-5 | Editor::Indent(bool,bool):4190
Indent DeleteChars(5, 0)
ToString: R#2,1-5 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-5 | Editor::Indent(bool,bool):4260
ToString: R#2,1-5 | ModelState::RememberSelectionForRedoOntoStack(int,const class Scintilla::Internal::Selection &,__int64):79
Tab  after: ToString: R#2,1-5 | Editor::KeyCommand(enum Scintilla::Message):4096
RefreshStyleData | ScintillaWin::PaintDC(struct HDC__ *):1406
SetRectangularRange | Editor::RefreshStyleData(const struct std::source_location):260
SetRectangularRange range: 1-0
SetRectangularRange range: 3-2
SetRectangularRange range: 6-5

So, the underline bug is not fixed: SetRectangularRange() may change selection ranges, and cause pdoc->DeleteChars() inside Indent() to actually delete characters.

[Zufu Liu]

Not find a fix, the bug can be reproduced in SciTE with similar code, e.g.:

case IDM_HELP: {
    wEditor.SetTabIndents(false);
    wEditor.SetUseTabs(true);
    wEditor.Tab();
    wEditor.SetUseTabs(props.GetInt("use.tabs", 1));
    wEditor.SetTabIndents(props.GetInt("tab.indents", 1));
    return;

[Neil Hodgson]

I think the basic issue here is between a rectangular selection and Editor::Indent not behaving in a 'rectangular' way. Each piece of the selection is treated as part of a multiple-selection. The selection type stays rectangular but the range is not updated to match the bounds of all of the selection pieces. So there is a mismatch between Selection::rangeRectangular and Selection::ranges.

Likely the best approach would be to work out how Tab should work in a rectangular context and implement that as a separate branch inside Editor::Indent. Less work would be repairing the relationship between the fields. Possibly by setting Selection::rangeRectangular to match the bounds of Selection::ranges then recalculating Selection::ranges.

The role of InvalidateStyleRedraw is to cause SetRectangularRange to occur, which does synchronize ranges from rangeRectangularbut that may be the wrong direction in this case, not matching user preference.

[Zufu Liu]

without SetRectangularRange, both caret and anchor positions moved to right, and keeps no text selected.

with SetRectangularRange, caret positions sticked at zero column, anchor positions moved to right, keeps first column selected, which is unexpected.

SetRectangularRange | Editor::ButtonUpWithModifiers(class Scintilla::Internal::Point,unsigned int,enum Scintilla::KeyMod):5269
ToString: R#2,0-2; 0,1,2 | Editor::SetRectangularRange(const struct std::source_location):581
SetRectangularRange range: 0
SetRectangularRange range: 1
SetRectangularRange range: 2
ToString: R#2,0-2; 0,1,2 | Editor::SetRectangularRange(const struct std::source_location):608
Tab before: ToString: R#2,0-2; 0,1,2 | Editor::KeyCommand(enum Scintilla::Message):4090
ToString: R#2,0-2; 0,1,2 | Editor::Indent(bool,bool):4192
Indent DeleteChars(0, 0)
ToString: R#2,0-2; 0,1,2 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
RefreshStyleData | Editor::NotifyModified(class Scintilla::Internal::Document *,class Scintilla::Internal::DocModification,void *):2919
SetRectangularRange | Editor::RefreshStyleData(const struct std::source_location):260
ToString: R#2,1-3; 0,2,3 | Editor::SetRectangularRange(const struct std::source_location):581
SetRectangularRange range: 1-0
SetRectangularRange range: 2v8-2
SetRectangularRange range: 3v8-3
ToString: R#2,1-3; 1-0,2v8-2,3v8-3 | Editor::SetRectangularRange(const struct std::source_location):608
ToString: R#2,1-3; 1,2v8-2,3v8-3 | Editor::Indent(bool,bool):4262
ToString: R#2,1-3; 1,2v8-2,3v8-3 | Editor::Indent(bool,bool):4192
Indent DeleteChars(2, 0)
ToString: R#2,1-3; 1,2v8-2,3v8-3 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-4; 1,3,4v8-4 | Editor::Indent(bool,bool):4262
ToString: R#2,1-4; 1,3,4v8-4 | Editor::Indent(bool,bool):4192
Indent DeleteChars(4, 0)
ToString: R#2,1-4; 1,3,4v8-4 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-4; 1,3,5 | Editor::Indent(bool,bool):4262
ToString: R#2,1-4; 1,3,5 | ModelState::RememberSelectionForRedoOntoStack(int,const class Scintilla::Internal::Selection &,__int64):79
Tab  after: ToString: R#2,1-4; 1,3,5 | Editor::KeyCommand(enum Scintilla::Message):4098
RefreshStyleData | ScintillaWin::PaintDC(struct HDC__ *):1406
SetRectangularRange | Editor::RefreshStyleData(const struct std::source_location):260
ToString: R#2,1-4; 1,3,5 | Editor::SetRectangularRange(const struct std::source_location):581
SetRectangularRange range: 1-0
SetRectangularRange range: 3-2
SetRectangularRange range: 5-4
ToString: R#2,1-4; 1-0,3-2,5-4 | Editor::SetRectangularRange(const struct std::source_location):608
Tab before: ToString: R#2,1-4; 1-0,3-2,5-4 | Editor::KeyCommand(enum Scintilla::Message):4090
ToString: R#2,1-4; 1-0,3-2,5-4 | Editor::Indent(bool,bool):4192
Indent DeleteChars(0, 1)
ToString: R#2,1-4; 1-0,3-2,5-4 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
NotifyModified DeleteText 0, 1
ToString: R#2,1-4; 1-0,3-2,5-4 | Editor::NotifyModified(class Scintilla::Internal::Document *,class Scintilla::Internal::DocModification,void *):2868
ToString: R#2,0-3; 0,2-1,4-3 | Editor::NotifyModified(class Scintilla::Internal::Document *,class Scintilla::Internal::DocModification,void *):2872
RefreshStyleData | Editor::NotifyModified(class Scintilla::Internal::Document *,class Scintilla::Internal::DocModification,void *):2919
SetRectangularRange | Editor::RefreshStyleData(const struct std::source_location):260
ToString: R#2,0-3; 0,2-1,4-3 | Editor::SetRectangularRange(const struct std::source_location):581
SetRectangularRange range: 0
SetRectangularRange range: 1
SetRectangularRange range: 3
ToString: R#2,0-3; 0,1,3 | Editor::SetRectangularRange(const struct std::source_location):608
ToString: R#2,0-3; 0,1,3 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-4; 1,2,4 | Editor::Indent(bool,bool):4262
ToString: R#2,1-4; 1,2,4 | Editor::Indent(bool,bool):4192
Indent DeleteChars(2, 0)
ToString: R#2,1-4; 1,2,4 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-5; 1,3,5 | Editor::Indent(bool,bool):4262
ToString: R#2,1-5; 1,3,5 | Editor::Indent(bool,bool):4192
Indent DeleteChars(5, 0)
ToString: R#2,1-5; 1,3,5 | ModelState::RememberSelectionForUndo(int,const class Scintilla::Internal::Selection &):64
ToString: R#2,1-5; 1,3,6 | Editor::Indent(bool,bool):4262
ToString: R#2,1-5; 1,3,6 | ModelState::RememberSelectionForRedoOntoStack(int,const class Scintilla::Internal::Selection &,__int64):79
Tab  after: ToString: R#2,1-5; 1,3,6 | Editor::KeyCommand(enum Scintilla::Message):4098
RefreshStyleData | ScintillaWin::PaintDC(struct HDC__ *):1406
SetRectangularRange | Editor::RefreshStyleData(const struct std::source_location):260
ToString: R#2,1-5; 1,3,6 | Editor::SetRectangularRange(const struct std::source_location):581
SetRectangularRange range: 1-0
SetRectangularRange range: 3-2
SetRectangularRange range: 6-5
ToString: R#2,1-5; 1-0,3-2,6-5 | Editor::SetRectangularRange(const struct std::source_location):608

[Neil Hodgson]

The rectangular range is moved for each tab insertion (which is reasonable) but is then used to recreate the ranges. At the beginning the rectangular range is 0-2, the first line has a tab inserted so the rectangular range is 1-3 which means it is from column 1 on line 0 back to column 0 on line 2, so now contains space (real for line 0, virtual for others).

For rectangular selections, ranges shouldn't be recalculated from rangeRectangular for each line. This could be achieved by copying the initial selection and reverse iterating (to avoid problems from text moving away from the copy). However, that would get in the way of the other cases and complicate the code.

[Neil Hodgson]

One possible technique is to temporarily treat the selection as multiple stream selections then convert back to rectangular at end of Indent:

diff -r a754ea3c2f77 src/Editor.cxx
--- a/src/Editor.cxx    Mon Oct 13 13:27:07 2025 +1100
+++ b/src/Editor.cxx    Sat Oct 18 08:40:23 2025 +1100
@@ -4155,6 +4155,10 @@

 void Editor::Indent(bool forwards, bool lineIndent) {
    UndoGroup ug(pdoc);
+   const bool wasRectangular = sel.IsRectangular();
+   if (wasRectangular) {
+       sel.selType = Selection::SelTypes::stream;
+   }
    for (size_t r=0; r<sel.Count(); r++) {
        const Sci::Line lineOfAnchor =
            pdoc->SciLineFromPosition(sel.Range(r).anchor.Position());
@@ -4231,6 +4235,11 @@
            }
        }
    }
+   if (wasRectangular) {
+       sel.selType = Selection::SelTypes::rectangle;
+       sel.Rectangular() = SelectionRange(sel.Range(sel.Count() - 1).anchor, sel.Range(0).anchor);
+       SetRectangularRange();
+   }
    ContainerNeedsUpdate(Update::Selection);
 }

This may need additional work for other conditions like 'thin' selections.

[Zufu Liu]

That works, changes with less code also works:

@@ -4155,6 +4155,10 @@

 void Editor::Indent(bool forwards, bool lineIndent) {
    UndoGroup ug(pdoc);
+   const Selection::SelTypes selType = sel.selType;
+   if (sel.IsRectangular()) {
+       sel.selType = Selection::SelTypes::stream;
+   }
    for (size_t r=0; r<sel.Count(); r++) {
        const Sci::Line lineOfAnchor =
            pdoc->SciLineFromPosition(sel.Range(r).anchor.Position());
@@ -4231,6 +4235,8 @@
            }
        }
    }
+   sel.selType = selType;
+   ThinRectangularRange();
    ContainerNeedsUpdate(Update::Selection);
 }

[Zufu Liu]

Just needs ThinRectangularRange();, no need to change selType.

[Neil Hodgson]

There are more problems when there is text on the lines, deleting text. This is fixed with the above change but not with just calling ThinRectangularRange.

[Zufu Liu]

Oh, not tested that.

[Zufu Liu]

Sorry, I'm not work on this bug. I just feel both the changes have Implicit assumption about selections and selType retains during Indent() operation, but origin code does not require the assumption.

[Neil Hodgson]

The scope of this change isn't great but I think it is still positive as it stops some cases where user text was deleted.

I got side tracked when trying to write a test but see different results in a mechanical test than a user test.

[YX Hao]

I'm not as familiar with the project code as you are. I commented out SciCall_SetTabIndents() and SciCall_SetUseTabs(), leaving only SciCall_Tab(). It worked fine (in use Tabs mode). Then narrowed down to the handling of Message::SetUseTabs and found that InvalidateStyleRedraw() was unnecessary. After removing it, it did work, so I didn't look into it.

[Neil Hodgson]

Committed with [403f36], [4200ef], [fb680a].

Changes Editor::Indent as described above and adds a test. Changes Python test script to make it easier to run inside Visual C++ debugger. Removes InvalidateStyleRedraw as not needed.