#1453 SCI_STYLESETINVISIBLEREP needed and implemented

[Ferdinand Oeinck]

With SCI_STYLESETVISIBLE certain text segments can be made invisible. Editing a document with invisible text segments is rather confusing. The cursor can be left or right of an invisible segment. But the user sees the cursor in both cases at the same position while the editing possibilties are different (ie not being able to paste text).

In my application I use scintilla as an editor component and have implemented the command SCI_STYLESETINVISIBLEREP which sets an UTF8 string as a representation for invisible text segments. This UTF8 string, such as for example an 'operator bullet' indicates to the user the position of the invisible segement while the cursor can be placed at both ends of the representsation. Thus making an end to the confusion. The representation text has the same style as the hidden text segment.

If there is need for this feature I'm pleased to contribute it to this great community.

[Neil Hodgson]

It is likely such a feature would be worthwhile so please publish the implementation.

[Ferdinand Oeinck]

I've not changed Scintilla.iface
Attached is the diff from the scintilla530.zip source

[Neil Hodgson]

Do you use different representations for different styles? It seems to me that having a single representation for invisible text would be adequate

[Ferdinand Oeinck]

I've written a lexilla parser for XTG files. Different xtg commands get different lexilla styles. To represent an xtg paragraph or character style command I need another colour than for example the xtg optional hyphen command. The representation of the latter should also be as small as possible ('operator bullet'). The representation of the other lexilla styles can be bigger ('circle or 'bullet').
See the image attached. Below is the origional text, above some styles are made invisible. You image that editing the text in the above view is much easier than in the view below?

[Neil Hodgson]

I suppose applying a small font size to bullet doesn't look as good?

Style goes to a lot of effort to minimize memory by not storing a string for each font name and this change enlarges the size of Style by 25%. See ViewStyle::fontNames but I'm not sure that is the best approach here. Another possibility is to reuse the character representation implementation which would also allow changing the colour or drawing the representation inside a blob.

The representation layout code should be moved inside the parallelized LayoutSegments function as execution has returned to a single thread at its current location. Its also not caching the resulting width which may be recalculated many times - see the way character representation widths can be retrieved from the cache pCache->MeasureWidths.

To follow Scintilla conventions, 'representation' should be spelled out: StyleSetInvisibleRepresentation, no underscores in variable names, and no tagging of UTF-8 strings like _utf8. Set APIs should also have corresponding Get APIs.

[Ferdinand Oeinck]

I've changed the code like you asked. The size of Style now has not grown at all! Only the first character of the UTF8 string will be stored in max 5 chars, which are on VS2022 already present as padding. Hope you like it.

[Neil Hodgson]

The test against styleInvisibleRepresentation always succeeds as its a pointer. You probably wanted styleInvisibleRepresentation[0].

C:\u\hg\scintilla\src\EditView.cxx(438,50): warning G205D2BA6: address of array 'vstyle.styles[ll->styles[ts.start]].styleInvisibleRepresentation' will always evaluate to 'true' [clang-diagnostic-pointer-bool-conversion]
                } else if (vstyle.styles[ll->styles[ts.start]].styleInvisibleRepresentation) {
                       ~~  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
C:\u\hg\scintilla\src\EditView.cxx(2247,41): warning G205D2BA6: address of array 'vsDraw.styles[styleMain].styleInvisibleRepresentation' will always evaluate to 'true' [clang-diagnostic-pointer-bool-conversion]
                                } else if (vsDraw.styles[styleMain].styleInvisibleRepresentation) {
                                       ~~  ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~

Since the setter is calling UTF8Classifyit should probably check the result for the UTF8MaskInvalid flag and do nothing when invalid.

There should be a test case like this in the test directory just to check the APIs work:

diff -r bafdddcfcbb0 test/simpleTests.py
--- a/test/simpleTests.py   Fri Sep 30 09:31:14 2022 +1000
+++ b/test/simpleTests.py   Sat Oct 01 09:10:49 2022 +1000
@@ -1956,6 +1956,7 @@
        self.ed.EmptyUndoBuffer()
        self.testColour = 0x171615
        self.testFont = b"Georgia"
+       self.testRepresentation = "\N{BULLET}".encode("utf-8")

    def tearDown(self):
        self.ed.StyleResetDefault()
@@ -2018,6 +2019,10 @@
        self.ed.StyleSetVisible(self.ed.STYLE_DEFAULT, 0)
        self.assertEquals(self.ed.StyleGetVisible(self.ed.STYLE_DEFAULT), 0)

+   def testInvisibleRepresentation(self):
+       self.assertEquals(self.ed.StyleGetInvisibleRepresentation(self.ed.STYLE_DEFAULT), b"")
+       self.ed.StyleSetInvisibleRepresentation(self.ed.STYLE_DEFAULT, self.testRepresentation)
+       self.assertEquals(self.ed.StyleGetInvisibleRepresentation(self.ed.STYLE_DEFAULT), self.testRepresentation)
+       self.ed.StyleSetInvisibleRepresentation(self.ed.STYLE_DEFAULT, b"\000")
+       self.assertEquals(self.ed.StyleGetInvisibleRepresentation(self.ed.STYLE_DEFAULT), b"")
+
    def testChangeable(self):
        self.assertEquals(self.ed.StyleGetChangeable(self.ed.STYLE_DEFAULT), 1)
        self.ed.StyleSetChangeable(self.ed.STYLE_DEFAULT, 0)

There are some spare API numbers 2256 and 2257 near other style APIs. Changing Scintilla.iface is worthwhile as running scripts/ScintillaAPIFacer.py fixes everything up:

# Set the invisible representation for a style.
set void StyleSetInvisibleRepresentation=2256(int style, string representation)

# Get the invisible representation for a style.
get int StyleGetInvisibleRepresentation=2257(int style, stringresult representation)

styleInvisibleRepresentation is a member of Style so should be shortened to invisibleRepresentation and can be fully initialized to all NULs with {}:

    invisibleRepresentation{} {

There should be a short explanation of the APIs in doc/ScintillaDoc.html mentioning that only a single character is supported.

[Ferdinand Oeinck]

Attached are the latest changes.

[Neil Hodgson]

The representation is displayed in the style of the invisible text but its measured in StyleControlChar which results in wrong positions like this picture where strings are represented by ● and overlap the ,. Control characters are styled to be tiny to illustrate the issue with style.*.36=font:Verdana,size:1 in SciTE.

The comment for measuring was copied from the control character code so mentions ts.representation->stringRep but it should be mentioning text or invisibleRepresentation.

In Editor.cxx the StyleGetChangeable case was damaged by the patch.

The documentation uses invisibleRepresentation as the name of the parameter but Scintilla.iface uses representation. The parameter names should be consistent and this is checked by scripts/CheckMentioned.py

[Ferdinand Oeinck]

I changed all that and additionally removed the memcpy in Editor.cxx and guarded against an invalid UTF8 character. I also removed the // fo comments and the line rcSegment.left += 0; // 2

[Neil Hodgson]

Committed with minor changes adding 'const' and formatting braces to follow conventions as [d72de4].
Also added invisiblerepresentation style attribute to SciTE as [c4c724].

[Neil Hodgson]

Committed with minor changes adding 'const' and formatting braces to follow conventions as [d72de4].
Also added invisiblerepresentation style attribute to SciTE as [c4c724].