#2483 (Win32) Sourcing autocompletion candidates from DBCS files has regressed since 5.5.6

[rdipardo]

Using Scintilla 5.5.6/7 with DirectWrite enabled, autocompletion lists can display illegible text if both:

1. the host PC's ANSI code page is a DBCS encoding, and;

2. completion candidates are sourced from a document in the same DBCS encoding as the ACP.

It seems the popup list's character set remains in the singe-byte default setting (influenced by the document's ACP encoding 🤔?), rather than a character set appropriate for the active SC_TECHNOLOGY_* setting.

Putting the parent Scintilla window into GDI mode resolves the issue.

Dynamically changing the popup's character set based on the document's encoding also appears to be effective.

To reproduce

• On a Windows PC, set the system's ANSI code page to Chinese Simplified (GB2312), so that the following Registry query returns the value shown below:

reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage /v ACP

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage
    ACP    REG_SZ    936

• Create a file with some Chinese in the same encoding, e.g.,

with open('sample.txt', 'wb') as f:
    f.write(bytes('zh-中文', '936'))

• Configure SciTE with the following properties:

code.page=936
character.set=132
technology=1
autocompleteword.automatic=1

• Open the sample file with wscite.exe and trigger the autocompletion list:

Cross-posted from https://github.com/notepad-plus-plus/notepad-plus-plus/issues/16719

[Zufu Liu]

A possible fix is directly passing code page (instead of Unicode mode) to list box, but this maybe a broken change on platform interface.

 ac.Start(wMain, idAutoComplete, sel.MainCaret(), PointMainCaret(),

-           lenEntered, lineHeight, IsUnicodeMode(), technology, options);
+           lenEntered, lineHeight, CodePage(), technology, options);

[Zufu Liu]

Or ListOptions could be extended to add codePage property like SurfaceMode.

[Neil Hodgson]

May be worthwhile although the ordering (SetOptions is called before Create which has to be maintained for compatibility) should be considered in ListBoxX::Create.

[Neil Hodgson]

This patch to Scintilla fixes Notepad++ in some circumstances although I don't really understand how it is determining code pages and character sets so may not handle other cases. The patch is also OK in SciTE.

[Zufu Liu]

It works the same way as EditModel::CurrentSurfaceMode() for Editor::CreateDrawingSurface(), code page 0 is CP_ACP.

unicodeMode filed and parameter can be removed.

[Neil Hodgson]

A SurfaceMode could be added to ListOptions but that would imply that bidiR2L may be implemented which looks unlikely due to lack of interest in bidirectional work. SurfaceMode may grow more complex.

If CodePageTransfer.patch is committed, there are various pieces of dead code that could be trimmed (FontWin::GetCharacterSet...) .

[Neil Hodgson]

code.page=936
character.set=132

Code page 936 is GB 2312 but that doesn't match character set 132. GB2312_CHARSET from WinGDI.h (and thus SC_CHARSET_GB2312) is 134.

When the property is corrected to character.set=134 I see:

[rdipardo]

So, yes, the issue comes entirely from a misconfigured character set, not technology after all. There must be a downstream reason why the char sets mismatch when N++ has D2D enabled.. Thanks anyway for looking at it.