#2403 Add an autocomplete popup mode doing no processing

[Jiri Techet]

I'm implementing a LSP client for Geany (I might start a few more tickets starting exactly this way) and for autocompletion, language servers typically return autocompletion items together with their ordering which might not be in the alphabetical order. Also, these items may not start with the prefix the user has just written - what has been written can be in the middle of the autocompleted symbol for instance.

For Scintilla this means using SC_ORDER_CUSTOM. The problem is with lengthEntered of SCI_AUTOCSHOW. When specified, Scintilla filteres-out entries of the autocompletion list based on the entered prefix so for LSP it filters-out those entries not starting with the prefix. Of course, one can always set lengthEntered to 0 but then the autocompletion list jumps for each entered letter to the right and also flickers because it seems to be cancelled and re-displayed (at least on GTK3).

I don't know if it has ever been discussed or if there's some workaround but wouldn't it be possible to introduce some mode which uses lengthEntered only for positioning the popup window but not for filtering the list? (with LSP one sends a new autocompletion request to the server for each entered character so all the filtering and logic behind what's displayed in the list is on the server side)

[Jiri Techet]

OK, so I think I was wrong - Scintilla doesn't filter the list based on the entered prefix when using SC_ORDER_CUSTOM, not sure what I did.

But there's another problem - it doesn't always select the first item item in the list which is what I'd like it to do. For instance, when I type "g_", the list looks like

get_ident_prefixlen
G_ENCODE_VERSION
G_LOG_DOMAIN
g_string_append_c <--- selected
get_autocomplete_icon

It clearly tries to select the one starting with the typed prefix but with LSP one gets the edit description from the server with the range of the text that should be replaced and the new text so when one selects for instance get_ident_prefixlen, the typed "g_" gets replaced. The selection is a bit unexpected because one's muscle memory is to scroll down and here one has to scroll up and also look what's before.

Can this behavior be somehow changed?

[Neil Hodgson]

Use AutoCSelect:

function UnorderedAC()
    editor.AutoCOrder = SC_ORDER_CUSTOM 
    editor.AutoCSeparator = 32 -- space
    editor:AutoCShow(2, "get_ident_prefixlen G_ENCODE_VERSION G_LOG_DOMAIN g_string_append_c get_autocomplete_icon")
    editor:AutoCSelect("get_ident_prefixlen")
end
-- Test g_

[Jiri Techet]

Thanks, it works so the first item is selected. One can still briefly see the "wrong" selection before it jumps to the first item while typing which isn't completely ideal.

Anyway, this is definitely the least eye-straining option right now and and kind of acceptable.

[Neil Hodgson]

When specified, Scintilla filteres-out entries of the autocompletion list based on the entered prefix

This doesn't sound correct to me. Are you sure it isn't application logic performing the filter?

[Jiri Techet]

OK, so I think I was wrong - Scintilla doesn't filter the list based on the entered prefix when using SC_ORDER_CUSTOM, not sure what I did.

Yeah, I was wrong, see above. The selection of the non-first row in the list is still present though.

[Neil Hodgson]

There can be another autocompletion option (for AutoCSetOptions) to avoid the move to current selection behaviour. Perhaps SC_AUTOCOMPLETE_SELECT_FIRST_ITEM=2

[Jiri Techet]

Great, does something like the attached patch look OK to you? It fixes the problem for me.

Just a question - should SCI_AUTOCSETOPTIONS behave as a bit mask if e.g. someone wanted to select SC_AUTOCOMPLETE_FIXED_SIZE and SC_AUTOCOMPLETE_SELECT_FIRST_ITEM at the same time? It would mean changing some AutoCompleteOption variables and parameters to int and using & for checking the values.

[Neil Hodgson]

The change affects more than I expected from your description. I thought you wanted to just change initial display behaviour but changing AutoComplete::Select changes behaviour on user actions like typing and even explicit API calls like SCI_AUTOCSELECT so its more of a permanent mode and I don't understand the benefit.

Choosing the first item just at start can probably be implemented by changing the code at the end of ScintillaBase::AutoCompleteStart.

SCI_AUTOCSETOPTIONS should be a bit-mask. Its more work to add whole new APIs than an enumeration value. Autocompletion has a lot more options than initially expected.

The canonical source of features is include/Scintilla.iface which is where the new state should be defined. Then scripts/ScintillaAPIFacer.py is run to update all dependent files.

[Jiri Techet]

Ah, no, that wasn't my intention, I should have checked how it's called in more detail. I'll have a look at ScintillaBase::AutoCompleteStart as you suggest.

[Jiri Techet]

I had a look at the code in ScintillaBase and what I'd like to actually achieve is that AutoCompleteMoveToCurrentWord() doesn't get executed in this mode or that it's replaced by code always selecting the first item.

I've attached a screenshot better illustrating the problem. Basically, clangd, but other LSP servers too, use some form of fuzzy-matching where the user doesn't have to write the word exactly, but, instead, the word may be from the middle of the autocompleted list or even slightly misspelled.

In the example from the screenshot, even though the user has written "comp", the server returned a non-alphabetically sorted list of words where the first word is "AutoComplete". Now when you type another letter, "l", the editor sends a new asynchronous request to the server for the updated autocompletion list. Menwhile, before the response is received from the server, Scintilla automatically tries to select the word from the list based on alphabetic sorting, so in this case it would select "CompletionMethods" (assuming SCI_AUTOCSETIGNORECASE is on). Then, the editor receives the response from the LSP server and the editor selects the first item which leads to the temporary flicker of unwanted "CompletionMethods" selection.

I tried to workaround this by selecting the first item in the list using SCI_AUTOCSELECT in SCN_CHARADDED but Scintilla apparently selects the other item at a later point so this didn't work.

Anyway, back to the patch - would replacing AutoCompleteMoveToCurrentWord() with first item selectin be acceptable in this mode? I think it's necessary in all three cases - popup appearance, addition, and deletion.

[Neil Hodgson]

would replacing AutoCompleteMoveToCurrentWord() with first item selectin be acceptable in this mode?

It appears to me that this change makes it not really autocompletion but its unclear just which features are supposed to remain. Maybe this should be considered a separate feature like user lists.

[Jiri Techet]

But this is still normal autocompletion - just without Scintilla automatically selecting the item based on alphabetic sorting but leaving the selection at the first item (with the assumption that the provided list is sorted so that the items in this order make sense, but it's the responsibility of the calling code).

I think this is a very common way autocompletion works in many editors - when you try vscode for instance, the autocompletion list also isn't sorted alphabetically and as you type, the first item is being selected and only the contents of the autocompletion list changes as you keep typing.

[Neil Hodgson]

It seems to me that this will continue to evolve, affecting other aspects (such as auto-hide and choose-single), and I'd prefer to wait until its finished. You should work on changes locally until then.

[Jiri Techet]

OK, I created the attached patch and I'm not sure if anything more is actually needed:

1. SCI_AUTOCSETAUTOHIDE doesn't seem to make sense in this mode as "there are no viable matches" never happens - LSP servers allow some form of fuzzy input so even when you misspell something, it still appears in the autocompletion list and may still be used for autocompletion.
2. I tried SCI_AUTOCSETCHOOSESINGLE but the result is extremely confusing and unpredictable. I can imagine it works for alphabetically sorted list of items where you keep narrowing-down the list to a single item so you know have some idea what gets selected at the end. With LSP, you never know what the server returns in the next request so you don't know what will be the single item and when this moment comes and the result is just a big surprise. This is doable though so if required, I can add it.

I tested this patch only with Geany (with the LSP plugin) only, hope it works with SciTE too.

[Neil Hodgson]

Patch should add documentation to doc/ScintillaDoc.html.

The way to write bit flag tests without excessive casting is the generic FlagSet:

if (FlagSet(ac.options, AutoCompleteOption::SelectFirstItem))

[Jiri Techet]

Sorry for the long delay, I was busy with other things.

I'm not very familiar with Scintilla tests - should this be done using the python scripts or the cxx tests under unit? Are there already some tests related to autocompletion I could have a look at for inspiration?

[Neil Hodgson]

There are autocompletion tests in the Python script scintilla/test/simpleTests.py that can be extended to cover this. These tests currently only run on Win32.

The C++ unit tests are for data structures and don't have access to a GUI so can't be used for this feature.

[Jiri Techet]

OK, I'll try to revive my Windows machine...

[Jiri Techet]

I've just created a new patch, does it look OK?

[Neil Hodgson]

Committed as [ad3520].

[Jiri Techet]

Great, thanks!