#1407 Emacs like TeX Input Method for Symbols

[honfui]

In Atom, particularly Juno, or VSCode via plugin "LaTeX Input:, special characters, e.g. beta (β) could be entered into the editor with "\beta", followed by "Tab" key. Can this feature be added into SciTE?

Currently, symbols (or smiley) are entered by from/copy web and paste it. Second method is using Windows+"." keys combinations for Win10 (of course UTF8 encoding file only). Thank you.

[honfui]

From Emacs documentation -> https://www.emacswiki.org/emacs/TeXInputMethod

[Neil Hodgson]

Sounds like the abbreviations feature. Edit | Expand Abbreviation (Ctrl+B) with definitions set up in the abbreviations file Options | Open Abbreviations File.

[Zufu Liu]

I have an implementation at https://github.com/zufuliu/notepad2/blob/master/scintilla/win32/ScintillaWin.cxx#L1334
You can try it in Notepad2 after enable LaTeX input method (under Settings menu).

See https://github.com/zufuliu/notepad2/issues/289 and https://github.com/zufuliu/notepad2/issues/296

[honfui]

I tried for 20 minutes the abbreviations approach, but didn't manage to get it working. It shows that it is not easy-to-use feature. Symbol or emoji are standard - thus is the reason (I'm guessing) Emacs created this feature.

The implementation by Notepad2 is the rather easy to use. Thanks Liu Zufu.

[Neil Hodgson]

Attached an example abbrev.properties file.

[Zufu Liu]

How about implement this inside Scintilla? my ScintillaWin::HandleLaTeXTabCompletion() is almost platform independent (it currently call MultiByteFromWideChar() to convert the result, see https://github.com/zufuliu/notepad2/blob/master/scintilla/include/LaTeXInput.h#L57).

The changes would be:

1. Add script to generate all data, I made https://github.com/zufuliu/notepad2/blob/master/scintilla/scripts/LaTeXInput.py, which will generates data from https://docs.julialang.org/en/v1.7-dev/manual/unicode-input/ and https://github.com/iamcal/emoji-data/blob/master/emoji_pretty.json
2. Add flag for SCK_TAB, I use TabCompletion::Latex https://github.com/zufuliu/notepad2/blob/master/scintilla/win32/ScintillaWin.cxx#L2362
3. Update code for Message::Tab, to handle LaTeX conversion.
4. Add API to get LaTeX or emoji input sequences, so app can show them when appropriate.

My current implementation have 67 KiB data (2508 LaTeX, 46 KiB; 1373 emoji, 26 KiB), bigger than Unicode category data, if size matters, a preprocessor can be added to make this feature optional by default.

[Neil Hodgson]

This appears to be better as application functionality. An application can set the definitions appropriate to their users, make them context sensitive, and allow users to add to and modify the definitions.

[honfui]

Thanks for the sample abbreviation. It works for simple notation. Complicated notation such as subscript_x ("_x") or superscript_y ("\^y") are harder. My sample abbreviation is below.

\beta=β
\alpha=α
\fire=🔥
\^0=⁰
\_1=₁
\_x=x

E.g. β⁰ + α₁ + Yx

[Zufu Liu]

A daft implementation for SciTE, show input sequences on auto-completion list is not implemented.

following sequences are not supported (maybe others that contains special punctuation).

^= ⁼ U+0207C Superscript Equals Sign
_= ₌ U+0208C Subscript Equals Sign

[Zufu Liu]

Added auto-completion, the code needs some tweaks, e.g. StringList wl(true); in SciTEBase::StartAutoCompleteWord() can be optimized out.

[Neil Hodgson]

There's a memory out-of-bounds write so it needs something like:

SA::Position start = std::max<SA::Position>(0, main - (sizeof(buffer) - 2));

Another way to access emoji and mathematical symbols is through the Windows Touch keyboard which has the advantage of showing the characters and requires less memorization.

[Zufu Liu]

Good catch. updated patch with hard-coded mapping for

^= ⁼ U+0207C Superscript Equals Sign
_= ₌ U+0208C Subscript Equals Sign

There are other problems: (1) SciTEGTK.cxx needs update to add new menus. (2) "Open LaTeXInput.properties" should be hided from Options menu.

[Zufu Liu]

(3) An property for visible item count on auto-completion list other than the 16 for wEditor.AutoCSetMaxHeight.

[Neil Hodgson]

The file can be used with the current abbreviations feature so why is there a need for duplicate functionality? The file can be simply placed on SciTE files (https://github.com/moltenform/scite-files) for users to download which also removes it from being a maintenance burden for SciTE.

It may be worth adding an autocompletion when Insert Abbreviation doesn't find a match.

^= ⁼ U+0207C Superscript Equals Sign
_= ₌ U+0208C Subscript Equals Sign

Some form of escaping could be added but it should have minimal impact. Perhaps allowing \u followed by exactly 4 hexadecimal digits would be sufficiently unique to not match other keys (there is a \u in the list). This could be applied to keys just in the abbreviations but may be safe for keys in all properties. It can't be applied to values (or whole lines) in properties files as they often parse escapes themselves in a particular manner.

[Zufu Liu]

moved the content into abbrev.properties, the GTK changes is not tested.

[Neil Hodgson]

[e5dc12] allows Unicode escapes \x, \u, and \U to be used for keys in property files, so enabling abbreviations like ^=. The escapes must be followed by 2, 4, or 8 hex characters (matching the x, u, or U) to be converted. Any invalid sequence is treated verbatim. The code point is converted to UTF-8.

Thus, these properties:

v\x3Dv=1
v\u003Dx=2
v\U0000003Dy=3
# Out-of-hex-range so not unescaped
v\x3Gz=4

Will cause this behaviour:

echo $(v=v) $(v=x) $(v=y) $(v\x3Gz)
>echo 1 2 3 4
1 2 3 4

[Zufu Liu]

Updated script to generate hex escape \x3D for =:

# U+0207C Superscript Equals Sign
\^\x3D=⁼
# U+0208C Subscript Equals Sign
\_\x3D=₌

[Neil Hodgson]

Escape feature included in 5.1.2 release.

[Neil Hodgson]

To minimize differences, parts of the patch have been committed.

[edb2cd] allows specifying number of items in autocompletion lists but changes the property to "autocomplete.visible.item.count" for consistency and set the default to 9 as that is the current behaviour.

[8701f0] adds a (reduced) AutoCompleteWordList class and uses it for Complete Word. Marked AutoCompleteWordList::Get as const and initialized minWordLength to SIZE_MAX to avoid casting.

Still not understanding why the patch diverges from current behaviour like binding to Tab instead of Ctrl+B. If users don't like key bindings they can change them.

[honfui]

Thanks both for the great effort. Tested the abbrev.properties approach (downloaded). The superscript doesn't work or not available, e.g. \^2 or \^y.

[honfui]

Tested again, superscript does work. My apology.

[Neil Hodgson]

A problem with the large abbreviations file on GTK is that the pop-up combo-box in the dialog contains so many values that it produces a window taller than 32K pixels which fails due to maximum window size. It also takes a long time so can appear to hang.

It would be possible to use a scrolling list instead of a pop-up. Its a little surprising that GTK doesn't automatically use a scrolling window as it does for menus that are too tall.