#2038 Inline IME mode

[Zufu Liu]

Is there any special treatment for

sptr_t ScintillaWin::HandleCompositionInline(uptr_t, sptr_t lParam) {
    view.imeCaretBlockOverride = false;
    if (lParam & GCS_COMPSTR) {
        if (KoreanIME()) {
            view.imeCaretBlockOverride = true;
        }
    } else if (lParam & GCS_RESULTSTR) {
        AddWString(imc.GetCompositionString(GCS_RESULTSTR));
    }
}

not using the pair KoreanIME() || imeInteraction == imeInline like other usages.

By the way, it's seems better to switch the pair to imeInteraction == imeInline || KoreanIME() when imeInteraction is already set to inline.

[Zufu Liu]

The changes.

By the way, these stuff (in PlatWin.cxx) is redundant and confusing with most other "Deleted" stuff at first glance.

    // Defaulted so BlobInline objects can be copied.
    BlobInline(const BlobInline &) = default;
    BlobInline(BlobInline &&) = default;
    BlobInline &operator=(const BlobInline &) = default;
    BlobInline &operator=(BlobInline &&) = default;

[Zufu Liu]

The imeInteraction for KoreanIME can even been cached on WM_CREATE and WM_INPUTLANGCHANGE.

[Zufu Liu]

There is a issue about inline IME mode on Japanese IME at
https://github.com/rizonesoft/Notepad3/issues/632#issuecomment-416623554

[Zufu Liu]

For Chinese Pinyin IME, inline mode will trigger SCN_CHARADDED, which will popup auto-completion for some editor.

[Zufu Liu]

Find a document on Chinese, Japanese, Korean IME at
https://docs.microsoft.com/en-us/windows/desktop/DxTechArts/installing-and-using-input-method-editors

Seems Korean IME always use a block caret, but not Chinese and Japanese.

So the origin issue ("imeCaretBlockOverride") may invalid.

[Neil Hodgson]

The Korean language handling was by johnsonj. If they don't reply to this ticket, you should try writing to the mailing list. I don't use IMEs normally and most of my IME experience is with Japanese.
https://groups.google.com/forum/#!forum/scintilla-interest

Are you just complaining about the inconsistency or do you want a different caret appearance for another language? I'd have preferred avoiding the block appearance for any language but its apparently normal for Korean. There is an enormous amount of mailing list discussion about IME issues. A starting point is
https://groups.google.com/d/topic/scintilla-interest/J16BtP6UrA0/discussion

A statement like

better to switch the pair to imeInteraction == imeInline || KoreanIME()

should indicate why its better. If its just to avoid calling a platform API then the benefits need to be clear. If its just performance then its unimportant in the context of receiving an IME-specific message. Caching values can lead to stale information.

[Zufu Liu]

OK, I don't care about the block caret.

From Windows document and other report, in many appications (e.g. Notepad, Wordpad, Office, Chrome, etc.) inline IME mode is always set on East Asian system.
So imeInteraction == imeInline is true on some settings, when it's not, there is no difference between imeInteraction == imeInline || KoreanIME() and KoreanIME() || imeInteraction == imeInline.

The Chinese Pinyin IME inline mode seems a bug.

See above screenshot ("pinyin-ime-inline.PNG"), the single quote char (') cause the incomplete input (before composition) been rendered as string (Notepad2 enabled auto close quotes, so there is a extra quote after w; in SciTE it's unterminated character literal).

Single quote char is a separator in Pinyin IME, the most famous example is "Xi'an" (西安) https://en.wikipedia.org/wiki/Xi%27an

In above example, it was auto added by the IME.

[Neil Hodgson]

The example is of a C source code file. In such a file "'" is the start of a character literal. If you don't want the IME text to actually be interpreted as part of the document then don't use an inline IME.

[Zufu Liu]

I can suppress auto completion on SCN_CHARADDED by check

BOOL IsInlineIMEActive(void) {
    BOOL result = FALSE;
    if (bUseInlineIME) {
        HIMC himc = ImmGetContext(hwndEdit);
        result = ImmGetOpenStatus(himc);
        ImmReleaseContext(hwndEdit, himc);
    }
    return result;
}

Don't known whether it better to put it into Scintilla to not fire SCN_CHARADDED or not.

SCN_CHARADDED should fired, otherwise auto indentation on \r\n will not work.

[Zufu Liu]

Above IsInlineIMEActive check will break auto-completion when inline IME is in English mode (which is common in Chinese, Japanese and Korean IME).

It can be fixed with (maybe need to enable other "conversion mode", see https://docs.microsoft.com/en-us/windows/desktop/Intl/ime-conversion-mode-values):

BOOL IsInlineIMEActive(void) {
    BOOL result = FALSE;
    if (bUseInlineIME) {
        HIMC himc = ImmGetContext(hwndEdit);
        if (himc) {
            if (ImmGetOpenStatus(himc)) {
                DWORD dwConversion = IME_CMODE_ALPHANUMERIC, dwSentence = 0;
                if (ImmGetConversionStatus(himc, &dwConversion, &dwSentence)) {
                    result = !(dwConversion == IME_CMODE_ALPHANUMERIC);
                }
            }
            ImmReleaseContext(hwndEdit, himc);
        }
    }
    return result;
}

Hi Neil, I think this at least is a document issue. I suggest add a sentence for SC_IME_INLINE in the document:

Inline IME mode will trigger `SCN_CHARADDED` notification before input text been composited, which may cause unexpected behavior if application handle `SCN_CHARADDED` (for example for auto-completion);
If IME input text (before composition) contains single quote (which is common in Chinese Pinyin IME), which may break existing style (for example terminates single quoted string at the middle).

The Pinyin IME input text been rendered as string is not important, because Chinese text is usually only used in string and comment, the input text will be rendered using same style as it's surrounded style, so will not observed normally unless single quote is used to quote string like in Python, JavaScript, etc.

[Neil Hodgson]

Added mention of SCN_CHARADDED in [892c36].

[Zufu Liu]

The hard-coded KoreanIME() is buggy, especially the block caret.

if (KoreanIME()) {
      view.imeCaretBlockOverride = true;
}

When Korean IME is in English mode, the input code page is still 949 (UHC) or 1361 (Johab).

I doubt Korean people like block caret in English mode input.

Instead of hard-coded KoreanIME() and check it every time on typing, I don't know why they not set imeInteraction to inline, and introduced a new settings like Use Block Caret For Inline IME.

[Neil Hodgson]

This was implemented by a Korean person and it does what they want.

[Zufu Liu]

I suggest add an option for Use Block Caret For Inline IME with three values:

• Not use block caret

• Always use block caret

• Only use block caret in non-English (native) mode

[Neil Hodgson]

Why? Do you want the block caret in a language you use?

[Zufu Liu]

No, I don't want block caret, just make it configurable instead of hard-coded. For people not use Korean IME, it's zero cost.

For my owner experience, change IME between native mode and English mode, the input code page is not changed, but the conversion mode is changed.

In English mode, HandleCompositionInline() not been called (tested with several Pinyin IMEs). Original code about block caret seems right.

Korean IME has English, Roman, Hangul and Hanja mode. Need someone use Korean IME to tell us they use block caret in all the three (Roman, Hangul and Hanja) modes?
Or HandleCompositionInline() will not been called in some modes?

[Neil Hodgson]

No, I don't want block caret, just make it configurable instead of hard-coded. For people not use Korean IME, it's zero cost.

Your suggested change is incompatible as some value of the setting has to be made the default and each choice is different from current behaviour.

There has to be some benefit to any change so someone has to want the block caret for their language or there has to be a Korean user that does not want the block caret.

[johnsonj]

There is a historical cause for KoreanIME().

The inline ime was initially developed only for Korean.
KoreanIME() and imeCaretBlockOverride for indicating composition mode werd introduced. It surprisingly had growed to international(especially CJK) ime with TentativeUndo() by Neil.

But it was made disabled by default since we were not sure it would work good.
And inline ime neeed to be tested.
So I wanted Inline ime should have been tested through KoreanIME().
since Korean dislike windowed ime.
And thankfully, Neil accepted my hard corded purpose.
Now Scintilla's inline ime is around 4 years old.

That means KoreanIME() proves my purpose.
No problems has been reported yet from Korean.

Congratuations Notepad3 has inline ime option.

[Zufu Liu]

Is block caret used in all mode (Roman, Hangul and Hanja) in Korean IME? Though I can install a Korean IME, but don't known how to used it.

[johnsonj]

Just press Hanja key in composition mode.
You will see Choice combo box just like Hanze or Kanjii.

For your interest:
https://www.youtube.com/channel/UC8QidmSCiJqHDKplRK8NHGw/videos?view=0&sort=dd&shelf_id=0

[Neil Hodgson]

Not seeing anything to do here so closing.

[Zufu Liu]

The IsInlineIMEActive() check is buggy: after full composition (GCS_RESULTSTR), it's still return true.

Suggest add a status variable surround the AddCharUTF(); loop inside HandleCompositionInline(), (there already has one: recordingMacro).
So (SCN_CHARADDED) in Editor::AddCharUTF() not fired before full composition.

My previous comment is wrong, type punctuation direct result in GCS_RESULTSTR (no composition).
So the unexpected SCN_CHARADDED notification can be suppressed and has not side effect (except for the extra check) compared to window mode IME.

[Neil Hodgson]

The patch never changes inlineIMEComposition.

It appears the intention is to disable SCN_CHARADDED during IME input but that woud be a regression. Applications should be able to perform actions such as displaying autompletion lists during IME input. If applications want to handle IME input differently then they can detect whether the IME is open.

[Zufu Liu]

Patch updated, two lines added in HandleCompositionInline().

So let current code not changed or can a new option be added to enable suppress SCN_CHARADDED during IME input if application not want to handle IME input differently?