#1881 [macOS 10.12] malloc failed while using with IME

[Tse Kit Yam]

Description:
This problem only happens on macOS >= 10.12.
When I try to input some Chinese characters to my doc, I found that malloc failed error is printed on the log

ScintillaTest(90859,0x1005c83c0) malloc: *** mach_vm_map(size=18446744073709535232) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

Steps to reproduce:

1. Enable Cangjie input source.
   System Preferences -> Keyboard -> + -> Chinese, Traditional -> Cangjie -> Add
2. Compile ScintilaTest app
3. Switch to Cangjie input source
4. Select the end of the text, press H, which repersents 竹 in Cangjie

Actual result:
竹 with red underline appends to the end of the text. malloc failed error message printed on the Xcode output

Expected result:
竹 with red underline appends to the end of the text, and no error will be seen in Xcode output.

[Neil Hodgson]

If you run with Address Sanitizer, a more detailed stack trace appears (attached) much of which is from IMKInputSession. Near the failure are symbols like attributedSubstringFromRange and if you place breakpoints in SCIContentView.attributedSubstringForProposedRange and other NSTextInputClient code you'll see many hits during the interpretKeyEvents.

The malloc failure was for 0xfffffffffffffff0 bytes so appears to be a wrap-around with a negative size. The ranges handed to NSTextInputClient methods are sometimes unreasonable and near NSNotFound. Logging calls and checking the behaviour of unexpected ranges may be worthwhile. Another thing to check for is that the correct width of '竹' (3 bytes in UTF-8) is being used.

I can't reproduce the problem with Japanese.

[Neil Hodgson]

This has been reported in other applications including Chrome.
https://bugs.chromium.org/p/chromium/issues/detail?id=654695

The Chrome bug fix interprets the documentation for selectedRange differently as requiring return of { NSNotFound, 0 } when the selection is empty. Also interprets the behaviour of attributedSubstringForProposedRange differently by quickly returning with nil for ranges outside the document.

These changes are in [8054db] which may fix the bug. However, this has altered the calls made to SCIContentView from the IME significantly and there may be additional issues.

[Neil Hodgson]

The change to selectedRange has broken the accented character picker used for European languages. That is when a character is held down to reveal a menu of accented variants of that character.

[Neil Hodgson]

Committed fix as [29534c] to restore the accented character picker.

This change special-cases the "Cangjie" input source to return NSNotFound for empty selections. If other Chinese input sources also see crashes then they should be reported so the special casing can be extended.