#503 frequent KeyDot failed, UnicodeError in interactive

[kxroberto]

build214

Firing event 'KeyDot' failed.
Traceback (most recent call last):
File "C:\Python23\Lib\site-packages\pythonwin\pywin\scintilla\bindings.py", line 145, in fire
rc = binding.handler(*args)
File "C:\Python23\Lib\site-packages\pythonwin\pywin\scintilla\view.py", line 307, in KeyDotEvent
self._AutoComplete()
File "C:\Python23\Lib\site-packages\pythonwin\pywin\scintilla\view.py", line 467, in _AutoComplete
text = self.GetTextRange(self.LineIndex(minline),endpos)
File "C:\Python23\Lib\site-packages\pythonwin\pywin\scintilla\control.py", line 362, in GetTextRange
ret = ret.decode(default_scintilla_encoding)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 5946: unexpected code byte

[Roger Upole]

The underlying problem is how Scintilla handles (or fails to handle)
multibyte characters when in utf-8 mode. None of the text handling
messages (SCI_GETTEXTRANGE in this case) take continuation characters
into account. If you ask for say 10 characters, what you get back is
10 bytes, which may contain partial characters at the start or the end
of the buffer.
The only way I can see to deal with discrete characters are the
SCI_POSITIONBEFORE and SCI_POSITIONAFTER messages, which means all operations would have to be done one character at a time.

[kxroberto]

the start/end for this GetTextRange come unchecked from LineIndex() 's.
The lineno's unchecked from _GetClassInfoFromBrowser
Maybe just a line is -1 or so.
If always complete valid lines are decoded completely, may there should be no such problem?

[Mark Hammond]

Yeah - we are sending the GETTESTRANGE message, but the start and end pos have both come directly from LineIndex (EM_LINEINDEX). This is basically the same strategy taken by GetLine():

start = self.LineIndex(line)
end = self.LineIndex(line+1)
return self.GetTextRange(start, end)

the docs for EM_GETTEXTRANGE say end if "Character position immediately following the last character in the range.", so this looks correct.

The above all assumes scintilla doesn't have a bug in some corner case:)

It also seems strange the exception refers to 0xff - best I can tell, 0xff is not a leading utf8 byte.

So I guess I need instructions for a repro so I can see what is going on via the debugger!

[Michael Taves]

I frequently (daily) have a similar error, typically while editing code or comments in Python scripts where I happen to type letters followed by the "." character. This edited error was 2000 lines long:

Firing event 'KeyDot' failed.
Traceback (most recent call last):
File "C:\Python25\Lib\site-packages\pythonwin\pywin\scintilla\bindings.py", line 142, in fire
rc = binding.handler(*args)
File "C:\Python25\Lib\site-packages\pythonwin\pywin\scintilla\view.py", line 307, in KeyDotEvent
self._AutoComplete()
File "C:\Python25\Lib\site-packages\pythonwin\pywin\scintilla\view.py", line 465, in _AutoComplete
minline, maxline, curclass = self._GetClassInfoFromBrowser()
File "C:\Python25\Lib\site-packages\pythonwin\pywin\scintilla\view.py", line 520, in _GetClassInfoFromBrowser
if not browser.list: return (minline,maxline,None) # Not initialized
File "C:\Python25\Lib\site-packages\pythonwin\pywin\tools\hierlist.py", line 62, in __getattr__
return getattr(self, attr)
File "C:\Python25\Lib\site-packages\pythonwin\pywin\tools\hierlist.py", line 62, in __getattr__
return getattr(self, attr)
80 lines deleted
File "C:\Python25\Lib\site-packages\pythonwin\pywin\tools\hierlist.py", line 62, in __getattr__
return getattr(self, attr)
File "C:\Python25\Lib\site-packages\pythonwin\pywin\tools\hierlist.py", line 60, in __getattr__
return getattr(self.listControl, attr)
RuntimeError: maximum recursion depth exceeded

I'm using pywin32 build 214, WinXP, Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)]

[kxroberto]

I have still this same "maximum recursion" kind of error in build 216.

The solution probably:
browser.list => browser.listControl
And the recursive "return getattr(self, attr)" in HierList.__getattr__ probably makes no sense. (if attr in .__dict__ then .__getattr__ is not called. )

---

--- _orig/view.py 2011-02-25 11:04:38 +0000
+++ ./view.py 2011-11-22 14:46:18 +0000
@@ -510,7 +510,7 @@
pass
if browser is None:
return (minline,maxline,None) # Current window has no browser
- if not browser.list: return (minline,maxline,None) # Not initialized
+ if not browser.listControl: return (minline,maxline,None) # Not initialized
path = self.GetDocument().GetPathName()
if not path: return (minline,maxline,None) # No current path

--- _orig/hierlist.py 2009-02-16 17:43:26 +0000
+++ ./hierlist.py 2011-11-22 14:43:35 +0000
@@ -56,10 +56,10 @@
self.filledItemHandlesMap = {}
self.bitmapMask = bitmapMask
def __getattr__(self, attr):
- try:
+ ##try:
return getattr(self.listControl, attr)
- except AttributeError:
- return getattr(self, attr)
+ ##except AttributeError:
+ ## return getattr(self, attr)

def ItemFromHandle(self, handle):
return self.itemHandleMap[handle]

[Mika]

I get always this error only when class browser in left panel is visible.

My workaround is just to outcomment the line from where error comes e.g.
#if not browser.list: return (minline,maxline,None)
Seems to work without error after that. I'm just wondering what functionality I'm missing when out-commenting the line.

AFAIK : The error comes because __getattr__ of browser is overloaded and when there is "browser." in code it starts infinite loop.