pywin32-bugs — Automated tracker notifications (not for bug reports!)

[pywin32-bugs] [ pywin32-Bugs-1976957 ] Column guides cause errors(can)

[SourceForge.net <no...@so...>]

Bugs item #1976957, was opened at 2008-05-28 10:20
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1976957&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: pythonwin
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Hugh S. Myers (hsmyers)
Assigned to: Nobody/Anonymous (nobody)
Summary: Column guides cause errors(can)

Initial Comment:
Use of column guides can cause bogus indentation errors. Pattern yet to
be determined.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1976957&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1976955 ] Runaway printer

[SourceForge.net <no...@so...>]

Bugs item #1976955, was opened at 2008-05-28 10:19
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1976955&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: pythonwin
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Hugh S. Myers (hsmyers)
Assigned to: Nobody/Anonymous (nobody)
Summary: Runaway printer

Initial Comment:
On a 43 line file, printer printed 29 pages, mostly blank(occasional
page and title info across top--- somewhat random). Good news is printer
performance is consistent with print preview!


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1976955&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1976953 ] Whitesace toggle problem

[SourceForge.net <no...@so...>]

Bugs item #1976953, was opened at 2008-05-28 10:18
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1976953&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: pythonwin
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Hugh S. Myers (hsmyers)
Assigned to: Nobody/Anonymous (nobody)
Summary: Whitesace toggle problem

Initial Comment:
  View Whitespace started up enabled when file loaded. Had I left it on when I shut down the editor I would understand(somewhat) the persistence. However I turned it off before I left. This now happens every time I load the file. Shouldn't.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1976953&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1976952 ] printer Error message?

[SourceForge.net <no...@so...>]

Bugs item #1976952, was opened at 2008-05-28 10:16
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1976952&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: pythonwin
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Hugh S. Myers (hsmyers)
Assigned to: Nobody/Anonymous (nobody)
Summary: printer Error message?

Initial Comment:
Just got this:

C:\Python25\Lib\site-packages\pythonwin\pywin\scintilla\view.py:641: DeprecationWarning: 'L' format requires 0 <= number <= 4294967295
  fr = struct.pack('LLIIIIIIIIll', hdcRender, hdcFormat, rc[0], rc[1], rc[2], rc[3], rc[0], rc[1], rc[2], rc[3], pageStart, lengthDoc)

Can't be good...

--hsm


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1976952&group_id=78018

[pywin32-bugs] [ pywin32-Patches-1962146 ] win32file.TransmitFile and ConnectEx patch

[SourceForge.net <no...@so...>]

Patches item #1962146, was opened at 2008-05-12 06:06
Message generated for change (Comment added) made by ionel_mc
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551956&aid=1962146&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: ionel (ionel_mc)
Assigned to: Nobody/Anonymous (nobody)
Summary: win32file.TransmitFile and ConnectEx patch

Initial Comment:
sample usage:

import win32file, socket, pywintypes
s = socket.socket()
f = file('somefile','rb')
s.connect(('192.168.111.128',10002))
ol = pywintypes.OVERLAPPED()
print win32file.TransmitFile(s, win32file._get_osfhandle(f.fileno()), 0, 0, ol, 0, "prepended data", "appended data")
length = win32file.GetOverlappedResult(s.fileno(), ol, 1)
print length

I hope I haven't done the wrong thing but for the sake of simplicity I'm providing a patch the includes the previous ConnectEx patch (hopefully it'll get accepted :)

----------------------------------------------------------------------

>Comment By: ionel (ionel_mc)
Date: 2008-05-28 02:56

Message:
Logged In: YES 
user_id=1189761
Originator: YES

File Added: win32file.i-8.patch

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-28 02:27

Message:
Logged In: YES 
user_id=771074
Originator: NO

Regarding the way getaddrinfo is called, it appears that the
family, socket type, and protocol are hardcoded.  Shouldn't it be
using the values from the Python socket object, since these are
specified when it's created ?

Also, Unicode is accepted for the host, but not the port.  This
isn't a show stopper, but it would be better to be consistent.


----------------------------------------------------------------------

Comment By: ionel (ionel_mc)
Date: 2008-05-13 21:27

Message:
Logged In: YES 
user_id=1189761
Originator: YES

File Added: win32file.i-7.patch

----------------------------------------------------------------------

Comment By: ionel (ionel_mc)
Date: 2008-05-13 21:16

Message:
Logged In: YES 
user_id=1189761
Originator: YES

File Added: win32file.i-6.patch

----------------------------------------------------------------------

Comment By: ionel (ionel_mc)
Date: 2008-05-13 18:51

Message:
Logged In: YES 
user_id=1189761
Originator: YES

sure, close it.
I'll post a message on the list in a few moments.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-13 16:22

Message:
Logged In: YES 
user_id=14198
Originator: NO

Sorry for the confusion (and for missing the test!)  To be clear, the
"problem" is simply my lack of cycles to try and digest what exactly the
API allows for versus what your patch does.  I'm not suggesting anything
you are doing is wrong, just that its not immediately obvious to me (hence
my comments re TransmitFile, which a patch for *is* likely to be "obvious"
- but as you mentioned you combined them, I didn't look closely).  You are
correct though that the arg handling is my primary concern, and if you are
really keen to get this in ASAP, would you be so kind as to mail the
python-win32 list asking for other comments and pointing at this bug? 
There are a few people there who's opinion I would defer to - and they may
have more time at the moment to have a look (I'm really hoping to get build
211 out any day now!)

Also, if you are intent on getting these in the same build, should I close
the other patch?

Thanks!

----------------------------------------------------------------------

Comment By: ionel (ionel_mc)
Date: 2008-05-13 13:09

Message:
Logged In: YES 
user_id=1189761
Originator: YES

I'd rather have them both in the same build. Also, on the arg handling
thing I imagine you didn't like the addrinfo stuff that I did there. It
might look like a mess but i have looked in the socketmodule.c from the
python dist and there's nothing i can use from there (there is a
getsockaddrarg that is used in the connects, it does about the same thing
that i do but on a larger _messy_ scale with more cases like unix sockets
that don't apply here).

Um, I did attach a test_transmitfile.py - should that code really be in
test_win32file.py ?


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-13 09:29

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks.  Unfortunately, I'm going to need to research the ConnextEx stuff
a little more to convince myself the arg handling is reasonable - but it
will be accepted one way or another :)  However, as TransmitFile is likely
to be a much simpler patch, if you supplied a patch for that alone
(including tests ideally - see test_win32file.py for stuff you can maybe
borrow?) I could get it in much sooner (and it would therefore be possible
to make build 211).  Otherwise I'll wrap both this and your ConnectEx
patches up when I find time to tweak.

Thanks again.

----------------------------------------------------------------------

Comment By: ionel (ionel_mc)
Date: 2008-05-12 06:06

Message:
Logged In: YES 
user_id=1189761
Originator: YES

File Added: test_transmitfile.py

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551956&aid=1962146&group_id=78018

[pywin32-bugs] [ pywin32-Patches-1962146 ] win32file.TransmitFile and ConnectEx patch

[SourceForge.net <no...@so...>]

Patches item #1962146, was opened at 2008-05-11 22:06
Message generated for change (Comment added) made by rupole
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551956&aid=1962146&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: ionel (ionel_mc)
Assigned to: Nobody/Anonymous (nobody)
Summary: win32file.TransmitFile and ConnectEx patch

Initial Comment:
sample usage:

import win32file, socket, pywintypes
s = socket.socket()
f = file('somefile','rb')
s.connect(('192.168.111.128',10002))
ol = pywintypes.OVERLAPPED()
print win32file.TransmitFile(s, win32file._get_osfhandle(f.fileno()), 0, 0, ol, 0, "prepended data", "appended data")
length = win32file.GetOverlappedResult(s.fileno(), ol, 1)
print length

I hope I haven't done the wrong thing but for the sake of simplicity I'm providing a patch the includes the previous ConnectEx patch (hopefully it'll get accepted :)

----------------------------------------------------------------------

>Comment By: Roger Upole (rupole)
Date: 2008-05-27 18:27

Message:
Logged In: YES 
user_id=771074
Originator: NO

Regarding the way getaddrinfo is called, it appears that the
family, socket type, and protocol are hardcoded.  Shouldn't it be
using the values from the Python socket object, since these are
specified when it's created ?

Also, Unicode is accepted for the host, but not the port.  This
isn't a show stopper, but it would be better to be consistent.


----------------------------------------------------------------------

Comment By: ionel (ionel_mc)
Date: 2008-05-13 13:27

Message:
Logged In: YES 
user_id=1189761
Originator: YES

File Added: win32file.i-7.patch

----------------------------------------------------------------------

Comment By: ionel (ionel_mc)
Date: 2008-05-13 13:16

Message:
Logged In: YES 
user_id=1189761
Originator: YES

File Added: win32file.i-6.patch

----------------------------------------------------------------------

Comment By: ionel (ionel_mc)
Date: 2008-05-13 10:51

Message:
Logged In: YES 
user_id=1189761
Originator: YES

sure, close it.
I'll post a message on the list in a few moments.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-13 08:22

Message:
Logged In: YES 
user_id=14198
Originator: NO

Sorry for the confusion (and for missing the test!)  To be clear, the
"problem" is simply my lack of cycles to try and digest what exactly the
API allows for versus what your patch does.  I'm not suggesting anything
you are doing is wrong, just that its not immediately obvious to me (hence
my comments re TransmitFile, which a patch for *is* likely to be "obvious"
- but as you mentioned you combined them, I didn't look closely).  You are
correct though that the arg handling is my primary concern, and if you are
really keen to get this in ASAP, would you be so kind as to mail the
python-win32 list asking for other comments and pointing at this bug? 
There are a few people there who's opinion I would defer to - and they may
have more time at the moment to have a look (I'm really hoping to get build
211 out any day now!)

Also, if you are intent on getting these in the same build, should I close
the other patch?

Thanks!

----------------------------------------------------------------------

Comment By: ionel (ionel_mc)
Date: 2008-05-13 05:09

Message:
Logged In: YES 
user_id=1189761
Originator: YES

I'd rather have them both in the same build. Also, on the arg handling
thing I imagine you didn't like the addrinfo stuff that I did there. It
might look like a mess but i have looked in the socketmodule.c from the
python dist and there's nothing i can use from there (there is a
getsockaddrarg that is used in the connects, it does about the same thing
that i do but on a larger _messy_ scale with more cases like unix sockets
that don't apply here).

Um, I did attach a test_transmitfile.py - should that code really be in
test_win32file.py ?


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-13 01:29

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks.  Unfortunately, I'm going to need to research the ConnextEx stuff
a little more to convince myself the arg handling is reasonable - but it
will be accepted one way or another :)  However, as TransmitFile is likely
to be a much simpler patch, if you supplied a patch for that alone
(including tests ideally - see test_win32file.py for stuff you can maybe
borrow?) I could get it in much sooner (and it would therefore be possible
to make build 211).  Otherwise I'll wrap both this and your ConnectEx
patches up when I find time to tweak.

Thanks again.

----------------------------------------------------------------------

Comment By: ionel (ionel_mc)
Date: 2008-05-11 22:06

Message:
Logged In: YES 
user_id=1189761
Originator: YES

File Added: test_transmitfile.py

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551956&aid=1962146&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1960311 ] 'None' returned for COM string constants

[SourceForge.net <no...@so...>]

Bugs item #1960311, was opened at 2008-05-09 01:01
Message generated for change (Comment added) made by mhammond
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1960311&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: com
Group: None
>Status: Closed
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Mark Hammond (mhammond)
Assigned to: Mark Hammond (mhammond)
Summary: 'None' returned for COM string constants

Initial Comment:
String constants in an IDL file have none returned.  Eg, an IDL file has:

  module Constants {
    const char CharTest = -1;
    const LPWSTR StringTest = L"Hello Loraine";
  };

and the generated .py file will have:

  class constants:
    CharTest   =-1         # from enum Constants
    StringTest ='None'     # from enum Constants

Attaching a patch that demonstrates the problem, but I don't see the problem yet.


----------------------------------------------------------------------

>Comment By: Mark Hammond (mhammond)
Date: 2008-05-24 12:56

Message:
Logged In: YES 
user_id=14198
Originator: YES

Turns out this was a regression caused by #1651025.

Checking in TestSources/PyCOMTest/PyCOMTest.idl; new revision: 1.18;
previous revision: 1.17
Checking in win32com/client/genpy.py; new revision: 1.54; previous
revision: 1.53
Checking in win32com/src/extensions/PyVARDESC.cpp; new revision: 1.5;
previous revision: 1.4
Checking in win32com/test/testPyComTest.py; new revision: 1.31; previous
revision: 1.30


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1960311&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1931877 ] Vista 64-bit install

[SourceForge.net <no...@so...>]

Bugs item #1931877, was opened at 2008-04-01 21:34
Message generated for change (Comment added) made by sf-robot
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1931877&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: installation
Group: None
>Status: Closed
Resolution: None
Priority: 5
Private: No
Submitted By: Adam Getchell (acgetchell)
Assigned to: Nobody/Anonymous (nobody)
Summary: Vista 64-bit install

Initial Comment:
Fails with:

Can't load Python for pre-install script

----------------------------------------------------------------------

>Comment By: SourceForge Robot (sf-robot)
Date: 2008-05-18 19:20

Message:
Logged In: YES 
user_id=1312539
Originator: NO

This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 14 days (the time period specified by
the administrator of this Tracker).

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 04:09

Message:
Logged In: YES 
user_id=14198
Originator: NO

Build 211 is due any day/week now and I'm confident this is fixed there -
please reopen if that is not the case.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1931877&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1941211 ] VB structure error - "Old format or invalid type library"

[SourceForge.net <no...@so...>]

Bugs item #1941211, was opened at 2008-04-12 23:03
Message generated for change (Comment added) made by sf-robot
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1941211&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
>Status: Closed
Resolution: None
Priority: 5
Private: No
Submitted By: Alex Omoto (moto23)
Assigned to: Nobody/Anonymous (nobody)
Summary: VB structure error - "Old format or invalid type library"

Initial Comment:
Hi,

I have the following VB and python code to test VB structures:

VB - ClassLibrary1:

Public Structure test
    Public var1 As Integer
    Public var2 As Double
    Public var3 As String
End Structure

Public Class Class1

End Class

Python:
import win32com.client
a = win32com.client.Dispatch('ClassLibrary1.Class1')
b = win32com.client.Record('test', a)


When var3 is a string, I get the error shown below.  I have no problems when var3 is int or double. I'm using ActiveState Python 2.4.5.14.


pywintypes.com_error: (-2147319783, 'Old format or invalid type library.', None, None)


Thanks,
Alex



----------------------------------------------------------------------

>Comment By: SourceForge Robot (sf-robot)
Date: 2008-05-18 19:20

Message:
Logged In: YES 
user_id=1312539
Originator: NO

This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 14 days (the time period specified by
the administrator of this Tracker).

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 04:08

Message:
Logged In: YES 
user_id=14198
Originator: NO

I'm sorry, but there isn't enough info here for me to repro the problem. 
Note that there are VB6 tests, so you probably need to modify them to
demonstrate the problem before I will be able to take any action.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1941211&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1448160 ] Win2003 dual-core crash on startup

[SourceForge.net <no...@so...>]

Bugs item #1448160, was opened at 2006-03-11 22:09
Message generated for change (Comment added) made by sf-robot
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1448160&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: pythonwin
Group: None
>Status: Closed
Resolution: None
Priority: 5
Private: No
Submitted By: Darren (zaphod72)
Assigned to: Nobody/Anonymous (nobody)
Summary: Win2003 dual-core crash on startup

Initial Comment:
I have a dual-core Windows Server 2003 and a single-
core server also running Windows Server 2003.

On the dual-core pythonwin crashes on startup, shortly 
after displaying the UI.

On the single-core it works fine.

Here's the crash info from WinDbg -
(ee8.8b8): Access violation - code c0000005 (!!! 
second chance !!!)
eax=00000000 ebx=773e20e0 ecx=000080a2 edx=001668d8 
esi=0013fcf8 edi=7ffdf6cc
eip=00a0180c esp=0013fcd4 ebp=0013fd04 iopl=0         
nv up ei pl zr na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  
gs=0000             efl=00010246
00a0180c 58               pop     eax

And the stack trace -
ChildEBP RetAddr  Args to Child              
WARNING: Frame IP not in any known module. Following 
frames may be wrong.
0013fcd0 77387050 000080a2 00000000 00000000 0xa0180c
0013fd04 77387dc3 00a0180c 000080a2 00000000 USER32!
UserCallDDECallback+0x23
0013fd3c 77387101 0015ca78 000080a2 00000000 USER32!
DoCallback+0x6a
0013fd74 7738e3f0 0015ca78 000004c8 0000c02c USER32!
ProcessRegistrationMessage+0xa4
0013fd8c 7739c3b7 001500ce 000004c8 0000c02c USER32!
DDEMLMotherWndProc+0x2c
0013fdb8 7739c484 7738e3ad 001500ce 000004c8 USER32!
InternalCallWinProc+0x28
0013fe30 7739c73c 00000000 7738e3ad 001500ce USER32!
UserCallWinProcCheckWow+0x151
0013fe98 7738e406 00155998 00000001 00000000 USER32!
DispatchMessageWorker+0x327
0013fea8 7c169076 00155998 00155998 00403040 USER32!
DispatchMessageA+0xf
0013feb8 7c16913e 00403040 1e055870 0013ffc0 MFC71!
AfxInternalPumpMessage+0x3e
0013fed4 1e2b6767 00403040 ffffffff 00000000 MFC71!
CWinThread::Run+0x54
0013ffc0 77e523cd 00000000 00000000 7ffdc000 win32ui!
Python_callback+0x1247
0013fff0 00000000 0040186c 00000000 78746341 kernel32!
BaseProcessStart+0x23


I didn't try to build the source. But if I can get the 
pdb files then I'd be happy to try and get some more 
information (darrenk at yahoo).
Thanks

----------------------------------------------------------------------

>Comment By: SourceForge Robot (sf-robot)
Date: 2008-05-18 19:20

Message:
Logged In: YES 
user_id=1312539
Originator: NO

This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 14 days (the time period specified by
the administrator of this Tracker).

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 04:11

Message:
Logged In: YES 
user_id=14198
Originator: NO

I can't repro this on a dual-core vista 64 system, and its getting old now
- please reopen if there are still problems (build 211 is due very soon
now)

----------------------------------------------------------------------

Comment By: kxroberto (kxroberto)
Date: 2006-11-18 03:49

Message:
Logged In: YES 
user_id=972995
Originator: NO

Think the probability is very high that this will be solved with patch of
bug #1590399 also. there were GIL-Problems and INC/DEC's of refcounts
outside of the GIL which are non-atomic on dual cores.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2006-09-22 05:49

Message:
Logged In: YES 
user_id=14198

I'm guessing this will work with -nodde specified on the
command-line.  I'd love to remove that - sadly the DDE code
is fairly intractable :(  I wouldn't know where to ask you
to start with the .pdb files

----------------------------------------------------------------------

Comment By: kxroberto (kxroberto)
Date: 2006-03-14 08:57

Message:
Logged In: YES 
user_id=972995

Good to get more about this dual core bug (following
#1442426 & #1441884 )

Can you maybe try this bug also with py2.3.5 / build 205 ? 
(build 207 doesn't work for py2.3). You can simply run a
second py2.3 installation parallel on your computer. Such
test would exclude the py2.4/MFC7.1 scheme as reason for the
crash. 
I sent you that win32ui with .pdb:  That should provide
detailed stack traces.

---
related:

#1442426 ,  #1441884

http://sourceforge.net/tracker/index.php?func=detail&aid=1441884&group_id=5470&atid=105470

---

I got similar user reports for a win32ui-app (build
203/MFC4.2/py2.3.5; even without pythonwin IDE). 
Unfortunately I got no stack traces.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1448160&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1874873 ] Vista Install

[SourceForge.net <no...@so...>]

Bugs item #1874873, was opened at 2008-01-18 09:01
Message generated for change (Comment added) made by sf-robot
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1874873&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
>Status: Closed
Resolution: None
Priority: 5
Private: No
Submitted By: robert (w8kk)
Assigned to: Nobody/Anonymous (nobody)
Summary: Vista Install

Initial Comment:
pywin32-210.win32-py2.5.exe will not install under Vista.  The messsage issued is "pywin32-210.win32-py2.5.exe has stopped working"

----------------------------------------------------------------------

>Comment By: SourceForge Robot (sf-robot)
Date: 2008-05-18 19:20

Message:
Logged In: YES 
user_id=1312539
Originator: NO

This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 14 days (the time period specified by
the administrator of this Tracker).

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 04:09

Message:
Logged In: YES 
user_id=14198
Originator: NO

Build 211 is due any day/week now and I'm confident this is fixed there -
please reopen if that is not the case.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1874873&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-17 07:11
Message generated for change (Comment added) made by mhammond
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Mark Hammond (mhammond)
Date: 2008-05-18 11:22

Message:
Logged In: YES 
user_id=14198
Originator: NO

Bugger - that's what I get for rushing :(  There is still something wrong
here though (my example doesn't seem to read MBCS that represents the
unicode that we wrote, but that isn't related to the bug being discussed
here.)

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-17 19:04

Message:
Logged In: YES 
user_id=771074
Originator: NO

You're calling RegEnumKey which enumerates subkeys,
and there are none.  Note that the error here is
ERROR_NO_MORE_ITEMS, rather than ERROR_MORE_DATA.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-17 15:33

Message:
Logged In: YES 
user_id=14198
Originator: NO

I've reproduced this (although sadly with a few hacks).  The trick is (a)
to use a character that doesn't fit in 1 MBCS byte and (b) possibly this
value needs to be the longest in the key.

The following code *should* then demonstrate the problem:
---
import win32api, win32con

test_key_name = "SOFTWARE\\Python Registry Test Key - Delete Me"
root_key = win32con.HKEY_CURRENT_USER
key = win32api.RegCreateKey(root_key, test_key_name)
try:
    win32api.RegSetValueEx(key, u"\u0106", 0, win32con.REG_SZ, u"\u0106")
    print win32api.RegEnumKey(key, 0)
finally:
    key.Close()
    win32api.RegDeleteKey(root_key, test_key_name)
---
However, that is not enough - for some reason, using that same character,
I see:

>>> len(u"\u0106".encode("mbcs"))
1

Which seems quite incorrect to me.  So, I just hacked win32apimodule.cpp
to check for a unicode value and call RegSetValueExW() - and as soon as I
did that, I saw:

Traceback (most recent call last):
  File "\temp\delme.py", line 9, in <module>
print win32api.RegEnumKey(key, 0)
pywintypes.error: (259, 'RegEnumKey', 'No more data is available.')

A look in the debugger shows that the max size returned is zero - which we
add 1 to for the terminator!  Hence this return value.  Sadly, my hack to
RegSetValueEx() is incomplete - PyWinObject_AsRegistryValue() can't handle
both unicode and ansi.

Roger (or anyone) - any clue why I can't seem to make
'len(u"\uxxxx".encode("mbcs"))' return >1 for any value of xxxx?


----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-17 13:35

Message:
Logged In: YES 
user_id=771074
Originator: NO

The size of TCHAR is actually determined at compile time.  It's either 1
or 2
bytes depending solely on whether UNICODE is defined, and does not depend
on
the platform on which it's compiled or run.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-16 23:18

Message:
Logged In: YES 
user_id=13396
Originator: YES

I am attaching a file that contains some changes to the implementations of
RegEnumKey, RegEnumKeyEx, and RegEnumValue - all of which are affected by
this ERROR_MORE_DATA situation.  I have modified the implementation to
expect ERROR_MORE_DATA.

I have looked at the implementation of RegENumValue and friends in the CVS
code.  They still won't work because they presume to know the size of a
TCHAR at compile time, and generally the code is not compiled on an
internationalized box.  I tried a similar implementation that uses the wide
method, and it suffers from the same problem.  I have not tried the link to
the latest version of pywin32, but I will try that today and see if it
resolves the problem.  If it is derived from the code in the latest CVS, it
most likely will not.
File Added: registry_fixes.cpp

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-16 08:35

Message:
Logged In: YES 
user_id=14198
Originator: NO

Hrm - I guess an RDP session might be our last option.  Before that, it
would be nice if you can check the latest pywin32 has the same issue, and
if possible, verify that when you get ERROR_MORE_DATA, the actual buffer
size isn't being returned in one of the pointers - that too would allow us
to avoid any hard-coded limits.

Thanks.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 23:55

Message:
Logged In: YES 
user_id=13396
Originator: YES

Ah. The .reg file I attached earlier fails properly on a Korean / Japanese
system.  It seems to work fine on an English system.  I suspect this is due
to some underlying difference in the windows registry handling code.  I
don't think it is possible to reproduce this w/o a version of Windows that
is localized in this way. 

I could probably make an RDP session to such a system available if that
would help you or someone else verify this.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 23:25

Message:
Logged In: YES 
user_id=14198
Originator: NO

> The test cases won't fail on Latin character sets because they are
> typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The
Japanese
> and Korean encodings are not that simple. 

Yeah - I was trying to say that if we could use the native Unicode
registry functions to write Unicode values that can't be represented in
latin1/mbcs, then we would probably be well on the way to reproducing this.
 But best I can see, that's not currently possible with pywin32/_winreg -
although I can't see why win32api and/or _winreg can't be modified to
handle Unicode natively in that case.  In the meantime, maybe a .reg file
could be used to simulate the same thing - as you mention, the key is
probably to end up with characters that can't be represented in 1 byte
inside our English registry for test purposes.

Thanks!

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 22:39

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 22:38

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 11:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-15 10:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 08:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 00:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 22:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 21:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-16 16:11
Message generated for change (Comment added) made by rupole
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Roger Upole (rupole)
Date: 2008-05-17 04:04

Message:
Logged In: YES 
user_id=771074
Originator: NO

You're calling RegEnumKey which enumerates subkeys,
and there are none.  Note that the error here is
ERROR_NO_MORE_ITEMS, rather than ERROR_MORE_DATA.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-17 00:33

Message:
Logged In: YES 
user_id=14198
Originator: NO

I've reproduced this (although sadly with a few hacks).  The trick is (a)
to use a character that doesn't fit in 1 MBCS byte and (b) possibly this
value needs to be the longest in the key.

The following code *should* then demonstrate the problem:
---
import win32api, win32con

test_key_name = "SOFTWARE\\Python Registry Test Key - Delete Me"
root_key = win32con.HKEY_CURRENT_USER
key = win32api.RegCreateKey(root_key, test_key_name)
try:
    win32api.RegSetValueEx(key, u"\u0106", 0, win32con.REG_SZ, u"\u0106")
    print win32api.RegEnumKey(key, 0)
finally:
    key.Close()
    win32api.RegDeleteKey(root_key, test_key_name)
---
However, that is not enough - for some reason, using that same character,
I see:

>>> len(u"\u0106".encode("mbcs"))
1

Which seems quite incorrect to me.  So, I just hacked win32apimodule.cpp
to check for a unicode value and call RegSetValueExW() - and as soon as I
did that, I saw:

Traceback (most recent call last):
  File "\temp\delme.py", line 9, in <module>
print win32api.RegEnumKey(key, 0)
pywintypes.error: (259, 'RegEnumKey', 'No more data is available.')

A look in the debugger shows that the max size returned is zero - which we
add 1 to for the terminator!  Hence this return value.  Sadly, my hack to
RegSetValueEx() is incomplete - PyWinObject_AsRegistryValue() can't handle
both unicode and ansi.

Roger (or anyone) - any clue why I can't seem to make
'len(u"\uxxxx".encode("mbcs"))' return >1 for any value of xxxx?


----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-16 22:35

Message:
Logged In: YES 
user_id=771074
Originator: NO

The size of TCHAR is actually determined at compile time.  It's either 1
or 2
bytes depending solely on whether UNICODE is defined, and does not depend
on
the platform on which it's compiled or run.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-16 08:18

Message:
Logged In: YES 
user_id=13396
Originator: YES

I am attaching a file that contains some changes to the implementations of
RegEnumKey, RegEnumKeyEx, and RegEnumValue - all of which are affected by
this ERROR_MORE_DATA situation.  I have modified the implementation to
expect ERROR_MORE_DATA.

I have looked at the implementation of RegENumValue and friends in the CVS
code.  They still won't work because they presume to know the size of a
TCHAR at compile time, and generally the code is not compiled on an
internationalized box.  I tried a similar implementation that uses the wide
method, and it suffers from the same problem.  I have not tried the link to
the latest version of pywin32, but I will try that today and see if it
resolves the problem.  If it is derived from the code in the latest CVS, it
most likely will not.
File Added: registry_fixes.cpp

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 17:35

Message:
Logged In: YES 
user_id=14198
Originator: NO

Hrm - I guess an RDP session might be our last option.  Before that, it
would be nice if you can check the latest pywin32 has the same issue, and
if possible, verify that when you get ERROR_MORE_DATA, the actual buffer
size isn't being returned in one of the pointers - that too would allow us
to avoid any hard-coded limits.

Thanks.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 08:55

Message:
Logged In: YES 
user_id=13396
Originator: YES

Ah. The .reg file I attached earlier fails properly on a Korean / Japanese
system.  It seems to work fine on an English system.  I suspect this is due
to some underlying difference in the windows registry handling code.  I
don't think it is possible to reproduce this w/o a version of Windows that
is localized in this way. 

I could probably make an RDP session to such a system available if that
would help you or someone else verify this.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 08:25

Message:
Logged In: YES 
user_id=14198
Originator: NO

> The test cases won't fail on Latin character sets because they are
> typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The
Japanese
> and Korean encodings are not that simple. 

Yeah - I was trying to say that if we could use the native Unicode
registry functions to write Unicode values that can't be represented in
latin1/mbcs, then we would probably be well on the way to reproducing this.
 But best I can see, that's not currently possible with pywin32/_winreg -
although I can't see why win32api and/or _winreg can't be modified to
handle Unicode natively in that case.  In the meantime, maybe a .reg file
could be used to simulate the same thing - as you mention, the key is
probably to end up with characters that can't be represented in 1 byte
inside our English registry for test purposes.

Thanks!

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 07:39

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 07:38

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-14 20:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-14 19:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-14 17:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 09:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 07:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 06:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-17 07:11
Message generated for change (Comment added) made by mhammond
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Mark Hammond (mhammond)
Date: 2008-05-17 15:33

Message:
Logged In: YES 
user_id=14198
Originator: NO

I've reproduced this (although sadly with a few hacks).  The trick is (a)
to use a character that doesn't fit in 1 MBCS byte and (b) possibly this
value needs to be the longest in the key.

The following code *should* then demonstrate the problem:
---
import win32api, win32con

test_key_name = "SOFTWARE\\Python Registry Test Key - Delete Me"
root_key = win32con.HKEY_CURRENT_USER
key = win32api.RegCreateKey(root_key, test_key_name)
try:
    win32api.RegSetValueEx(key, u"\u0106", 0, win32con.REG_SZ, u"\u0106")
    print win32api.RegEnumKey(key, 0)
finally:
    key.Close()
    win32api.RegDeleteKey(root_key, test_key_name)
---
However, that is not enough - for some reason, using that same character,
I see:

>>> len(u"\u0106".encode("mbcs"))
1

Which seems quite incorrect to me.  So, I just hacked win32apimodule.cpp
to check for a unicode value and call RegSetValueExW() - and as soon as I
did that, I saw:

Traceback (most recent call last):
  File "\temp\delme.py", line 9, in <module>
print win32api.RegEnumKey(key, 0)
pywintypes.error: (259, 'RegEnumKey', 'No more data is available.')

A look in the debugger shows that the max size returned is zero - which we
add 1 to for the terminator!  Hence this return value.  Sadly, my hack to
RegSetValueEx() is incomplete - PyWinObject_AsRegistryValue() can't handle
both unicode and ansi.

Roger (or anyone) - any clue why I can't seem to make
'len(u"\uxxxx".encode("mbcs"))' return >1 for any value of xxxx?


----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-17 13:35

Message:
Logged In: YES 
user_id=771074
Originator: NO

The size of TCHAR is actually determined at compile time.  It's either 1
or 2
bytes depending solely on whether UNICODE is defined, and does not depend
on
the platform on which it's compiled or run.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-16 23:18

Message:
Logged In: YES 
user_id=13396
Originator: YES

I am attaching a file that contains some changes to the implementations of
RegEnumKey, RegEnumKeyEx, and RegEnumValue - all of which are affected by
this ERROR_MORE_DATA situation.  I have modified the implementation to
expect ERROR_MORE_DATA.

I have looked at the implementation of RegENumValue and friends in the CVS
code.  They still won't work because they presume to know the size of a
TCHAR at compile time, and generally the code is not compiled on an
internationalized box.  I tried a similar implementation that uses the wide
method, and it suffers from the same problem.  I have not tried the link to
the latest version of pywin32, but I will try that today and see if it
resolves the problem.  If it is derived from the code in the latest CVS, it
most likely will not.
File Added: registry_fixes.cpp

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-16 08:35

Message:
Logged In: YES 
user_id=14198
Originator: NO

Hrm - I guess an RDP session might be our last option.  Before that, it
would be nice if you can check the latest pywin32 has the same issue, and
if possible, verify that when you get ERROR_MORE_DATA, the actual buffer
size isn't being returned in one of the pointers - that too would allow us
to avoid any hard-coded limits.

Thanks.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 23:55

Message:
Logged In: YES 
user_id=13396
Originator: YES

Ah. The .reg file I attached earlier fails properly on a Korean / Japanese
system.  It seems to work fine on an English system.  I suspect this is due
to some underlying difference in the windows registry handling code.  I
don't think it is possible to reproduce this w/o a version of Windows that
is localized in this way. 

I could probably make an RDP session to such a system available if that
would help you or someone else verify this.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 23:25

Message:
Logged In: YES 
user_id=14198
Originator: NO

> The test cases won't fail on Latin character sets because they are
> typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The
Japanese
> and Korean encodings are not that simple. 

Yeah - I was trying to say that if we could use the native Unicode
registry functions to write Unicode values that can't be represented in
latin1/mbcs, then we would probably be well on the way to reproducing this.
 But best I can see, that's not currently possible with pywin32/_winreg -
although I can't see why win32api and/or _winreg can't be modified to
handle Unicode natively in that case.  In the meantime, maybe a .reg file
could be used to simulate the same thing - as you mention, the key is
probably to end up with characters that can't be represented in 1 byte
inside our English registry for test purposes.

Thanks!

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 22:39

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 22:38

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 11:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-15 10:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 08:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 00:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 22:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 21:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-16 16:11
Message generated for change (Comment added) made by rupole
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Roger Upole (rupole)
Date: 2008-05-16 22:35

Message:
Logged In: YES 
user_id=771074
Originator: NO

The size of TCHAR is actually determined at compile time.  It's either 1
or 2
bytes depending solely on whether UNICODE is defined, and does not depend
on
the platform on which it's compiled or run.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-16 08:18

Message:
Logged In: YES 
user_id=13396
Originator: YES

I am attaching a file that contains some changes to the implementations of
RegEnumKey, RegEnumKeyEx, and RegEnumValue - all of which are affected by
this ERROR_MORE_DATA situation.  I have modified the implementation to
expect ERROR_MORE_DATA.

I have looked at the implementation of RegENumValue and friends in the CVS
code.  They still won't work because they presume to know the size of a
TCHAR at compile time, and generally the code is not compiled on an
internationalized box.  I tried a similar implementation that uses the wide
method, and it suffers from the same problem.  I have not tried the link to
the latest version of pywin32, but I will try that today and see if it
resolves the problem.  If it is derived from the code in the latest CVS, it
most likely will not.
File Added: registry_fixes.cpp

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 17:35

Message:
Logged In: YES 
user_id=14198
Originator: NO

Hrm - I guess an RDP session might be our last option.  Before that, it
would be nice if you can check the latest pywin32 has the same issue, and
if possible, verify that when you get ERROR_MORE_DATA, the actual buffer
size isn't being returned in one of the pointers - that too would allow us
to avoid any hard-coded limits.

Thanks.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 08:55

Message:
Logged In: YES 
user_id=13396
Originator: YES

Ah. The .reg file I attached earlier fails properly on a Korean / Japanese
system.  It seems to work fine on an English system.  I suspect this is due
to some underlying difference in the windows registry handling code.  I
don't think it is possible to reproduce this w/o a version of Windows that
is localized in this way. 

I could probably make an RDP session to such a system available if that
would help you or someone else verify this.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 08:25

Message:
Logged In: YES 
user_id=14198
Originator: NO

> The test cases won't fail on Latin character sets because they are
> typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The
Japanese
> and Korean encodings are not that simple. 

Yeah - I was trying to say that if we could use the native Unicode
registry functions to write Unicode values that can't be represented in
latin1/mbcs, then we would probably be well on the way to reproducing this.
 But best I can see, that's not currently possible with pywin32/_winreg -
although I can't see why win32api and/or _winreg can't be modified to
handle Unicode natively in that case.  In the meantime, maybe a .reg file
could be used to simulate the same thing - as you mention, the key is
probably to end up with characters that can't be represented in 1 byte
inside our English registry for test purposes.

Thanks!

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 07:39

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 07:38

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-14 20:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-14 19:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-14 17:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 09:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 07:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 06:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-16 21:11
Message generated for change (Comment added) made by neverjade
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Christopher Nelson (neverjade)
Date: 2008-05-16 13:18

Message:
Logged In: YES 
user_id=13396
Originator: YES

I am attaching a file that contains some changes to the implementations of
RegEnumKey, RegEnumKeyEx, and RegEnumValue - all of which are affected by
this ERROR_MORE_DATA situation.  I have modified the implementation to
expect ERROR_MORE_DATA.

I have looked at the implementation of RegENumValue and friends in the CVS
code.  They still won't work because they presume to know the size of a
TCHAR at compile time, and generally the code is not compiled on an
internationalized box.  I tried a similar implementation that uses the wide
method, and it suffers from the same problem.  I have not tried the link to
the latest version of pywin32, but I will try that today and see if it
resolves the problem.  If it is derived from the code in the latest CVS, it
most likely will not.
File Added: registry_fixes.cpp

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 22:35

Message:
Logged In: YES 
user_id=14198
Originator: NO

Hrm - I guess an RDP session might be our last option.  Before that, it
would be nice if you can check the latest pywin32 has the same issue, and
if possible, verify that when you get ERROR_MORE_DATA, the actual buffer
size isn't being returned in one of the pointers - that too would allow us
to avoid any hard-coded limits.

Thanks.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 13:55

Message:
Logged In: YES 
user_id=13396
Originator: YES

Ah. The .reg file I attached earlier fails properly on a Korean / Japanese
system.  It seems to work fine on an English system.  I suspect this is due
to some underlying difference in the windows registry handling code.  I
don't think it is possible to reproduce this w/o a version of Windows that
is localized in this way. 

I could probably make an RDP session to such a system available if that
would help you or someone else verify this.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 13:25

Message:
Logged In: YES 
user_id=14198
Originator: NO

> The test cases won't fail on Latin character sets because they are
> typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The
Japanese
> and Korean encodings are not that simple. 

Yeah - I was trying to say that if we could use the native Unicode
registry functions to write Unicode values that can't be represented in
latin1/mbcs, then we would probably be well on the way to reproducing this.
 But best I can see, that's not currently possible with pywin32/_winreg -
although I can't see why win32api and/or _winreg can't be modified to
handle Unicode natively in that case.  In the meantime, maybe a .reg file
could be used to simulate the same thing - as you mention, the key is
probably to end up with characters that can't be represented in 1 byte
inside our English registry for test purposes.

Thanks!

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 12:39

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 12:38

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 01:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-15 00:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-14 22:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 19:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 19:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 12:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 11:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-17 07:11
Message generated for change (Comment added) made by mhammond
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Mark Hammond (mhammond)
Date: 2008-05-16 08:35

Message:
Logged In: YES 
user_id=14198
Originator: NO

Hrm - I guess an RDP session might be our last option.  Before that, it
would be nice if you can check the latest pywin32 has the same issue, and
if possible, verify that when you get ERROR_MORE_DATA, the actual buffer
size isn't being returned in one of the pointers - that too would allow us
to avoid any hard-coded limits.

Thanks.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 23:55

Message:
Logged In: YES 
user_id=13396
Originator: YES

Ah. The .reg file I attached earlier fails properly on a Korean / Japanese
system.  It seems to work fine on an English system.  I suspect this is due
to some underlying difference in the windows registry handling code.  I
don't think it is possible to reproduce this w/o a version of Windows that
is localized in this way. 

I could probably make an RDP session to such a system available if that
would help you or someone else verify this.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 23:25

Message:
Logged In: YES 
user_id=14198
Originator: NO

> The test cases won't fail on Latin character sets because they are
> typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The
Japanese
> and Korean encodings are not that simple. 

Yeah - I was trying to say that if we could use the native Unicode
registry functions to write Unicode values that can't be represented in
latin1/mbcs, then we would probably be well on the way to reproducing this.
 But best I can see, that's not currently possible with pywin32/_winreg -
although I can't see why win32api and/or _winreg can't be modified to
handle Unicode natively in that case.  In the meantime, maybe a .reg file
could be used to simulate the same thing - as you mention, the key is
probably to end up with characters that can't be represented in 1 byte
inside our English registry for test purposes.

Thanks!

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 22:39

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 22:38

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 11:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-15 10:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 08:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 00:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 22:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 21:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-16 21:11
Message generated for change (Comment added) made by neverjade
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 13:55

Message:
Logged In: YES 
user_id=13396
Originator: YES

Ah. The .reg file I attached earlier fails properly on a Korean / Japanese
system.  It seems to work fine on an English system.  I suspect this is due
to some underlying difference in the windows registry handling code.  I
don't think it is possible to reproduce this w/o a version of Windows that
is localized in this way. 

I could probably make an RDP session to such a system available if that
would help you or someone else verify this.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 13:25

Message:
Logged In: YES 
user_id=14198
Originator: NO

> The test cases won't fail on Latin character sets because they are
> typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The
Japanese
> and Korean encodings are not that simple. 

Yeah - I was trying to say that if we could use the native Unicode
registry functions to write Unicode values that can't be represented in
latin1/mbcs, then we would probably be well on the way to reproducing this.
 But best I can see, that's not currently possible with pywin32/_winreg -
although I can't see why win32api and/or _winreg can't be modified to
handle Unicode natively in that case.  In the meantime, maybe a .reg file
could be used to simulate the same thing - as you mention, the key is
probably to end up with characters that can't be represented in 1 byte
inside our English registry for test purposes.

Thanks!

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 12:39

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 12:38

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 01:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-15 00:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-14 22:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 19:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 19:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 12:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 11:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-17 07:11
Message generated for change (Comment added) made by mhammond
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 23:25

Message:
Logged In: YES 
user_id=14198
Originator: NO

> The test cases won't fail on Latin character sets because they are
> typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The
Japanese
> and Korean encodings are not that simple. 

Yeah - I was trying to say that if we could use the native Unicode
registry functions to write Unicode values that can't be represented in
latin1/mbcs, then we would probably be well on the way to reproducing this.
 But best I can see, that's not currently possible with pywin32/_winreg -
although I can't see why win32api and/or _winreg can't be modified to
handle Unicode natively in that case.  In the meantime, maybe a .reg file
could be used to simulate the same thing - as you mention, the key is
probably to end up with characters that can't be represented in 1 byte
inside our English registry for test purposes.

Thanks!

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 22:39

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 22:38

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 11:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-15 10:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 08:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 00:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 22:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 21:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-16 21:11
Message generated for change (Comment added) made by neverjade
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 12:39

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 12:38

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 01:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-15 00:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-14 22:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 19:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 19:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 12:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 11:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-16 21:11
Message generated for change (Comment added) made by neverjade
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 12:38

Message:
Logged In: YES 
user_id=13396
Originator: YES

The test cases won't fail on Latin character sets because they are
typically 1-byte granular, so 1 TCHAR ends up being 1 byte.  The Japanese
and Korean encodings are not that simple.  I am also unhappy with setting
to a maximum value, so I have revised RegEnumKey and friends to loop when
detecting ERROR_MORE_DATA.  I will also change that for RegEnumValue.

The Windows API seems a bit fuzzy.  However, QueryInfoKey supposedly
returns the maximum size of a key and value name in Unicode characters,
whereas EnumKey/EnumValue say that they return the exact size in TCHARS. 
TCHARS are not necessarily unicode characters, which makes it frustrating
to interpret.  There are a variety of unicode encodings, and they are not
necessarily byte_width * 2.  For example, in the output I pasted below the
key was apparently 8 TCHARS long, but that translated to a buffer of 12
bytes plus a NULL sequence.

I will look at the CVS version and the link you pasted below and see if I
can reproduce the problem with those versions.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 01:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-15 00:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-14 22:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 19:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 19:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 12:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 11:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-17 07:11
Message generated for change (Comment added) made by mhammond
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 11:03

Message:
Logged In: YES 
user_id=14198
Originator: NO

You might like to try out
starship.python.net/crew/mhammond/pywin32-210.9.win32-py2.5.exe and see if
you can still repro it.

----------------------------------------------------------------------

Comment By: Roger Upole (rupole)
Date: 2008-05-15 10:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 08:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 00:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 22:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 21:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-16 16:11
Message generated for change (Comment added) made by rupole
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Roger Upole (rupole)
Date: 2008-05-14 19:03

Message:
Logged In: YES 
user_id=771074
Originator: NO

In CVS, win32apimodule.cpp has already been updated to calculate
the name buffer size in TCHARs, and can be compiled with
UNICODE defined to call the wide-character versions of the
API functions.  With the current CVS code (compiled with
or without UNICODE), I don't get an error while reading the
registry data you posted.


----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-14 17:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 09:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 08:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 07:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 06:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-17 07:11
Message generated for change (Comment added) made by mhammond
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Mark Hammond (mhammond)
Date: 2008-05-15 08:42

Message:
Logged In: YES 
user_id=14198
Originator: NO

Thanks for the update, but I'm afraid I will not have time for a few days,
but a couple of thoughts: does the test case fail on English windows?  If
the issue is simply that the number of bytes returned is sometimes too
small, presumably when multi-byte characters exist, it should be possible
to repro by writing Unicode that has a multi-byte representation and trying
to read it back.  

When you say:
> If you read the documentation for that RegEnumValue closely, you will
> realize that it doesn't actually return what you think.

can you be more specific?  I'd be happy with code that handled the
ERROR_MORE_DATA case, but I'm not happy with setting a hard-coded limit to
either key or value size, and ideally I'd like *some* indication why the
code in question isn't working correctly - as we are discussing the API
itself here, I bet we are not the first people to strike this issue.  Hence
I'm quite keen to see this fail myself (if for no better reason than I can
then submit a patch upstream for Python's _winreg module - they will insist
on a test)

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 05:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-15 00:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 23:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 22:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 21:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

[pywin32-bugs] [ pywin32-Bugs-1944375 ] PyRegEnumValue fails on i18n systems

[SourceForge.net <no...@so...>]

Bugs item #1944375, was opened at 2008-04-16 21:11
Message generated for change (Comment added) made by neverjade
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: win32
Group: None
Status: Open
Resolution: Rejected
Priority: 5
Private: No
Submitted By: Christopher Nelson (neverjade)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyRegEnumValue fails on i18n systems

Initial Comment:
If you try to enumerate the contents of the registry key:

"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control Panel\\Cursors\\Schemes"

pywin32 will fail with a (234, "PyRegEnumValue", ""), which means that RegEnumValue returned with ERROR_MORE_DATA.

The method pywin32 currently uses to detect the size of the value name buffer does not work properly on Windows  2003 X64 Japanese.  

To fix this bug you must make the code look like the follwing:

// @pymethod (string,object,type)|win32api|RegEnumValue|Enumerates values of the specified open registry key. The function retrieves the name of one subkey each time it is called.

static PyObject *

PyRegEnumValue( PyObject *self, PyObject *args )

{

	// This value is taken from MSDN docs.

	const DWORD maxValueNameSize=16384;



	HKEY hKey;

	PyObject *obKey;

	int index;

	long rc;

	TCHAR retValueBuf[maxValueNameSize];

	BYTE *retDataBuf;

	DWORD retValueSize = maxValueNameSize;

	DWORD retDataSize=0;

	DWORD typ;

	

	// @pyparm <o PyHKEY>/int|key||An already open key, or any one of the following win32con constants:<nl>HKEY_CLASSES_ROOT<nl>HKEY_CURRENT_USER<nl>HKEY_LOCAL_MACHINE<nl>HKEY_USERS

	// @pyparm int|index||The index of the key to retrieve.



	if (!PyArg_ParseTuple(args, "Oi:PyRegEnumValue", &obKey, &index))

		return NULL;

	if (!PyWinObject_AsHKEY(obKey, &hKey))

		return NULL;

	

	// @pyseeapi PyRegEnumValue

	PyW32_BEGIN_ALLOW_THREADS

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, NULL, &retDataSize);

	PyW32_END_ALLOW_THREADS

		

	// Reset because the call above messed it up.

	retValueSize=maxValueNameSize;

	// Don't need to increment because the size returned from RegEnumValue includes any needed terminators.

	retDataBuf= (BYTE * )alloca(retDataSize);

	if ((retDataBuf==NULL)){

		PyErr_NoMemory();

		return NULL;

		}

	

	rc=RegEnumValue(hKey, index, retValueBuf, &retValueSize, NULL, &typ, retDataBuf, &retDataSize);	 



	if (rc!=ERROR_SUCCESS)

	{		

		return ReturnAPIError("PyRegEnumValue", rc);

	}



	PyObject *obData=PyWinObject_FromRegistryValue(retDataBuf, retDataSize, typ);

	if (obData==NULL)

	{

		return NULL;

	}

	PyObject *retVal = Py_BuildValue("NOi", PyWinObject_FromTCHAR(retValueBuf), obData, typ);

	

	Py_DECREF(obData);

	

	return retVal;

	// @comm This function is typically called repeatedly, until an exception is raised, indicating no more values.

}

----------------------------------------------------------------------

>Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 19:43

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached the .cpp file containing the program I used to generate
these results.
File Added: main.cpp

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 19:42

Message:
Logged In: YES 
user_id=13396
Originator: YES

I believe there are two problems:

(1) MS's registry code works strangely
(2) python and pywin32 assume that the functions are returning sizes in
bytes, whereas the API documentation clearly says unicode characters (or in
some places, TCHARS.)

This happens to work by accident on Latin character set systems b/c they
usually use utf-8 or similar 8-byte systems. Korean and Japanese systems do
not.
The following is output of a C++ program I wrote to examine the
functionality of the registry functions on a 64-bit japanese localized
system:

C:\temp>\\tsclient\dev\registry_test.exe
info:          number of subkeys: 2
   maximum length of subkey name: 8
                number of values: 0
    maximum length of value name: 0
info: read key name:
  size before call: 9
   size after call: 5
     buffer growth: 0
          contents: agent
warn: the buffer allocated was not large enough:
  size before call: 9
   size after call: 9
       resizing to: 10
warn: the buffer allocated was not large enough:
  size before call: 10
   size after call: 10
       resizing to: 12
warn: the buffer allocated was not large enough:
  size before call: 12
   size after call: 12
       resizing to: 15
info: read key name:
  size before call: 15
   size after call: 12
     buffer growth: 3
          contents: \x8c\xc2\x82\xcc\x83t\x83@\x83C\x83\x8b
info: done

== Some explanation:

Size before call is the size of the buffer we allocated in order retrieve
the results of RegEnumKeyEx.
Size after call is what RegEnumKeyEx wrote back INTO the buffer size
parameter after the RegEnumKeyEx call.
Resizing to is the size in bytes of the buffer that has been enlarged to
potentially fit the contents.
buffer growth is the increment over the value returned by RegEnumKeyEx in
the lpcbName buffer.


----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 14:33

Message:
Logged In: YES 
user_id=13396
Originator: YES

I should also clarify - this is a problem with key and value NAMES. Not
the data.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:24

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have attached a registry key that causes the failure that I posted below
for your debugging pleasure.

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:23

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: opsware_reg.dat

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:22

Message:
Logged In: YES 
user_id=13396
Originator: YES

I have also discovered the same problem in RegEnumKey today.  The error
occurs this way:

Testing:  SOFTWARE\Opsware
=========================
agent
Traceback (most recent call last):
  File "c:\temp\testreg3.py", line 21, in ?
    print win32api.RegEnumKey(r2, index)
pywintypes.error: (234, 'RegEnumKey',
'\x83f\x81[\x83^\x82\xaa\x82\xb3\x82\xe7\x
82\xc9\x82\xa0\x82\xe8\x82\xdc\x82\xb7\x81B')

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:12

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: PyRegEnumValue.c

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:09

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_pywin32_reg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:04

Message:
Logged In: YES 
user_id=13396
Originator: YES

File Added: test_winreg.py

----------------------------------------------------------------------

Comment By: Christopher Nelson (neverjade)
Date: 2008-05-14 13:03

Message:
Logged In: YES 
user_id=13396
Originator: YES

Mark, please read the bug description. This happens on i18n systems.  For
example, Windows 2003 x64 localized in Japanese or Korean.  I am sorry that
you haven't had any bug reports on this, but I can guarantee you that if
you run that code on a w2k3 Japanese or Korean localized system, you will
hit that bug.

With respect to you comment that 

// Reset because the call above messed it up.
retValueSize=maxValueNameSize;

is wrong, I would agree with you, except that I watched the original code
not work.  I spent a number of hours tracking this problem down in our
production code, and it came back to this function in pywin32. FWIW,
_winreg manifests the same problem, in the same way.

If you read the documentation for that RegEnumValue closely, you will
realize that it doesn't actually return what you think.  Also, on i18n
systems, the ascii version of this call simply doesn't work correctly. 
Partly this is due to size limitations of the API call.  In any case, the
MSDN documentation does not try to get the key's name size, it simply uses
this value.  I suspect this is because they know that the ascii version of
this call does not work properly.

I am attaching the scripts that fail below.










----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-05 12:59

Message:
Logged In: YES 
user_id=14198
Originator: NO

I had another look at this and sadly can't repro it with that key on
win2003.  Either way, the patch as it stands needs work - eg, the code and
comment:

  // Reset because the call above messed it up.
  retValueSize=maxValueNameSize;

is wrong - the whole point of the call above was to determine the size -
so the correct fix given that approach would be to remove the first call
completely.  It also seems this would fail to work for binary values with
more than 16384 bytes from working, which best I can tell does now.  So
while I don't doubt the Japanese version of 2003 fails, I'm rejecting this
as it stands still welcome more input on the best way to approach this -
eg, the smallest script that fails for you would help - I'm assuming it
would be:

  import win32api, win32con
  key=win32api.RegOpenKey(win32con.HKEY_LOCAL_MACHINE,
                         
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Control
Panel\\Cursors\\Schemes",
                          win32con.KEY_READ)
  print win32api.RegEnumValue(key, 0)

FWIW, Python's _winreg module still has identical code to win32api and no
open bugs on a similar issue, so no one has reported a problem there either
(but that still doesn't mean one doesn't exist :)

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2008-05-04 11:07

Message:
Logged In: YES 
user_id=14198
Originator: NO

Could you please attach either a patch, or the complete source file with
the new function (all the indentation is lost above)

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=551954&aid=1944375&group_id=78018