[Tcl-bugs] [ tktoolkit-Bugs-3480471 ] tk_getOpenFile crashes on Win64

[SourceForge.net <no...@so...>]

Bugs item #3480471, was opened at 2012-01-27 01:50
Message generated for change (Comment added) made by nijtmans
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=112997&aid=3480471&group_id=12997

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: 68. Win Window Operations
Group: current: 8.5.10
Status: Open
Resolution: None
Priority: 9
Private: No
Submitted By: Alexander Haas (halex-ecs)
>Assigned to: Jan Nijtmans (nijtmans)
Summary: tk_getOpenFile crashes on Win64 

Initial Comment:
On Windows x64, when running a 64bit wish 8.5.10 and after allocating and using lots of memory (above 2GB), wish crashes immediately when selecting a File in a tk_getOpenFile dialog. We tracked the problem down to the function OFNHookProcW in  tkWinDialog.c.

Referencing tk 8.5.10, in line 1035-1040 of tkWinDialog.c, there is the following piece of code:

SendMessageW(hdlg, CDM_GETFOLDERPATH, dirsize, (int) buffer);
buffer += dirsize;
SendMessageW(hdlg, CDM_GETSPEC, selsize, (int) buffer);

The crash is caught by the debugger in the line
buffer += dirsize;
but the mistake resides in the lines above and below:
The last argument of SendMessageW is declared  as LPARAM
(see http://msdn.microsoft.com/en-us/library/windows/desktop/ms644950(v=vs.85).aspx), which is defined as LONG_PTR which is defined as __int64 when building with 64bit (see http://msdn.microsoft.com/en-us/library/aa383751(v=VS.85).aspx)

When buffer starts above 2GB adress (this makes it a little difficult to reproduce this error), the cast to int cuts the upper part of the adress and things become very wrong and wish crashes.

Changing those lines to 
SendMessageW(hdlg, CDM_GETFOLDERPATH, dirsize, (__int64) buffer);
buffer += dirsize;
SendMessageW(hdlg, CDM_GETSPEC, selsize, (__int64) buffer);

makes everything fine.
The correct all-platform fix should cast to (LONG_PTR), imho.

This error also occours in tk 8.5.11 (although these lines are different, but PTR2INT makes the same mistake as the int-cast) and in tk 8.6b2



----------------------------------------------------------------------

>Comment By: Jan Nijtmans (nijtmans)
Date: 2012-01-27 03:51

Message:
I will have a look at this

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=112997&aid=3480471&group_id=12997