tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] TIP 716 ready for comments

[Jan N. <jan...@gm...>]

Op vr 25 apr 2025 om 19:37 schreef Harald Oehlmann:
> P.S. I have retried my test with the updated branch. This does not make
> any difference (and should IMHO not). The result is ok and now, we have
> migration text ;-).

1) So, did you see the following test failures?
2) Does TIP #716 tell anything about this?

Hope I have your attention now ;-)
            Jan NIjtmans

P.S.: Github ACTIONS shows the same failure:
         <https://github.com/tcltk/tcl/actions/runs/14635843756/job/41066664724>

109==== cmdAH-16.2 Tcl_FileObjCmd: readable FAILED
110==== Contents of test case:
111file readable $gorpfile
112---- Test setup failed:
113D:\a\tcl\tcl\win\górp.file: no such file or directory
114---- errorInfo(setup): D:\a\tcl\tcl\win\górp.file: no such file or directory
115 while executing
116"testchmod 0o444 $gorpfile"
117 ("uplevel" body line 1)
118 invoked from within
119"uplevel 1 $setup"
120---- errorCode(setup): POSIX ENOENT {no such file or directory}
121==== cmdAH-16.2 FAILED
122
123
124
125==== cmdAH-17.2 Tcl_FileObjCmd: writable FAILED
126==== Contents of test case:
127file writable $gorpfile
128---- Test setup failed:
129D:\a\tcl\tcl\win\górp.file: no such file or directory
130---- errorInfo(setup): D:\a\tcl\tcl\win\górp.file: no such file or directory
131 while executing
132"testchmod 0o555 $gorpfile"
133 ("uplevel" body line 1)
134 invoked from within
135"uplevel 1 $setup"
136---- errorCode(setup): POSIX ENOENT {no such file or directory}
137==== cmdAH-17.2 FAILED
138
139
140
141==== cmdAH-17.3 Tcl_FileObjCmd: writable FAILED
142==== Contents of test case:
143file writable $gorpfile
144---- Test setup failed:
145D:\a\tcl\tcl\win\górp.file: no such file or directory
146---- errorInfo(setup): D:\a\tcl\tcl\win\górp.file: no such file or directory
147 while executing
148"testchmod 0o222 $gorpfile"
149 ("uplevel" body line 1)
150 invoked from within
151"uplevel 1 $setup"
152---- errorCode(setup): POSIX ENOENT {no such file or directory}
153==== cmdAH-17.3 FAILED
154
155cmdIL.test

Re: [TCLCORE] TIP 716 ready for comments

[Harald O. <har...@el...>]

Am 26.04.2025 um 00:12 schrieb Jan Nijtmans:
> Op vr 25 apr 2025 om 16:06 schreef Ashok Nadkarni <apn...@ya...>:
>>
>> And other posts on the same (DB2 drivers not supporting UTF-8) - SQLAllocHandle of the driver on SQL_HANDLE_ENV failed when connecting to IBM-DB2 database - MATLAB Answers - MATLAB Central and windows - DB2 service does not start - Stack Overflow
> 
> Those posts tell me something. First, compare the story with this Tcl ticket:
>      <https://core.tcl-lang.org/tcl/info/0b9332722a>
> Tcl had the same problem 5 years ago. It got the ACP value, prepended it
> with "cp" and then used it as an encoding name. There is no "cp65001"
> encoding, it should be handled as "utf-8".
> 
> My guess is that the IBM dll does the same thing. No surprise
> the initialisation fails. They really should fix this bug, it shouldn't
> be difficult. Is it already reported to them? The DB2 dll
> simply doesn't work in any utf-8 environment.
> 
> Does that make sense?
> 
>          Jan Nijtmans
> 
Dear Jan,
indeed, it might be the same issue related to the ticket, who knows.
The magic manifest key is a partial work-around for programs using the 8 
bit Windows API. It was introduced in Windows 10.
It only affects the ANSI/char/MCBS Windows API which is superseeded 
since 1993 by the wide Windows API.

What does this workaround do:
- switch some (not all) ANSI API functions from the local encoding 
(cp1252 in Western Europe) to use UTF-8 as char encoding
- switch the reporting of the native codepage
- as the manifest is in the exe, it applies to all loaded DLLs

Unfortunately, this breaks a lot of legacy DLLs. They are not fixed as 
the source code or maintainer is not available.
In addition, there is no reason to fix. The DLLs work.

The upper workaround is only useful (if for any case), if a small 
application with only WinAPI dependencies should be easily ported.

It is not useful for TCL as it breaks legacy DLLs.
There is no useful application, as TCL uses the wide Windows API.

Ashok gets a bit nerved, as this was changed without a TIP.
Here, I am not his opinion. The intention was clear and when reading 
from it, I was initialy also in favor: Unicode? Yes, we want it.
Now, we are wiser and we should revert this change.
Ashok goes the formal way by a TIP, what is noble.
IMHO, it could just be reverted as a not valid bugfix.

I really appreciate all your work.
But putting energy in here is IMHO lost and may better be investigated 
in the great three tickets by Christian ;-).

Thanks for all and take care,
Harald

Re: [TCLCORE] TIP 716 ready for comments

[Jan N. <jan...@gm...>]

Op vr 25 apr 2025 om 16:06 schreef Ashok Nadkarni <apn...@ya...>:
>
> And other posts on the same (DB2 drivers not supporting UTF-8) - SQLAllocHandle of the driver on SQL_HANDLE_ENV failed when connecting to IBM-DB2 database - MATLAB Answers - MATLAB Central and windows - DB2 service does not start - Stack Overflow

Those posts tell me something. First, compare the story with this Tcl ticket:
    <https://core.tcl-lang.org/tcl/info/0b9332722a>
Tcl had the same problem 5 years ago. It got the ACP value, prepended it
with "cp" and then used it as an encoding name. There is no "cp65001"
encoding, it should be handled as "utf-8".

My guess is that the IBM dll does the same thing. No surprise
the initialisation fails. They really should fix this bug, it shouldn't
be difficult. Is it already reported to them? The DB2 dll
simply doesn't work in any utf-8 environment.

Does that make sense?

        Jan Nijtmans

Re: [TCLCORE] TIP 716 ready for comments

[Harald O. <har...@el...>]

Am 16.04.2025 um 14:03 schrieb Jan Nijtmans:
> Op wo 16 apr 2025 om 13:35 schreef Harald Oehlmann:
>> The migration:
>> https://core.tcl-lang.org/tcl/wiki?name=Migrating+C+extensions+to+Tcl+9&p
>> does not tell anything on this...
> 
> So, that should be extended. All is described here:
>       <https://core.tcl-lang.org/tips/doc/trunk/tip/548.md>
The migration page is now extended by the change of the MS-Windows 
system encoding:

https://core.tcl-lang.org/tcl/info/e16612341c1416c5

and the deprecated Tcl_Winxxx functions:

https://core.tcl-lang.org/tcl/info/d6ff4a080d8f9e87

I try to use full examples. The Tcl_Winxxx functions were never fully 
documented. I hope, this works.

About TIP 548:
https://core.tcl-lang.org/tips/doc/trunk/tip/548.md

I am hesitant to document more. Basically, I don't understand the text.
Or it is incomplete. The type "wchar_t" is 16 bit on Windows, but may be 
another size on other platforms. But this is a platform issue, not a TCL 
issue.
I am also not happy by the word "utf-8". Here, not the utf-8 encoding is 
ment, but the TCL variant replacing the 0 Codepoint. Or is this not the 
case, e.g. the functions may not handle 0 bytes.
Also, there is nothing written about eventual surrogates for the 16 bit 
functions.

More questions than answers as usual....

Thanks for all,
Harald

P.S. I have retried my test with the updated branch. This does not make 
any difference (and should IMHO not). The result is ok and now, we have 
migration text ;-).

Re: [TCLCORE] TIP 716 ready for comments

[da S. P. J <pet...@fl...>]

I will be so happy to see the backside of Tcl_DStringToExternal and its hyena cousins. ☺

From: Jan Nijtmans <jan...@gm...>
Date: Friday, April 25, 2025 at 02:38
To: tcl...@li... <tcl...@li...>
Subject: Re: [TCLCORE] TIP 716 ready for comments
Op di 15 apr 2025 om 21: 02 schreef apnmbx-public--- via Tcl-Core <tcl-core@ lists. sourceforge. net>: > The rules are (as per TIP 716): > > Use the Unicode Windows API as Tcl/Tk already do. Then these issues do not arise. > If


Op di 15 apr 2025 om 21:02 schreef apnmbx-public--- via Tcl-Core

<tcl...@li...>:

> The rules are (as per TIP 716):

>

> Use the Unicode Windows API as Tcl/Tk already do. Then these issues do not arise.

> If you have to use ANSI API’s, use Windows encoding settings, not Tcl encoding settings.



So, let's establish that TIP #716 changes the rules!



I did some investigation which extensions will be broken by TIP #716.

But - first -

let's remember the rules how they were in  Tcl 8.x.



1) Use the Unicode Windows API as Tcl/Tk already do. Then these issues

do not arise:

       Tcl_WinUtfToTChar(str, strlen(str), &ds);

       _wfopen((WCHAR *)Tcl_DStringValue(ds), L"r")



2) If you have to use ANSI API’s, use the following:

       Tcl_DStringInit(&ds);

       Tcl_DStringToExtenal(NULL, str, -1, &ds);

       fopen(Tcl_DStringValue(ds), "r")



3) If you only care about ASCII characters, just do:

       fopen(str, "r")



In Tcl 9.0.0/9.0.3, all 3 rules start working correctly with UTF-8

(except when using

the GDI API but there is no known extension using that). Whatever

choice is made, 1), 2) or 3), all work fine with Tcl 9.0.0/9.0.1.  If

TIP #716 is

accepted, 2) and 3) won't work correctly with characters outside ASCII any more.

Which extensions will suffer from that? I found the following:



    blt/src/bltWinImage.c (line 1194)

    expect/exp_main_sub.c (line 813)

    pikchr/pikchr.c (line 8136)

    tkimg/sgi/sgi.c (line 1512)

    VecTcl/WavReader/generic/wavreader.c (line 88)



So, as long as BLT images, pikchr, sgi or waveform filenames

use ASCII characters only, nothing to worry about. When running

in Tcl 9.0.0 or 9.0.1, nothing to worry about either. But with

TIP #716, the rules change, and 2) or 3) can no longer handle

characters outside ASCII.



Hope this helps,

        Jan Nijtmans





_______________________________________________

Tcl-Core mailing list

Tcl...@li...

https://urldefense.com/v3/__https://lists.sourceforge.net/lists/listinfo/tcl-core__;!!MvWE!FTxd301TfyO5GgcGfdBSuSnO-jh1562z8eQNudPLkWdmxnKFL9fojX7lFMF-seoyoO-rIpaQa8FobgbaN2F45Mu0k43tNA$<https://urldefense.com/v3/__https:/lists.sourceforge.net/lists/listinfo/tcl-core__;!!MvWE!FTxd301TfyO5GgcGfdBSuSnO-jh1562z8eQNudPLkWdmxnKFL9fojX7lFMF-seoyoO-rIpaQa8FobgbaN2F45Mu0k43tNA$>

Re: [TCLCORE] TIP 716 ready for comments

[Ashok N. <apn...@ya...>]

And other posts on the same (DB2 drivers not supporting UTF-8) - SQLAllocHandle of the driver on SQL_HANDLE_ENV failed when connecting to IBM-DB2 database - MATLAB Answers - MATLAB Central<https://www.mathworks.com/matlabcentral/answers/1786300-sqlallochandle-of-the-driver-on-sql_handle_env-failed-when-connecting-to-ibm-db2-database> and windows - DB2 service does not start - Stack Overflow<https://stackoverflow.com/questions/60296057/db2-service-does-not-start>

It would really help folks making up their mind on TIP 716 (including me!) if you could spell out your motivation for adding the manifest entry to Tcl. Perhaps there is an overriding benefit to it but without a TIP or any kind of discussion prior to its introduction in Tcl 9, there is no way for anyone to know.

/Ashok






________________________________
From: Ashok Nadkarni via Tcl-Core
Sent: Friday, April 25, 2025 7:27 PM
To: Jan Nijtmans; tcl...@li...
Subject: Re: [TCLCORE] TIP 716 ready for comments

Jan,

Sorry, could not respond earlier as have been traveling all week.

The DB2 error occurs at initialization to the db on a call

conn->rc = SQLAllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, &henv);

Conn->rc is the result code  (-1 indicating failure), henv would normally contain the returned handle.
Any attempt to retrieve additional error information with SQLRegDiagRec fails. I am not sure why but could be because the error retrieval mechanisms also cannot handle a UTF-8 code page.

So to elaborate on Harald's correct interpretation of my original post, the failure mode does not involve any Tcl code at all. No data is passed so no encoding mismatch is involved. Your conjecture about Tcl passing it UTF-8 encoded data is off base. It never gets that far to even attempt that (not even as far as SqlConnect).

Your comment that Tcl should respect the IBM API is spot on. Except Tcl 9 does not! It is expected that the user's real code page will be retrieved with GetACP() and Tcl 9's manifest hijacks that API to force a UTF-8 return with no recourse or workaround available. How would you suggest the above basic and fundamental API call be written to work around the issue even if the extension writer is willing?

To remove Tcl from the equation, also see this issued reported with R 4.2 and DB2 drivers - RODBC odbcConnect fails for ibm db2 connection in Prairie Trillium · Issue #10509 · rstudio/rstudio<https://github.com/rstudio/rstudio/issues/10509>. Guess what R 4.2 and Tcl 9 have in common? The UTF-8 manifest. The motivation for R 4.2 using the manifest is spelled out in the link I included in the TIP. It is completely inapplicable to Tcl for reasons I also outlined in the TIP.

I have not looked at the link you included but the failing extension code demonstrating the error is attached.

/Ashok


________________________________
From: Jan Nijtmans

My guess is that the API used by this IBM DLL is expecting
cp1252, but Tcl is providing/retrieving utf-8-encoded data
for it. Do you know where the Tcl wrapper-code for this
IBM dll is? I think there's nothing wrong with the IBM dll,
It's just that Tcl should respect the IBM API.

Is the code here?
      https://github.com/memmertoIBM/db2tcl/

Regards,
        Jan Nijtmans


_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] TIP 716 ready for comments

[Ashok N. <apn...@ya...>]

Jan,

Sorry, could not respond earlier as have been traveling all week.

The DB2 error occurs at initialization to the db on a call

conn->rc = SQLAllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, &henv);

Conn->rc is the result code  (-1 indicating failure), henv would normally contain the returned handle.
Any attempt to retrieve additional error information with SQLRegDiagRec fails. I am not sure why but could be because the error retrieval mechanisms also cannot handle a UTF-8 code page.

So to elaborate on Harald's correct interpretation of my original post, the failure mode does not involve any Tcl code at all. No data is passed so no encoding mismatch is involved. Your conjecture about Tcl passing it UTF-8 encoded data is off base. It never gets that far to even attempt that (not even as far as SqlConnect).

Your comment that Tcl should respect the IBM API is spot on. Except Tcl 9 does not! It is expected that the user's real code page will be retrieved with GetACP() and Tcl 9's manifest hijacks that API to force a UTF-8 return with no recourse or workaround available. How would you suggest the above basic and fundamental API call be written to work around the issue even if the extension writer is willing?

To remove Tcl from the equation, also see this issued reported with R 4.2 and DB2 drivers - RODBC odbcConnect fails for ibm db2 connection in Prairie Trillium · Issue #10509 · rstudio/rstudio<https://github.com/rstudio/rstudio/issues/10509>. Guess what R 4.2 and Tcl 9 have in common? The UTF-8 manifest. The motivation for R 4.2 using the manifest is spelled out in the link I included in the TIP. It is completely inapplicable to Tcl for reasons I also outlined in the TIP.

I have not looked at the link you included but the failing extension code demonstrating the error is attached.

/Ashok


________________________________
From: Jan Nijtmans

My guess is that the API used by this IBM DLL is expecting
cp1252, but Tcl is providing/retrieving utf-8-encoded data
for it. Do you know where the Tcl wrapper-code for this
IBM dll is? I think there's nothing wrong with the IBM dll,
It's just that Tcl should respect the IBM API.

Is the code here?
      https://github.com/memmertoIBM/db2tcl/

Regards,
        Jan Nijtmans


_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] TIP 716 ready for comments

[Harald O. <har...@el...>]

Am 25.04.2025 um 11:46 schrieb Jan Nijtmans:
> Op vr 25 apr 2025 om 09:47 schreef Harald Oehlmann:
>> Dear Jan,
>> thanks for the action. I will test later or next week.
>> The example given by Ashok was a driver to the IBM data base which was
>> loaded as a dll into TCL. This 3rd party driver did not work any more
>> due to the manifest change.
> 
> My guess is that the API used by this IBM DLL is expecting
> cp1252, but Tcl is providing/retrieving utf-8-encoded data
> for it. Do you know where the Tcl wrapper-code for this
> IBM dll is? I think there's nothing wrong with the IBM dll,
> It's just that Tcl should respect the IBM API.
> 
> Is the code here?
>        https://github.com/memmertoIBM/db2tcl/
> 
> Regards,
>          Jan Nijtmans
To my knowledge, the dll does not work any more internally, as the 
interface DLL to the DB depends on it.
The issue is not related to TCL expect that the manifest value makes it 
not working any more.
To my knowledge, the concerned user case has no direct TCL interface in 
the sense, that data from TCL is transfered to the data base and vice 
versa. That is why it took so long to find the issue.

Harald

Re: [TCLCORE] TIP 716 ready for comments

[Jan N. <jan...@gm...>]

Op vr 25 apr 2025 om 09:47 schreef Harald Oehlmann:
> Dear Jan,
> thanks for the action. I will test later or next week.
> The example given by Ashok was a driver to the IBM data base which was
> loaded as a dll into TCL. This 3rd party driver did not work any more
> due to the manifest change.

My guess is that the API used by this IBM DLL is expecting
cp1252, but Tcl is providing/retrieving utf-8-encoded data
for it. Do you know where the Tcl wrapper-code for this
IBM dll is? I think there's nothing wrong with the IBM dll,
It's just that Tcl should respect the IBM API.

Is the code here?
      https://github.com/memmertoIBM/db2tcl/

Regards,
        Jan Nijtmans

Re: [TCLCORE] TIP 716 ready for comments

[Harald O. <har...@el...>]

Am 25.04.2025 um 09:37 schrieb Jan Nijtmans:
> Op di 15 apr 2025 om 21:02 schreef apnmbx-public--- via Tcl-Core
> <tcl...@li...>:
>> The rules are (as per TIP 716):
>>
>> Use the Unicode Windows API as Tcl/Tk already do. Then these issues do not arise.
>> If you have to use ANSI API’s, use Windows encoding settings, not Tcl encoding settings.
> 
> So, let's establish that TIP #716 changes the rules!
> 
> I did some investigation which extensions will be broken by TIP #716.
> But - first -
> let's remember the rules how they were in  Tcl 8.x.
> 
> 1) Use the Unicode Windows API as Tcl/Tk already do. Then these issues
> do not arise:
>         Tcl_WinUtfToTChar(str, strlen(str), &ds);
>         _wfopen((WCHAR *)Tcl_DStringValue(ds), L"r")
> 
> 2) If you have to use ANSI API’s, use the following:
>         Tcl_DStringInit(&ds);
>         Tcl_DStringToExtenal(NULL, str, -1, &ds);
>         fopen(Tcl_DStringValue(ds), "r")
> 
> 3) If you only care about ASCII characters, just do:
>         fopen(str, "r")
> 
> In Tcl 9.0.0/9.0.3, all 3 rules start working correctly with UTF-8
> (except when using
> the GDI API but there is no known extension using that). Whatever
> choice is made, 1), 2) or 3), all work fine with Tcl 9.0.0/9.0.1.  If
> TIP #716 is
> accepted, 2) and 3) won't work correctly with characters outside ASCII any more.
> Which extensions will suffer from that? I found the following:
> 
>      blt/src/bltWinImage.c (line 1194)
>      expect/exp_main_sub.c (line 813)
>      pikchr/pikchr.c (line 8136)
>      tkimg/sgi/sgi.c (line 1512)
>      VecTcl/WavReader/generic/wavreader.c (line 88)
> 
> So, as long as BLT images, pikchr, sgi or waveform filenames
> use ASCII characters only, nothing to worry about. When running
> in Tcl 9.0.0 or 9.0.1, nothing to worry about either. But with
> TIP #716, the rules change, and 2) or 3) can no longer handle
> characters outside ASCII.
> 
> Hope this helps,
>          Jan Nijtmans
Dear Jan,
thanks for the action. I will test later or next week.
The example given by Ashok was a driver to the IBM data base which was 
loaded as a dll into TCL. This 3rd party driver did not work any more 
due to the manifest change.
There is nothing TCL related here. The only issue is, that any load DLL 
is influenced by the manifest.
This is only fixable by removing the UTF-8 manifest.
There is no problem with this, as TCL does not rely on it.
And no compatible extion should rely on it.
It is a work around, not a feature.

Thanks for all,
Harald

Re: [TCLCORE] TIP 716 ready for comments

[Jan N. <jan...@gm...>]

Op di 15 apr 2025 om 21:02 schreef apnmbx-public--- via Tcl-Core
<tcl...@li...>:
> The rules are (as per TIP 716):
>
> Use the Unicode Windows API as Tcl/Tk already do. Then these issues do not arise.
> If you have to use ANSI API’s, use Windows encoding settings, not Tcl encoding settings.

So, let's establish that TIP #716 changes the rules!

I did some investigation which extensions will be broken by TIP #716.
But - first -
let's remember the rules how they were in  Tcl 8.x.

1) Use the Unicode Windows API as Tcl/Tk already do. Then these issues
do not arise:
       Tcl_WinUtfToTChar(str, strlen(str), &ds);
       _wfopen((WCHAR *)Tcl_DStringValue(ds), L"r")

2) If you have to use ANSI API’s, use the following:
       Tcl_DStringInit(&ds);
       Tcl_DStringToExtenal(NULL, str, -1, &ds);
       fopen(Tcl_DStringValue(ds), "r")

3) If you only care about ASCII characters, just do:
       fopen(str, "r")

In Tcl 9.0.0/9.0.3, all 3 rules start working correctly with UTF-8
(except when using
the GDI API but there is no known extension using that). Whatever
choice is made, 1), 2) or 3), all work fine with Tcl 9.0.0/9.0.1.  If
TIP #716 is
accepted, 2) and 3) won't work correctly with characters outside ASCII any more.
Which extensions will suffer from that? I found the following:

    blt/src/bltWinImage.c (line 1194)
    expect/exp_main_sub.c (line 813)
    pikchr/pikchr.c (line 8136)
    tkimg/sgi/sgi.c (line 1512)
    VecTcl/WavReader/generic/wavreader.c (line 88)

So, as long as BLT images, pikchr, sgi or waveform filenames
use ASCII characters only, nothing to worry about. When running
in Tcl 9.0.0 or 9.0.1, nothing to worry about either. But with
TIP #716, the rules change, and 2) or 3) can no longer handle
characters outside ASCII.

Hope this helps,
        Jan Nijtmans

Re: [TCLCORE] TIP 716 ready for comments

[Jan N. <jan...@gm...>]

Op di 22 apr 2025 om 10:04 schreef Harald Oehlmann:
> Jan,
> thanks! I have tested it already. It had no influence on GDI, what was
> explained by Ashok.
> The long awaited "encoding user" command is effective and will allow to
> be compatible with 8.6 by:
>         source -encoding [encoding user] $file
> what is a big step forward.
> I think, the removal of utf-8 from the certificate is a good thing.
> This should also happen for "wish" which may be explicitly stated in the
> TIP.

Yes, I am aware that you tested it already. But I'm asking to
retest, because "trunk" has been merged in the TIP branch.
This has effect on some testcases, you'll see.

B.T.W: I'm all in favour of a new "encoding user" subcommand,
but the other parts of TIP #716 have some side-effects which
are not mentioned in the TIP text. I can explain, but it's
better you see it with your own eyes first.

Thanks!
          Jan Nijtmans

Re: [TCLCORE] TIP 716 ready for comments

[Harald O. <har...@el...>]

Am 21.04.2025 um 15:36 schrieb Jan Nijtmans:
> Op ma 21 apr 2025 om 07:19 schreef Ashok Nadkarni via Tcl-Core:
>>
>> I plan a CFV on TIP 716 at the end of this month.
> 
> Harald, could you try the TIP #716 branch on the Windows environments you have?
> 
> Thanks!
>       Jan Nijtmans
Jan,
thanks! I have tested it already. It had no influence on GDI, what was 
explained by Ashok.
The long awaited "encoding user" command is effective and will allow to 
be compatible with 8.6 by:
	source -encoding [encoding user] $file
what is a big step forward.
I think, the removal of utf-8 from the certificate is a good thing.
This should also happen for "wish" which may be explicitly stated in the 
TIP.

Thanks for all,
Harald

Re: [TCLCORE] Looking for help accessing Tk internals on Windows

[Andrew C <jx...@gm...>]

That did the trick! Many thanks Jan.

On Mon, Apr 21, 2025, 9:34 a.m. Jan Nijtmans <jan...@gm...> wrote:

> Op ma 21 apr 2025 om 14:03 schreef Andrew C:
> >
> > Hello everyone!
> >
> > I'm modifying Tkhtml and am trying to use Cairo to add border-radius
> support but am hitting some roadblocks.
> >
> > On Linux, it is easy; I simply give Cairo the Xlib Pixmap to draw on.
> However, on Windows, I need to give Cairo a Win32 HDC. Is there any way to
> get a Pixmap's corresponding HDC on Windows?
> >
> > Something like TkWinGetDrawableDC might be exactly what I need, but
> using it gives me a slew of errors, including "error: 'TkIntPlatStubs' has
> no member named 'tkWinGetDrawableDC'".
> >
> > Any thoughts would be greatly appreciated.
>
> Try to add this at the beginning of the file:
>     #include "tkWinInt.h"
> That should help.
>
> Take care!
>       Jan Nijtmans
>

Re: [TCLCORE] Looking for help accessing Tk internals on Windows

[Arjen M. <Arj...@de...>]

Hi Andrew,

I just checked the source code and that particular entry is there in the tkIntPlatStubs structure, but only if the preprocessor macros _WIN32 or __CYGWIN__ are defined. Are you sure that these macros are indeed defined?

Regards,

Arjen

From: Andrew C <jx...@gm...>
Sent: maandag 21 april 2025 18:03
To: tcl...@li...
Subject: [TCLCORE] Looking for help accessing Tk internals on Windows

Caution: This message was sent from outside of Deltares. Please do not click links or open attachments unless you recognize the source of this email and know the content is safe. Please report all suspicious emails to "Ser...@de...<mailto:Ser...@de...>" as an attachment.

Hello everyone!

I'm modifying Tkhtml and am trying to use Cairo to add border-radius support but am hitting some roadblocks.

On Linux, it is easy; I simply give Cairo the Xlib Pixmap to draw on. However, on Windows, I need to give Cairo a Win32 HDC. Is there any way to get a Pixmap's corresponding HDC on Windows?

Something like TkWinGetDrawableDC might be exactly what I need, but using it gives me a slew of errors, including "error: 'TkIntPlatStubs' has no member named 'tkWinGetDrawableDC'".

Any thoughts would be greatly appreciated.

Thank you!
Andrew
DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

Re: [TCLCORE] TIP 716 ready for comments

[Jan N. <jan...@gm...>]

Op ma 21 apr 2025 om 07:19 schreef Ashok Nadkarni via Tcl-Core:
>
> I plan a CFV on TIP 716 at the end of this month.

Harald, could you try the TIP #716 branch on the Windows environments you have?

Thanks!
     Jan Nijtmans

Re: [TCLCORE] Looking for help accessing Tk internals on Windows

[Jan N. <jan...@gm...>]

Op ma 21 apr 2025 om 14:03 schreef Andrew C:
>
> Hello everyone!
>
> I'm modifying Tkhtml and am trying to use Cairo to add border-radius support but am hitting some roadblocks.
>
> On Linux, it is easy; I simply give Cairo the Xlib Pixmap to draw on. However, on Windows, I need to give Cairo a Win32 HDC. Is there any way to get a Pixmap's corresponding HDC on Windows?
>
> Something like TkWinGetDrawableDC might be exactly what I need, but using it gives me a slew of errors, including "error: 'TkIntPlatStubs' has no member named 'tkWinGetDrawableDC'".
>
> Any thoughts would be greatly appreciated.

Try to add this at the beginning of the file:
    #include "tkWinInt.h"
That should help.

Take care!
      Jan Nijtmans

[TCLCORE] Looking for help accessing Tk internals on Windows

[Andrew C <jx...@gm...>]

Hello everyone!

I'm modifying Tkhtml and am trying to use Cairo to add border-radius
support but am hitting some roadblocks.

On Linux, it is easy; I simply give Cairo the Xlib Pixmap to draw on.
However, on Windows, I need to give Cairo a Win32 HDC. Is there any way to
get a Pixmap's corresponding HDC on Windows?

Something like TkWinGetDrawableDC might be exactly what I need, but using
it gives me a slew of errors, including "error: 'TkIntPlatStubs' has no
member named 'tkWinGetDrawableDC'".

Any thoughts would be greatly appreciated.

Thank you!
Andrew

Re: [TCLCORE] TIP 716 ready for comments

[Ashok N. <apn...@ya...>]

I plan a CFV on TIP 716 at the end of this month.

Before then, I'm particularly interested in any opinions on the questions listed in the TIP regarding TCL_EXEC_ENCODING environment variable. Of course, that is only relevant for those in favor of the rest of the TIP!

Thanks,
Ashok


________________________________
From: apnmbx-public--- via Tcl-Core
Sent: Monday, April 14, 2025 10:21 AM
To: tcl...@li...
Subject: [TCLCORE] TIP 716 ready for comments


TIP 716: New command "encoding user", remove UTF-8 manifest setting on Windows<https://core.tcl-lang.org/tips/doc/trunk/tip/716.md> is ready for comments. It proposes reversion of a change made in 9.0 to tclsh and wish Windows manifests while keeping compatibility with 9.0.{0,1}.



Apologies for my usual verbosity, but when I brought this up in the mailing list prior to the previous release, the comments wandered into why UTF-8 should be the default. I've tried to better explain that is not the issue.



I will point out that the original change to the manifest, which made UTF-8 the default on Win 10 1903+ and Win 11, should have been TIP'ed as it overrides user settings and is a change in behavior of a public API and command. It's water under the bridge now that 9.0 has shipped so the TIP maintains status quo and only changes the implementation. It also adds a new encoding user command and an -encoding option to exec as a workaround for compatibility issues introduced by forcing a UTF-8 default.



Note the implementation in the tip-716 branch is mostly complete but not ready for review. I am only looking for comments on the proposal before proceeding further with tests and docs.



/Ashok

Re: [TCLCORE] Project tk_collect_test_utils, report and invitation for final review

[elns <el...@xs...>]

L.S.

There is one more week of opportunity to submit comments.
So, the deadline is at:

      clock format 1745841600

(being 27-apr-2025 24:00 in the last time zone of our planet.)

By the way:
I applied a few more improvements since the initial request for comments.
Therefore, all check-in's since that initial request carry the tag FINAL_REVIEW.
Please choose the latest check-in if you decide to review.

Best regards,
Erik Leunissen
--

Re: [TCLCORE] Speed/memory tradeoffs on list operations

[Ashok N. <apn...@ya...>]

Hi Brian,

Thanks for your comments. I remember you mentioning the tradeoff before as well.

As it turns out, for the existing list commands, where the objects already exist, the indexing overhead of abstract lists is minor enough to not be an issue while memory savings are still present. So I'm experimenting with just the built-in list commands for now.

We'll see how that goes and whether it is worthwhile.

/Ashok


________________________________
From: Brian Griffin


Hi Ashok.,

Nice analysis! Thanks!

I've long been aware of the time/space tradeoff here. That's a decision one needs to make when implementing specific applications. I've always thought that abstract lists are best suited in cases where the dataset is effectively large, and already optimized for efficiency on some axis. And, the consumption is likely (but not necessarily) sparse.

One example is an acyclic graph that represents a very large number of paths, with a much smaller number of nodes. The use case in practice may only explore some subset of the paths. The abstract list interface can provide access to the graph using existing Tcl script level commands, i.e., not  having to create a new custom set of commands to access the custom graph data.

I didn't answer your question directly. I think it depends on the specific application case.


That's my 2¢
-Brian


/Ashok
_______________________________________________
Tcl-Core mailing list
Tcl...@li...<mailto:Tcl...@li...>
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Tcl_WideInt and long long

[Jan N. <jan...@gm...>]

Op wo 16 apr 2025 om 21:59 schreef Donald G Porter:
> If it is true, what do people think about work for Tcl 9.1 to tear
> down constructions that suggest otherwise, like conditional compilation
> on the value TCL_WIDE_INT_TYPE, etc. ?  What do people think about
> including that simplification work also in Tcl 9.0 patch releases?

You mean, something like this?
    <https://core.tcl-lang.org/tcl/info/tclwideint-is-longlong>

In my opinion, overriding it using TCL_WIDE_INT_TYPE
(where sizeof(TCL_WIDE_INT_TYPE) != sizeof(long long))
creates a binary incompatible version of Tcl. We should
not allow it. It is not even documented anywhere
(it shouldn't be!). That means, we can safely remove it.

Hope this helps,
       Jan Nijtmans

Re: [TCLCORE] Speed/memory tradeoffs on list operations

[Brian G. <bri...@ea...>]

On Apr 17, 2025, at 04:18, apnmbx-public--- via Tcl-Core <tcl...@li...> wrote:

All,

While Tcl 9 supports strings, lists etc. greater than 2**32, even some
basic operations on data of much smaller lengths has practical
difficulties around execution time and memory space. I am experimenting
with approaches to improve efficiency and looking for opinions and some
guidance on time and memory tradeoffs, starting with list operations
(both existing ones like lreverse as well as new ones like merge, zip).

The most obvious way to improve performance is of course to implement
more commands in C. Consider the operation of "zipping" lists written
in script as "lmap x $l1 y $l2 {list $x $y}". Measuring speed (timerate)
and memory (/proc/self/statm) when zipping lists of size 100000 shows:

Memory pre: 3070 1752 1083 1 0 752 0
lmap $a $b: 46985.7 µs/# 22 # 21.283 #/sec 1033.686 net-ms
Memory post lmap: 7701 6293 1083 1 0 5383 0

The same operation implemented as a "lzip" command in C, is obviously
faster.

Memory pre: 3314 2066 1102 1 0 992 0
lzip $a $b: 13895.7 µs/# 72 # 71.965 #/sec 1000.491 net-ms
Memory post lzip: 7747 6417 1102 1 0 5425 0

Note however the memory requirements are (naturally) the same.

Going further, implementing lzip as a TIP 636 abstract list with an
obvious implementation,

Memory pre: 3314 2076 1112 1 0 992 0
lzip $a $b: 0.270993 µs/# 3298677 # 3690129 #/sec 893.919 net-ms
Memory post lzip: 3314 2076 1112 1 0 992 0

Notice (a) orders of magnitude faster (in fact constant time
irrespective of list length), and (b) *zero* memory cost as the list
arguments' storage is simply referenced.

Alas, there is no free lunch.

The cost of indexing and iterating over the list as "foreach x $l {}" is
significantly higher with abstract lists.

traditional list:
iterate: 18256.1 µs/# 55 # 54.776 #/sec 1004.083 net-ms
index: 0.056268 µs/# 10757211 # 17772142 #/sec 605.285 net-ms
Memory post pure iterate: 7747 6483 1102 1 0 5425 0

abstract list:
iterate: 29596.3 µs/# 34 # 33.788 #/sec 1006.274 net-ms
index: 0.169833 µs/# 4926811 # 5888123 #/sec 836.737 net-ms
Memory post pure iterate: 3314 2076 1112 1 0 992 0

Thus cost of construction+iteration is roughly a wash with the abstract
list implementation coming out slightly ahead. However, if iterating
multiple times, the traditional lists will be much more performant.

The difference (which shows up even for lseq) arises because the
non-abstract implementation is basically a direct access into the C
array holding the elements whereas abstract lists have to construct
elements on request.

Note however, the memory disparity remains.

[The above measured using extension. There might be some improvement in
the core with some byte code support but I do not think it would be
substantial].

Given the lack of application benchmarks, it's difficult to make a call
one way or another as to real-life impact. May be the much lower memory
requirements means better cache properties? Conversely, if application
retains constructed elements retrieved during iteration, perhaps the
memory savings are illusory?

Of course, it is possible to choose at runtime between the two internal
representations (abstract/non-abstract) based on the length of the lists
so only "large" lists longer than some threshold would be abstracted.

I am looking for opinions as to all this would be worth
doing ("this" being abstract lists implementing zip, lreverse, concat,
merge, range, and other list operations). There are really two
questions:

1. Is it worth moving above operations into C?

2. If so, are abstract lists worth the construction+memory vs iteration
speed tradeoff?

I'm inclined to say yes to both, particularly as abstract lists will
allow operations on lists too large for the other methods (what was the
point of increasing size limits otherwise?).

Alternatively, we could do just 1. benefiting speed but not memory.

Or neither, which would save me a bunch of work and avoid code bloat in
the core.

Thoughts anyone?

Hi Ashok.,

Nice analysis! Thanks!

I've long been aware of the time/space tradeoff here. That's a decision one needs to make when implementing specific applications. I've always thought that abstract lists are best suited in cases where the dataset is effectively large, and already optimized for efficiency on some axis. And, the consumption is likely (but not necessarily) sparse.

One example is an acyclic graph that represents a very large number of paths, with a much smaller number of nodes. The use case in practice may only explore some subset of the paths. The abstract list interface can provide access to the graph using existing Tcl script level commands, i.e., not  having to create a new custom set of commands to access the custom graph data.

I didn't answer your question directly. I think it depends on the specific application case.


That's my 2¢
-Brian


/Ashok
_______________________________________________
Tcl-Core mailing list
Tcl...@li...<mailto:Tcl...@li...>
https://lists.sourceforge.net/lists/listinfo/tcl-core

[TCLCORE] Speed/memory tradeoffs on list operations

[<apn...@ya...>]

All,

 

While Tcl 9 supports strings, lists etc. greater than 2**32, even some

basic operations on data of much smaller lengths has practical

difficulties around execution time and memory space. I am experimenting

with approaches to improve efficiency and looking for opinions and some

guidance on time and memory tradeoffs, starting with list operations

(both existing ones like lreverse as well as new ones like merge, zip).

 

The most obvious way to improve performance is of course to implement

more commands in C. Consider the operation of "zipping" lists written

in script as "lmap x $l1 y $l2 {list $x $y}". Measuring speed (timerate)

and memory (/proc/self/statm) when zipping lists of size 100000 shows:

 

Memory pre: 3070 1752 1083 1 0 752 0

lmap $a $b: 46985.7 µs/# 22 # 21.283 #/sec 1033.686 net-ms

Memory post lmap: 7701 6293 1083 1 0 5383 0

 

The same operation implemented as a "lzip" command in C, is obviously

faster.

 

Memory pre: 3314 2066 1102 1 0 992 0

lzip $a $b: 13895.7 µs/# 72 # 71.965 #/sec 1000.491 net-ms

Memory post lzip: 7747 6417 1102 1 0 5425 0

 

Note however the memory requirements are (naturally) the same.

 

Going further, implementing lzip as a TIP 636 abstract list with an

obvious implementation,

 

Memory pre: 3314 2076 1112 1 0 992 0

lzip $a $b: 0.270993 µs/# 3298677 # 3690129 #/sec 893.919 net-ms

Memory post lzip: 3314 2076 1112 1 0 992 0

 

Notice (a) orders of magnitude faster (in fact constant time

irrespective of list length), and (b) *zero* memory cost as the list

arguments' storage is simply referenced.

 

Alas, there is no free lunch.

 

The cost of indexing and iterating over the list as "foreach x $l {}" is

significantly higher with abstract lists.

 

traditional list:

iterate: 18256.1 µs/# 55 # 54.776 #/sec 1004.083 net-ms

index: 0.056268 µs/# 10757211 # 17772142 #/sec 605.285 net-ms

Memory post pure iterate: 7747 6483 1102 1 0 5425 0

 

abstract list:

iterate: 29596.3 µs/# 34 # 33.788 #/sec 1006.274 net-ms

index: 0.169833 µs/# 4926811 # 5888123 #/sec 836.737 net-ms

Memory post pure iterate: 3314 2076 1112 1 0 992 0

 

Thus cost of construction+iteration is roughly a wash with the abstract

list implementation coming out slightly ahead. However, if iterating

multiple times, the traditional lists will be much more performant.

 

The difference (which shows up even for lseq) arises because the

non-abstract implementation is basically a direct access into the C

array holding the elements whereas abstract lists have to construct

elements on request.

 

Note however, the memory disparity remains.

 

[The above measured using extension. There might be some improvement in

the core with some byte code support but I do not think it would be

substantial].

 

Given the lack of application benchmarks, it's difficult to make a call

one way or another as to real-life impact. May be the much lower memory

requirements means better cache properties? Conversely, if application

retains constructed elements retrieved during iteration, perhaps the

memory savings are illusory?

 

Of course, it is possible to choose at runtime between the two internal

representations (abstract/non-abstract) based on the length of the lists

so only "large" lists longer than some threshold would be abstracted.

 

I am looking for opinions as to all this would be worth

doing ("this" being abstract lists implementing zip, lreverse, concat,

merge, range, and other list operations). There are really two

questions:

 

1. Is it worth moving above operations into C?

 

2. If so, are abstract lists worth the construction+memory vs iteration

speed tradeoff?

 

I'm inclined to say yes to both, particularly as abstract lists will

allow operations on lists too large for the other methods (what was the

point of increasing size limits otherwise?).

 

Alternatively, we could do just 1. benefiting speed but not memory.

 

Or neither, which would save me a bunch of work and avoid code bloat in

the core.

 

Thoughts anyone?

 

/Ashok

Re: [TCLCORE] Tcl_WideInt and long long

[<apn...@ya...>]

Thanks for bringing this up. It has always been a source of confusion for me so I'm all for eliminating it. See my question and Jan's response at https://sourceforge.net/p/tcl/mailman/tcl-core/thread/00c701db4146%24f0ec3650%24d2c4a2f0%24%40yahoo.com/#msg58846548

I am a little hesitant about defining Tcl_WideInt as int64_t (if I understood Donal correctly) rather than long long, in case some platform typedefs int64_t as long. Even though that is more elegant and cleaner in some respect, I suspect it would cause gcc warnings in extensions that are passing Tcl_WideInt to (external functions) that expect long long * (see above thread) or use it with %lld.

/Ashok

-----Original Message-----
From: Donald G Porter via Tcl-Core <tcl...@li...> 
Sent: Thursday, April 17, 2025 12:55 AM
To: Tcl List Core <tcl...@li...>
Subject: [TCLCORE] Tcl_WideInt and long long


This question may be fully addressed and answered in some TIP, and if
so it's a sufficient answer to point me to it.

In Tcl 9, is it true that C type Tcl_WideInt is always the same as
C type (long long int) ?  Likewise Tcl_WideUInt is always same as
(unsigned long long int) ?

If it is not true, what are the exceptions and why is it important to
continue supporting them?

If it is true, what do people think about work for Tcl 9.1 to tear
down constructions that suggest otherwise, like conditional compilation
on the value TCL_WIDE_INT_TYPE, etc. ?  What do people think about
including that simplification work also in Tcl 9.0 patch releases?

-- 
| Don Porter            Applied and Computational Mathematics Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|



_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core