tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] Review request: [0439e1e1a3]

[Reinhard M. <rei...@m4...>]

Am 08.07.2024 11:39, schrieb Jan Nijtmans:

> My proposal:
>     % set x [lseq 1 11111111111111];set y 5
>     5
>     % expr {$x+$y}
>     can't use a list as operand of "+"

If possible without a considerable performance hit, I'd suggest to add 
information about which operand it was to the error message:

     % expr {$x+$y}
     can't use a list as the left operand of "+"
     % expr {$y+$x}
     can't use a list as the right operand of "+"

cu
Reinhard

[TCLCORE] Review request: [0439e1e1a3]

[Jan N. <jan...@gm...>]

Hi all,

This ticket is meant for functions which require a
single numerical argument. If they are provided
a (potentially very long) list, it might take a long
time to generate the string representation of the
list, only discovering afterwards it won't work.

Example:
    $ tclsh9.0
    % set x [lseq 1 11111111111111];set y 5
    5
    % expr {$x+$y}
(this simply hangs for a long time)

My proposal:
    % set x [lseq 1 11111111111111];set y 5
    5
    % expr {$x+$y}
    can't use a list as operand of "+"

That means, if the argument can be represented
as a list (with >1 elements), it won't be printed as
part of the error-message any more.

Please have a look:
    <https://core.tcl-lang.org/tcl/tktview/0439e1e1a3>

If there are no objections, I'll commit this in
a few days.

Thanks!
        Jan NIjtmans

[TCLCORE] July 9 2024 Tcl monthly virtual meetup notice

[Steve L. <st...@di...>]

A reminder that the July Tcl virtual meetup will be  held Tuesday July 9th 2024 at [clock format 1720573200] - Tuesday 6pm US West, 8pm US Central, 9pm US East, Wednesday 1am UTC, 2am UK, 3am Western Europe, 6:30am India, 9am Australia West / Singapore / China, 10am Japan, 11am Australia East, 1pm New Zealand

Details (including how to connect) are available via https://wiki.tcl-lang.org/page/Monthly+Virtual+Meetup

-- Steve

Re: [TCLCORE] Migration tool for Tcl 9

[<apn...@ya...>]

V0.2 of the migration tool has been released following feedback from Paul and Andreas (thanks both).

 

https://github.com/apnadkarni/tcl9-migrate/releases

 

Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md

 

Additional feedback on false positive/negatives appreciated.

 

/Ashok

 

From: apn...@ya... <apn...@ya...> On Behalf Of apn...@ya...
Sent: Tuesday, July 2, 2024 10:44 AM
To: apn...@ya...; 'Paul Obermeier' <pa...@po...>; tcl...@li...
Subject: RE: [TCLCORE] Migration tool for Tcl 9

 

Oops, I mean false positive, not false negative 😊 It should not flag a warning.

 

From: apnmbx-public--- via Tcl-Core <tcl...@li... <mailto:tcl...@li...> > 
Sent: Tuesday, July 2, 2024 8:01 AM
To: 'Paul Obermeier' <pa...@po... <mailto:pa...@po...> >; tcl...@li... <mailto:tcl...@li...> 
Subject: Re: [TCLCORE] Migration tool for Tcl 9

 

Yes, that would seem to be a false negative. Does not seem to account for reference to array elements when the array variable is declared. Should be fixable..I think.

 

Thanks for the feedback

 

From: Paul Obermeier <pa...@po... <mailto:pa...@po...> > 
Sent: Tuesday, July 2, 2024 3:14 AM
To: tcl...@li... <mailto:tcl...@li...> 
Subject: Re: [TCLCORE] Migration tool for Tcl 9

 

For the following simple test script:
namespace eval poImg {
    variable DrawPos

    set DrawPos(x) 0
    set DrawPos(y) 0
}

I get the following warnings from tcl9-migrate:
Line   4: W Variable "DrawPos(x)" possibly set without a variable or global declaration within namespace ::poImg. [UNDECLARED]
Line   5: W Variable "DrawPos(y)" possibly set without a variable or global declaration within namespace ::poImg. [UNDECLARED]

Is this a false negative?

Paul

Am 01.07.2024 um 22:42 schrieb Paul Obermeier:

Hi Ashok,

thanks for this great tool.

I found several Tcl9 related issues (esp. [RELATIVENS] and [UNDECLARED]) in my Tcl sources,
which are otherwise difficult to find.
The runtime checker works, too. 

Best regards,
Paul

Am 01.07.2024 um 09:44 schrieb apnmbx-public--- via Tcl-Core:

All,

 

I've been working on a tool to help with migration of Tcl 8 scripts to Tcl 9. It includes a static checker based on Nagelfar as well as a runtime shim that fixes up incompatibilities like i/o encoding errors, tilde expansion, package requirements etc., allowing the scripts to be loaded while emitting warnings.

 

Repository: https://github.com/apnadkarni/tcl9-migrate

Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md

Download: https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1

 

I'm looking for testers and feedback on both the tool as well as documentation (explanation of the warnings). I'd appreciate testing with your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above repository.

 

Be warned it has (and always will have) false positives and false negatives but the hope is it will nevertheless be useful in saving time.

 

Thanks

 

/Ashok

 

_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core





_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core

 

[TCLCORE] Tklib 0.8 Released

[Andreas K. <and...@gm...>]

See
	https://core.tcl-lang.org/tklib/technote/f6f9eed465b76f73fd066a65e4275b9552a66
4c7

for the release note with details and links to archives.

Alternate archive site at

	https://sourceforge.net/projects/tcllib/files/tklib/0.8/

Since the release candidate this had Ashok's migration checker applied.

Updated references and information on

	http://www.tcl-lang.org/
	http://www.tcl-lang.org/software/tklib/
	https://core.tcl-lang.org/tklib/doc/trunk/embedded/index.md
	https://core.tcl-lang.org/tklib/wiki?name=Releases
	https://wiki.tcl-lang.org/page/tklib

See (some of) you in Vienna.

-- 
Happy Tcling,
	Andreas Kupries <and...@gm...>
			<https://core.tcl-lang.org/akupries/>
			<https://akupries.tclers.tk/>
	Developer @	SUSE Software Solutions Germany GmbH
-------------------------------------------------------------------------------

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Jan N. <jan...@gm...>]

Op vr 5 jul 2024 om 22:27 schreef Kevin Walzer:

> I still believe it’s best to stick with this now and not revert the
> manifest. My reasoning is that the most opportune moment for this kind of
> disruptive change is now, when all the other disruptive changes are being
> introduced. And I also think there is value in Tcl being ahead of all the
> other languages—for once.
>

+1

That said, there is currently a bug in the handling of "info hostname"
on windows: I would expect this command to give the same answer,
independent of the "encoding system" setting. That's currently not
the case. It should be possible to set "encoding system cp1252",
even in Tcl 9.0, to change back the system encoding.

We can handle this simply as a bug to be solved. This
"info hostname" is the only bug of this type I'm aware of.
I don't think this is urgent enough to delay 9.0b3 for.....

Regards,
      Jan Nijtmans

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Poor Y. <org...@po...>]

On 2024-07-05 17:05, apnmbx-public--- via Tcl-Core wrote:
> Kevin, I think you are viewing the Build 1903 crossover in reverse.
> The compatibility issue (like the one illustrated below) is not with
> older builds, it's with tclsh on Windows builds AFTER (and including)
> 1903. So 1903 being old is not relevant. That is only the starting
> point for when the issue arises.
> 
> Georg, I'm afraid if it was in fact a win-win situation, we would not
> be debating this :-)
> 
> Compare tclsh 8.6
> 
> c:\src>findstr abc x.x | tclsh readpipe.tcl
> 
> abcé
> 
> with tclsh 9 (on the same Win10 system post-build 1903)
> 
> c:\src>findstr abc x.x | d:\tcl\90\x64-debug\bin\tclsh90.exe
> readpipe.tcl
> 
> error reading "stdin": invalid or incomplete multibyte or wide
> character

This is an example of how Microsoft doesn't even follow its own advice
to developers to set the code page for all applications to utf-8.  All
things considered, the best move for Tcl here is not to wait for
Microsoft to get its act together with all its own utilities, but to
move ahead with standardizing on utf-8 as the universal system encoding.
Application developers can adjust it manually if they feel the need.

-- 
Yorick

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Kevin W. <kw...@co...>]

<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><img width="1" height="1" src='https://fedbdhd.r.af.d.sendibt2.com/tr/op/zj2dblFoBeUc2w_mCSJHWuNLsuksYtOQQquJla78TR6toqx8Glf2lVP3-ZSVlh8CxbOSAXf3WZH6HYY-W-MDVzL8Rw6PnDimysOmjyABIDq-9kQdOdjZfLWTT_zvJH_jltOk-yl8XvsfZuVgNk6r0rmsbCBPc-70Njf63rylB5wo6zpFAeZWq5p7jR4n5ItaImCeAPjZUM_cxBNvfVz979os-e_XQ8Vg' /><div dir="ltr"></div><div dir="ltr">Ashok,</div><div dir="ltr">&nbsp;&nbsp;</div><div dir="ltr">Ok, I understand your point better about the timing. I still believe it’s best to stick with this now and not revert the manifest. My reasoning is that the most opportune moment for this kind of disruptive change is now, when all the other disruptive changes are being introduced. And I also think there is value in Tcl being ahead of all the other languages—for once.&nbsp;</div><div dir="ltr"><br></div><div dir="ltr">Thanks,</div><div dir="ltr">Kevin</div><div dir="ltr"><br><blockquote type="cite">On Jul 5, 2024, at 10:06 AM, apnmbx-public--- via Tcl-Core &lt;tcl...@li...&gt; wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta name="Generator" content="Microsoft Word 15 (filtered medium)"><style>@font-face { font-family: "Cambria Math"; }
@font-face { font-family: Calibri; }
@font-face { font-family: Aptos; }
@font-face { font-family: Consolas; }
p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in; font-size: 11pt; font-family: Aptos, sans-serif; }
a:link, span.MsoHyperlink { color: rgb(70, 120, 134); text-decoration: underline; }
pre { margin: 0in; font-size: 10pt; font-family: "Courier New"; }
span.HTMLPreformattedChar { font-family: Consolas, serif; }
span.EmailStyle22 { font-family: Aptos, sans-serif; color: windowtext; }
.MsoChpDefault { font-size: 10pt; }
@page WordSection1 { size: 8.5in 11in; margin: 1in; }
div.WordSection1 { page: WordSection1; }</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--><div class="WordSection1"><p class="MsoNormal">Kevin, I think you are viewing the Build 1903 crossover in reverse. The compatibility issue (like the one illustrated below) is not with older builds, it's with tclsh on Windows builds AFTER (and including) 1903. So 1903 being old is not relevant. That is only the starting point for when the issue arises.<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">Georg, I'm afraid if it was in fact a win-win situation, we would not be debating this :-)<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">Compare tclsh 8.6<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">c:\src&gt;findstr abc x.x | tclsh readpipe.tcl<o:p></o:p></p><p class="MsoNormal">abcé<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">with tclsh 9 (on the same Win10 system post-build 1903)<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">c:\src&gt;findstr abc x.x | d:\tcl\90\x64-debug\bin\tclsh90.exe readpipe.tcl<o:p></o:p></p><p class="MsoNormal">error reading "stdin": invalid or incomplete multibyte or wide character<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">(Note this is NOT an issue related to strict profiles. Without strict profiles, the data would be garbled, instead of error being raised. Even worse.)<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">Summarizing…<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">There is no question at some point the move to UTF8 has to be made.<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">There is no question that at that time there will be a lot of pain involved for users.<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">The question then is whether we inflict the pain now or later and which causes less disruption.<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">My position is that the move should be made when a majority of Windows systems are running with UTF-8 as the user code page. Doing it now, when most (*all* on my systems) applications including Windows command line tools still use the user's code page will lead to a lot more issues like the above compared to doing it later, when tools and applications will be spitting out UTF8 either because the user code page is UTF-8 or the tools themselves have adapted.<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">That is the debate.<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal">/Ashok<o:p></o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><div><div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal" style="margin-left:.5in"><b><span lang="EN-US" style="font-family:&quot;Calibri&quot;,sans-serif;mso-ligatures:none;mso-fareast-language:EN-IN">From:</span></b><span lang="EN-US" style="font-family:&quot;Calibri&quot;,sans-serif;mso-ligatures:none;mso-fareast-language:EN-IN"> Georg Lehner &lt;jor...@ma...&gt; <br><b>Sent:</b> Friday, July 5, 2024 3:30 PM<br><b>To:</b> tcl...@li...<br><b>Subject:</b> Re: [TCLCORE] Propose reverting [encoding system] on Windows<o:p></o:p></span></p></div></div><p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p><p style="margin-left:.5in">My 2c,<span style="font-size:12.0pt;mso-ligatures:none;mso-fareast-language:EN-IN"><o:p></o:p></span></p><p style="margin-left:.5in">Since always one had to handle the encoding and end-of-line translation explicitly for cross platform compatibility.<o:p></o:p></p><p style="margin-left:.5in">What I understand from your contribution is, that this will remain so no matter if the ActiveCodePage manifest property is set or not, Does not surprise me, we are including legacy systems in our scope.<o:p></o:p></p><p style="margin-left:.5in">New thing seems to be: If Windows users on newer systems already use UTF-8 as coding system, they'd benefit with respect to cross platform compatibility from the ActiveCodePage setting in tclsh/wish for all things done from now in the future .<o:p></o:p></p><p style="margin-left:.5in">Seems like a win to me.<o:p></o:p></p><p style="margin-left:.5in">Best Regards,<o:p></o:p></p><p style="margin-left:.5in">&nbsp; Georg<o:p></o:p></p><div><p class="MsoNormal" style="margin-left:.5in">On 6/30/24 17:39, apnmbx-public--- via Tcl-Core wrote:<o:p></o:p></p></div><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"><p class="MsoNormal" style="margin-left:.5in">I realize everyone is tired of encoding issues but I feel this is important given the consequences ...<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">One of the changes in Tcl 9 is to add a manifest property on Windows which effectively sets [encoding system] to UTF-8 irrespective of the user's code page setting. This manifest only has effect on Windows versions Windows 10 Build 1903 and later. Earlier Windows systems ignore the setting and Tcl 9 will use the user's code page setting on those platforms.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">This change, presumably made as part of TIP 587, has consequences serious enough to be reverted in my opinion.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">Below "file" refers to files containing non-ASCII content and that it is read/written with the (default) system encoding. And "cannot be read" means either an encoding error is thrown or data is garbled depending on the combination of Tcl version, platform version and system encoding (!).<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">- A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice versa even on *on the same exact system* on modern (build 1903 or later) Windows systems. So for example, on US English (cp1252) systems "set fd [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in foo.txt that will raise an error in Tcl 9's readFile (or the longer open/read/close equivalent). This is not acceptable in my view.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">- Along the same lines, a file written by a 9.x tclsh on pre-1903 Windows 10 cannot be read by the *<b>same</b>* 9.x tclsh on the *<b>same</b>* system after a *<b>Windows</b>* update. Also not acceptable.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">- A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 9.x tclsh on Windows 10 1903 and vice versa even when the two share a code page. May be less serious but still undesirable.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">- A third-party application (not using tclsh) that uses as its scripting language will not be able to read files written by tclsh and vice versa unless it also includes the utf-8 magic line in its manifest. This latter is unlikely as it would raise the same compatibility problems for them as for tclsh above. Also undesirable.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">In my view, these are all serious compatibility problems.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">The original motivation for the change in [encoding system] presumably came from this page - <a href="https://fedbdhd.r.af.d.sendibt2.com/tr/cl/7Kjat8GNFoMy9_sqpFWUe5Jd1PTD1sqXt9kays-ZSTbtdtrPn0I7OtyBpqesJU9DnArtN5YgwOalGzJL2P6uwZPUWPfKqwNGqz8k4JQk_GhTW1gFWRofTyYAmPhqy43E9xw-HqU1wONEKmPEf7xKUqAWsTfxezvfPkaoubm7mHZs_8SdXADjLaz2B42OKnVc1N6isCIntxwfKTGyPSUsXxCSga69jEqD1uDkVi-nJ7oVI050uLqaENKR-rYjxCSd7vmO90dpRJMtMgkzaKDHQazGlHgJ7vdJ5jXYbGGmAkh-bozV7I2_oLgZpEL5mMECwyIj8oxNzjsx2PwlS4eEU3OaUKrBgcKscBvGB-pIwjuRKUsLQZo">https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page</a>. However, note the target audience is really applications that use the ANSI Api's and its usefulness is limited for applications (like Tcl) that use the wide character API's. Moreover, as stated on that page, if targeting platforms prior to Build 1903, "you must handle legacy code page detection and conversion as usual".<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in"><b>Given the above, I would propose reverting [encoding system] on Windows to go back to defaulting to the encoding as per the user's code page setting as in Tcl 8.</b> Note this is not a code change, it is a change to the manifest resource in tclsh (and wish) to remove the ActiveCodePage property. The right place to move forward with UTF-8 as the default encoding belongs to the *applications* written in Tcl, not Tcl itself.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">(Note: I am not proposing changing the encoding used by the source command)<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">What would be drawbacks of reverting this change? One could argue that it would reduce compatibility with other platforms like Unix which use UTF-8 as the system encoding. However, I would argue that<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">- this is no different from what's the case with 8.x today<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">- sharing of files between platforms really has to include explicit configuration of encoding as good practice and most cross-platform applications will do that<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">- the issues listed above for a single-platform, on the very same system even, are much more serious<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">Reverting would probably need to be TIP'ed but I wanted to get feedback before embarking on that path.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">With regards to Unix, I do not have enough experience with encodings on Unix to know if similar issues would arise there. The [encoding system] in Tcl 9 has changed from iso8859-1 to utf-8. Someone should examine this closer.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">Comments please. I’d prefer to avoid TIP overhead if someone can spot holes in the above.<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">/Ashok<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;mso-ligatures:none;mso-fareast-language:EN-IN"><br><br><br><o:p></o:p></span></p><pre style="margin-left:.5in">_______________________________________________<o:p></o:p></pre><pre style="margin-left:.5in">Tcl-Core mailing list<o:p></o:p></pre><pre style="margin-left:.5in"><a href="mailto:Tcl...@li...">Tcl...@li...</a><o:p></o:p></pre><pre style="margin-left:.5in"><a href="https://fedbdhd.r.af.d.sendibt2.com/tr/cl/oULk3jl1CnN_IiXTc0fUo83MSG_EY7x8vX4Qu4n5Qxanvh4n2E9z3cHOe4P927KJPrgCpgAZYDLz8x-kyEWdYfEvFC22qPrtP29flAmKpa7jfQWnoQnFXk_SVTLRZ4Fm-i3by-aot1c8ToZoVNmZXkSqF7wKg6ydHpBz3vVUX4bNf7DQoEP73RwETeec1bJJhFl_EzjuDOngPm5D90lIzBiddazZxFE-P-L5FfPNCY_XXYlRj7LsZkt7ObNKZLadZPBecwF_dsT0tfvLGoejhoOWkw-7YwHn0D1AdynXwlCA_xhhYbJl63I5YU2HVBZY_g">https://lists.sourceforge.net/lists/listinfo/tcl-core</a><o:p></o:p></pre></blockquote></div><span>_______________________________________________</span><br><span>Tcl-Core mailing list</span><br><span>Tcl...@li...</span><br><span>https://lists.sourceforge.net/lists/listinfo/tcl-core</span><br></div></blockquote></body></html>

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[<apn...@ya...>]

Kevin, I think you are viewing the Build 1903 crossover in reverse. The compatibility issue (like the one illustrated below) is not with older builds, it's with tclsh on Windows builds AFTER (and including) 1903. So 1903 being old is not relevant. That is only the starting point for when the issue arises.

 

Georg, I'm afraid if it was in fact a win-win situation, we would not be debating this :-)

 

Compare tclsh 8.6

 

c:\src>findstr abc x.x | tclsh readpipe.tcl

abcé

 

with tclsh 9 (on the same Win10 system post-build 1903)

 

c:\src>findstr abc x.x | d:\tcl\90\x64-debug\bin\tclsh90.exe readpipe.tcl

error reading "stdin": invalid or incomplete multibyte or wide character

 

(Note this is NOT an issue related to strict profiles. Without strict profiles, the data would be garbled, instead of error being raised. Even worse.)

 

Summarizing…

 

There is no question at some point the move to UTF8 has to be made.

 

There is no question that at that time there will be a lot of pain involved for users.

 

The question then is whether we inflict the pain now or later and which causes less disruption.

 

My position is that the move should be made when a majority of Windows systems are running with UTF-8 as the user code page. Doing it now, when most (*all* on my systems) applications including Windows command line tools still use the user's code page will lead to a lot more issues like the above compared to doing it later, when tools and applications will be spitting out UTF8 either because the user code page is UTF-8 or the tools themselves have adapted.

 

That is the debate.

 

/Ashok

 

 

 

 

From: Georg Lehner <jor...@ma...> 
Sent: Friday, July 5, 2024 3:30 PM
To: tcl...@li...
Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows

 

My 2c,

Since always one had to handle the encoding and end-of-line translation explicitly for cross platform compatibility.

What I understand from your contribution is, that this will remain so no matter if the ActiveCodePage manifest property is set or not, Does not surprise me, we are including legacy systems in our scope.

New thing seems to be: If Windows users on newer systems already use UTF-8 as coding system, they'd benefit with respect to cross platform compatibility from the ActiveCodePage setting in tclsh/wish for all things done from now in the future .

Seems like a win to me.

Best Regards,

  Georg

On 6/30/24 17:39, apnmbx-public--- via Tcl-Core wrote:

I realize everyone is tired of encoding issues but I feel this is important given the consequences ...

 

One of the changes in Tcl 9 is to add a manifest property on Windows which effectively sets [encoding system] to UTF-8 irrespective of the user's code page setting. This manifest only has effect on Windows versions Windows 10 Build 1903 and later. Earlier Windows systems ignore the setting and Tcl 9 will use the user's code page setting on those platforms.

 

This change, presumably made as part of TIP 587, has consequences serious enough to be reverted in my opinion.

 

Below "file" refers to files containing non-ASCII content and that it is read/written with the (default) system encoding. And "cannot be read" means either an encoding error is thrown or data is garbled depending on the combination of Tcl version, platform version and system encoding (!).

 

- A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice versa even on *on the same exact system* on modern (build 1903 or later) Windows systems. So for example, on US English (cp1252) systems "set fd [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in foo.txt that will raise an error in Tcl 9's readFile (or the longer open/read/close equivalent). This is not acceptable in my view.

 

- Along the same lines, a file written by a 9.x tclsh on pre-1903 Windows 10 cannot be read by the *same* 9.x tclsh on the *same* system after a *Windows* update. Also not acceptable.

 

- A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 9.x tclsh on Windows 10 1903 and vice versa even when the two share a code page. May be less serious but still undesirable.

 

- A third-party application (not using tclsh) that uses as its scripting language will not be able to read files written by tclsh and vice versa unless it also includes the utf-8 magic line in its manifest. This latter is unlikely as it would raise the same compatibility problems for them as for tclsh above. Also undesirable.

 

In my view, these are all serious compatibility problems.

 

The original motivation for the change in [encoding system] presumably came from this page - https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page. However, note the target audience is really applications that use the ANSI Api's and its usefulness is limited for applications (like Tcl) that use the wide character API's. Moreover, as stated on that page, if targeting platforms prior to Build 1903, "you must handle legacy code page detection and conversion as usual".

 

Given the above, I would propose reverting [encoding system] on Windows to go back to defaulting to the encoding as per the user's code page setting as in Tcl 8. Note this is not a code change, it is a change to the manifest resource in tclsh (and wish) to remove the ActiveCodePage property. The right place to move forward with UTF-8 as the default encoding belongs to the *applications* written in Tcl, not Tcl itself.

 

(Note: I am not proposing changing the encoding used by the source command)

 

What would be drawbacks of reverting this change? One could argue that it would reduce compatibility with other platforms like Unix which use UTF-8 as the system encoding. However, I would argue that

 

- this is no different from what's the case with 8.x today

 

- sharing of files between platforms really has to include explicit configuration of encoding as good practice and most cross-platform applications will do that

 

- the issues listed above for a single-platform, on the very same system even, are much more serious

 

Reverting would probably need to be TIP'ed but I wanted to get feedback before embarking on that path.

 

With regards to Unix, I do not have enough experience with encodings on Unix to know if similar issues would arise there. The [encoding system] in Tcl 9 has changed from iso8859-1 to utf-8. Someone should examine this closer.

 

Comments please. I’d prefer to avoid TIP overhead if someone can spot holes in the above.

 

/Ashok

 






_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Kevin W. <kw...@co...>]

One more point: Windows build 1903 dates from 2019. So we are talking about a release that occurred five years ago, not one made last year. That makes me even more comfortable with proceeding. 

> On Jul 5, 2024, at 8:20 AM, Kevin Walzer <kw...@co...> wrote:
> 
> 
> 
> I reviewed TIP 587 and it is very clear that this is a step that will break compatibility. My inclination is to leave it as is. And what’s wrong with Tcl being ahead of the curve for once? Let Python, Rust, and others catch up to us.
> 
> > On Jul 5, 2024, at 6:34 AM, Steve Landers wrote:
> >
> > 
> > Your discomfort is infectious, even thought I don’t have a dog in this fight. I would be much more comfortable announcing such a change for a subsequent release and then sticking to that timetable. Even if it was an upcoming point release, which we will surely need after 9.0 is released.
> >
> > -- Steve
> >> On 5 Jul 2024 at 2:40 PM +0800, apnmbx-public--- via Tcl-Core , wrote:
> >> Christian,
> >>
> >> I did not ignore programming languages, I *explicitly* listed Python in my email. It does not include a manifest for UTF-8. I admittedly did not look at their future plans, only their *current* binaries. The PEP you mentioned targets Python *3.15*, slated for release in October *2026*, more than 2 years from now. I have no issues with a UTF-8 default for Tcl 2026, the Windows world would have moved by then to a UTF-8 code page by default even in system settings. As an aside, note the "loud" warnings in that PEP about the compatibility!
> >>
> >> [To reiterate - I do appreciate the rationale behind moving to UTF-8. I am just not comfortable with the timing of it for reasons of compatibility.]
> >>
> >> /Ashok
> >>
> >> -----Original Message-----
> >> From: Christian Gollwitzer via Tcl-Core
> >> Sent: Friday, July 5, 2024 11:17 AM
> >> To: tcl...@li...
> >> Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows
> >>
> >>> Am 05.07.24 um 04:34 schrieb apnmbx-public--- via Tcl-Core:
> >>> Alexander,
> >>>
> >>> The issue is not about the ability to use UTF-8 in applications as you've described. Obviously Tcl has had that for decades.
> >>>
> >>> The issue is what [encoding system] is initialized to in tclsh/wish (and thereby controls default encodings on channels). In Tcl 8, it was initialized based on user's settings. In Tcl 9, this remains the case on all platforms EXCEPT on Win 10 build 1903 or later where it is hardcoded to be UTF-8 irrespective of user settings. Apart from the compatibility concerns that I raised in my original post, my latest gripe was that Tcl is the *only* application I've found that does this.
> >>
> >> You've looked at applications, look at programming languages:
> >>
> >> https://peps.python.org/pep-0686/
> >>
> >> In the rationale there, Node.js, Go, Rust, and Java are also mentioned.
> >>
> >> I'm not sure about the "manifest" thing and what implications that has.
> >>
> >> Christian
> >>>
> >>> /Ashok
> >>>
> >>> -----Original Message-----
> >>> From: Alexander Schöpe
> >>> Sent: Friday, July 5, 2024 12:46 AM
> >>> To: apn...@ya...
> >>> Cc: Harald Oehlmann ; tcl...@li...
> >>> Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows
> >>>
> >>> I don’t understand the problem.
> >>>
> >>>> Am 04.07.2024 um 19:30 schrieb apnmbx-public--- via Tcl-Core :
> >>>>
> >>>> While the primary argument folks seem to have made is that the world is moving to utf-8, I suspect this has come from a Unix perspective.
> >>>
> >>> UTF-8 is used in various cases and contexts in Windows 11 to ensure compatibility and internationalization. Here are some specific scenarios where UTF-8 is employed:
> >>>
> >>> Text Files and Documents: Many applications, including text editors and office applications, support saving and loading files in UTF-8 format. This ensures that text in various languages is correctly displayed.
> >>>
> >>> Web Browsers and Web Applications: Web browsers like Microsoft Edge, Chrome, and Firefox use UTF-8 to render web pages. This is the standard character set for HTML5 and is used by most modern websites.
> >>>
> >>> Console Applications: In the Command Prompt and PowerShell, the code page can be changed to UTF-8 to ensure the correct display of Unicode characters. This is particularly useful for applications that need to display international characters or emojis.
> >>>
> >>> Programming Environments and Scripts: Development environments like Visual Studio Code and other IDEs support UTF-8 as the standard character set for source code files. Many programming languages and scripting environments (e.g., Python, JavaScript) also use UTF-8 to encode and process strings.
> >>>
> >>> Network Communication and APIs: Many network protocols and APIs use UTF-8 to encode data. This includes HTTP requests and responses, JSON and XML data, and other internet-based communications.
> >>>
> >>> International Software and Localization: Applications intended for the international market often use UTF-8 to ensure that all characters are correctly displayed, regardless of language or region.
> >>>
> >>> Emails and Messaging: Email clients and messaging applications use UTF-8 to ensure that messages with international content are correctly encoded and displayed.
> >>>
> >>> Databases: Many relational and NoSQL databases support UTF-8 to store and retrieve strings in various languages.
> >>>
> >>> The use of UTF-8 ensures that applications and systems are internationally compatible and can correctly display a wide range of characters and symbols. This is particularly important in a globalized world where software is often used in multiple languages and regions.
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Tcl-Core mailing list
> >>> Tcl...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/tcl-core
> >>
> >>
> >>
> >> _______________________________________________
> >> Tcl-Core mailing list
> >> Tcl...@li...
> >> https://lists.sourceforge.net/lists/listinfo/tcl-core
> >>
> >>
> >>
> >> _______________________________________________
> >> Tcl-Core mailing list
> >> Tcl...@li...
> >> https://lists.sourceforge.net/lists/listinfo/tcl-core
> > _______________________________________________
> > Tcl-Core mailing list
> > Tcl...@li...
> > https://lists.sourceforge.net/lists/listinfo/tcl-core
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Kevin W. <kw...@co...>]

<div><img width="1" height="1" src='https://fedbdhd.r.af.d.sendibt2.com/tr/op/xOA0OHFRABrJlcTO4N0eYtsgSBmhqvtoqcPVAHlg_cTkg7LTcH3PY5zeOHdna35HvKHGK0yV32pw6c9b3UG-aqoMwItsvFupwNOkzXu-oLqZcVyu-ko1VFDS3acyfWyiKrMwRtI32sm9c-LiC84_8NCbfEsDZBS0Igob_zN85Pvw-37cV1AK4rQS3_XBMYhrO29GyL92fk1UXXuUMJX4M7-YETmo8koN' /></div>I reviewed TIP 587 and it is very clear that this is a step that will break compatibility. My inclination is to leave it as is. And what’s wrong with Tcl being ahead of the curve for once? Let Python, Rust, and others catch up to us. <br/><br/>> On Jul 5, 2024, at 6:34 AM, Steve Landers <st...@di...> wrote:<br/>> <br/>> <br/>> Your discomfort is infectious, even thought I don’t have a dog in this fight. I would be much more comfortable announcing such a change for a subsequent release and then sticking to that timetable. Even if it was an upcoming point release, which we will surely need after 9.0 is released.<br/>> <br/>> -- Steve<br/>>> On 5 Jul 2024 at 2:40 PM +0800, apnmbx-public--- via Tcl-Core <tcl...@li...>, wrote:<br/>>> Christian,<br/>>> <br/>>> I did not ignore programming languages, I *explicitly* listed Python in my email. It does not include a manifest for UTF-8. I admittedly did not look at their future plans, only their *current* binaries. The PEP you mentioned targets Python *3.15*, slated for release in October *2026*, more than 2 years from now. I have no issues with a UTF-8 default for Tcl 2026, the Windows world would have moved by then to a UTF-8 code page by default even in system settings. As an aside, note the "loud" warnings in that PEP about the compatibility!<br/>>> <br/>>> [To reiterate - I do appreciate the rationale behind moving to UTF-8. I am just not comfortable with the timing of it for reasons of compatibility.]<br/>>> <br/>>> /Ashok<br/>>> <br/>>> -----Original Message-----<br/>>> From: Christian Gollwitzer via Tcl-Core <tcl...@li...><br/>>> Sent: Friday, July 5, 2024 11:17 AM<br/>>> To: tcl...@li...<br/>>> Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows<br/>>> <br/>>>> Am 05.07.24 um 04:34 schrieb apnmbx-public--- via Tcl-Core:<br/>>>> Alexander,<br/>>>> <br/>>>> The issue is not about the ability to use UTF-8 in applications as you've described. Obviously Tcl has had that for decades.<br/>>>> <br/>>>> The issue is what [encoding system] is initialized to in tclsh/wish (and thereby controls default encodings on channels). In Tcl 8, it was initialized based on user's settings. In Tcl 9, this remains the case on all platforms EXCEPT on Win 10 build 1903 or later where it is hardcoded to be UTF-8 irrespective of user settings. Apart from the compatibility concerns that I raised in my original post, my latest gripe was that Tcl is the *only* application I've found that does this.<br/>>> <br/>>> You've looked at applications, look at programming languages:<br/>>> <br/>>> https://peps.python.org/pep-0686/<br/>>> <br/>>> In the rationale there, Node.js, Go, Rust, and Java are also mentioned.<br/>>> <br/>>> I'm not sure about the "manifest" thing and what implications that has.<br/>>> <br/>>> Christian<br/>>>> <br/>>>> /Ashok<br/>>>> <br/>>>> -----Original Message-----<br/>>>> From: Alexander Schöpe <a.s...@gm...><br/>>>> Sent: Friday, July 5, 2024 12:46 AM<br/>>>> To: apn...@ya...<br/>>>> Cc: Harald Oehlmann <har...@el...>; tcl...@li...<br/>>>> Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows<br/>>>> <br/>>>> I don’t understand the problem.<br/>>>> <br/>>>>> Am 04.07.2024 um 19:30 schrieb apnmbx-public--- via Tcl-Core <tcl...@li...>:<br/>>>>> <br/>>>>> While the primary argument folks seem to have made is that the world is moving to utf-8, I suspect this has come from a Unix perspective.<br/>>>> <br/>>>> UTF-8 is used in various cases and contexts in Windows 11 to ensure compatibility and internationalization. Here are some specific scenarios where UTF-8 is employed:<br/>>>> <br/>>>> Text Files and Documents: Many applications, including text editors and office applications, support saving and loading files in UTF-8 format. This ensures that text in various languages is correctly displayed.<br/>>>> <br/>>>> Web Browsers and Web Applications: Web browsers like Microsoft Edge, Chrome, and Firefox use UTF-8 to render web pages. This is the standard character set for HTML5 and is used by most modern websites.<br/>>>> <br/>>>> Console Applications: In the Command Prompt and PowerShell, the code page can be changed to UTF-8 to ensure the correct display of Unicode characters. This is particularly useful for applications that need to display international characters or emojis.<br/>>>> <br/>>>> Programming Environments and Scripts: Development environments like Visual Studio Code and other IDEs support UTF-8 as the standard character set for source code files. Many programming languages and scripting environments (e.g., Python, JavaScript) also use UTF-8 to encode and process strings.<br/>>>> <br/>>>> Network Communication and APIs: Many network protocols and APIs use UTF-8 to encode data. This includes HTTP requests and responses, JSON and XML data, and other internet-based communications.<br/>>>> <br/>>>> International Software and Localization: Applications intended for the international market often use UTF-8 to ensure that all characters are correctly displayed, regardless of language or region.<br/>>>> <br/>>>> Emails and Messaging: Email clients and messaging applications use UTF-8 to ensure that messages with international content are correctly encoded and displayed.<br/>>>> <br/>>>> Databases: Many relational and NoSQL databases support UTF-8 to store and retrieve strings in various languages.<br/>>>> <br/>>>> The use of UTF-8 ensures that applications and systems are internationally compatible and can correctly display a wide range of characters and symbols. This is particularly important in a globalized world where software is often used in multiple languages and regions.<br/>>>> <br/>>>> <br/>>>> <br/>>>> _______________________________________________<br/>>>> Tcl-Core mailing list<br/>>>> Tcl...@li...<br/>>>> https://lists.sourceforge.net/lists/listinfo/tcl-core<br/>>> <br/>>> <br/>>> <br/>>> _______________________________________________<br/>>> Tcl-Core mailing list<br/>>> Tcl...@li...<br/>>> https://lists.sourceforge.net/lists/listinfo/tcl-core<br/>>> <br/>>> <br/>>> <br/>>> _______________________________________________<br/>>> Tcl-Core mailing list<br/>>> Tcl...@li...<br/>>> https://lists.sourceforge.net/lists/listinfo/tcl-core<br/>> _______________________________________________<br/>> Tcl-Core mailing list<br/>> Tcl...@li...<br/>> https://lists.sourceforge.net/lists/listinfo/tcl-core<br/>

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Jan N. <jan...@gm...>]

Op do 4 jul 2024 om 19:31 schreef Ashok:
> Having looked into it deeper, I will just list couple of additional factors I have discovered.
...
> Also concerning, as the SDK documentation states, even the GDI subsystem does not support this manifest entry. I don't know to what extent that affects Tcl or Tk where wide character API's are used for the most part...

Let me answer this. Tk uses GDI, but only the wide character API. If
a wide character API is available, that always is a better choice than
depending on the system codepage.

Conclusion: no problem here.

Hope this helps,
       Jan Nijtmans

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Steve L. <st...@di...>]

Your discomfort is infectious, even thought I don’t have a dog in this fight. I would be much more comfortable announcing such a change for a subsequent release and then sticking to that timetable. Even if it was an upcoming point release, which we will surely need after 9.0 is released.

-- Steve
On 5 Jul 2024 at 2:40 PM +0800, apnmbx-public--- via Tcl-Core <tcl...@li...>, wrote:
> Christian,
>
> I did not ignore programming languages, I *explicitly* listed Python in my email. It does not include a manifest for UTF-8. I admittedly did not look at their future plans, only their *current* binaries. The PEP you mentioned targets Python *3.15*, slated for release in October *2026*, more than 2 years from now. I have no issues with a UTF-8 default for Tcl 2026, the Windows world would have moved by then to a UTF-8 code page by default even in system settings. As an aside, note the "loud" warnings in that PEP about the compatibility!
>
> [To reiterate - I do appreciate the rationale behind moving to UTF-8. I am just not comfortable with the timing of it for reasons of compatibility.]
>
> /Ashok
>
> -----Original Message-----
> From: Christian Gollwitzer via Tcl-Core <tcl...@li...>
> Sent: Friday, July 5, 2024 11:17 AM
> To: tcl...@li...
> Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows
>
> Am 05.07.24 um 04:34 schrieb apnmbx-public--- via Tcl-Core:
> > Alexander,
> >
> > The issue is not about the ability to use UTF-8 in applications as you've described. Obviously Tcl has had that for decades.
> >
> > The issue is what [encoding system] is initialized to in tclsh/wish (and thereby controls default encodings on channels). In Tcl 8, it was initialized based on user's settings. In Tcl 9, this remains the case on all platforms EXCEPT on Win 10 build 1903 or later where it is hardcoded to be UTF-8 irrespective of user settings. Apart from the compatibility concerns that I raised in my original post, my latest gripe was that Tcl is the *only* application I've found that does this.
>
> You've looked at applications, look at programming languages:
>
> https://peps.python.org/pep-0686/
>
> In the rationale there, Node.js, Go, Rust, and Java are also mentioned.
>
> I'm not sure about the "manifest" thing and what implications that has.
>
> Christian
> >
> > /Ashok
> >
> > -----Original Message-----
> > From: Alexander Schöpe <a.s...@gm...>
> > Sent: Friday, July 5, 2024 12:46 AM
> > To: apn...@ya...
> > Cc: Harald Oehlmann <har...@el...>; tcl...@li...
> > Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows
> >
> > I don’t understand the problem.
> >
> > > Am 04.07.2024 um 19:30 schrieb apnmbx-public--- via Tcl-Core <tcl...@li...>:
> > >
> > > While the primary argument folks seem to have made is that the world is moving to utf-8, I suspect this has come from a Unix perspective.
> >
> > UTF-8 is used in various cases and contexts in Windows 11 to ensure compatibility and internationalization. Here are some specific scenarios where UTF-8 is employed:
> >
> > Text Files and Documents: Many applications, including text editors and office applications, support saving and loading files in UTF-8 format. This ensures that text in various languages is correctly displayed.
> >
> > Web Browsers and Web Applications: Web browsers like Microsoft Edge, Chrome, and Firefox use UTF-8 to render web pages. This is the standard character set for HTML5 and is used by most modern websites.
> >
> > Console Applications: In the Command Prompt and PowerShell, the code page can be changed to UTF-8 to ensure the correct display of Unicode characters. This is particularly useful for applications that need to display international characters or emojis.
> >
> > Programming Environments and Scripts: Development environments like Visual Studio Code and other IDEs support UTF-8 as the standard character set for source code files. Many programming languages and scripting environments (e.g., Python, JavaScript) also use UTF-8 to encode and process strings.
> >
> > Network Communication and APIs: Many network protocols and APIs use UTF-8 to encode data. This includes HTTP requests and responses, JSON and XML data, and other internet-based communications.
> >
> > International Software and Localization: Applications intended for the international market often use UTF-8 to ensure that all characters are correctly displayed, regardless of language or region.
> >
> > Emails and Messaging: Email clients and messaging applications use UTF-8 to ensure that messages with international content are correctly encoded and displayed.
> >
> > Databases: Many relational and NoSQL databases support UTF-8 to store and retrieve strings in various languages.
> >
> > The use of UTF-8 ensures that applications and systems are internationally compatible and can correctly display a wide range of characters and symbols. This is particularly important in a globalized world where software is often used in multiple languages and regions.
> >
> >
> >
> > _______________________________________________
> > Tcl-Core mailing list
> > Tcl...@li...
> > https://lists.sourceforge.net/lists/listinfo/tcl-core
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Georg L. <jor...@ma...>]

My 2c,

Since always one had to handle the encoding and end-of-line translation 
explicitly for cross platform compatibility.

What I understand from your contribution is, that this will remain so no 
matter if the ActiveCodePage manifest property is set or not, Does not 
surprise me, we are including legacy systems in our scope.

New thing seems to be: If Windows users on newer systems already use 
UTF-8 as coding system, they'd benefit with respect to cross platform 
compatibility from the ActiveCodePage setting in tclsh/wish for all 
things done from now in the future .

Seems like a win to me.

Best Regards,

   Georg

On 6/30/24 17:39, apnmbx-public--- via Tcl-Core wrote:
>
> I realize everyone is tired of encoding issues but I feel this is 
> important given the consequences ...
>
> One of the changes in Tcl 9 is to add a manifest property on Windows 
> which effectively sets [encoding system] to UTF-8 irrespective of the 
> user's code page setting. This manifest only has effect on Windows 
> versions Windows 10 Build 1903 and later. Earlier Windows systems 
> ignore the setting and Tcl 9 will use the user's code page setting on 
> those platforms.
>
> This change, presumably made as part of TIP 587, has consequences 
> serious enough to be reverted in my opinion.
>
> Below "file" refers to files containing non-ASCII content and that it 
> is read/written with the (default) system encoding. And "cannot be 
> read" means either an encoding error is thrown or data is garbled 
> depending on the combination of Tcl version, platform version and 
> system encoding (!).
>
> - A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice 
> versa even on *on the same exact system* on modern (build 1903 or 
> later) Windows systems. So for example, on US English (cp1252) systems 
> "set fd [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will 
> result in foo.txt that will raise an error in Tcl 9's readFile (or the 
> longer open/read/close equivalent). This is not acceptable in my view.
>
> - Along the same lines, a file written by a 9.x tclsh on pre-1903 
> Windows 10 cannot be read by the **same** 9.x tclsh on the **same** 
> system after a **Windows** update. Also not acceptable.
>
> - A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 
> 9.x tclsh on Windows 10 1903 and vice versa even when the two share a 
> code page. May be less serious but still undesirable.
>
> - A third-party application (not using tclsh) that uses as its 
> scripting language will not be able to read files written by tclsh and 
> vice versa unless it also includes the utf-8 magic line in its 
> manifest. This latter is unlikely as it would raise the same 
> compatibility problems for them as for tclsh above. Also undesirable.
>
> In my view, these are all serious compatibility problems.
>
> The original motivation for the change in [encoding system] presumably 
> came from this page - 
> https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page. 
> However, note the target audience is really applications that use the 
> ANSI Api's and its usefulness is limited for applications (like Tcl) 
> that use the wide character API's. Moreover, as stated on that page, 
> if targeting platforms prior to Build 1903, "you must handle legacy 
> code page detection and conversion as usual".
>
> *Given the above, I would propose reverting [encoding system] on 
> Windows to go back to defaulting to the encoding as per the user's 
> code page setting as in Tcl 8.* Note this is not a code change, it is 
> a change to the manifest resource in tclsh (and wish) to remove the 
> ActiveCodePage property. The right place to move forward with UTF-8 as 
> the default encoding belongs to the *applications* written in Tcl, not 
> Tcl itself.
>
> (Note: I am not proposing changing the encoding used by the source 
> command)
>
> What would be drawbacks of reverting this change? One could argue that 
> it would reduce compatibility with other platforms like Unix which use 
> UTF-8 as the system encoding. However, I would argue that
>
> - this is no different from what's the case with 8.x today
>
> - sharing of files between platforms really has to include explicit 
> configuration of encoding as good practice and most cross-platform 
> applications will do that
>
> - the issues listed above for a single-platform, on the very same 
> system even, are much more serious
>
> Reverting would probably need to be TIP'ed but I wanted to get 
> feedback before embarking on that path.
>
> With regards to Unix, I do not have enough experience with encodings 
> on Unix to know if similar issues would arise there. The [encoding 
> system] in Tcl 9 has changed from iso8859-1 to utf-8. Someone should 
> examine this closer.
>
> Comments please. I’d prefer to avoid TIP overhead if someone can spot 
> holes in the above.
>
> /Ashok
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[<apn...@ya...>]

Christian,

I did not ignore programming languages, I *explicitly* listed Python in my email. It does not include a manifest for UTF-8. I admittedly did not look at their future plans, only their *current* binaries. The PEP you mentioned targets Python *3.15*, slated for release in October *2026*, more than 2 years from now. I have no issues with a UTF-8 default for Tcl 2026, the Windows world would have moved by then to a UTF-8 code page by default even in system settings. As an aside, note the "loud" warnings in that PEP about the compatibility!

[To reiterate - I do appreciate the rationale behind moving to UTF-8. I am just not comfortable with the timing of it for reasons of compatibility.]

/Ashok

-----Original Message-----
From: Christian Gollwitzer via Tcl-Core <tcl...@li...> 
Sent: Friday, July 5, 2024 11:17 AM
To: tcl...@li...
Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows

Am 05.07.24 um 04:34 schrieb apnmbx-public--- via Tcl-Core:
> Alexander,
>
> The issue is not about the ability to use UTF-8 in applications as you've described. Obviously Tcl has had that for decades.
>
> The issue is what [encoding system] is initialized to in tclsh/wish (and thereby controls default encodings on channels). In Tcl 8, it was initialized based on user's settings. In Tcl 9, this remains the case on all platforms EXCEPT on Win 10 build 1903 or later where it is hardcoded to be UTF-8 irrespective of user settings. Apart from the compatibility concerns that I raised in my original post, my latest gripe was that Tcl is the *only* application I've found that does this.

You've looked at applications, look at programming languages:

https://peps.python.org/pep-0686/

In the rationale there, Node.js, Go, Rust, and Java are also mentioned.

I'm not sure about the "manifest" thing and what implications that has.

      Christian
>
> /Ashok
>
> -----Original Message-----
> From: Alexander Schöpe <a.s...@gm...>
> Sent: Friday, July 5, 2024 12:46 AM
> To: apn...@ya...
> Cc: Harald Oehlmann <har...@el...>; tcl...@li...
> Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows
>
> I don’t understand the problem.
>
>> Am 04.07.2024 um 19:30 schrieb apnmbx-public--- via Tcl-Core <tcl...@li...>:
>>
>> While the primary argument folks seem to have made is that the world is moving to utf-8, I suspect this has come from a Unix perspective.
>
> UTF-8 is used in various cases and contexts in Windows 11 to ensure compatibility and internationalization. Here are some specific scenarios where UTF-8 is employed:
>
> Text Files and Documents: Many applications, including text editors and office applications, support saving and loading files in UTF-8 format. This ensures that text in various languages is correctly displayed.
>
> Web Browsers and Web Applications: Web browsers like Microsoft Edge, Chrome, and Firefox use UTF-8 to render web pages. This is the standard character set for HTML5 and is used by most modern websites.
>
> Console Applications: In the Command Prompt and PowerShell, the code page can be changed to UTF-8 to ensure the correct display of Unicode characters. This is particularly useful for applications that need to display international characters or emojis.
>
> Programming Environments and Scripts: Development environments like Visual Studio Code and other IDEs support UTF-8 as the standard character set for source code files. Many programming languages and scripting environments (e.g., Python, JavaScript) also use UTF-8 to encode and process strings.
>
> Network Communication and APIs: Many network protocols and APIs use UTF-8 to encode data. This includes HTTP requests and responses, JSON and XML data, and other internet-based communications.
>
> International Software and Localization: Applications intended for the international market often use UTF-8 to ensure that all characters are correctly displayed, regardless of language or region.
>
> Emails and Messaging: Email clients and messaging applications use UTF-8 to ensure that messages with international content are correctly encoded and displayed.
>
> Databases: Many relational and NoSQL databases support UTF-8 to store and retrieve strings in various languages.
>
> The use of UTF-8 ensures that applications and systems are internationally compatible and can correctly display a wide range of characters and symbols. This is particularly important in a globalized world where software is often used in multiple languages and regions.
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core



_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Christian G. <aur...@gm...>]

Am 05.07.24 um 04:34 schrieb apnmbx-public--- via Tcl-Core:
> Alexander,
>
> The issue is not about the ability to use UTF-8 in applications as you've described. Obviously Tcl has had that for decades.
>
> The issue is what [encoding system] is initialized to in tclsh/wish (and thereby controls default encodings on channels). In Tcl 8, it was initialized based on user's settings. In Tcl 9, this remains the case on all platforms EXCEPT on Win 10 build 1903 or later where it is hardcoded to be UTF-8 irrespective of user settings. Apart from the compatibility concerns that I raised in my original post, my latest gripe was that Tcl is the *only* application I've found that does this.

You've looked at applications, look at programming languages:

https://peps.python.org/pep-0686/

In the rationale there, Node.js, Go, Rust, and Java are also mentioned.

I'm not sure about the "manifest" thing and what implications that has.

      Christian
>
> /Ashok
>
> -----Original Message-----
> From: Alexander Schöpe <a.s...@gm...>
> Sent: Friday, July 5, 2024 12:46 AM
> To: apn...@ya...
> Cc: Harald Oehlmann <har...@el...>; tcl...@li...
> Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows
>
> I don’t understand the problem.
>
>> Am 04.07.2024 um 19:30 schrieb apnmbx-public--- via Tcl-Core <tcl...@li...>:
>>
>> While the primary argument folks seem to have made is that the world is moving to utf-8, I suspect this has come from a Unix perspective.
>
> UTF-8 is used in various cases and contexts in Windows 11 to ensure compatibility and internationalization. Here are some specific scenarios where UTF-8 is employed:
>
> Text Files and Documents: Many applications, including text editors and office applications, support saving and loading files in UTF-8 format. This ensures that text in various languages is correctly displayed.
>
> Web Browsers and Web Applications: Web browsers like Microsoft Edge, Chrome, and Firefox use UTF-8 to render web pages. This is the standard character set for HTML5 and is used by most modern websites.
>
> Console Applications: In the Command Prompt and PowerShell, the code page can be changed to UTF-8 to ensure the correct display of Unicode characters. This is particularly useful for applications that need to display international characters or emojis.
>
> Programming Environments and Scripts: Development environments like Visual Studio Code and other IDEs support UTF-8 as the standard character set for source code files. Many programming languages and scripting environments (e.g., Python, JavaScript) also use UTF-8 to encode and process strings.
>
> Network Communication and APIs: Many network protocols and APIs use UTF-8 to encode data. This includes HTTP requests and responses, JSON and XML data, and other internet-based communications.
>
> International Software and Localization: Applications intended for the international market often use UTF-8 to ensure that all characters are correctly displayed, regardless of language or region.
>
> Emails and Messaging: Email clients and messaging applications use UTF-8 to ensure that messages with international content are correctly encoded and displayed.
>
> Databases: Many relational and NoSQL databases support UTF-8 to store and retrieve strings in various languages.
>
> The use of UTF-8 ensures that applications and systems are internationally compatible and can correctly display a wide range of characters and symbols. This is particularly important in a globalized world where software is often used in multiple languages and regions.
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[<apn...@ya...>]

Alexander,

The issue is not about the ability to use UTF-8 in applications as you've described. Obviously Tcl has had that for decades.

The issue is what [encoding system] is initialized to in tclsh/wish (and thereby controls default encodings on channels). In Tcl 8, it was initialized based on user's settings. In Tcl 9, this remains the case on all platforms EXCEPT on Win 10 build 1903 or later where it is hardcoded to be UTF-8 irrespective of user settings. Apart from the compatibility concerns that I raised in my original post, my latest gripe was that Tcl is the *only* application I've found that does this.

/Ashok

-----Original Message-----
From: Alexander Schöpe <a.s...@gm...> 
Sent: Friday, July 5, 2024 12:46 AM
To: apn...@ya...
Cc: Harald Oehlmann <har...@el...>; tcl...@li...
Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows

I don’t understand the problem. 

> Am 04.07.2024 um 19:30 schrieb apnmbx-public--- via Tcl-Core <tcl...@li...>:
> 
> While the primary argument folks seem to have made is that the world is moving to utf-8, I suspect this has come from a Unix perspective.

UTF-8 is used in various cases and contexts in Windows 11 to ensure compatibility and internationalization. Here are some specific scenarios where UTF-8 is employed:

Text Files and Documents: Many applications, including text editors and office applications, support saving and loading files in UTF-8 format. This ensures that text in various languages is correctly displayed.

Web Browsers and Web Applications: Web browsers like Microsoft Edge, Chrome, and Firefox use UTF-8 to render web pages. This is the standard character set for HTML5 and is used by most modern websites.

Console Applications: In the Command Prompt and PowerShell, the code page can be changed to UTF-8 to ensure the correct display of Unicode characters. This is particularly useful for applications that need to display international characters or emojis.

Programming Environments and Scripts: Development environments like Visual Studio Code and other IDEs support UTF-8 as the standard character set for source code files. Many programming languages and scripting environments (e.g., Python, JavaScript) also use UTF-8 to encode and process strings.

Network Communication and APIs: Many network protocols and APIs use UTF-8 to encode data. This includes HTTP requests and responses, JSON and XML data, and other internet-based communications.

International Software and Localization: Applications intended for the international market often use UTF-8 to ensure that all characters are correctly displayed, regardless of language or region.

Emails and Messaging: Email clients and messaging applications use UTF-8 to ensure that messages with international content are correctly encoded and displayed.

Databases: Many relational and NoSQL databases support UTF-8 to store and retrieve strings in various languages.

The use of UTF-8 ensures that applications and systems are internationally compatible and can correctly display a wide range of characters and symbols. This is particularly important in a globalized world where software is often used in multiple languages and regions.

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Alexander S. <a.s...@gm...>]

I don’t understand the problem. 

> Am 04.07.2024 um 19:30 schrieb apnmbx-public--- via Tcl-Core <tcl...@li...>:
> 
> While the primary argument folks seem to have made is that the world is moving to utf-8, I suspect this has come from a Unix perspective.

UTF-8 is used in various cases and contexts in Windows 11 to ensure compatibility and internationalization. Here are some specific scenarios where UTF-8 is employed:

Text Files and Documents: Many applications, including text editors and office applications, support saving and loading files in UTF-8 format. This ensures that text in various languages is correctly displayed.

Web Browsers and Web Applications: Web browsers like Microsoft Edge, Chrome, and Firefox use UTF-8 to render web pages. This is the standard character set for HTML5 and is used by most modern websites.

Console Applications: In the Command Prompt and PowerShell, the code page can be changed to UTF-8 to ensure the correct display of Unicode characters. This is particularly useful for applications that need to display international characters or emojis.

Programming Environments and Scripts: Development environments like Visual Studio Code and other IDEs support UTF-8 as the standard character set for source code files. Many programming languages and scripting environments (e.g., Python, JavaScript) also use UTF-8 to encode and process strings.

Network Communication and APIs: Many network protocols and APIs use UTF-8 to encode data. This includes HTTP requests and responses, JSON and XML data, and other internet-based communications.

International Software and Localization: Applications intended for the international market often use UTF-8 to ensure that all characters are correctly displayed, regardless of language or region.

Emails and Messaging: Email clients and messaging applications use UTF-8 to ensure that messages with international content are correctly encoded and displayed.

Databases: Many relational and NoSQL databases support UTF-8 to store and retrieve strings in various languages.

The use of UTF-8 ensures that applications and systems are internationally compatible and can correctly display a wide range of characters and symbols. This is particularly important in a globalized world where software is often used in multiple languages and regions.

Re: [TCLCORE] Major update to the macOS port of Tk

[Jan N. <jan...@gm...>]

Op wo 3 jul 2024 om 18:12 schreef Marc Culler:
> The five of us are confident that this is a significant improvement. Some highlights are:
>     * Better performance - CPU-intensive apps use about 20% fewer CPU cycles
>     * Better conformance - Most runs of the regression test suite on macOS 14 show no failures.  This is not close to being the case for the trunk branch.
>     * Fewer graphical artifacts (and work continues on those that remain.)
>     * Fewer crashes (none that we know of at the moment.)

GITHUB recently removed the macos-11 runner, so we moved
to a later one. Since then some Tk test-cases started failing.

Not so with the "cgimage_with_crossing" branch, all
MacOS (and other) test-cases are green there:
    <https://github.com/tcltk/tk/actions/runs/9773220535>

That makes my light green as well ;-)

Regards,
     Jan Nijtmans

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[<apn...@ya...>]

The consensus in prior discussions, that I had reluctantly accepted, was to keep the status quo in Tcl 9 of hard coding the code page to be utf-8 via the manifest. This despite the fact that I did not hear of any actual concrete issues with reverting and keeping the code page detection same as in Tcl 8 while I had listed several issues in hardcoding it to UTF-8. The rationale behind moving to UTF-8 was that the world is moving to it, we should too, compatibility with Unix etc.

Having looked into it deeper, I will just list couple of additional factors I have discovered.

While the primary argument folks seem to have made is that the world is moving to utf-8, I suspect this has come from a Unix perspective. In particular, the *only* executables that hardcoded the code page to utf-8 via the manifest were tclsh90 and wish90 when I searched (strings+grep) across 3 Win10/11 systems including the entire Windows system directory, multiple browsers, Office, Visual Studio, Python, Ruby and a host of smaller applications. The only one that even had an activeCodePage manifest entry was Firefox and even there it was set to Legacy and not UTF-8.

This makes me nervous. Is Tcl supposed to push the Windows world to UTF-8? Does it make sense for Tcl to be the odd man out and to exactly what benefit?

[Note applications like notepad do support UTF-8 and encodings but that is by application configurable choice and encoding guessing, not with a manifest entry. That should be the case with Tcl applications as well. The push to UTF-8 should come from Tcl applications, not Tcl itself.]

Also concerning, as the SDK documentation states, even the GDI subsystem does not support this manifest entry. I don't know to what extent that affects Tcl or Tk where wide character API's are used for the most part, but the fact that a major Windows subsystem does not support this per-process code pages setting seemingly gives lie to UTF-8 improving compatibility with other applications (on Windows).

I only raise this in case some minds might be changed. Else I don't plan to pursue this further as reverting anything should require a high bar in terms of consensus.

/Ashok


-----Original Message-----
From: Harald Oehlmann <har...@el...> 
Sent: Thursday, July 4, 2024 6:02 PM
To: tcl...@li...
Subject: Re: [TCLCORE] Propose reverting [encoding system] on Windows

Am 04.07.2024 um 14:17 schrieb Jan Nijtmans:
> Op zo 30 jun 2024 om 23:18 schreef Jan Nijtmans <jan...@gm...>:
>> It shouldn't be too difficult to get the old code-page from the
>> registry, it can be found in the following registry key:
>>       HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage\ACP
> 
> This is how to retrieve the code page it had in Tcl 8.x:
> 
>      $ tclsh90
>      % encoding system
>      utf-8
>      % package require registry
>      1.3.7
>      % registry get
> HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\Nls\\CodePage
> ACP
>      1252
> 
> Hope this helps,
>          Jan Nijtmans

Thanks, Jan. As you mentioned, a recipe should be added to the migration 
notes. Here it is with some additional tests and fallback to 8859-1

proc winNativeCodepageGet {} {
	if {[catch {
		package require registry
		set cp [registry get 
HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\Nls\\CodePage ACP]
	}]} {return "iso8859-1"}
	if {$cp eq "65001"} {return "utf-8"}
	set cp "cp$cp"
	if {$cp in [encoding names]} {return $cp}
	return "iso8859-1"
}

Take care,
Harald

Re: [TCLCORE] Tcl_ObjType public surface

[Marc C. <cul...@gm...>]

On Thu, Jul 4, 2024 at 4:40 AM Harald Oehlmann <har...@el...>
wrote:

Marc has tried the new access function Tcl_GetValueFromObj with positive
> results:
>

Correction: I tried Tcl_GetNumberFromObj.

There is no such thing (yet) as Tcl_GetValueFromObj.   Creating that
function would require a new TIP, as Jan says in the ticket.

- Marc

Re: [TCLCORE] [MACTCL] Major update to the macOS port of Tk

[Marc C. <mar...@gm...>]

I think you are proposing to implement every widget as a subview of the
contentView, since a CALayer can only be attached to an NSView.  Apple says
in their documentation that performance is bad if a contentView has too
many subviews.  Tk windows can contain a huge number of widgets, especially
when an extension like tablelist is used.  So using a subview for every
widget is a bad idea.  (Feel free to try it, if you think Apple is wrong.)

On the other hand, one would not expect to have huge numbers of OpenGL or
Metal widgets in a window.  Such a widget could be implemented as a subview
which is managed by a Tk widget.  That is what is done it Togl and TkGL.

- Marc



On Thu, Jul 4, 2024 at 3:39 AM Uwe Kirschner <Uwe...@gm...> wrote:

> Hello Marc,
> why not have one CALayer per widget?
>
> That way one could easily implement OpenGL and Metal widgets
> (CAOpenGLLayer, and CAMetalLayer)
>
> In the one CALayer-per-toplevel scenario an OpenGL or Metal widget would
> need to first render to texture, then copy the texture to a rectangle in
> the toplevel's CGImage, and finally the toplevel's CALayer would have the
> system composit the CGImage onscreen. That seems wasteful in a situation
> where speed matters most...
>
> I work with a UIKit implementation for macOS called Chameleon which uses
> CALayer to implement UIView on AppKit (this was long before Apple started
> Catalyst). It works great and UIView can be thought of as the equivalent to
> a tk widget. That's why I believe one CALayer per tk widget is the way to
> go...
>
> Also, this would open the route to tcl/tk on iPadOS and visionOS (if the
> new tk drops HIToolbox).
>
>
> best regards,
> Uwe
>
>
> On 3. Jul 2024, at 18:12, Marc Culler <mar...@gm...> wrote:
>
> 
> Tk Timeline watchers may have noticed a long chain of commits to the
> cgimage_with_crossing branch over the last month.  This announcement is to
> explain what has been going on there.  In brief, that branch contains a
> major change to how drawing works in the macOS port of Tk.  This change
> makes drawing in the macOS port work in a way which is much closer to how
> it works in the other platforms, and much closer to what the generic Tk
> code expects. The result is better performance and increased stability.
>
> The work is based on an idea of Christopher Chavez's.  He realized that
> Apple's Appkit design allowed for an alternative drawing strategy, and
> wrote a proof-of-concept implementation.  I had created a branch containing
> his implementation a couple of years ago.  Recently I merged that branch
> with Tk 9 and then spent a few weeks ironing out wrinkles.  Nicolas Bats,
> Torsten Berg, Steve Landers, and Kevin Walzer kindly offered to help me by
> testing the new implementation on their apps, some of which provide very
> challenging environments for Tk.  The result of this work is in the
> cgimage-with-crossing branch.  (The branch also includes a fix for ticket
> [22349fc78a] which is about <Enter> and <Leave> events - hence the
> reference to crossing in the branch tag.)
>
> The five of us are confident that this is a significant improvement. Some
> highlights are:
>     * Better performance - CPU-intensive apps use about 20% fewer CPU
> cycles
>     * Better conformance - Most runs of the regression test suite on macOS
> 14 show no failures.  This is not close to being the case for the trunk
> branch.
>     * Fewer graphical artifacts (and work continues on those that remain.)
>     * Fewer crashes (none that we know of at the moment.)
>
> Given that the branch has been tested on large and complex apps - all of
> which are working well - this is a major step forwards and we’d like to see
> it tested more widely.  To that end, although we are late in the release
> cycle we are comfortable recommending that this be merged into trunk and
> made part of beta3.  That’s the only way we can guarantee wider testing. Of
> course, if any issues crop up they will be addressed in a timely manner.
> If you are a macOS tkchat user please download and try
> https://www.codebykevin.com/TkChat_Setup_1.5.2.dmg   If you want to try
> the new macOS Tk with your own code and are comfortable building it
> yourself please checkout and build the Tk cgimage_with_crossing branch
>
> The rest of this message gives some more details about the changes.
>
> The main difference is that the new drawing strategy does away with the
> asynchronous drawing via the drawRect method.  A macOS app which uses
> drawRect is supposed to do all of its drawing within the drawRect method,
> which can only be called by the NSApplication object.  The app can request
> that drawRect be called by setting the viewNeedsDisplay flag, but can not
> call drawRect.  The original design of the macOS Aqua port only partially
> complied with this strategy.  In fact it was possible to obtain a valid
> graphics context for drawing to the backing layer of a window's contentView
> even outside of the drawRect method.  The original port would do this, e.g.
> in a widget display proc which is being run as an idle task.  But it would
> also draw in the drawRect method by first generating expose events for all
> widgets that need updates, then processing those expose events to create
> idle tasks, then processing all of those idle tasks immediately.
>
> Obviously this scheme was not what Tk expected.  Nor was it what Apple
> expected.  And Apple put an abrupqt end to half of it with the release of
> macOS Mojave, which made it impossible to obtain a valid graphics context
> outside of the drawRect method. When Tk was first built on Mojave, it
> produced nothing but totally black windows.  Eventually this was worked
> around by arranging that any drawing operation which failed due to an
> invalid graphics context would be rescheduled and also to modify drawRect
> to make it (attempt to) run all of those rescheduled idle tasks.
>
> With the new macOS code it is once again true that Tk always has a valid
> graphics context available and hence can draw at any time.  It draws into a
> CGBitmapImageContext which serves as the backing store for a Tk Window.
> Instead of calling drawRect, it is configured with Apple's other option
> which is to call updateLayer instead.  In the new setup updateLayer
> installs the CGImage as the CALayer of the ContentView of the NSWindow
> which hosts the Tk toplevel, causing the contents of the image to appear in
> the window.  It also creates a new CGImage containing a copy of the window
> in which subsequent drawing operations take place.  This allows drawing
> operations to take place at the expected time and in the expected order.
> That makes for better stability and less unpredictability.  In addition, it
> removes the overhead that arises from attempting a drawing operation,
> having it fail due to an invalid graphics context, rescheduling and
> rerunning the operation.  This accounts for the reduced CPU usage.
>
> - Marc
> _______________________________________________
> Tcl-mac mailing list
> tc...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-mac
>
>

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Harald O. <har...@el...>]

Am 04.07.2024 um 14:17 schrieb Jan Nijtmans:
> Op zo 30 jun 2024 om 23:18 schreef Jan Nijtmans <jan...@gm...>:
>> It shouldn't be too difficult to get the old code-page from the
>> registry, it can be found in the following registry key:
>>       HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage\ACP
> 
> This is how to retrieve the code page it had in Tcl 8.x:
> 
>      $ tclsh90
>      % encoding system
>      utf-8
>      % package require registry
>      1.3.7
>      % registry get
> HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\Nls\\CodePage
> ACP
>      1252
> 
> Hope this helps,
>          Jan Nijtmans

Thanks, Jan. As you mentioned, a recipe should be added to the migration 
notes. Here it is with some additional tests and fallback to 8859-1

proc winNativeCodepageGet {} {
	if {[catch {
		package require registry
		set cp [registry get 
HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\Nls\\CodePage ACP]
	}]} {return "iso8859-1"}
	if {$cp eq "65001"} {return "utf-8"}
	set cp "cp$cp"
	if {$cp in [encoding names]} {return $cp}
	return "iso8859-1"
}

Take care,
Harald

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Jan N. <jan...@gm...>]

Op zo 30 jun 2024 om 23:18 schreef Jan Nijtmans <jan...@gm...>:
> It shouldn't be too difficult to get the old code-page from the
> registry, it can be found in the following registry key:
>      HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage\ACP

This is how to retrieve the code page it had in Tcl 8.x:

    $ tclsh90
    % encoding system
    utf-8
    % package require registry
    1.3.7
    % registry get
HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\Nls\\CodePage
ACP
    1252

Hope this helps,
        Jan Nijtmans

Re: [TCLCORE] Tcl_ObjType public surface

[Harald O. <har...@el...>]

Am 04.07.2024 um 09:39 schrieb Poor Yorick:
> Speaking of the long march to correct the mistakes of the past, it would
> also be good not to repeat the mistakes of the past.  One of the big
> mistakes of the past was exposing the details of the Tcl_Obj structure
> in tcl.h.  Don Porter of others have worked for years to move in the
> direction of correcting that mistake.  Now, Tcl is posed to repeat that
> mistake in the release of version 9 by expanding the public surface of the
> Tcl_ObjType structure.  It would be wise to correct that before the
> release of Tcl 9, and instead provide a functional interface for
> registering the implementations of functions that are now available for
> operating on a Tcl_Obj having a certain type.
> 

Dear Nathan,
thank you for the last two posts.
It is absolutely right to avoid the access to internal structures.
Marc has tried the new access function Tcl_GetValueFromObj with positive 
results:

https://core.tcl-lang.org/tcl/info/b5bd08df8d61b4ea

This is a great step and there is a good possibility for a more general 
solution.

Thanks for all,
Harald