tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Harald O. <har...@el...>]

Am 25.05.2025 um 13:40 schrieb Gustaf Neumann (sslmail):
> 
> 
>> I am only sponsoring "after for monotonic clock on WIndows". This is a critical patch absolutely needed and hits each user each day, probably without noticing the reason.
> 
> maybe i have missed the argument: why does it hit windows users more than unix/macos users?
> -g
I did not say, that it hits Linux users more often than Windows users.
I can not say anything on Unix or Mac-OS.
I can only say, that it hits Windows users.

Any insight on platform Unix/Linux or MacOS welcome. I can not help.

Sorry,
Harald

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Gustaf N. (sslmail) <ne...@wu...>]

> I am only sponsoring "after for monotonic clock on WIndows". This is a critical patch absolutely needed and hits each user each day, probably without noticing the reason.

maybe i have missed the argument: why does it hit windows users more than unix/macos users?
-g

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Harald O. <har...@el...>]

Am 24.05.2025 um 21:14 schrieb Christian Werner:
> On 05/24/2025 07:49 PM, Harald Oehlmann wrote:
> 
>> On multiple platforms: no comment, as there is no sponsor... I don't 
>> think, the merge done by me would even compile on Linux/MacOS, but who 
>> knows...
> 
> Fine, then. So now we are at these two obstacles (or are there more?):
> 
> * how the clock clocks after we closed and re-opened the lid of our 
> valued laptops
> * how clock monotonery may have impacts on the 20+year old definition of 
> "interp limit …"
>    which in my irrelevant opinion is a failed design from the start
> 
> If you are willing to sacrifice the fundamental problems of the 30+year 
> old implementation
> to these points, I wish you good luck with carrying forward all the 
> problems we'd like to
> have eagerly solved since decades. And I will gladly deviate from this 
> path with my own forkery.
> 
> Thanks for your attention,
> Christian
I can only say, that I do my best of the 3 wished actions:
- the numeric patch is in
- the blockcursor was even noticed by Francois, mega-great!
- the monotonic clock may come on Windows, other platforms: no sponsor, 
sorry
- SQLite break and dict patch in

About the "limit", I just have to handle the comment by Ashok, and I 
don't understand it. So, I tend to back this out, as there is again, no 
sponsor... I am only sponsoring "after for monotonic clock on WIndows". 
This is a critical patch absolutely needed and hits each user each day, 
probably without noticing the reason.

Thanks for all your continuous work.

Harald

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Christian W. <Chr...@t-...>]

On 05/24/2025 09:14 PM, Christian Werner wrote:

> …
> Thanks for your attention, …

And I had this week the idea to TIP a "return -code continue" in "after" handlers which
by default remember their "after" delay argument and do the Right Thing(tm) to reschedule
the command for this return code in order to allow for (admittedly slow) rate monotonic
control loops.

I shall dream, periodically,
Christian

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Christian W. <Chr...@t-...>]

On 05/24/2025 07:49 PM, Harald Oehlmann wrote:

> On multiple platforms: no comment, as there is no sponsor... I don't think, the merge done by me would even compile on Linux/MacOS, but who knows...

Fine, then. So now we are at these two obstacles (or are there more?):

* how the clock clocks after we closed and re-opened the lid of our valued laptops
* how clock monotonery may have impacts on the 20+year old definition of "interp limit …"
   which in my irrelevant opinion is a failed design from the start

If you are willing to sacrifice the fundamental problems of the 30+year old implementation
to these points, I wish you good luck with carrying forward all the problems we'd like to
have eagerly solved since decades. And I will gladly deviate from this path with my own forkery.

Thanks for your attention,
Christian

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Harald O. <har...@el...>]

Ashok,
thanks for the review, great. Perhaps, Christian may comment, if the 
"interp limit" includes the time in suspension.
If yes, I think, this would not be good.

I have no idea, how to use "interp limit". I never use "interp", as it 
causes mostly trouble. Great Emiliano has found issues with "interp" and 
"tk photo" which stops me on this path...

On multiple platforms: no comment, as there is no sponsor... I don't 
think, the merge done by me would even compile on Linux/MacOS, but who 
knows...

Thanks for all,
Harald

Am 24.05.2025 um 07:43 schrieb apnmbx-public--- via Tcl-Core:
> Harald,
> 
> The TIP only mentions system suspension in passing. It appears from the implementation on Windows that the time in suspension is also included in the counted interval. This should be clarified and if that is the case, perhaps it is not suitable for use with interp execution time limits. May be this is not important ...
> 
> Like Sergey, I am hesitant about different behavior between Windows and other platforms. Looking at the tickets, I am still unclear as to whether the issue for other platforms is technical or is it just a matter of someone taking the responsibility to port and validate the patches for other platforms. Is anyone working on the other platforms (a question for others, not Harald) ?
> 
> /Ashok
> 
> -----Original Message-----
> From: Harald Oehlmann <har...@el...>
> Sent: Friday, May 23, 2025 11:58 AM
> To: Dipl. Ing. Sergey G. Brester <se...@us...>
> Cc: Tcl Core List <tcl...@li...>
> Subject: Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only
> 
> Dear Sergey,
> 
> I appreciate your review and valuable opinion.
> Why Windows only is affected, I don't know. I can not help here. But it
> is a fact, that there is opposition to see this as a bugfix or no
> interest or both, as the patch is not reviewed or commented. The patch
> is very complicated compared to the 3 lines on Windows, true...
> For me, Windows only is the only way to bring this forward.
> 
> About the "summer time change". The concerned systems run without NTP
> and the change is done manually. So, the clock is manually turned back
> by one hour once per year.
> This may be detailed in the tip.
> 
> Thanks for all,
> Harald
> 
> 
> Am 22.05.2025 um 17:38 schrieb Dipl. Ing. Sergey G. Brester:
>> I don't know why it shall be limited to Windows only, because all 3
>> points are affecting *nix in the same way:
>>
>>     * manual time adjustment (is possible with `date -s`)
>>     * suspension of the system (suspend of VM, hibernation, etc)
>>     * automatic time adjustment (via ntpd and co)
>>
>> Also there is a mistake in the TIP - "specially on day time switching
>> +/- 1 hour". Because DST-switches normally doesn't adjust the clock in
>> UTC (GMT), which is running continuously. Everything what asking [clock
>> seconds], [clock milliseconds], [clock microseconds] (or related C-API
>> functions) etc is not affected by DST-switches at all.
>> The DST-switch happens only in related timezone (table of offsets stored
>> in tzdata) and doesn't exist without TZ context (pure UTC time).
>>
>> Just for illustration:
>>
>> # forward DST jump in CET-TZ:
>> % clock format [expr {1743296399 + 0}] -timezone :Europe/Berlin
>> Sun Mar 30 01:59:59 CET 2025
>> % clock format [expr {1743296399 + 1}] -timezone :Europe/Berlin
>> Sun Mar 30 03:00:00 CEST 2025
>>
>> # backward DST jump in CET-TZ:
>> % clock format [expr {1761440399 + 0}] -timezone :Europe/Berlin
>> Sun Oct 26 02:59:59 CEST 2025
>> % clock format [expr {1761440399 + 1}] -timezone :Europe/Berlin
>> Sun Oct 26 02:00:00 CET 2025
>>
>> The issues are only manual or automatic time adjustments affecting UTC-
>> time, and suspension/hibernation.
>> However as already said it definitely affecting all systems that using
>> wall clock (and need a switch to monotonic time).
>>
>> Regards,
>> Sergey.
>>
>> 21.05.2025 12:35, Harald Oehlmann wrote:
>>
>>> Dear TCL team,
>>>
>>> please allow me to propose a TIP to cure one of my biggest issues on TCL and MS-Windows: after does not fire if the wall clock changes.
>>>
>>> The TIP text is here:
>>> https://core.tcl-lang.org/tips/doc/trunk/tip/723.md <https://core.tcl-lang.org/tips/doc/trunk/tip/723.md>
>>>
>>> I am sorry to write it, as it is written (e.g. no respect on TIP233, Windows only). But those are the conclusions taken on recent discussions on the forelast biweekly telco and on the related bug tickets and on the core list.
>>>
>>> I hope we will find a better solution, but this is my "minimum viable outcome" to this very urgent and annoying issue.
>>>
>>> Thanks for all and take care,
>>> Harald
> 

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Gustaf N. (sslmail) <ne...@wu...>]

> On 24.05.2025, at 07:33, apnmbx-public--- via Tcl-Core <tcl...@li...> wrote:
> 
> Like Sergey, I am hesitant about different behavior between Windows and other platforms. 

+1

trying to provide a platform consistent behavior is one of the assets of Tcl since ever.
-g

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Christian W. <Chr...@t-...>]

On 05/24/2025 07:43 AM, apnmbx-public--- via Tcl-Core wrote:

Ashok, Harald, all

> …
> Like Sergey, I am hesitant about different behavior between Windows and other platforms. Looking at the tickets, I am still unclear as to whether the issue for other platforms is technical or is it just a matter of someone taking the responsibility to port and validate the patches for other platforms. Is anyone working on the other platforms (a question for others, not Harald) ?

let me add my two hundredths of the currency unit of your choice: Sergey's branch and my patches, both address
POSIX platforms as well. And both approaches suffer from lack of review and determination. Technically, it is
even feasible for those UN*Xen which don't fully support _POSIX_MONOTONIC_CLOCK (if at all) to perform a mixture
of build time and runtime checks in order to provide the best clock possible for the platform.

Again, let me emphasize that AndroWish and its siblings support the approach layed out in my patches since about
September 2016. Thus IMO it were a pity if we need to wait for another major release to have this feature.

Time will tell,
Christian

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[<apn...@ya...>]

Harald,

The TIP only mentions system suspension in passing. It appears from the implementation on Windows that the time in suspension is also included in the counted interval. This should be clarified and if that is the case, perhaps it is not suitable for use with interp execution time limits. May be this is not important ...

Like Sergey, I am hesitant about different behavior between Windows and other platforms. Looking at the tickets, I am still unclear as to whether the issue for other platforms is technical or is it just a matter of someone taking the responsibility to port and validate the patches for other platforms. Is anyone working on the other platforms (a question for others, not Harald) ?

/Ashok

-----Original Message-----
From: Harald Oehlmann <har...@el...> 
Sent: Friday, May 23, 2025 11:58 AM
To: Dipl. Ing. Sergey G. Brester <se...@us...>
Cc: Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

Dear Sergey,

I appreciate your review and valuable opinion.
Why Windows only is affected, I don't know. I can not help here. But it 
is a fact, that there is opposition to see this as a bugfix or no 
interest or both, as the patch is not reviewed or commented. The patch 
is very complicated compared to the 3 lines on Windows, true...
For me, Windows only is the only way to bring this forward.

About the "summer time change". The concerned systems run without NTP 
and the change is done manually. So, the clock is manually turned back 
by one hour once per year.
This may be detailed in the tip.

Thanks for all,
Harald


Am 22.05.2025 um 17:38 schrieb Dipl. Ing. Sergey G. Brester:
> I don't know why it shall be limited to Windows only, because all 3 
> points are affecting *nix in the same way:
> 
>    * manual time adjustment (is possible with `date -s`)
>    * suspension of the system (suspend of VM, hibernation, etc)
>    * automatic time adjustment (via ntpd and co)
> 
> Also there is a mistake in the TIP - "specially on day time switching 
> +/- 1 hour". Because DST-switches normally doesn't adjust the clock in 
> UTC (GMT), which is running continuously. Everything what asking [clock 
> seconds], [clock milliseconds], [clock microseconds] (or related C-API 
> functions) etc is not affected by DST-switches at all.
> The DST-switch happens only in related timezone (table of offsets stored 
> in tzdata) and doesn't exist without TZ context (pure UTC time).
> 
> Just for illustration:
> 
> # forward DST jump in CET-TZ:
> % clock format [expr {1743296399 + 0}] -timezone :Europe/Berlin
> Sun Mar 30 01:59:59 CET 2025
> % clock format [expr {1743296399 + 1}] -timezone :Europe/Berlin
> Sun Mar 30 03:00:00 CEST 2025
> 
> # backward DST jump in CET-TZ:
> % clock format [expr {1761440399 + 0}] -timezone :Europe/Berlin
> Sun Oct 26 02:59:59 CEST 2025
> % clock format [expr {1761440399 + 1}] -timezone :Europe/Berlin
> Sun Oct 26 02:00:00 CET 2025
> 
> The issues are only manual or automatic time adjustments affecting UTC- 
> time, and suspension/hibernation.
> However as already said it definitely affecting all systems that using 
> wall clock (and need a switch to monotonic time).
> 
> Regards,
> Sergey.
> 
> 21.05.2025 12:35, Harald Oehlmann wrote:
> 
>> Dear TCL team,
>>
>> please allow me to propose a TIP to cure one of my biggest issues on TCL and MS-Windows: after does not fire if the wall clock changes.
>>
>> The TIP text is here:
>> https://core.tcl-lang.org/tips/doc/trunk/tip/723.md <https://core.tcl-lang.org/tips/doc/trunk/tip/723.md>
>>
>> I am sorry to write it, as it is written (e.g. no respect on TIP233, Windows only). But those are the conclusions taken on recent discussions on the forelast biweekly telco and on the related bug tickets and on the core list.
>>
>> I hope we will find a better solution, but this is my "minimum viable outcome" to this very urgent and annoying issue.
>>
>> Thanks for all and take care,
>> Harald

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[<apn...@ya...>]

Harald,

The TIP only mentions system suspension in passing. It appears from the implementation on Windows that the time in suspension is also included in the counted interval. This should be clarified and if that is the case, perhaps it is not suitable for use with interp execution time limits. May be this is not important ...

Like Sergey, I am hesitant about different behavior between Windows and other platforms. Looking at the tickets, I am still unclear as to whether the issue for other platforms is technical or is it just a matter of someone taking the responsibility to port and validate the patches for other platforms. Is anyone working on the other platforms (a question for others, not Harald) ?

/Ashok

-----Original Message-----
From: Harald Oehlmann <har...@el...> 
Sent: Friday, May 23, 2025 11:58 AM
To: Dipl. Ing. Sergey G. Brester <se...@us...>
Cc: Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

Dear Sergey,

I appreciate your review and valuable opinion.
Why Windows only is affected, I don't know. I can not help here. But it 
is a fact, that there is opposition to see this as a bugfix or no 
interest or both, as the patch is not reviewed or commented. The patch 
is very complicated compared to the 3 lines on Windows, true...
For me, Windows only is the only way to bring this forward.

About the "summer time change". The concerned systems run without NTP 
and the change is done manually. So, the clock is manually turned back 
by one hour once per year.
This may be detailed in the tip.

Thanks for all,
Harald


Am 22.05.2025 um 17:38 schrieb Dipl. Ing. Sergey G. Brester:
> I don't know why it shall be limited to Windows only, because all 3 
> points are affecting *nix in the same way:
> 
>    * manual time adjustment (is possible with `date -s`)
>    * suspension of the system (suspend of VM, hibernation, etc)
>    * automatic time adjustment (via ntpd and co)
> 
> Also there is a mistake in the TIP - "specially on day time switching 
> +/- 1 hour". Because DST-switches normally doesn't adjust the clock in 
> UTC (GMT), which is running continuously. Everything what asking [clock 
> seconds], [clock milliseconds], [clock microseconds] (or related C-API 
> functions) etc is not affected by DST-switches at all.
> The DST-switch happens only in related timezone (table of offsets stored 
> in tzdata) and doesn't exist without TZ context (pure UTC time).
> 
> Just for illustration:
> 
> # forward DST jump in CET-TZ:
> % clock format [expr {1743296399 + 0}] -timezone :Europe/Berlin
> Sun Mar 30 01:59:59 CET 2025
> % clock format [expr {1743296399 + 1}] -timezone :Europe/Berlin
> Sun Mar 30 03:00:00 CEST 2025
> 
> # backward DST jump in CET-TZ:
> % clock format [expr {1761440399 + 0}] -timezone :Europe/Berlin
> Sun Oct 26 02:59:59 CEST 2025
> % clock format [expr {1761440399 + 1}] -timezone :Europe/Berlin
> Sun Oct 26 02:00:00 CET 2025
> 
> The issues are only manual or automatic time adjustments affecting UTC- 
> time, and suspension/hibernation.
> However as already said it definitely affecting all systems that using 
> wall clock (and need a switch to monotonic time).
> 
> Regards,
> Sergey.
> 
> 21.05.2025 12:35, Harald Oehlmann wrote:
> 
>> Dear TCL team,
>>
>> please allow me to propose a TIP to cure one of my biggest issues on TCL and MS-Windows: after does not fire if the wall clock changes.
>>
>> The TIP text is here:
>> https://core.tcl-lang.org/tips/doc/trunk/tip/723.md <https://core.tcl-lang.org/tips/doc/trunk/tip/723.md>
>>
>> I am sorry to write it, as it is written (e.g. no respect on TIP233, Windows only). But those are the conclusions taken on recent discussions on the forelast biweekly telco and on the related bug tickets and on the core list.
>>
>> I hope we will find a better solution, but this is my "minimum viable outcome" to this very urgent and annoying issue.
>>
>> Thanks for all and take care,
>> Harald

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[<apn...@ya...>]

Thanks, Jan. Will start a vote shortly.

 

/Ashok

 

From: Jan Nijtmans <jan...@gm...> 
Sent: Friday, May 23, 2025 8:04 PM
To: apn...@ya...
Cc: Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] About TIP 716 vs 718 vs other options

 

Op vr 23 mei 2025 om 05:04 schreef apnmbx-public:

TIP 716 is in its final proposed form. Will plan a CFV in a few days once people have chewed on the 716 vs 718 discussion.

 

Changes in the TIP and implementation:

 

- the code page is obtained from the registry and not GetACP. See Jan's ticket at https://core.tcl-lang.org/tcl/info/c776eb586d08c2f7

 

- the TCL_EXEC_ENCODING environment has been removed as no one has commented on it. Note the -encoding option to exec remains.

 

With those 2 changes (thanks!) my main issues with TIP #716 disappeared.

I can go along with it, so we don't need to waste too much time discussing the

difference between the two TIP's. I won't start a vote for #718.

 

Hope this helps,

      Jan Nijtmans

 

 

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[Jan N. <jan...@gm...>]

Op vr 23 mei 2025 om 05:04 schreef apnmbx-public:

> TIP 716 is in its final proposed form. Will plan a CFV in a few days once
> people have chewed on the 716 vs 718 discussion.
>
>
>
> Changes in the TIP and implementation:
>
>
>
> - the code page is obtained from the registry and not GetACP. See Jan's
> ticket at https://core.tcl-lang.org/tcl/info/c776eb586d08c2f7
>
>
>
> - the TCL_EXEC_ENCODING environment has been removed as no one has
> commented on it. Note the -encoding option to exec remains.
>

With those 2 changes (thanks!) my main issues with TIP #716 disappeared.
I can go along with it, so we don't need to waste too much time discussing
the
difference between the two TIP's. I won't start a vote for #718.

Hope this helps,
      Jan Nijtmans

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[Harald O. <har...@el...>]

Dear Ashok, dear Jan,
thanks for all the action, great.

Here are my two pence:

- on exec codepage with environment variable: if I remember right, I had 
commented this. IMHO, environment variables are difficult on Windows and 
they open another door of external influence which may break program 
stability. I thing, the script-level only is ok.

- on TIP 716/718: I think, a mouse is getting an elephant. I prefer 716. 
I don't like the hack-style of the manifest which half-fixes broken 
design and brings legacy dlls (which is the target) in danger for no 
gain. Thus: 716++. I like, that you two work together. The best idea 
should win, independent on the author.

- About the list stuff: great stuff, ship it!

THanks for all,
Harald


Am 23.05.2025 um 05:03 schrieb apnmbx-public--- via Tcl-Core:
> TIP 716 is in its final proposed form. Will plan a CFV in a few days 
> once people have chewed on the 716 vs 718 discussion.
> 
> Changes in the TIP and implementation:
> 
> - the code page is obtained from the registry and not GetACP. See Jan's 
> ticket at https://core.tcl-lang.org/tcl/info/c776eb586d08c2f7 <https:// 
> core.tcl-lang.org/tcl/info/c776eb586d08c2f7>
> 
> - the TCL_EXEC_ENCODING environment has been removed as no one has 
> commented on it. Note the -encoding option to exec remains.
> 
> - a section comparing TIP 718 has been added to the TIP. In particular, 
> in addition to what's been already said in this thread, it responds to 
> the section in 718 about the convenience of its TclWinGetUserEncoding 
> function. TL;DR the equivalent function is also available in the 716 
> implementation but #if 0'ed because I am not sure it is the right API. 
> See https://core.tcl-lang.org/tips/doc/trunk/tip/716.md#RelationtoTIP718 
> <https://core.tcl-lang.org/tips/doc/trunk/tip/716.md#RelationtoTIP718> 
> for details.
> 
> /Ashok
> 
> *From:*Kevin Walzer <kw...@54...>
> *Sent:* Thursday, May 22, 2025 1:33 AM
> *To:* apn...@ya...
> *Cc:* Marc Culler <cul...@gm...>; Tcl Core List <tcl- 
> co...@li...>
> *Subject:* Re: [TCLCORE] About TIP 716 vs 718 vs other options
> 
> Image removed by sender.
> 
> Agree that TIP 716 makes the most sense.
> 
> 
> 
>     On May 21, 2025, at 11:59 AM, apnmbx-public--- via Tcl-Core <tcl-
>     co...@li...> wrote:
> 
>     
> 
>     Marc,
> 
>     Thanks for taking the time to read and comment.
> 
>     FWIW, I agree 9.1 should formally standardize on UTF-8 on other
>     platforms as well. It just does not make sense to force UTF-8 only
>     on Windows.
> 
>     It needs a TIP though.
> 
>     /Ashok
> 
>     *From:*Marc Culler <cul...@gm...>
>     *Sent:* Wednesday, May 21, 2025 7:39 PM
>     *To:* apn...@ya...
>     *Cc:* Tcl Core List <tcl...@li...>
>     *Subject:* Re: [TCLCORE] About TIP 716 vs 718 vs other options
> 
>     Based on Ashok's description of the situation, TIP #716 seems to me
>     to be the best choice.  Shipping two tclsh executables seems likely
>     to cause confusion and problems, and I am also skeptical about
>     "automatic" support.  The first two options look to me like cans of
>     worms to be opened in the future.
> 
>     Not that it matters for this discussion, but I am in favor of
>     "forcing" utf-8 on our users.  Standardization is very important,
>     and it is too late for any other encoding to become the "standard".
> 
>     - Marc
> 
>     On Tue, May 20, 2025 at 9:41 PM apnmbx-public--- via Tcl-Core <tcl-
>     co...@li... <mailto:tcl...@li...>>
>     wrote:
> 
>         All,
> 
>         We need to make a collective decision regarding the default
>         encoding issues on Windows. In the last meeting, I promised to
>         write up a summary of the status and considerations for the
>         problem being addressed by TIP's 716 and 718. Here it is along
>         with the choices we have as to the next step.
> 
>         Although specific to Windows, the basic issue does not need any
>         deep understanding of Windows so I hope folks will read through
>         and opine on the matter without copping out that they do not
>         have Windows knowledge :-).
> 
>         For those who have not read 716 or 718 (particularly the
>         Background section in 716) or did not find the explanations
>         clear, here is an example scenario the TIP's are independently
>         addressing that I hope will shed some light.
> 
>         A client library, linked to a Tcl extension, shares data (config
>         info, db content etc.) with another program such as a server.
>         The encoding used for the data is that returned by the Windows
>         GetACP() call which returns the user's code page. Note that
>         neither the library nor the server side is aware of Tcl's
>         existence in any form except for the library being linked to a
>         Tcl extension loaded into tclsh.
> 
>         This scenario worked fine in Tcl 8.6 because both the client and
>         server ends saw the same encoding value returned from GetACP().
> 
>         Tcl 9.0 broke the scenario because of the addition of the
>         activeCodePage entry in the executable manifest for tclsh and
>         wish. *The presence of this setting means all calls to GetACP()
>         from that process, including those from the loaded client
>         library, will return the code page utf-8 irrespective of the
>         user setting.* The breakage may be because now the library
>         encodes using utf-8 and the server component encodes using the
>         user's real code page. Or it may be because the library cannot
>         handle non-DBCS multibyte encodings. Does not really matter. The
>         primary point is that the issue cannot be fixed by modifying or
>         recompiling the extension.
> 
>         Moving forward, we can deal with this problem in one of several
>         ways:
> 
>         (1) Remove the activeCodePage entry from the manifest with no
>         other changes. This will revert 9.x behavior on Windows to that
>         of 8.6 (and also consistent with non-Windows platforms since
>         user settings are not ignored on them).
> 
>         (2) Ignore the issue. In scenarios where this is an issue, users
>         can continue to use 8.6 or build tclsh/wish themselves after
>         removing the activeCodePage entry from the manifest.
> 
>         (3) TIP 716 as detailed below.
> 
>         (4) TIP 718 as detailed below.
> 
>         Going with (1) means that 9.0.2 will not be compatible with
>         9.0.1 with respect to default encodings. For example, non-ASCII
>         files written with writeFile in 9.0.1 will not be readable using
>         readFile in 9.0.2 and vice versa. This will be a pain point
>         users. Further, if as is possible, UTF-8 is made the default for
>         Tcl on all platforms at some point in the future, there will be
>         whole another round of incompatibility headaches. So as much as
>         I wish the UTF-8 defaulting had never been made without
>         discussion, 9.0 is already out there so I am not in favour of (1).
> 
>         Option (2) may be a candidate for discussion. We can just tell
>         users in this situation to stick to 8.6 or build their own Tcl
>         with the understanding that it will have similar compatibility
>         issues with respect to "standard" Tcl as in (1) above. Not our
>         problem so to speak. But also to be considered is the fact that
>         the TPC/HammerDB is currently one of the most popular Tcl
>         applications based on public download numbers. Similar
>         situations may arise with other large applications. I am
>         therefore skeptical that this is a viable solution.
> 
>         Details behind options (3) and (4) are in the respective TIP's.
>         Leaving aside the minor "reconcilable" differences with respect
>         to implementation of [encoding user] or [exec -encoding], here
>         is the crucial difference between the two.
> 
>         * TIP 716 removes the manifest entry from tclsh and wish. GetACP
>         then returns the user's code page setting from the registry
>         allowing the HammerDB-like applications to run. At the same
>         time, in order to keep compatibility with 9.0.0/1, it hardcodes
>         utf-8 as the default encoding returned by [encoding system] et
>         al. In my opinion, if we indeed wanted to force utf-8 onto
>         users, this is the way it should have been done in the first place.
> 
>         * TIP 718 deals with the problem by proposing to ship two Tcl
>         shells, the current tclsh and a second one, tclshc, which has
>         the manifest removed. The tclsh shell will behave identically to
>         the one in 9.0.0/1. tclshc will behave substantially similar to
>         TIP 716 and can be used by HammerDB etc. which do not work with
>         tclsh.
> 
>         The primary hesitation I have with 718 is this dual shell
>         approach and the potential for confusion. Somehow a user needs
>         to know to use tclsh for all scripts. Except (for example) when
>         accessing DB2. And X, Y and Z as the case may be. How are they
>         to make that determination? Will a scripted application author
>         now have to tell the user which Tcl shell to use? In fairness,
>         most scripts will work with either. Still, this is a point for
>         potential confusion. It is also the case that extension writers
>         will now have test their extensions with both variations (not
>         that they will!).
> 
>         The advantage of tclsh (but not tclshc) in 718 over 716 (I am
>         paraphrasing Jan here from the TIP 718 rationale, so he can
>         clarify if I got it wrong) is that extensions that make use of
>         the Windows ANSI API will automatically get UTF-8 support
>         thereby supporting the entire Unicode range. With TIP 716 as
>         well as 8.6, the ANSI API's will work only as long as the
>         characters are supported by the user's code page.
> 
>         My counterpoint to this argument is that (a) extensions should
>         not be using Windows ANSI API's in the first place, they should
>         use the Unicode API's, (b) since 8.6 did not support this
>         utf-8+ANSI API combination, such extensions are likely rare or
>         old and in need of update anyways, and (c) the "automatic"
>         support is likely overstated as not all API's and libraries
>         support UTF-8 even when set as the code page (see TIP 716).
> 
>         The question to answer is then whether this perceived benefit of
>         718 is overrides the downside to shipping two tclsh variations.
> 
>         A lesser issue is that 718 does not include the -encoding option
>         to exec as I believe Jan thinks it should be a separate TIP.
>         That is not difficult to resolve if we agree the option is needed.
> 
>         The decisions for the community to make now are which of the
>         four options listed above make sense moving forward. Important
>         that we make a decision for 9.0.2 as the longer we wait, the
>         more 9.0.0/1 applications are going to be out there that might
>         face this issue.
> 
>         /Ashok
> 

Re: [TCLCORE] OpenACS/TCL/Tk conference 2025 in Bologna/Italy/Europe on 10/11th of July 2025

[Harald O. <har...@el...>]

Dear OpenACS/TCL/Tk enthusiasts,

the hotel recommendation to "Royel Hotel Carlton" will end 1st of June.

https://openacs.km.at/evaluate/org/129998253/conferencenews/

The hotel should be called "Tickle Basecamp", but this will change ;-).

The 23th of June 17:00 will be the program telco. Expect a program the 
24th of June.

See you all there !

Take care,
Harald

Am 21.05.2025 um 10:51 schrieb Harald Oehlmann:
> Dear TCL/Tk enthusiasts,
> please allow me to invite you all to the
> OpenACS/TCL/Tk conference 2025 in Bologna/Italy/Europe
> 
> https://openacs.km.at/evaluate/org/129998253/conferencenews/
> 
> it is a user meeting and we all show our own solutions.
> As the minimum participants of 20 is reached, I can confirm that it will 
> take place.
> 
> The so far presentations on my radar are:
> 
> - Tcl and Tk pannel
> - Androwish Coffee machine
> - Csabas new slider checkbox
> - webcam barcode reading by me
> 
> So, there is still a lot of space for your contribution.
> Any contribution is welcome, 5 minutes, no slides, is ok too.
> We need you !
> The interesting "abstract" entry field on the web site requires 200 
> characters. I have filled it with "1234567890..." for my own 
> contribution...
> 
> There is no online participation. It may happen, that the event is 
> streamed. The chat will be monitored.
> 
> It would be great to see you in Bologna!
> Harald
> On behalf of the organisation committee
> 

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Harald O. <har...@el...>]

Dear Sergey,

I appreciate your review and valuable opinion.
Why Windows only is affected, I don't know. I can not help here. But it 
is a fact, that there is opposition to see this as a bugfix or no 
interest or both, as the patch is not reviewed or commented. The patch 
is very complicated compared to the 3 lines on Windows, true...
For me, Windows only is the only way to bring this forward.

About the "summer time change". The concerned systems run without NTP 
and the change is done manually. So, the clock is manually turned back 
by one hour once per year.
This may be detailed in the tip.

Thanks for all,
Harald


Am 22.05.2025 um 17:38 schrieb Dipl. Ing. Sergey G. Brester:
> I don't know why it shall be limited to Windows only, because all 3 
> points are affecting *nix in the same way:
> 
>    * manual time adjustment (is possible with `date -s`)
>    * suspension of the system (suspend of VM, hibernation, etc)
>    * automatic time adjustment (via ntpd and co)
> 
> Also there is a mistake in the TIP - "specially on day time switching 
> +/- 1 hour". Because DST-switches normally doesn't adjust the clock in 
> UTC (GMT), which is running continuously. Everything what asking [clock 
> seconds], [clock milliseconds], [clock microseconds] (or related C-API 
> functions) etc is not affected by DST-switches at all.
> The DST-switch happens only in related timezone (table of offsets stored 
> in tzdata) and doesn't exist without TZ context (pure UTC time).
> 
> Just for illustration:
> 
> # forward DST jump in CET-TZ:
> % clock format [expr {1743296399 + 0}] -timezone :Europe/Berlin
> Sun Mar 30 01:59:59 CET 2025
> % clock format [expr {1743296399 + 1}] -timezone :Europe/Berlin
> Sun Mar 30 03:00:00 CEST 2025
> 
> # backward DST jump in CET-TZ:
> % clock format [expr {1761440399 + 0}] -timezone :Europe/Berlin
> Sun Oct 26 02:59:59 CEST 2025
> % clock format [expr {1761440399 + 1}] -timezone :Europe/Berlin
> Sun Oct 26 02:00:00 CET 2025
> 
> The issues are only manual or automatic time adjustments affecting UTC- 
> time, and suspension/hibernation.
> However as already said it definitely affecting all systems that using 
> wall clock (and need a switch to monotonic time).
> 
> Regards,
> Sergey.
> 
> 21.05.2025 12:35, Harald Oehlmann wrote:
> 
>> Dear TCL team,
>>
>> please allow me to propose a TIP to cure one of my biggest issues on TCL and MS-Windows: after does not fire if the wall clock changes.
>>
>> The TIP text is here:
>> https://core.tcl-lang.org/tips/doc/trunk/tip/723.md <https://core.tcl-lang.org/tips/doc/trunk/tip/723.md>
>>
>> I am sorry to write it, as it is written (e.g. no respect on TIP233, Windows only). But those are the conclusions taken on recent discussions on the forelast biweekly telco and on the related bug tickets and on the core list.
>>
>> I hope we will find a better solution, but this is my "minimum viable outcome" to this very urgent and annoying issue.
>>
>> Thanks for all and take care,
>> Harald

Re: [TCLCORE] New internal list representations

[<apn...@ya...>]

Last call on the internal list rep changes below. Brian's looked it over. If
any one else concerns please speak up now. Else will merge into trunk for
9.1a0 in the next couple of days.

 

/Ashok

 

From: apnadkarni--- via Tcl-Core <tcl...@li...> 
Sent: Thursday, May 15, 2025 9:53 PM
To: 'Tcl Core' <tcl...@li...>
Subject: [TCLCORE] New internal list representations

 

https://core.tcl-lang.org/tcl/wiki?name=New+abstract+list+representations
<https://core.tcl-lang.org/tcl/wiki?name=New+abstract+list+representations&p
> &p is a proposal to implement some list operations using new internal list
representations enabled by Brian's TIP 636. The wiki post has the full
details but TL;DR memory efficiency at a small (probably fixable) cost in
iteration speed for large lists. The post has the numbers.

 

The apn-tip636-appl branch contains the implementation of the proposal. The
listTypes.test file contains the test cases to exercise the three new
internal list types as well as the existing ones. Passes -enable-symbols=mem
and valgrind OMM. gcov shows 91% code and 100% function coverage of the new
implementation. Missing coverage is for "panic blocks" and validation which
is not exercised as it is already done by upper callers.

 

Not planning a TIP as there are no changes at the script or C API level.

 

Please review (both proposal and code) and raise any objections you may
have. (It's not that much code to review and pretty isolated thanks to 636).

 

/Ashok

 

 

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[<apn...@ya...>]

TIP 716 is in its final proposed form. Will plan a CFV in a few days once people have chewed on the 716 vs 718 discussion.

 

Changes in the TIP and implementation:

 

- the code page is obtained from the registry and not GetACP. See Jan's ticket at https://core.tcl-lang.org/tcl/info/c776eb586d08c2f7

 

- the TCL_EXEC_ENCODING environment has been removed as no one has commented on it. Note the -encoding option to exec remains.

 

- a section comparing TIP 718 has been added to the TIP. In particular, in addition to what's been already said in this thread, it responds to the section in 718 about the convenience of its TclWinGetUserEncoding function. TL;DR the equivalent function is also available in the 716 implementation but #if 0'ed because I am not sure it is the right API. See https://core.tcl-lang.org/tips/doc/trunk/tip/716.md#RelationtoTIP718 for details.

 

/Ashok

 

From: Kevin Walzer <kw...@54...> 
Sent: Thursday, May 22, 2025 1:33 AM
To: apn...@ya...
Cc: Marc Culler <cul...@gm...>; Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] About TIP 716 vs 718 vs other options

 



Agree that TIP 716 makes the most sense. 





On May 21, 2025, at 11:59 AM, apnmbx-public--- via Tcl-Core <tcl...@li...> wrote:



Marc,

 

Thanks for taking the time to read and comment.

 

FWIW, I agree 9.1 should formally standardize on UTF-8 on other platforms as well. It just does not make sense to force UTF-8 only on Windows.

 

It needs a TIP though.

 

/Ashok

 

 

 

From: Marc Culler <cul...@gm...> 
Sent: Wednesday, May 21, 2025 7:39 PM
To: apn...@ya...
Cc: Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] About TIP 716 vs 718 vs other options

 

Based on Ashok's description of the situation, TIP #716 seems to me to be the best choice.  Shipping two tclsh executables seems likely to cause confusion and problems, and I am also skeptical about "automatic" support.  The first two options look to me like cans of worms to be opened in the future.

 

Not that it matters for this discussion, but I am in favor of "forcing" utf-8 on our users.  Standardization is very important, and it is too late for any other encoding to become the "standard".

 

- Marc

 

On Tue, May 20, 2025 at 9:41 PM apnmbx-public--- via Tcl-Core <tcl...@li... <mailto:tcl...@li...> > wrote:

All,

 

We need to make a collective decision regarding the default encoding issues on Windows. In the last meeting, I promised to write up a summary of the status and considerations for the problem being addressed by TIP's 716 and 718. Here it is along with the choices we have as to the next step.

 

Although specific to Windows, the basic issue does not need any deep understanding of Windows so I hope folks will read through and opine on the matter without copping out that they do not have Windows knowledge :-).

 

For those who have not read 716 or 718 (particularly the Background section in 716) or did not find the explanations clear, here is an example scenario the TIP's are independently addressing that I hope will shed some light.

 

A client library, linked to a Tcl extension, shares data (config info, db content etc.) with another program such as a server. The encoding used for the data is that returned by the Windows GetACP() call which returns the user's code page. Note that neither the library nor the server side is aware of Tcl's existence in any form except for the library being linked to a Tcl extension loaded into tclsh.

 

This scenario worked fine in Tcl 8.6 because both the client and server ends saw the same encoding value returned from GetACP().

 

Tcl 9.0 broke the scenario because of the addition of the activeCodePage entry in the executable manifest for tclsh and wish. The presence of this setting means all calls to GetACP() from that process, including those from the loaded client library, will return the code page utf-8 irrespective of the user setting. The breakage may be because now the library encodes using utf-8 and the server component encodes using the user's real code page. Or it may be because the library cannot handle non-DBCS multibyte encodings. Does not really matter. The primary point is that the issue cannot be fixed by modifying or recompiling the extension.

 

Moving forward, we can deal with this problem in one of several ways:

 

(1) Remove the activeCodePage entry from the manifest with no other changes. This will revert 9.x behavior on Windows to that of 8.6 (and also consistent with non-Windows platforms since user settings are not ignored on them).

 

(2) Ignore the issue. In scenarios where this is an issue, users can continue to use 8.6 or build tclsh/wish themselves after removing the activeCodePage entry from the manifest.

 

(3) TIP 716 as detailed below.

 

(4) TIP 718 as detailed below.

 

Going with (1) means that 9.0.2 will not be compatible with 9.0.1 with respect to default encodings. For example, non-ASCII files written with writeFile in 9.0.1 will not be readable using readFile in 9.0.2 and vice versa. This will be a pain point users. Further, if as is possible, UTF-8 is made the default for Tcl on all platforms at some point in the future, there will be whole another round of incompatibility headaches. So as much as I wish the UTF-8 defaulting had never been made without discussion, 9.0 is already out there so I am not in favour of (1).

 

Option (2) may be a candidate for discussion. We can just tell users in this situation to stick to 8.6 or build their own Tcl with the understanding that it will have similar compatibility issues with respect to "standard" Tcl as in (1) above. Not our problem so to speak. But also to be considered is the fact that the TPC/HammerDB is currently one of the most popular Tcl applications based on public download numbers. Similar situations may arise with other large applications. I am therefore skeptical that this is a viable solution.

 

Details behind options (3) and (4) are in the respective TIP's. Leaving aside the minor "reconcilable" differences with respect to implementation of [encoding user] or [exec -encoding], here is the crucial difference between the two.

 

* TIP 716 removes the manifest entry from tclsh and wish. GetACP then returns the user's code page setting from the registry allowing the HammerDB-like applications to run. At the same time, in order to keep compatibility with 9.0.0/1, it hardcodes utf-8 as the default encoding returned by [encoding system] et al. In my opinion, if we indeed wanted to force utf-8 onto users, this is the way it should have been done in the first place.

 

* TIP 718 deals with the problem by proposing to ship two Tcl shells, the current tclsh and a second one, tclshc, which has the manifest removed. The tclsh shell will behave identically to the one in 9.0.0/1. tclshc will behave substantially similar to TIP 716 and can be used by HammerDB etc. which do not work with tclsh.

 

The primary hesitation I have with 718 is this dual shell approach and the potential for confusion. Somehow a user needs to know to use tclsh for all scripts. Except (for example) when accessing DB2. And X, Y and Z as the case may be. How are they to make that determination? Will a scripted application author now have to tell the user which Tcl shell to use? In fairness, most scripts will work with either. Still, this is a point for potential confusion. It is also the case that extension writers will now have test their extensions with both variations (not that they will!).

 

The advantage of tclsh (but not tclshc) in 718 over 716 (I am paraphrasing Jan here from the TIP 718 rationale, so he can clarify if I got it wrong) is that extensions that make use of the Windows ANSI API will automatically get UTF-8 support thereby supporting the entire Unicode range. With TIP 716 as well as 8.6, the ANSI API's will work only as long as the characters are supported by the user's code page.

 

My counterpoint to this argument is that (a) extensions should not be using Windows ANSI API's in the first place, they should use the Unicode API's, (b) since 8.6 did not support this utf-8+ANSI API combination, such extensions are likely rare or old and in need of update anyways, and (c) the "automatic" support is likely overstated as not all API's and libraries support UTF-8 even when set as the code page (see TIP 716).

 

The question to answer is then whether this perceived benefit of 718 is overrides the downside to shipping two tclsh variations.

 

A lesser issue is that 718 does not include the -encoding option to exec as I believe Jan thinks it should be a separate TIP. That is not difficult to resolve if we agree the option is needed.

 

The decisions for the community to make now are which of the four options listed above make sense moving forward. Important that we make a decision for 9.0.2 as the longer we wait, the more 9.0.0/1 applications are going to be out there that might face this issue.

 

/Ashok

_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core <https://fedbdhd.r.bh.d.sendibt3.com/tr/cl/dN96ZIXkuEXleedxhrmxUa9SsHYQ5pGEMHlQ6MmZobGmdF1QwvSu6ONBozUHXxcnGGtRlmGXHg46XdECmJr00-SATPFcfGskPV62GDTb6NUEJtyWrdBS7lftPd6KBaZiXEceJP6GaUWNgBcOMQeWzzBb9LEa4pZkTyyWuw2qD4URCkRwbecry05LcrKdNwZMsZeVi7EVgymYhhizEIVwttoruoq_NR0yzrTnrRL7NZw3BdhptnAeP7pCHHb2l1RoaCAc-meYgBmAc3Gw3f9PRgFQ_UHdgyBAvfvIzxfuRTDEYsV-X8sRRSRK> 

_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Dipl. I. S. G. B. <se...@us...>]

 

I don't know why it shall be limited to Windows only, because all 3
points are affecting *nix in the same way: 

 * manual time adjustment (is possible with `date -s`)
 * suspension of the system (suspend of VM, hibernation, etc)
 * automatic time adjustment (via ntpd and co) 

Also there is a mistake in the TIP - "specially on day time switching
+/- 1 hour". Because DST-switches normally doesn't adjust the clock in
UTC (GMT), which is running continuously. Everything what asking [clock
seconds], [clock milliseconds], [clock microseconds] (or related C-API
functions) etc is not affected by DST-switches at all.
The DST-switch happens only in related timezone (table of offsets stored
in tzdata) and doesn't exist without TZ context (pure UTC time).

Just for illustration: 

# forward DST jump in CET-TZ:
% clock format [expr {1743296399 + 0}] -timezone :Europe/Berlin
Sun Mar 30 01:59:59 CET 2025
% clock format [expr {1743296399 + 1}] -timezone :Europe/Berlin
Sun Mar 30 03:00:00 CEST 2025

# backward DST jump in CET-TZ:
% clock format [expr {1761440399 + 0}] -timezone :Europe/Berlin
Sun Oct 26 02:59:59 CEST 2025
% clock format [expr {1761440399 + 1}] -timezone :Europe/Berlin
Sun Oct 26 02:00:00 CET 2025 

The issues are only manual or automatic time adjustments affecting
UTC-time, and suspension/hibernation.
However as already said it definitely affecting all systems that using
wall clock (and need a switch to monotonic time).

Regards,
Sergey. 

21.05.2025 12:35, Harald Oehlmann wrote: 

> Dear TCL team,
> 
> please allow me to propose a TIP to cure one of my biggest issues on TCL and MS-Windows: after does not fire if the wall clock changes.
> 
> The TIP text is here:
> https://core.tcl-lang.org/tips/doc/trunk/tip/723.md [1]
> 
> I am sorry to write it, as it is written (e.g. no respect on TIP233, Windows only). But those are the conclusions taken on recent discussions on the forelast biweekly telco and on the related bug tickets and on the core list.
> 
> I hope we will find a better solution, but this is my "minimum viable outcome" to this very urgent and annoying issue.
> 
> Thanks for all and take care,
> Harald
> 
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core [2]
 

Links:
------
[1] https://core.tcl-lang.org/tips/doc/trunk/tip/723.md
[2] https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[Kevin W. <kw...@co...>]

<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><img width="1" height="1" src="https://fedbdhd.r.tsp1-brevo.net/tr/op/HiJ4uqGoaWJaQppIUaYaT1sIcrvSIyZwyuMilMtUiR0aKFHmblboWPgIHuTNDuGh4KbnD-1QGiOukkOKVNEd6Oyl84_2qhwAI-iHVPL4PhCuEdmQ9RMG-SOjCF8wS2ZrOiowaRLafy_q9aYa0ktqibrZjg2FmeYSLL1-83MXzjlC7M0KQ5Vgj_kVQFkmPtP3vBB4tJC69j5qLCVJvnfCo5FoXKpU" style="mso-hide:all"/><div dir="ltr"></div><div dir="ltr">Agree that TIP 716 makes the most sense.&nbsp;</div><div dir="ltr"><br><blockquote type="cite">On May 21, 2025, at 11:59 AM, apnmbx-public--- via Tcl-Core &lt;tcl...@li...&gt; wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta name="Generator" content="Microsoft Word 15 (filtered medium)"><style>@font-face { font-family: "Cambria Math"; }
@font-face { font-family: Calibri; }
@font-face { font-family: Aptos; }
p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in; font-size: 12pt; font-family: Aptos, sans-serif; }
a:link, span.MsoHyperlink { color: blue; text-decoration: underline; }
span.EmailStyle18 { font-family: Aptos, sans-serif; color: windowtext; }
.MsoChpDefault { font-size: 11pt; }
@page WordSection1 { size: 8.5in 11in; margin: 1in; }
div.WordSection1 { page: WordSection1; }</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--><div class="WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Marc,<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Thanks for taking the time to read and comment.<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">FWIW, I agree 9.1 should formally standardize on UTF-8 on other platforms as well. It just does not make sense to force UTF-8 only on Windows.<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">It needs a TIP though.<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">/Ashok<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal" style="margin-left:.5in"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Marc Culler &lt;cul...@gm...&gt; <br><b>Sent:</b> Wednesday, May 21, 2025 7:39 PM<br><b>To:</b> apn...@ya...<br><b>Cc:</b> Tcl Core List &lt;tcl...@li...&gt;<br><b>Subject:</b> Re: [TCLCORE] About TIP 716 vs 718 vs other options<o:p></o:p></span></p></div><p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p><div><div><p class="MsoNormal" style="margin-left:.5in">Based on Ashok's description of the situation, TIP #716 seems to me to be the best choice.&nbsp; Shipping two tclsh executables seems likely to cause confusion and problems, and I am also skeptical about "automatic" support.&nbsp; The first two options look to me like cans of worms to be opened in the future.<o:p></o:p></p></div><div><p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p></div><div><p class="MsoNormal" style="margin-left:.5in">Not that it matters for this discussion, but I am in favor of "forcing" utf-8 on our users.&nbsp; Standardization is very important, and it is too late for any other encoding to become the "standard".<o:p></o:p></p></div><div><p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p></div><div><p class="MsoNormal" style="margin-left:.5in">- Marc<o:p></o:p></p></div></div><p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p><div><div><p class="MsoNormal" style="margin-left:.5in">On Tue, May 20, 2025 at 9:41<span style="font-family:&quot;Arial&quot;,sans-serif"> </span>PM apnmbx-public--- via Tcl-Core &lt;<a href="mailto:tcl...@li...">tcl...@li...</a>&gt; wrote:<o:p></o:p></p></div><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"><div><div><div><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">All,<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">We need to make a collective decision regarding the default encoding issues on Windows. In the last meeting, I promised to write up a summary of the status and considerations for the problem being addressed by TIP's 716 and 718. Here it is along with the choices we have as to the next step.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Although specific to Windows, the basic issue does not need any deep understanding of Windows so I hope folks will read through and opine on the matter without copping out that they do not have Windows knowledge :-).<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">For those who have not read 716 or 718 (particularly the Background section in 716) or did not find the explanations clear, here is an example scenario the TIP's are independently addressing that I hope will shed some light.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">A client library, linked to a Tcl extension, shares data (config info, db content etc.) with another program such as a server. The encoding used for the data is that returned by the Windows GetACP() call which returns the user's code page. Note that neither the library nor the server side is aware of Tcl's existence in any form except for the library being linked to a Tcl extension loaded into tclsh.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">This scenario worked fine in Tcl 8.6 because both the client and server ends saw the same encoding value returned from GetACP().<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Tcl 9.0 broke the scenario because of the addition of the activeCodePage entry in the executable manifest for tclsh and wish. <b>The presence of this setting means all calls to GetACP() from that process, including those from the loaded client library, will return the code page utf-8 irrespective of the user setting.</b> The breakage may be because now the library encodes using utf-8 and the server component encodes using the user's real code page. Or it may be because the library cannot handle non-DBCS multibyte encodings. Does not really matter. The primary point is that the issue cannot be fixed by modifying or recompiling the extension.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Moving forward, we can deal with this problem in one of several ways:<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">(1) Remove the activeCodePage entry from the manifest with no other changes. This will revert 9.x behavior on Windows to that of 8.6 (and also consistent with non-Windows platforms since user settings are not ignored on them).<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">(2) Ignore the issue. In scenarios where this is an issue, users can continue to use 8.6 or build tclsh/wish themselves after removing the activeCodePage entry from the manifest.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">(3) TIP 716 as detailed below.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">(4) TIP 718 as detailed below.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Going with (1) means that 9.0.2 will not be compatible with 9.0.1 with respect to default encodings. For example, non-ASCII files written with writeFile in 9.0.1 will not be readable using readFile in 9.0.2 and vice versa. This will be a pain point users. Further, if as is possible, UTF-8 is made the default for Tcl on all platforms at some point in the future, there will be whole another round of incompatibility headaches. So as much as I wish the UTF-8 defaulting had never been made without discussion, 9.0 is already out there so I am not in favour of (1).<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Option (2) may be a candidate for discussion. We can just tell users in this situation to stick to 8.6 or build their own Tcl with the understanding that it will have similar compatibility issues with respect to "standard" Tcl as in (1) above. Not our problem so to speak. But also to be considered is the fact that the TPC/HammerDB is currently one of the most popular Tcl applications based on public download numbers. Similar situations may arise with other large applications. I am therefore skeptical that this is a viable solution.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Details behind options (3) and (4) are in the respective TIP's. Leaving aside the minor "reconcilable" differences with respect to implementation of [encoding user] or [exec -encoding], here is the crucial difference between the two.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">* TIP 716 removes the manifest entry from tclsh and wish. GetACP then returns the user's code page setting from the registry allowing the HammerDB-like applications to run. At the same time, in order to keep compatibility with 9.0.0/1, it hardcodes utf-8 as the default encoding returned by [encoding system] et al. In my opinion, if we indeed wanted to force utf-8 onto users, this is the way it should have been done in the first place.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">* TIP 718 deals with the problem by proposing to ship two Tcl shells, the current tclsh and a second one, tclshc, which has the manifest removed. The tclsh shell will behave identically to the one in 9.0.0/1. tclshc will behave substantially similar to TIP 716 and can be used by HammerDB etc. which do not work with tclsh.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">The primary hesitation I have with 718 is this dual shell approach and the potential for confusion. Somehow a user needs to know to use tclsh for all scripts. Except (for example) when accessing DB2. And X, Y and Z as the case may be. How are they to make that determination? Will a scripted application author now have to tell the user which Tcl shell to use? In fairness, most scripts will work with either. Still, this is a point for potential confusion. It is also the case that extension writers will now have test their extensions with both variations (not that they will!).<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">The advantage of tclsh (but not tclshc) in 718 over 716 (I am paraphrasing Jan here from the TIP 718 rationale, so he can clarify if I got it wrong) is that extensions that make use of the Windows ANSI API will automatically get UTF-8 support thereby supporting the entire Unicode range. With TIP 716 as well as 8.6, the ANSI API's will work only as long as the characters are supported by the user's code page.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">My counterpoint to this argument is that (a) extensions should not be using Windows ANSI API's in the first place, they should use the Unicode API's, (b) since 8.6 did not support this utf-8+ANSI API combination, such extensions are likely rare or old and in need of update anyways, and (c) the "automatic" support is likely overstated as not all API's and libraries support UTF-8 even when set as the code page (see TIP 716).<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">The question to answer is then whether this perceived benefit of 718 is overrides the downside to shipping two tclsh variations.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">A lesser issue is that 718 does not include the -encoding option to exec as I believe Jan thinks it should be a separate TIP. That is not difficult to resolve if we agree the option is needed.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">The decisions for the community to make now are which of the four options listed above make sense moving forward. Important that we make a decision for 9.0.2 as the longer we wait, the more 9.0.0/1 applications are going to be out there that might face this issue.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">/Ashok<o:p></o:p></p></div></div><p class="MsoNormal" style="margin-left:.5in">_______________________________________________<br>Tcl-Core mailing list<br><a href="mailto:Tcl...@li..." target="_blank">Tcl...@li...</a><br><a href="https://fedbdhd.r.tsp1-brevo.net/tr/cl/GDjC159F-zWfJNLNMCAqZ91dwRCeOIsWJSi_VS176PHSGOFK37nxkz_5OctJBFT5cjRe_INbl-chJmI3RTGuMm3jsKkuCxlGzJQ_Y5dmzBFqacZGnmcQtGMIJ41LmQ9Owu9jQ_-m_H0TW2rRq9pra2o4-U2zuoD7ah58ZK88lW6LvPFyGjn_khXGIp5xWkVq_UiSbwnAjfBmwIKCv5ZE1IrLDz8ldijcGeAIHsT5ruyqxcbp_IG95_Qa-yG-P_E-NIP_8zlskmtU54Xr4CjzubySWjWuWfyEYnR-1IVPBzRPUHuc31RyhiygEtwHrw" target="_blank">https://lists.sourceforge.net/lists/listinfo/tcl-core</a><o:p></o:p></p></div></blockquote></div></div><span>_______________________________________________</span><br><span>Tcl-Core mailing list</span><br><span>Tcl...@li...</span><br><span>https://lists.sourceforge.net/lists/listinfo/tcl-core</span><br></div></blockquote></body></html>

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[<apn...@ya...>]

Marc,

 

Thanks for taking the time to read and comment.

 

FWIW, I agree 9.1 should formally standardize on UTF-8 on other platforms as well. It just does not make sense to force UTF-8 only on Windows.

 

It needs a TIP though.

 

/Ashok

 

 

 

From: Marc Culler <cul...@gm...> 
Sent: Wednesday, May 21, 2025 7:39 PM
To: apn...@ya...
Cc: Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] About TIP 716 vs 718 vs other options

 

Based on Ashok's description of the situation, TIP #716 seems to me to be the best choice.  Shipping two tclsh executables seems likely to cause confusion and problems, and I am also skeptical about "automatic" support.  The first two options look to me like cans of worms to be opened in the future.

 

Not that it matters for this discussion, but I am in favor of "forcing" utf-8 on our users.  Standardization is very important, and it is too late for any other encoding to become the "standard".

 

- Marc

 

On Tue, May 20, 2025 at 9:41 PM apnmbx-public--- via Tcl-Core <tcl...@li... <mailto:tcl...@li...> > wrote:

All,

 

We need to make a collective decision regarding the default encoding issues on Windows. In the last meeting, I promised to write up a summary of the status and considerations for the problem being addressed by TIP's 716 and 718. Here it is along with the choices we have as to the next step.

 

Although specific to Windows, the basic issue does not need any deep understanding of Windows so I hope folks will read through and opine on the matter without copping out that they do not have Windows knowledge :-).

 

For those who have not read 716 or 718 (particularly the Background section in 716) or did not find the explanations clear, here is an example scenario the TIP's are independently addressing that I hope will shed some light.

 

A client library, linked to a Tcl extension, shares data (config info, db content etc.) with another program such as a server. The encoding used for the data is that returned by the Windows GetACP() call which returns the user's code page. Note that neither the library nor the server side is aware of Tcl's existence in any form except for the library being linked to a Tcl extension loaded into tclsh.

 

This scenario worked fine in Tcl 8.6 because both the client and server ends saw the same encoding value returned from GetACP().

 

Tcl 9.0 broke the scenario because of the addition of the activeCodePage entry in the executable manifest for tclsh and wish. The presence of this setting means all calls to GetACP() from that process, including those from the loaded client library, will return the code page utf-8 irrespective of the user setting. The breakage may be because now the library encodes using utf-8 and the server component encodes using the user's real code page. Or it may be because the library cannot handle non-DBCS multibyte encodings. Does not really matter. The primary point is that the issue cannot be fixed by modifying or recompiling the extension.

 

Moving forward, we can deal with this problem in one of several ways:

 

(1) Remove the activeCodePage entry from the manifest with no other changes. This will revert 9.x behavior on Windows to that of 8.6 (and also consistent with non-Windows platforms since user settings are not ignored on them).

 

(2) Ignore the issue. In scenarios where this is an issue, users can continue to use 8.6 or build tclsh/wish themselves after removing the activeCodePage entry from the manifest.

 

(3) TIP 716 as detailed below.

 

(4) TIP 718 as detailed below.

 

Going with (1) means that 9.0.2 will not be compatible with 9.0.1 with respect to default encodings. For example, non-ASCII files written with writeFile in 9.0.1 will not be readable using readFile in 9.0.2 and vice versa. This will be a pain point users. Further, if as is possible, UTF-8 is made the default for Tcl on all platforms at some point in the future, there will be whole another round of incompatibility headaches. So as much as I wish the UTF-8 defaulting had never been made without discussion, 9.0 is already out there so I am not in favour of (1).

 

Option (2) may be a candidate for discussion. We can just tell users in this situation to stick to 8.6 or build their own Tcl with the understanding that it will have similar compatibility issues with respect to "standard" Tcl as in (1) above. Not our problem so to speak. But also to be considered is the fact that the TPC/HammerDB is currently one of the most popular Tcl applications based on public download numbers. Similar situations may arise with other large applications. I am therefore skeptical that this is a viable solution.

 

Details behind options (3) and (4) are in the respective TIP's. Leaving aside the minor "reconcilable" differences with respect to implementation of [encoding user] or [exec -encoding], here is the crucial difference between the two.

 

* TIP 716 removes the manifest entry from tclsh and wish. GetACP then returns the user's code page setting from the registry allowing the HammerDB-like applications to run. At the same time, in order to keep compatibility with 9.0.0/1, it hardcodes utf-8 as the default encoding returned by [encoding system] et al. In my opinion, if we indeed wanted to force utf-8 onto users, this is the way it should have been done in the first place.

 

* TIP 718 deals with the problem by proposing to ship two Tcl shells, the current tclsh and a second one, tclshc, which has the manifest removed. The tclsh shell will behave identically to the one in 9.0.0/1. tclshc will behave substantially similar to TIP 716 and can be used by HammerDB etc. which do not work with tclsh.

 

The primary hesitation I have with 718 is this dual shell approach and the potential for confusion. Somehow a user needs to know to use tclsh for all scripts. Except (for example) when accessing DB2. And X, Y and Z as the case may be. How are they to make that determination? Will a scripted application author now have to tell the user which Tcl shell to use? In fairness, most scripts will work with either. Still, this is a point for potential confusion. It is also the case that extension writers will now have test their extensions with both variations (not that they will!).

 

The advantage of tclsh (but not tclshc) in 718 over 716 (I am paraphrasing Jan here from the TIP 718 rationale, so he can clarify if I got it wrong) is that extensions that make use of the Windows ANSI API will automatically get UTF-8 support thereby supporting the entire Unicode range. With TIP 716 as well as 8.6, the ANSI API's will work only as long as the characters are supported by the user's code page.

 

My counterpoint to this argument is that (a) extensions should not be using Windows ANSI API's in the first place, they should use the Unicode API's, (b) since 8.6 did not support this utf-8+ANSI API combination, such extensions are likely rare or old and in need of update anyways, and (c) the "automatic" support is likely overstated as not all API's and libraries support UTF-8 even when set as the code page (see TIP 716).

 

The question to answer is then whether this perceived benefit of 718 is overrides the downside to shipping two tclsh variations.

 

A lesser issue is that 718 does not include the -encoding option to exec as I believe Jan thinks it should be a separate TIP. That is not difficult to resolve if we agree the option is needed.

 

The decisions for the community to make now are which of the four options listed above make sense moving forward. Important that we make a decision for 9.0.2 as the longer we wait, the more 9.0.0/1 applications are going to be out there that might face this issue.

 

/Ashok

_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[Marc C. <cul...@gm...>]

Based on Ashok's description of the situation, TIP #716 seems to me to be
the best choice.  Shipping two tclsh executables seems likely to cause
confusion and problems, and I am also skeptical about "automatic" support.
The first two options look to me like cans of worms to be opened in the
future.

Not that it matters for this discussion, but I am in favor of "forcing"
utf-8 on our users.  Standardization is very important, and it is too late
for any other encoding to become the "standard".

- Marc

On Tue, May 20, 2025 at 9:41 PM apnmbx-public--- via Tcl-Core <
tcl...@li...> wrote:

> All,
>
>
>
> We need to make a collective decision regarding the default encoding
> issues on Windows. In the last meeting, I promised to write up a summary of
> the status and considerations for the problem being addressed by TIP's 716
> and 718. Here it is along with the choices we have as to the next step.
>
>
>
> Although specific to Windows, the basic issue does not need any deep
> understanding of Windows so I hope folks will read through and opine on the
> matter without copping out that they do not have Windows knowledge :-).
>
>
>
> For those who have not read 716 or 718 (particularly the Background
> section in 716) or did not find the explanations clear, here is an example
> scenario the TIP's are independently addressing that I hope will shed some
> light.
>
>
>
> A client library, linked to a Tcl extension, shares data (config info, db
> content etc.) with another program such as a server. The encoding used for
> the data is that returned by the Windows GetACP() call which returns the
> user's code page. Note that neither the library nor the server side is
> aware of Tcl's existence in any form except for the library being linked to
> a Tcl extension loaded into tclsh.
>
>
>
> This scenario worked fine in Tcl 8.6 because both the client and server
> ends saw the same encoding value returned from GetACP().
>
>
>
> Tcl 9.0 broke the scenario because of the addition of the activeCodePage
> entry in the executable manifest for tclsh and wish. *The presence of
> this setting means all calls to GetACP() from that process, including those
> from the loaded client library, will return the code page utf-8
> irrespective of the user setting.* The breakage may be because now the
> library encodes using utf-8 and the server component encodes using the
> user's real code page. Or it may be because the library cannot handle
> non-DBCS multibyte encodings. Does not really matter. The primary point is
> that the issue cannot be fixed by modifying or recompiling the extension.
>
>
>
> Moving forward, we can deal with this problem in one of several ways:
>
>
>
> (1) Remove the activeCodePage entry from the manifest with no other
> changes. This will revert 9.x behavior on Windows to that of 8.6 (and also
> consistent with non-Windows platforms since user settings are not ignored
> on them).
>
>
>
> (2) Ignore the issue. In scenarios where this is an issue, users can
> continue to use 8.6 or build tclsh/wish themselves after removing the
> activeCodePage entry from the manifest.
>
>
>
> (3) TIP 716 as detailed below.
>
>
>
> (4) TIP 718 as detailed below.
>
>
>
> Going with (1) means that 9.0.2 will not be compatible with 9.0.1 with
> respect to default encodings. For example, non-ASCII files written with
> writeFile in 9.0.1 will not be readable using readFile in 9.0.2 and vice
> versa. This will be a pain point users. Further, if as is possible, UTF-8
> is made the default for Tcl on all platforms at some point in the future,
> there will be whole another round of incompatibility headaches. So as much
> as I wish the UTF-8 defaulting had never been made without discussion, 9.0
> is already out there so I am not in favour of (1).
>
>
>
> Option (2) may be a candidate for discussion. We can just tell users in
> this situation to stick to 8.6 or build their own Tcl with the
> understanding that it will have similar compatibility issues with respect
> to "standard" Tcl as in (1) above. Not our problem so to speak. But also to
> be considered is the fact that the TPC/HammerDB is currently one of the
> most popular Tcl applications based on public download numbers. Similar
> situations may arise with other large applications. I am therefore
> skeptical that this is a viable solution.
>
>
>
> Details behind options (3) and (4) are in the respective TIP's. Leaving
> aside the minor "reconcilable" differences with respect to implementation
> of [encoding user] or [exec -encoding], here is the crucial difference
> between the two.
>
>
>
> * TIP 716 removes the manifest entry from tclsh and wish. GetACP then
> returns the user's code page setting from the registry allowing the
> HammerDB-like applications to run. At the same time, in order to keep
> compatibility with 9.0.0/1, it hardcodes utf-8 as the default encoding
> returned by [encoding system] et al. In my opinion, if we indeed wanted to
> force utf-8 onto users, this is the way it should have been done in the
> first place.
>
>
>
> * TIP 718 deals with the problem by proposing to ship two Tcl shells, the
> current tclsh and a second one, tclshc, which has the manifest removed. The
> tclsh shell will behave identically to the one in 9.0.0/1. tclshc will
> behave substantially similar to TIP 716 and can be used by HammerDB etc.
> which do not work with tclsh.
>
>
>
> The primary hesitation I have with 718 is this dual shell approach and the
> potential for confusion. Somehow a user needs to know to use tclsh for all
> scripts. Except (for example) when accessing DB2. And X, Y and Z as the
> case may be. How are they to make that determination? Will a scripted
> application author now have to tell the user which Tcl shell to use? In
> fairness, most scripts will work with either. Still, this is a point for
> potential confusion. It is also the case that extension writers will now
> have test their extensions with both variations (not that they will!).
>
>
>
> The advantage of tclsh (but not tclshc) in 718 over 716 (I am paraphrasing
> Jan here from the TIP 718 rationale, so he can clarify if I got it wrong)
> is that extensions that make use of the Windows ANSI API will automatically
> get UTF-8 support thereby supporting the entire Unicode range. With TIP 716
> as well as 8.6, the ANSI API's will work only as long as the characters are
> supported by the user's code page.
>
>
>
> My counterpoint to this argument is that (a) extensions should not be
> using Windows ANSI API's in the first place, they should use the Unicode
> API's, (b) since 8.6 did not support this utf-8+ANSI API combination, such
> extensions are likely rare or old and in need of update anyways, and (c)
> the "automatic" support is likely overstated as not all API's and libraries
> support UTF-8 even when set as the code page (see TIP 716).
>
>
>
> The question to answer is then whether this perceived benefit of 718 is
> overrides the downside to shipping two tclsh variations.
>
>
>
> A lesser issue is that 718 does not include the -encoding option to exec
> as I believe Jan thinks it should be a separate TIP. That is not difficult
> to resolve if we agree the option is needed.
>
>
>
> The decisions for the community to make now are which of the four options
> listed above make sense moving forward. Important that we make a decision
> for 9.0.2 as the longer we wait, the more 9.0.0/1 applications are going to
> be out there that might face this issue.
>
>
>
> /Ashok
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] New tip #722

[Donald P. <d.g...@co...>]

> If that makes "available" a misleading name, then it should change to something more accurate.

Perhaps [package names known] conveys better what the command does.

DGP

[TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Harald O. <har...@el...>]

Dear TCL team,

please allow me to propose a TIP to cure one of my biggest issues on TCL 
and MS-Windows: after does not fire if the wall clock changes.

The TIP text is here:
https://core.tcl-lang.org/tips/doc/trunk/tip/723.md

I am sorry to write it, as it is written (e.g. no respect on TIP233, 
Windows only). But those are the conclusions taken on recent discussions 
on the forelast biweekly telco and on the related bug tickets and on the 
core list.

I hope we will find a better solution, but this is my "minimum viable 
outcome" to this very urgent and annoying issue.

Thanks for all and take care,
Harald

[TCLCORE] OpenACS/TCL/Tk conference 2025 in Bologna/Italy/Europe on 10/11th of July 2025

[Harald O. <har...@el...>]

Dear TCL/Tk enthusiasts,
please allow me to invite you all to the
OpenACS/TCL/Tk conference 2025 in Bologna/Italy/Europe

https://openacs.km.at/evaluate/org/129998253/conferencenews/

it is a user meeting and we all show our own solutions.
As the minimum participants of 20 is reached, I can confirm that it will 
take place.

The so far presentations on my radar are:

- Tcl and Tk pannel
- Androwish Coffee machine
- Csabas new slider checkbox
- webcam barcode reading by me

So, there is still a lot of space for your contribution.
Any contribution is welcome, 5 minutes, no slides, is ok too.
We need you !
The interesting "abstract" entry field on the web site requires 200 
characters. I have filled it with "1234567890..." for my own contribution...

There is no online participation. It may happen, that the event is 
streamed. The chat will be monitored.

It would be great to see you in Bologna!
Harald
On behalf of the organisation committee

Re: [TCLCORE] New tip #722

[<apn...@ya...>]

Don,

Thanks for the exposition. You are right my suggestion would lead to a deep rabbit hole and should not be considered.

/Ashok

-----Original Message-----
From: Donald G Porter via Tcl-Core <tcl...@li...> 
Sent: Tuesday, May 20, 2025 5:39 PM
To: tcl...@li...
Subject: Re: [TCLCORE] New tip #722

On 5/19/25 22:34, apnmbx-public--- via Tcl-Core wrote:
> Since we are tinkering with [package names], I'd like to point out that the
> term "available" is a little misleading, even in the current manpage.
> Currently, [package names] only returns names *currently* known to Tcl, not
> those available on disk. One has to do the completely non-intuitive
>
> catch {package require nosuchpackage}; # Force a full search along path
> package names
>
> to retrieve the packages available to be loaded.
>
> So the question is,
>
> - will [package names available] continue the same behavior of not actually
> searching the package paths ?

Short answer:
[package names available] should only report names of packages with ifneeded scripts
already recorded in Tcl's memory.  This command should not prompt a search.

If that makes "available" a misleading name, then it should change to something more accurate.
> - if it does not do a full search, can we add a [package names refresh] that
> will replace the (ugly imo) catch {} ?
> - if it does do a full search, [package names] as currently implemented is
> no longer the union of [package names loaded] and [package names available]

These ideas are digging deeper into the [package unknown] problems, and that is a much
deeper rabbit hole than we want to explore in this TIP I think.

For those who have forgotten, or never learned, the [package] command is customizable.

The [package unknown] command permits an app or an installation to replace Tcl's
default package finder with another one better suited to any local method of package
storage.

The default package finder is [tclPkgUnkown].  It is implemented in a way that if
you ask it to go find package "foo", along the way it may discover and record the
ifneeded scripts of packages "bar" and "baz" as well.  ***This behavior is not in the
[package unknown] contract***  Custom [package unknown] replacements need not
and probably will not have that "discover all the packages" functionality.

When you invoke the incantation [package require nosuch], you are not using any
[package] command feature.  You are using a side effect of [tclPkgUnknown].

If we need a "report all the findable packages" functionality to exploit, step one will
have to be to extend or replace the [package unknown] mechanism. That's a bigger
project that will have slower progress and better not joined to this fairly simple proposal.

If your curiosity is sparked by any of this, you might enjoy an ancient presentation
on the subject.

https://math.nist.gov/~DPorter/tcltk/oscon/

-- 
| Don Porter            Applied and Computational Mathematics Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|



_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core