tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[Jan N. <jan...@gm...>]

Op vr 23 mei 2025 om 05:04 schreef apnmbx-public:

> TIP 716 is in its final proposed form. Will plan a CFV in a few days once
> people have chewed on the 716 vs 718 discussion.
>
>
>
> Changes in the TIP and implementation:
>
>
>
> - the code page is obtained from the registry and not GetACP. See Jan's
> ticket at https://core.tcl-lang.org/tcl/info/c776eb586d08c2f7
>
>
>
> - the TCL_EXEC_ENCODING environment has been removed as no one has
> commented on it. Note the -encoding option to exec remains.
>

With those 2 changes (thanks!) my main issues with TIP #716 disappeared.
I can go along with it, so we don't need to waste too much time discussing
the
difference between the two TIP's. I won't start a vote for #718.

Hope this helps,
      Jan Nijtmans

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[Harald O. <har...@el...>]

Dear Ashok, dear Jan,
thanks for all the action, great.

Here are my two pence:

- on exec codepage with environment variable: if I remember right, I had 
commented this. IMHO, environment variables are difficult on Windows and 
they open another door of external influence which may break program 
stability. I thing, the script-level only is ok.

- on TIP 716/718: I think, a mouse is getting an elephant. I prefer 716. 
I don't like the hack-style of the manifest which half-fixes broken 
design and brings legacy dlls (which is the target) in danger for no 
gain. Thus: 716++. I like, that you two work together. The best idea 
should win, independent on the author.

- About the list stuff: great stuff, ship it!

THanks for all,
Harald


Am 23.05.2025 um 05:03 schrieb apnmbx-public--- via Tcl-Core:
> TIP 716 is in its final proposed form. Will plan a CFV in a few days 
> once people have chewed on the 716 vs 718 discussion.
> 
> Changes in the TIP and implementation:
> 
> - the code page is obtained from the registry and not GetACP. See Jan's 
> ticket at https://core.tcl-lang.org/tcl/info/c776eb586d08c2f7 <https:// 
> core.tcl-lang.org/tcl/info/c776eb586d08c2f7>
> 
> - the TCL_EXEC_ENCODING environment has been removed as no one has 
> commented on it. Note the -encoding option to exec remains.
> 
> - a section comparing TIP 718 has been added to the TIP. In particular, 
> in addition to what's been already said in this thread, it responds to 
> the section in 718 about the convenience of its TclWinGetUserEncoding 
> function. TL;DR the equivalent function is also available in the 716 
> implementation but #if 0'ed because I am not sure it is the right API. 
> See https://core.tcl-lang.org/tips/doc/trunk/tip/716.md#RelationtoTIP718 
> <https://core.tcl-lang.org/tips/doc/trunk/tip/716.md#RelationtoTIP718> 
> for details.
> 
> /Ashok
> 
> *From:*Kevin Walzer <kw...@54...>
> *Sent:* Thursday, May 22, 2025 1:33 AM
> *To:* apn...@ya...
> *Cc:* Marc Culler <cul...@gm...>; Tcl Core List <tcl- 
> co...@li...>
> *Subject:* Re: [TCLCORE] About TIP 716 vs 718 vs other options
> 
> Image removed by sender.
> 
> Agree that TIP 716 makes the most sense.
> 
> 
> 
>     On May 21, 2025, at 11:59 AM, apnmbx-public--- via Tcl-Core <tcl-
>     co...@li...> wrote:
> 
>     
> 
>     Marc,
> 
>     Thanks for taking the time to read and comment.
> 
>     FWIW, I agree 9.1 should formally standardize on UTF-8 on other
>     platforms as well. It just does not make sense to force UTF-8 only
>     on Windows.
> 
>     It needs a TIP though.
> 
>     /Ashok
> 
>     *From:*Marc Culler <cul...@gm...>
>     *Sent:* Wednesday, May 21, 2025 7:39 PM
>     *To:* apn...@ya...
>     *Cc:* Tcl Core List <tcl...@li...>
>     *Subject:* Re: [TCLCORE] About TIP 716 vs 718 vs other options
> 
>     Based on Ashok's description of the situation, TIP #716 seems to me
>     to be the best choice.  Shipping two tclsh executables seems likely
>     to cause confusion and problems, and I am also skeptical about
>     "automatic" support.  The first two options look to me like cans of
>     worms to be opened in the future.
> 
>     Not that it matters for this discussion, but I am in favor of
>     "forcing" utf-8 on our users.  Standardization is very important,
>     and it is too late for any other encoding to become the "standard".
> 
>     - Marc
> 
>     On Tue, May 20, 2025 at 9:41 PM apnmbx-public--- via Tcl-Core <tcl-
>     co...@li... <mailto:tcl...@li...>>
>     wrote:
> 
>         All,
> 
>         We need to make a collective decision regarding the default
>         encoding issues on Windows. In the last meeting, I promised to
>         write up a summary of the status and considerations for the
>         problem being addressed by TIP's 716 and 718. Here it is along
>         with the choices we have as to the next step.
> 
>         Although specific to Windows, the basic issue does not need any
>         deep understanding of Windows so I hope folks will read through
>         and opine on the matter without copping out that they do not
>         have Windows knowledge :-).
> 
>         For those who have not read 716 or 718 (particularly the
>         Background section in 716) or did not find the explanations
>         clear, here is an example scenario the TIP's are independently
>         addressing that I hope will shed some light.
> 
>         A client library, linked to a Tcl extension, shares data (config
>         info, db content etc.) with another program such as a server.
>         The encoding used for the data is that returned by the Windows
>         GetACP() call which returns the user's code page. Note that
>         neither the library nor the server side is aware of Tcl's
>         existence in any form except for the library being linked to a
>         Tcl extension loaded into tclsh.
> 
>         This scenario worked fine in Tcl 8.6 because both the client and
>         server ends saw the same encoding value returned from GetACP().
> 
>         Tcl 9.0 broke the scenario because of the addition of the
>         activeCodePage entry in the executable manifest for tclsh and
>         wish. *The presence of this setting means all calls to GetACP()
>         from that process, including those from the loaded client
>         library, will return the code page utf-8 irrespective of the
>         user setting.* The breakage may be because now the library
>         encodes using utf-8 and the server component encodes using the
>         user's real code page. Or it may be because the library cannot
>         handle non-DBCS multibyte encodings. Does not really matter. The
>         primary point is that the issue cannot be fixed by modifying or
>         recompiling the extension.
> 
>         Moving forward, we can deal with this problem in one of several
>         ways:
> 
>         (1) Remove the activeCodePage entry from the manifest with no
>         other changes. This will revert 9.x behavior on Windows to that
>         of 8.6 (and also consistent with non-Windows platforms since
>         user settings are not ignored on them).
> 
>         (2) Ignore the issue. In scenarios where this is an issue, users
>         can continue to use 8.6 or build tclsh/wish themselves after
>         removing the activeCodePage entry from the manifest.
> 
>         (3) TIP 716 as detailed below.
> 
>         (4) TIP 718 as detailed below.
> 
>         Going with (1) means that 9.0.2 will not be compatible with
>         9.0.1 with respect to default encodings. For example, non-ASCII
>         files written with writeFile in 9.0.1 will not be readable using
>         readFile in 9.0.2 and vice versa. This will be a pain point
>         users. Further, if as is possible, UTF-8 is made the default for
>         Tcl on all platforms at some point in the future, there will be
>         whole another round of incompatibility headaches. So as much as
>         I wish the UTF-8 defaulting had never been made without
>         discussion, 9.0 is already out there so I am not in favour of (1).
> 
>         Option (2) may be a candidate for discussion. We can just tell
>         users in this situation to stick to 8.6 or build their own Tcl
>         with the understanding that it will have similar compatibility
>         issues with respect to "standard" Tcl as in (1) above. Not our
>         problem so to speak. But also to be considered is the fact that
>         the TPC/HammerDB is currently one of the most popular Tcl
>         applications based on public download numbers. Similar
>         situations may arise with other large applications. I am
>         therefore skeptical that this is a viable solution.
> 
>         Details behind options (3) and (4) are in the respective TIP's.
>         Leaving aside the minor "reconcilable" differences with respect
>         to implementation of [encoding user] or [exec -encoding], here
>         is the crucial difference between the two.
> 
>         * TIP 716 removes the manifest entry from tclsh and wish. GetACP
>         then returns the user's code page setting from the registry
>         allowing the HammerDB-like applications to run. At the same
>         time, in order to keep compatibility with 9.0.0/1, it hardcodes
>         utf-8 as the default encoding returned by [encoding system] et
>         al. In my opinion, if we indeed wanted to force utf-8 onto
>         users, this is the way it should have been done in the first place.
> 
>         * TIP 718 deals with the problem by proposing to ship two Tcl
>         shells, the current tclsh and a second one, tclshc, which has
>         the manifest removed. The tclsh shell will behave identically to
>         the one in 9.0.0/1. tclshc will behave substantially similar to
>         TIP 716 and can be used by HammerDB etc. which do not work with
>         tclsh.
> 
>         The primary hesitation I have with 718 is this dual shell
>         approach and the potential for confusion. Somehow a user needs
>         to know to use tclsh for all scripts. Except (for example) when
>         accessing DB2. And X, Y and Z as the case may be. How are they
>         to make that determination? Will a scripted application author
>         now have to tell the user which Tcl shell to use? In fairness,
>         most scripts will work with either. Still, this is a point for
>         potential confusion. It is also the case that extension writers
>         will now have test their extensions with both variations (not
>         that they will!).
> 
>         The advantage of tclsh (but not tclshc) in 718 over 716 (I am
>         paraphrasing Jan here from the TIP 718 rationale, so he can
>         clarify if I got it wrong) is that extensions that make use of
>         the Windows ANSI API will automatically get UTF-8 support
>         thereby supporting the entire Unicode range. With TIP 716 as
>         well as 8.6, the ANSI API's will work only as long as the
>         characters are supported by the user's code page.
> 
>         My counterpoint to this argument is that (a) extensions should
>         not be using Windows ANSI API's in the first place, they should
>         use the Unicode API's, (b) since 8.6 did not support this
>         utf-8+ANSI API combination, such extensions are likely rare or
>         old and in need of update anyways, and (c) the "automatic"
>         support is likely overstated as not all API's and libraries
>         support UTF-8 even when set as the code page (see TIP 716).
> 
>         The question to answer is then whether this perceived benefit of
>         718 is overrides the downside to shipping two tclsh variations.
> 
>         A lesser issue is that 718 does not include the -encoding option
>         to exec as I believe Jan thinks it should be a separate TIP.
>         That is not difficult to resolve if we agree the option is needed.
> 
>         The decisions for the community to make now are which of the
>         four options listed above make sense moving forward. Important
>         that we make a decision for 9.0.2 as the longer we wait, the
>         more 9.0.0/1 applications are going to be out there that might
>         face this issue.
> 
>         /Ashok
> 

Re: [TCLCORE] OpenACS/TCL/Tk conference 2025 in Bologna/Italy/Europe on 10/11th of July 2025

[Harald O. <har...@el...>]

Dear OpenACS/TCL/Tk enthusiasts,

the hotel recommendation to "Royel Hotel Carlton" will end 1st of June.

https://openacs.km.at/evaluate/org/129998253/conferencenews/

The hotel should be called "Tickle Basecamp", but this will change ;-).

The 23th of June 17:00 will be the program telco. Expect a program the 
24th of June.

See you all there !

Take care,
Harald

Am 21.05.2025 um 10:51 schrieb Harald Oehlmann:
> Dear TCL/Tk enthusiasts,
> please allow me to invite you all to the
> OpenACS/TCL/Tk conference 2025 in Bologna/Italy/Europe
> 
> https://openacs.km.at/evaluate/org/129998253/conferencenews/
> 
> it is a user meeting and we all show our own solutions.
> As the minimum participants of 20 is reached, I can confirm that it will 
> take place.
> 
> The so far presentations on my radar are:
> 
> - Tcl and Tk pannel
> - Androwish Coffee machine
> - Csabas new slider checkbox
> - webcam barcode reading by me
> 
> So, there is still a lot of space for your contribution.
> Any contribution is welcome, 5 minutes, no slides, is ok too.
> We need you !
> The interesting "abstract" entry field on the web site requires 200 
> characters. I have filled it with "1234567890..." for my own 
> contribution...
> 
> There is no online participation. It may happen, that the event is 
> streamed. The chat will be monitored.
> 
> It would be great to see you in Bologna!
> Harald
> On behalf of the organisation committee
> 

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Harald O. <har...@el...>]

Dear Sergey,

I appreciate your review and valuable opinion.
Why Windows only is affected, I don't know. I can not help here. But it 
is a fact, that there is opposition to see this as a bugfix or no 
interest or both, as the patch is not reviewed or commented. The patch 
is very complicated compared to the 3 lines on Windows, true...
For me, Windows only is the only way to bring this forward.

About the "summer time change". The concerned systems run without NTP 
and the change is done manually. So, the clock is manually turned back 
by one hour once per year.
This may be detailed in the tip.

Thanks for all,
Harald


Am 22.05.2025 um 17:38 schrieb Dipl. Ing. Sergey G. Brester:
> I don't know why it shall be limited to Windows only, because all 3 
> points are affecting *nix in the same way:
> 
>    * manual time adjustment (is possible with `date -s`)
>    * suspension of the system (suspend of VM, hibernation, etc)
>    * automatic time adjustment (via ntpd and co)
> 
> Also there is a mistake in the TIP - "specially on day time switching 
> +/- 1 hour". Because DST-switches normally doesn't adjust the clock in 
> UTC (GMT), which is running continuously. Everything what asking [clock 
> seconds], [clock milliseconds], [clock microseconds] (or related C-API 
> functions) etc is not affected by DST-switches at all.
> The DST-switch happens only in related timezone (table of offsets stored 
> in tzdata) and doesn't exist without TZ context (pure UTC time).
> 
> Just for illustration:
> 
> # forward DST jump in CET-TZ:
> % clock format [expr {1743296399 + 0}] -timezone :Europe/Berlin
> Sun Mar 30 01:59:59 CET 2025
> % clock format [expr {1743296399 + 1}] -timezone :Europe/Berlin
> Sun Mar 30 03:00:00 CEST 2025
> 
> # backward DST jump in CET-TZ:
> % clock format [expr {1761440399 + 0}] -timezone :Europe/Berlin
> Sun Oct 26 02:59:59 CEST 2025
> % clock format [expr {1761440399 + 1}] -timezone :Europe/Berlin
> Sun Oct 26 02:00:00 CET 2025
> 
> The issues are only manual or automatic time adjustments affecting UTC- 
> time, and suspension/hibernation.
> However as already said it definitely affecting all systems that using 
> wall clock (and need a switch to monotonic time).
> 
> Regards,
> Sergey.
> 
> 21.05.2025 12:35, Harald Oehlmann wrote:
> 
>> Dear TCL team,
>>
>> please allow me to propose a TIP to cure one of my biggest issues on TCL and MS-Windows: after does not fire if the wall clock changes.
>>
>> The TIP text is here:
>> https://core.tcl-lang.org/tips/doc/trunk/tip/723.md <https://core.tcl-lang.org/tips/doc/trunk/tip/723.md>
>>
>> I am sorry to write it, as it is written (e.g. no respect on TIP233, Windows only). But those are the conclusions taken on recent discussions on the forelast biweekly telco and on the related bug tickets and on the core list.
>>
>> I hope we will find a better solution, but this is my "minimum viable outcome" to this very urgent and annoying issue.
>>
>> Thanks for all and take care,
>> Harald

Re: [TCLCORE] New internal list representations

[<apn...@ya...>]

Last call on the internal list rep changes below. Brian's looked it over. If
any one else concerns please speak up now. Else will merge into trunk for
9.1a0 in the next couple of days.

 

/Ashok

 

From: apnadkarni--- via Tcl-Core <tcl...@li...> 
Sent: Thursday, May 15, 2025 9:53 PM
To: 'Tcl Core' <tcl...@li...>
Subject: [TCLCORE] New internal list representations

 

https://core.tcl-lang.org/tcl/wiki?name=New+abstract+list+representations
<https://core.tcl-lang.org/tcl/wiki?name=New+abstract+list+representations&p
> &p is a proposal to implement some list operations using new internal list
representations enabled by Brian's TIP 636. The wiki post has the full
details but TL;DR memory efficiency at a small (probably fixable) cost in
iteration speed for large lists. The post has the numbers.

 

The apn-tip636-appl branch contains the implementation of the proposal. The
listTypes.test file contains the test cases to exercise the three new
internal list types as well as the existing ones. Passes -enable-symbols=mem
and valgrind OMM. gcov shows 91% code and 100% function coverage of the new
implementation. Missing coverage is for "panic blocks" and validation which
is not exercised as it is already done by upper callers.

 

Not planning a TIP as there are no changes at the script or C API level.

 

Please review (both proposal and code) and raise any objections you may
have. (It's not that much code to review and pretty isolated thanks to 636).

 

/Ashok

 

 

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[<apn...@ya...>]

TIP 716 is in its final proposed form. Will plan a CFV in a few days once people have chewed on the 716 vs 718 discussion.

 

Changes in the TIP and implementation:

 

- the code page is obtained from the registry and not GetACP. See Jan's ticket at https://core.tcl-lang.org/tcl/info/c776eb586d08c2f7

 

- the TCL_EXEC_ENCODING environment has been removed as no one has commented on it. Note the -encoding option to exec remains.

 

- a section comparing TIP 718 has been added to the TIP. In particular, in addition to what's been already said in this thread, it responds to the section in 718 about the convenience of its TclWinGetUserEncoding function. TL;DR the equivalent function is also available in the 716 implementation but #if 0'ed because I am not sure it is the right API. See https://core.tcl-lang.org/tips/doc/trunk/tip/716.md#RelationtoTIP718 for details.

 

/Ashok

 

From: Kevin Walzer <kw...@54...> 
Sent: Thursday, May 22, 2025 1:33 AM
To: apn...@ya...
Cc: Marc Culler <cul...@gm...>; Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] About TIP 716 vs 718 vs other options

 



Agree that TIP 716 makes the most sense. 





On May 21, 2025, at 11:59 AM, apnmbx-public--- via Tcl-Core <tcl...@li...> wrote:



Marc,

 

Thanks for taking the time to read and comment.

 

FWIW, I agree 9.1 should formally standardize on UTF-8 on other platforms as well. It just does not make sense to force UTF-8 only on Windows.

 

It needs a TIP though.

 

/Ashok

 

 

 

From: Marc Culler <cul...@gm...> 
Sent: Wednesday, May 21, 2025 7:39 PM
To: apn...@ya...
Cc: Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] About TIP 716 vs 718 vs other options

 

Based on Ashok's description of the situation, TIP #716 seems to me to be the best choice.  Shipping two tclsh executables seems likely to cause confusion and problems, and I am also skeptical about "automatic" support.  The first two options look to me like cans of worms to be opened in the future.

 

Not that it matters for this discussion, but I am in favor of "forcing" utf-8 on our users.  Standardization is very important, and it is too late for any other encoding to become the "standard".

 

- Marc

 

On Tue, May 20, 2025 at 9:41 PM apnmbx-public--- via Tcl-Core <tcl...@li... <mailto:tcl...@li...> > wrote:

All,

 

We need to make a collective decision regarding the default encoding issues on Windows. In the last meeting, I promised to write up a summary of the status and considerations for the problem being addressed by TIP's 716 and 718. Here it is along with the choices we have as to the next step.

 

Although specific to Windows, the basic issue does not need any deep understanding of Windows so I hope folks will read through and opine on the matter without copping out that they do not have Windows knowledge :-).

 

For those who have not read 716 or 718 (particularly the Background section in 716) or did not find the explanations clear, here is an example scenario the TIP's are independently addressing that I hope will shed some light.

 

A client library, linked to a Tcl extension, shares data (config info, db content etc.) with another program such as a server. The encoding used for the data is that returned by the Windows GetACP() call which returns the user's code page. Note that neither the library nor the server side is aware of Tcl's existence in any form except for the library being linked to a Tcl extension loaded into tclsh.

 

This scenario worked fine in Tcl 8.6 because both the client and server ends saw the same encoding value returned from GetACP().

 

Tcl 9.0 broke the scenario because of the addition of the activeCodePage entry in the executable manifest for tclsh and wish. The presence of this setting means all calls to GetACP() from that process, including those from the loaded client library, will return the code page utf-8 irrespective of the user setting. The breakage may be because now the library encodes using utf-8 and the server component encodes using the user's real code page. Or it may be because the library cannot handle non-DBCS multibyte encodings. Does not really matter. The primary point is that the issue cannot be fixed by modifying or recompiling the extension.

 

Moving forward, we can deal with this problem in one of several ways:

 

(1) Remove the activeCodePage entry from the manifest with no other changes. This will revert 9.x behavior on Windows to that of 8.6 (and also consistent with non-Windows platforms since user settings are not ignored on them).

 

(2) Ignore the issue. In scenarios where this is an issue, users can continue to use 8.6 or build tclsh/wish themselves after removing the activeCodePage entry from the manifest.

 

(3) TIP 716 as detailed below.

 

(4) TIP 718 as detailed below.

 

Going with (1) means that 9.0.2 will not be compatible with 9.0.1 with respect to default encodings. For example, non-ASCII files written with writeFile in 9.0.1 will not be readable using readFile in 9.0.2 and vice versa. This will be a pain point users. Further, if as is possible, UTF-8 is made the default for Tcl on all platforms at some point in the future, there will be whole another round of incompatibility headaches. So as much as I wish the UTF-8 defaulting had never been made without discussion, 9.0 is already out there so I am not in favour of (1).

 

Option (2) may be a candidate for discussion. We can just tell users in this situation to stick to 8.6 or build their own Tcl with the understanding that it will have similar compatibility issues with respect to "standard" Tcl as in (1) above. Not our problem so to speak. But also to be considered is the fact that the TPC/HammerDB is currently one of the most popular Tcl applications based on public download numbers. Similar situations may arise with other large applications. I am therefore skeptical that this is a viable solution.

 

Details behind options (3) and (4) are in the respective TIP's. Leaving aside the minor "reconcilable" differences with respect to implementation of [encoding user] or [exec -encoding], here is the crucial difference between the two.

 

* TIP 716 removes the manifest entry from tclsh and wish. GetACP then returns the user's code page setting from the registry allowing the HammerDB-like applications to run. At the same time, in order to keep compatibility with 9.0.0/1, it hardcodes utf-8 as the default encoding returned by [encoding system] et al. In my opinion, if we indeed wanted to force utf-8 onto users, this is the way it should have been done in the first place.

 

* TIP 718 deals with the problem by proposing to ship two Tcl shells, the current tclsh and a second one, tclshc, which has the manifest removed. The tclsh shell will behave identically to the one in 9.0.0/1. tclshc will behave substantially similar to TIP 716 and can be used by HammerDB etc. which do not work with tclsh.

 

The primary hesitation I have with 718 is this dual shell approach and the potential for confusion. Somehow a user needs to know to use tclsh for all scripts. Except (for example) when accessing DB2. And X, Y and Z as the case may be. How are they to make that determination? Will a scripted application author now have to tell the user which Tcl shell to use? In fairness, most scripts will work with either. Still, this is a point for potential confusion. It is also the case that extension writers will now have test their extensions with both variations (not that they will!).

 

The advantage of tclsh (but not tclshc) in 718 over 716 (I am paraphrasing Jan here from the TIP 718 rationale, so he can clarify if I got it wrong) is that extensions that make use of the Windows ANSI API will automatically get UTF-8 support thereby supporting the entire Unicode range. With TIP 716 as well as 8.6, the ANSI API's will work only as long as the characters are supported by the user's code page.

 

My counterpoint to this argument is that (a) extensions should not be using Windows ANSI API's in the first place, they should use the Unicode API's, (b) since 8.6 did not support this utf-8+ANSI API combination, such extensions are likely rare or old and in need of update anyways, and (c) the "automatic" support is likely overstated as not all API's and libraries support UTF-8 even when set as the code page (see TIP 716).

 

The question to answer is then whether this perceived benefit of 718 is overrides the downside to shipping two tclsh variations.

 

A lesser issue is that 718 does not include the -encoding option to exec as I believe Jan thinks it should be a separate TIP. That is not difficult to resolve if we agree the option is needed.

 

The decisions for the community to make now are which of the four options listed above make sense moving forward. Important that we make a decision for 9.0.2 as the longer we wait, the more 9.0.0/1 applications are going to be out there that might face this issue.

 

/Ashok

_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core <https://fedbdhd.r.bh.d.sendibt3.com/tr/cl/dN96ZIXkuEXleedxhrmxUa9SsHYQ5pGEMHlQ6MmZobGmdF1QwvSu6ONBozUHXxcnGGtRlmGXHg46XdECmJr00-SATPFcfGskPV62GDTb6NUEJtyWrdBS7lftPd6KBaZiXEceJP6GaUWNgBcOMQeWzzBb9LEa4pZkTyyWuw2qD4URCkRwbecry05LcrKdNwZMsZeVi7EVgymYhhizEIVwttoruoq_NR0yzrTnrRL7NZw3BdhptnAeP7pCHHb2l1RoaCAc-meYgBmAc3Gw3f9PRgFQ_UHdgyBAvfvIzxfuRTDEYsV-X8sRRSRK> 

_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Dipl. I. S. G. B. <se...@us...>]

 

I don't know why it shall be limited to Windows only, because all 3
points are affecting *nix in the same way: 

 * manual time adjustment (is possible with `date -s`)
 * suspension of the system (suspend of VM, hibernation, etc)
 * automatic time adjustment (via ntpd and co) 

Also there is a mistake in the TIP - "specially on day time switching
+/- 1 hour". Because DST-switches normally doesn't adjust the clock in
UTC (GMT), which is running continuously. Everything what asking [clock
seconds], [clock milliseconds], [clock microseconds] (or related C-API
functions) etc is not affected by DST-switches at all.
The DST-switch happens only in related timezone (table of offsets stored
in tzdata) and doesn't exist without TZ context (pure UTC time).

Just for illustration: 

# forward DST jump in CET-TZ:
% clock format [expr {1743296399 + 0}] -timezone :Europe/Berlin
Sun Mar 30 01:59:59 CET 2025
% clock format [expr {1743296399 + 1}] -timezone :Europe/Berlin
Sun Mar 30 03:00:00 CEST 2025

# backward DST jump in CET-TZ:
% clock format [expr {1761440399 + 0}] -timezone :Europe/Berlin
Sun Oct 26 02:59:59 CEST 2025
% clock format [expr {1761440399 + 1}] -timezone :Europe/Berlin
Sun Oct 26 02:00:00 CET 2025 

The issues are only manual or automatic time adjustments affecting
UTC-time, and suspension/hibernation.
However as already said it definitely affecting all systems that using
wall clock (and need a switch to monotonic time).

Regards,
Sergey. 

21.05.2025 12:35, Harald Oehlmann wrote: 

> Dear TCL team,
> 
> please allow me to propose a TIP to cure one of my biggest issues on TCL and MS-Windows: after does not fire if the wall clock changes.
> 
> The TIP text is here:
> https://core.tcl-lang.org/tips/doc/trunk/tip/723.md [1]
> 
> I am sorry to write it, as it is written (e.g. no respect on TIP233, Windows only). But those are the conclusions taken on recent discussions on the forelast biweekly telco and on the related bug tickets and on the core list.
> 
> I hope we will find a better solution, but this is my "minimum viable outcome" to this very urgent and annoying issue.
> 
> Thanks for all and take care,
> Harald
> 
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core [2]
 

Links:
------
[1] https://core.tcl-lang.org/tips/doc/trunk/tip/723.md
[2] https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[Kevin W. <kw...@co...>]

<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><img width="1" height="1" src="https://fedbdhd.r.tsp1-brevo.net/tr/op/HiJ4uqGoaWJaQppIUaYaT1sIcrvSIyZwyuMilMtUiR0aKFHmblboWPgIHuTNDuGh4KbnD-1QGiOukkOKVNEd6Oyl84_2qhwAI-iHVPL4PhCuEdmQ9RMG-SOjCF8wS2ZrOiowaRLafy_q9aYa0ktqibrZjg2FmeYSLL1-83MXzjlC7M0KQ5Vgj_kVQFkmPtP3vBB4tJC69j5qLCVJvnfCo5FoXKpU" style="mso-hide:all"/><div dir="ltr"></div><div dir="ltr">Agree that TIP 716 makes the most sense.&nbsp;</div><div dir="ltr"><br><blockquote type="cite">On May 21, 2025, at 11:59 AM, apnmbx-public--- via Tcl-Core &lt;tcl...@li...&gt; wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta name="Generator" content="Microsoft Word 15 (filtered medium)"><style>@font-face { font-family: "Cambria Math"; }
@font-face { font-family: Calibri; }
@font-face { font-family: Aptos; }
p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in; font-size: 12pt; font-family: Aptos, sans-serif; }
a:link, span.MsoHyperlink { color: blue; text-decoration: underline; }
span.EmailStyle18 { font-family: Aptos, sans-serif; color: windowtext; }
.MsoChpDefault { font-size: 11pt; }
@page WordSection1 { size: 8.5in 11in; margin: 1in; }
div.WordSection1 { page: WordSection1; }</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--><div class="WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Marc,<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Thanks for taking the time to read and comment.<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">FWIW, I agree 9.1 should formally standardize on UTF-8 on other platforms as well. It just does not make sense to force UTF-8 only on Windows.<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">It needs a TIP though.<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">/Ashok<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p>&nbsp;</o:p></span></p><div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal" style="margin-left:.5in"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Marc Culler &lt;cul...@gm...&gt; <br><b>Sent:</b> Wednesday, May 21, 2025 7:39 PM<br><b>To:</b> apn...@ya...<br><b>Cc:</b> Tcl Core List &lt;tcl...@li...&gt;<br><b>Subject:</b> Re: [TCLCORE] About TIP 716 vs 718 vs other options<o:p></o:p></span></p></div><p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p><div><div><p class="MsoNormal" style="margin-left:.5in">Based on Ashok's description of the situation, TIP #716 seems to me to be the best choice.&nbsp; Shipping two tclsh executables seems likely to cause confusion and problems, and I am also skeptical about "automatic" support.&nbsp; The first two options look to me like cans of worms to be opened in the future.<o:p></o:p></p></div><div><p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p></div><div><p class="MsoNormal" style="margin-left:.5in">Not that it matters for this discussion, but I am in favor of "forcing" utf-8 on our users.&nbsp; Standardization is very important, and it is too late for any other encoding to become the "standard".<o:p></o:p></p></div><div><p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p></div><div><p class="MsoNormal" style="margin-left:.5in">- Marc<o:p></o:p></p></div></div><p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p><div><div><p class="MsoNormal" style="margin-left:.5in">On Tue, May 20, 2025 at 9:41<span style="font-family:&quot;Arial&quot;,sans-serif"> </span>PM apnmbx-public--- via Tcl-Core &lt;<a href="mailto:tcl...@li...">tcl...@li...</a>&gt; wrote:<o:p></o:p></p></div><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"><div><div><div><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">All,<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">We need to make a collective decision regarding the default encoding issues on Windows. In the last meeting, I promised to write up a summary of the status and considerations for the problem being addressed by TIP's 716 and 718. Here it is along with the choices we have as to the next step.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Although specific to Windows, the basic issue does not need any deep understanding of Windows so I hope folks will read through and opine on the matter without copping out that they do not have Windows knowledge :-).<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">For those who have not read 716 or 718 (particularly the Background section in 716) or did not find the explanations clear, here is an example scenario the TIP's are independently addressing that I hope will shed some light.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">A client library, linked to a Tcl extension, shares data (config info, db content etc.) with another program such as a server. The encoding used for the data is that returned by the Windows GetACP() call which returns the user's code page. Note that neither the library nor the server side is aware of Tcl's existence in any form except for the library being linked to a Tcl extension loaded into tclsh.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">This scenario worked fine in Tcl 8.6 because both the client and server ends saw the same encoding value returned from GetACP().<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Tcl 9.0 broke the scenario because of the addition of the activeCodePage entry in the executable manifest for tclsh and wish. <b>The presence of this setting means all calls to GetACP() from that process, including those from the loaded client library, will return the code page utf-8 irrespective of the user setting.</b> The breakage may be because now the library encodes using utf-8 and the server component encodes using the user's real code page. Or it may be because the library cannot handle non-DBCS multibyte encodings. Does not really matter. The primary point is that the issue cannot be fixed by modifying or recompiling the extension.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Moving forward, we can deal with this problem in one of several ways:<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">(1) Remove the activeCodePage entry from the manifest with no other changes. This will revert 9.x behavior on Windows to that of 8.6 (and also consistent with non-Windows platforms since user settings are not ignored on them).<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">(2) Ignore the issue. In scenarios where this is an issue, users can continue to use 8.6 or build tclsh/wish themselves after removing the activeCodePage entry from the manifest.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">(3) TIP 716 as detailed below.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">(4) TIP 718 as detailed below.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Going with (1) means that 9.0.2 will not be compatible with 9.0.1 with respect to default encodings. For example, non-ASCII files written with writeFile in 9.0.1 will not be readable using readFile in 9.0.2 and vice versa. This will be a pain point users. Further, if as is possible, UTF-8 is made the default for Tcl on all platforms at some point in the future, there will be whole another round of incompatibility headaches. So as much as I wish the UTF-8 defaulting had never been made without discussion, 9.0 is already out there so I am not in favour of (1).<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Option (2) may be a candidate for discussion. We can just tell users in this situation to stick to 8.6 or build their own Tcl with the understanding that it will have similar compatibility issues with respect to "standard" Tcl as in (1) above. Not our problem so to speak. But also to be considered is the fact that the TPC/HammerDB is currently one of the most popular Tcl applications based on public download numbers. Similar situations may arise with other large applications. I am therefore skeptical that this is a viable solution.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">Details behind options (3) and (4) are in the respective TIP's. Leaving aside the minor "reconcilable" differences with respect to implementation of [encoding user] or [exec -encoding], here is the crucial difference between the two.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">* TIP 716 removes the manifest entry from tclsh and wish. GetACP then returns the user's code page setting from the registry allowing the HammerDB-like applications to run. At the same time, in order to keep compatibility with 9.0.0/1, it hardcodes utf-8 as the default encoding returned by [encoding system] et al. In my opinion, if we indeed wanted to force utf-8 onto users, this is the way it should have been done in the first place.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">* TIP 718 deals with the problem by proposing to ship two Tcl shells, the current tclsh and a second one, tclshc, which has the manifest removed. The tclsh shell will behave identically to the one in 9.0.0/1. tclshc will behave substantially similar to TIP 716 and can be used by HammerDB etc. which do not work with tclsh.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">The primary hesitation I have with 718 is this dual shell approach and the potential for confusion. Somehow a user needs to know to use tclsh for all scripts. Except (for example) when accessing DB2. And X, Y and Z as the case may be. How are they to make that determination? Will a scripted application author now have to tell the user which Tcl shell to use? In fairness, most scripts will work with either. Still, this is a point for potential confusion. It is also the case that extension writers will now have test their extensions with both variations (not that they will!).<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">The advantage of tclsh (but not tclshc) in 718 over 716 (I am paraphrasing Jan here from the TIP 718 rationale, so he can clarify if I got it wrong) is that extensions that make use of the Windows ANSI API will automatically get UTF-8 support thereby supporting the entire Unicode range. With TIP 716 as well as 8.6, the ANSI API's will work only as long as the characters are supported by the user's code page.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">My counterpoint to this argument is that (a) extensions should not be using Windows ANSI API's in the first place, they should use the Unicode API's, (b) since 8.6 did not support this utf-8+ANSI API combination, such extensions are likely rare or old and in need of update anyways, and (c) the "automatic" support is likely overstated as not all API's and libraries support UTF-8 even when set as the code page (see TIP 716).<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">The question to answer is then whether this perceived benefit of 718 is overrides the downside to shipping two tclsh variations.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">A lesser issue is that 718 does not include the -encoding option to exec as I believe Jan thinks it should be a separate TIP. That is not difficult to resolve if we agree the option is needed.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">The decisions for the community to make now are which of the four options listed above make sense moving forward. Important that we make a decision for 9.0.2 as the longer we wait, the more 9.0.0/1 applications are going to be out there that might face this issue.<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">&nbsp;<o:p></o:p></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">/Ashok<o:p></o:p></p></div></div><p class="MsoNormal" style="margin-left:.5in">_______________________________________________<br>Tcl-Core mailing list<br><a href="mailto:Tcl...@li..." target="_blank">Tcl...@li...</a><br><a href="https://fedbdhd.r.tsp1-brevo.net/tr/cl/GDjC159F-zWfJNLNMCAqZ91dwRCeOIsWJSi_VS176PHSGOFK37nxkz_5OctJBFT5cjRe_INbl-chJmI3RTGuMm3jsKkuCxlGzJQ_Y5dmzBFqacZGnmcQtGMIJ41LmQ9Owu9jQ_-m_H0TW2rRq9pra2o4-U2zuoD7ah58ZK88lW6LvPFyGjn_khXGIp5xWkVq_UiSbwnAjfBmwIKCv5ZE1IrLDz8ldijcGeAIHsT5ruyqxcbp_IG95_Qa-yG-P_E-NIP_8zlskmtU54Xr4CjzubySWjWuWfyEYnR-1IVPBzRPUHuc31RyhiygEtwHrw" target="_blank">https://lists.sourceforge.net/lists/listinfo/tcl-core</a><o:p></o:p></p></div></blockquote></div></div><span>_______________________________________________</span><br><span>Tcl-Core mailing list</span><br><span>Tcl...@li...</span><br><span>https://lists.sourceforge.net/lists/listinfo/tcl-core</span><br></div></blockquote></body></html>

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[<apn...@ya...>]

Marc,

 

Thanks for taking the time to read and comment.

 

FWIW, I agree 9.1 should formally standardize on UTF-8 on other platforms as well. It just does not make sense to force UTF-8 only on Windows.

 

It needs a TIP though.

 

/Ashok

 

 

 

From: Marc Culler <cul...@gm...> 
Sent: Wednesday, May 21, 2025 7:39 PM
To: apn...@ya...
Cc: Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] About TIP 716 vs 718 vs other options

 

Based on Ashok's description of the situation, TIP #716 seems to me to be the best choice.  Shipping two tclsh executables seems likely to cause confusion and problems, and I am also skeptical about "automatic" support.  The first two options look to me like cans of worms to be opened in the future.

 

Not that it matters for this discussion, but I am in favor of "forcing" utf-8 on our users.  Standardization is very important, and it is too late for any other encoding to become the "standard".

 

- Marc

 

On Tue, May 20, 2025 at 9:41 PM apnmbx-public--- via Tcl-Core <tcl...@li... <mailto:tcl...@li...> > wrote:

All,

 

We need to make a collective decision regarding the default encoding issues on Windows. In the last meeting, I promised to write up a summary of the status and considerations for the problem being addressed by TIP's 716 and 718. Here it is along with the choices we have as to the next step.

 

Although specific to Windows, the basic issue does not need any deep understanding of Windows so I hope folks will read through and opine on the matter without copping out that they do not have Windows knowledge :-).

 

For those who have not read 716 or 718 (particularly the Background section in 716) or did not find the explanations clear, here is an example scenario the TIP's are independently addressing that I hope will shed some light.

 

A client library, linked to a Tcl extension, shares data (config info, db content etc.) with another program such as a server. The encoding used for the data is that returned by the Windows GetACP() call which returns the user's code page. Note that neither the library nor the server side is aware of Tcl's existence in any form except for the library being linked to a Tcl extension loaded into tclsh.

 

This scenario worked fine in Tcl 8.6 because both the client and server ends saw the same encoding value returned from GetACP().

 

Tcl 9.0 broke the scenario because of the addition of the activeCodePage entry in the executable manifest for tclsh and wish. The presence of this setting means all calls to GetACP() from that process, including those from the loaded client library, will return the code page utf-8 irrespective of the user setting. The breakage may be because now the library encodes using utf-8 and the server component encodes using the user's real code page. Or it may be because the library cannot handle non-DBCS multibyte encodings. Does not really matter. The primary point is that the issue cannot be fixed by modifying or recompiling the extension.

 

Moving forward, we can deal with this problem in one of several ways:

 

(1) Remove the activeCodePage entry from the manifest with no other changes. This will revert 9.x behavior on Windows to that of 8.6 (and also consistent with non-Windows platforms since user settings are not ignored on them).

 

(2) Ignore the issue. In scenarios where this is an issue, users can continue to use 8.6 or build tclsh/wish themselves after removing the activeCodePage entry from the manifest.

 

(3) TIP 716 as detailed below.

 

(4) TIP 718 as detailed below.

 

Going with (1) means that 9.0.2 will not be compatible with 9.0.1 with respect to default encodings. For example, non-ASCII files written with writeFile in 9.0.1 will not be readable using readFile in 9.0.2 and vice versa. This will be a pain point users. Further, if as is possible, UTF-8 is made the default for Tcl on all platforms at some point in the future, there will be whole another round of incompatibility headaches. So as much as I wish the UTF-8 defaulting had never been made without discussion, 9.0 is already out there so I am not in favour of (1).

 

Option (2) may be a candidate for discussion. We can just tell users in this situation to stick to 8.6 or build their own Tcl with the understanding that it will have similar compatibility issues with respect to "standard" Tcl as in (1) above. Not our problem so to speak. But also to be considered is the fact that the TPC/HammerDB is currently one of the most popular Tcl applications based on public download numbers. Similar situations may arise with other large applications. I am therefore skeptical that this is a viable solution.

 

Details behind options (3) and (4) are in the respective TIP's. Leaving aside the minor "reconcilable" differences with respect to implementation of [encoding user] or [exec -encoding], here is the crucial difference between the two.

 

* TIP 716 removes the manifest entry from tclsh and wish. GetACP then returns the user's code page setting from the registry allowing the HammerDB-like applications to run. At the same time, in order to keep compatibility with 9.0.0/1, it hardcodes utf-8 as the default encoding returned by [encoding system] et al. In my opinion, if we indeed wanted to force utf-8 onto users, this is the way it should have been done in the first place.

 

* TIP 718 deals with the problem by proposing to ship two Tcl shells, the current tclsh and a second one, tclshc, which has the manifest removed. The tclsh shell will behave identically to the one in 9.0.0/1. tclshc will behave substantially similar to TIP 716 and can be used by HammerDB etc. which do not work with tclsh.

 

The primary hesitation I have with 718 is this dual shell approach and the potential for confusion. Somehow a user needs to know to use tclsh for all scripts. Except (for example) when accessing DB2. And X, Y and Z as the case may be. How are they to make that determination? Will a scripted application author now have to tell the user which Tcl shell to use? In fairness, most scripts will work with either. Still, this is a point for potential confusion. It is also the case that extension writers will now have test their extensions with both variations (not that they will!).

 

The advantage of tclsh (but not tclshc) in 718 over 716 (I am paraphrasing Jan here from the TIP 718 rationale, so he can clarify if I got it wrong) is that extensions that make use of the Windows ANSI API will automatically get UTF-8 support thereby supporting the entire Unicode range. With TIP 716 as well as 8.6, the ANSI API's will work only as long as the characters are supported by the user's code page.

 

My counterpoint to this argument is that (a) extensions should not be using Windows ANSI API's in the first place, they should use the Unicode API's, (b) since 8.6 did not support this utf-8+ANSI API combination, such extensions are likely rare or old and in need of update anyways, and (c) the "automatic" support is likely overstated as not all API's and libraries support UTF-8 even when set as the code page (see TIP 716).

 

The question to answer is then whether this perceived benefit of 718 is overrides the downside to shipping two tclsh variations.

 

A lesser issue is that 718 does not include the -encoding option to exec as I believe Jan thinks it should be a separate TIP. That is not difficult to resolve if we agree the option is needed.

 

The decisions for the community to make now are which of the four options listed above make sense moving forward. Important that we make a decision for 9.0.2 as the longer we wait, the more 9.0.0/1 applications are going to be out there that might face this issue.

 

/Ashok

_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] About TIP 716 vs 718 vs other options

[Marc C. <cul...@gm...>]

Based on Ashok's description of the situation, TIP #716 seems to me to be
the best choice.  Shipping two tclsh executables seems likely to cause
confusion and problems, and I am also skeptical about "automatic" support.
The first two options look to me like cans of worms to be opened in the
future.

Not that it matters for this discussion, but I am in favor of "forcing"
utf-8 on our users.  Standardization is very important, and it is too late
for any other encoding to become the "standard".

- Marc

On Tue, May 20, 2025 at 9:41 PM apnmbx-public--- via Tcl-Core <
tcl...@li...> wrote:

> All,
>
>
>
> We need to make a collective decision regarding the default encoding
> issues on Windows. In the last meeting, I promised to write up a summary of
> the status and considerations for the problem being addressed by TIP's 716
> and 718. Here it is along with the choices we have as to the next step.
>
>
>
> Although specific to Windows, the basic issue does not need any deep
> understanding of Windows so I hope folks will read through and opine on the
> matter without copping out that they do not have Windows knowledge :-).
>
>
>
> For those who have not read 716 or 718 (particularly the Background
> section in 716) or did not find the explanations clear, here is an example
> scenario the TIP's are independently addressing that I hope will shed some
> light.
>
>
>
> A client library, linked to a Tcl extension, shares data (config info, db
> content etc.) with another program such as a server. The encoding used for
> the data is that returned by the Windows GetACP() call which returns the
> user's code page. Note that neither the library nor the server side is
> aware of Tcl's existence in any form except for the library being linked to
> a Tcl extension loaded into tclsh.
>
>
>
> This scenario worked fine in Tcl 8.6 because both the client and server
> ends saw the same encoding value returned from GetACP().
>
>
>
> Tcl 9.0 broke the scenario because of the addition of the activeCodePage
> entry in the executable manifest for tclsh and wish. *The presence of
> this setting means all calls to GetACP() from that process, including those
> from the loaded client library, will return the code page utf-8
> irrespective of the user setting.* The breakage may be because now the
> library encodes using utf-8 and the server component encodes using the
> user's real code page. Or it may be because the library cannot handle
> non-DBCS multibyte encodings. Does not really matter. The primary point is
> that the issue cannot be fixed by modifying or recompiling the extension.
>
>
>
> Moving forward, we can deal with this problem in one of several ways:
>
>
>
> (1) Remove the activeCodePage entry from the manifest with no other
> changes. This will revert 9.x behavior on Windows to that of 8.6 (and also
> consistent with non-Windows platforms since user settings are not ignored
> on them).
>
>
>
> (2) Ignore the issue. In scenarios where this is an issue, users can
> continue to use 8.6 or build tclsh/wish themselves after removing the
> activeCodePage entry from the manifest.
>
>
>
> (3) TIP 716 as detailed below.
>
>
>
> (4) TIP 718 as detailed below.
>
>
>
> Going with (1) means that 9.0.2 will not be compatible with 9.0.1 with
> respect to default encodings. For example, non-ASCII files written with
> writeFile in 9.0.1 will not be readable using readFile in 9.0.2 and vice
> versa. This will be a pain point users. Further, if as is possible, UTF-8
> is made the default for Tcl on all platforms at some point in the future,
> there will be whole another round of incompatibility headaches. So as much
> as I wish the UTF-8 defaulting had never been made without discussion, 9.0
> is already out there so I am not in favour of (1).
>
>
>
> Option (2) may be a candidate for discussion. We can just tell users in
> this situation to stick to 8.6 or build their own Tcl with the
> understanding that it will have similar compatibility issues with respect
> to "standard" Tcl as in (1) above. Not our problem so to speak. But also to
> be considered is the fact that the TPC/HammerDB is currently one of the
> most popular Tcl applications based on public download numbers. Similar
> situations may arise with other large applications. I am therefore
> skeptical that this is a viable solution.
>
>
>
> Details behind options (3) and (4) are in the respective TIP's. Leaving
> aside the minor "reconcilable" differences with respect to implementation
> of [encoding user] or [exec -encoding], here is the crucial difference
> between the two.
>
>
>
> * TIP 716 removes the manifest entry from tclsh and wish. GetACP then
> returns the user's code page setting from the registry allowing the
> HammerDB-like applications to run. At the same time, in order to keep
> compatibility with 9.0.0/1, it hardcodes utf-8 as the default encoding
> returned by [encoding system] et al. In my opinion, if we indeed wanted to
> force utf-8 onto users, this is the way it should have been done in the
> first place.
>
>
>
> * TIP 718 deals with the problem by proposing to ship two Tcl shells, the
> current tclsh and a second one, tclshc, which has the manifest removed. The
> tclsh shell will behave identically to the one in 9.0.0/1. tclshc will
> behave substantially similar to TIP 716 and can be used by HammerDB etc.
> which do not work with tclsh.
>
>
>
> The primary hesitation I have with 718 is this dual shell approach and the
> potential for confusion. Somehow a user needs to know to use tclsh for all
> scripts. Except (for example) when accessing DB2. And X, Y and Z as the
> case may be. How are they to make that determination? Will a scripted
> application author now have to tell the user which Tcl shell to use? In
> fairness, most scripts will work with either. Still, this is a point for
> potential confusion. It is also the case that extension writers will now
> have test their extensions with both variations (not that they will!).
>
>
>
> The advantage of tclsh (but not tclshc) in 718 over 716 (I am paraphrasing
> Jan here from the TIP 718 rationale, so he can clarify if I got it wrong)
> is that extensions that make use of the Windows ANSI API will automatically
> get UTF-8 support thereby supporting the entire Unicode range. With TIP 716
> as well as 8.6, the ANSI API's will work only as long as the characters are
> supported by the user's code page.
>
>
>
> My counterpoint to this argument is that (a) extensions should not be
> using Windows ANSI API's in the first place, they should use the Unicode
> API's, (b) since 8.6 did not support this utf-8+ANSI API combination, such
> extensions are likely rare or old and in need of update anyways, and (c)
> the "automatic" support is likely overstated as not all API's and libraries
> support UTF-8 even when set as the code page (see TIP 716).
>
>
>
> The question to answer is then whether this perceived benefit of 718 is
> overrides the downside to shipping two tclsh variations.
>
>
>
> A lesser issue is that 718 does not include the -encoding option to exec
> as I believe Jan thinks it should be a separate TIP. That is not difficult
> to resolve if we agree the option is needed.
>
>
>
> The decisions for the community to make now are which of the four options
> listed above make sense moving forward. Important that we make a decision
> for 9.0.2 as the longer we wait, the more 9.0.0/1 applications are going to
> be out there that might face this issue.
>
>
>
> /Ashok
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] New tip #722

[Donald P. <d.g...@co...>]

> If that makes "available" a misleading name, then it should change to something more accurate.

Perhaps [package names known] conveys better what the command does.

DGP

[TCLCORE] TIP 723: monotomic clock for after, independent of TIP 233 (Tcl_SetTimeProc/Tcl_QueryTimeProc), MS-Windows only

[Harald O. <har...@el...>]

Dear TCL team,

please allow me to propose a TIP to cure one of my biggest issues on TCL 
and MS-Windows: after does not fire if the wall clock changes.

The TIP text is here:
https://core.tcl-lang.org/tips/doc/trunk/tip/723.md

I am sorry to write it, as it is written (e.g. no respect on TIP233, 
Windows only). But those are the conclusions taken on recent discussions 
on the forelast biweekly telco and on the related bug tickets and on the 
core list.

I hope we will find a better solution, but this is my "minimum viable 
outcome" to this very urgent and annoying issue.

Thanks for all and take care,
Harald

[TCLCORE] OpenACS/TCL/Tk conference 2025 in Bologna/Italy/Europe on 10/11th of July 2025

[Harald O. <har...@el...>]

Dear TCL/Tk enthusiasts,
please allow me to invite you all to the
OpenACS/TCL/Tk conference 2025 in Bologna/Italy/Europe

https://openacs.km.at/evaluate/org/129998253/conferencenews/

it is a user meeting and we all show our own solutions.
As the minimum participants of 20 is reached, I can confirm that it will 
take place.

The so far presentations on my radar are:

- Tcl and Tk pannel
- Androwish Coffee machine
- Csabas new slider checkbox
- webcam barcode reading by me

So, there is still a lot of space for your contribution.
Any contribution is welcome, 5 minutes, no slides, is ok too.
We need you !
The interesting "abstract" entry field on the web site requires 200 
characters. I have filled it with "1234567890..." for my own contribution...

There is no online participation. It may happen, that the event is 
streamed. The chat will be monitored.

It would be great to see you in Bologna!
Harald
On behalf of the organisation committee

Re: [TCLCORE] New tip #722

[<apn...@ya...>]

Don,

Thanks for the exposition. You are right my suggestion would lead to a deep rabbit hole and should not be considered.

/Ashok

-----Original Message-----
From: Donald G Porter via Tcl-Core <tcl...@li...> 
Sent: Tuesday, May 20, 2025 5:39 PM
To: tcl...@li...
Subject: Re: [TCLCORE] New tip #722

On 5/19/25 22:34, apnmbx-public--- via Tcl-Core wrote:
> Since we are tinkering with [package names], I'd like to point out that the
> term "available" is a little misleading, even in the current manpage.
> Currently, [package names] only returns names *currently* known to Tcl, not
> those available on disk. One has to do the completely non-intuitive
>
> catch {package require nosuchpackage}; # Force a full search along path
> package names
>
> to retrieve the packages available to be loaded.
>
> So the question is,
>
> - will [package names available] continue the same behavior of not actually
> searching the package paths ?

Short answer:
[package names available] should only report names of packages with ifneeded scripts
already recorded in Tcl's memory.  This command should not prompt a search.

If that makes "available" a misleading name, then it should change to something more accurate.
> - if it does not do a full search, can we add a [package names refresh] that
> will replace the (ugly imo) catch {} ?
> - if it does do a full search, [package names] as currently implemented is
> no longer the union of [package names loaded] and [package names available]

These ideas are digging deeper into the [package unknown] problems, and that is a much
deeper rabbit hole than we want to explore in this TIP I think.

For those who have forgotten, or never learned, the [package] command is customizable.

The [package unknown] command permits an app or an installation to replace Tcl's
default package finder with another one better suited to any local method of package
storage.

The default package finder is [tclPkgUnkown].  It is implemented in a way that if
you ask it to go find package "foo", along the way it may discover and record the
ifneeded scripts of packages "bar" and "baz" as well.  ***This behavior is not in the
[package unknown] contract***  Custom [package unknown] replacements need not
and probably will not have that "discover all the packages" functionality.

When you invoke the incantation [package require nosuch], you are not using any
[package] command feature.  You are using a side effect of [tclPkgUnknown].

If we need a "report all the findable packages" functionality to exploit, step one will
have to be to extend or replace the [package unknown] mechanism. That's a bigger
project that will have slower progress and better not joined to this fairly simple proposal.

If your curiosity is sparked by any of this, you might enjoy an ancient presentation
on the subject.

https://math.nist.gov/~DPorter/tcltk/oscon/

-- 
| Don Porter            Applied and Computational Mathematics Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|



_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

[TCLCORE] About TIP 716 vs 718 vs other options

[<apn...@ya...>]

All,

 

We need to make a collective decision regarding the default encoding issues
on Windows. In the last meeting, I promised to write up a summary of the
status and considerations for the problem being addressed by TIP's 716 and
718. Here it is along with the choices we have as to the next step.

 

Although specific to Windows, the basic issue does not need any deep
understanding of Windows so I hope folks will read through and opine on the
matter without copping out that they do not have Windows knowledge :-).

 

For those who have not read 716 or 718 (particularly the Background section
in 716) or did not find the explanations clear, here is an example scenario
the TIP's are independently addressing that I hope will shed some light.

 

A client library, linked to a Tcl extension, shares data (config info, db
content etc.) with another program such as a server. The encoding used for
the data is that returned by the Windows GetACP() call which returns the
user's code page. Note that neither the library nor the server side is aware
of Tcl's existence in any form except for the library being linked to a Tcl
extension loaded into tclsh.

 

This scenario worked fine in Tcl 8.6 because both the client and server ends
saw the same encoding value returned from GetACP().

 

Tcl 9.0 broke the scenario because of the addition of the activeCodePage
entry in the executable manifest for tclsh and wish. The presence of this
setting means all calls to GetACP() from that process, including those from
the loaded client library, will return the code page utf-8 irrespective of
the user setting. The breakage may be because now the library encodes using
utf-8 and the server component encodes using the user's real code page. Or
it may be because the library cannot handle non-DBCS multibyte encodings.
Does not really matter. The primary point is that the issue cannot be fixed
by modifying or recompiling the extension.

 

Moving forward, we can deal with this problem in one of several ways:

 

(1) Remove the activeCodePage entry from the manifest with no other changes.
This will revert 9.x behavior on Windows to that of 8.6 (and also consistent
with non-Windows platforms since user settings are not ignored on them).

 

(2) Ignore the issue. In scenarios where this is an issue, users can
continue to use 8.6 or build tclsh/wish themselves after removing the
activeCodePage entry from the manifest.

 

(3) TIP 716 as detailed below.

 

(4) TIP 718 as detailed below.

 

Going with (1) means that 9.0.2 will not be compatible with 9.0.1 with
respect to default encodings. For example, non-ASCII files written with
writeFile in 9.0.1 will not be readable using readFile in 9.0.2 and vice
versa. This will be a pain point users. Further, if as is possible, UTF-8 is
made the default for Tcl on all platforms at some point in the future, there
will be whole another round of incompatibility headaches. So as much as I
wish the UTF-8 defaulting had never been made without discussion, 9.0 is
already out there so I am not in favour of (1).

 

Option (2) may be a candidate for discussion. We can just tell users in this
situation to stick to 8.6 or build their own Tcl with the understanding that
it will have similar compatibility issues with respect to "standard" Tcl as
in (1) above. Not our problem so to speak. But also to be considered is the
fact that the TPC/HammerDB is currently one of the most popular Tcl
applications based on public download numbers. Similar situations may arise
with other large applications. I am therefore skeptical that this is a
viable solution.

 

Details behind options (3) and (4) are in the respective TIP's. Leaving
aside the minor "reconcilable" differences with respect to implementation of
[encoding user] or [exec -encoding], here is the crucial difference between
the two.

 

* TIP 716 removes the manifest entry from tclsh and wish. GetACP then
returns the user's code page setting from the registry allowing the
HammerDB-like applications to run. At the same time, in order to keep
compatibility with 9.0.0/1, it hardcodes utf-8 as the default encoding
returned by [encoding system] et al. In my opinion, if we indeed wanted to
force utf-8 onto users, this is the way it should have been done in the
first place.

 

* TIP 718 deals with the problem by proposing to ship two Tcl shells, the
current tclsh and a second one, tclshc, which has the manifest removed. The
tclsh shell will behave identically to the one in 9.0.0/1. tclshc will
behave substantially similar to TIP 716 and can be used by HammerDB etc.
which do not work with tclsh.

 

The primary hesitation I have with 718 is this dual shell approach and the
potential for confusion. Somehow a user needs to know to use tclsh for all
scripts. Except (for example) when accessing DB2. And X, Y and Z as the case
may be. How are they to make that determination? Will a scripted application
author now have to tell the user which Tcl shell to use? In fairness, most
scripts will work with either. Still, this is a point for potential
confusion. It is also the case that extension writers will now have test
their extensions with both variations (not that they will!).

 

The advantage of tclsh (but not tclshc) in 718 over 716 (I am paraphrasing
Jan here from the TIP 718 rationale, so he can clarify if I got it wrong) is
that extensions that make use of the Windows ANSI API will automatically get
UTF-8 support thereby supporting the entire Unicode range. With TIP 716 as
well as 8.6, the ANSI API's will work only as long as the characters are
supported by the user's code page.

 

My counterpoint to this argument is that (a) extensions should not be using
Windows ANSI API's in the first place, they should use the Unicode API's,
(b) since 8.6 did not support this utf-8+ANSI API combination, such
extensions are likely rare or old and in need of update anyways, and (c) the
"automatic" support is likely overstated as not all API's and libraries
support UTF-8 even when set as the code page (see TIP 716).

 

The question to answer is then whether this perceived benefit of 718 is
overrides the downside to shipping two tclsh variations.

 

A lesser issue is that 718 does not include the -encoding option to exec as
I believe Jan thinks it should be a separate TIP. That is not difficult to
resolve if we agree the option is needed.

 

The decisions for the community to make now are which of the four options
listed above make sense moving forward. Important that we make a decision
for 9.0.2 as the longer we wait, the more 9.0.0/1 applications are going to
be out there that might face this issue.

 

/Ashok

Re: [TCLCORE] New tip #722

[Donald G P. <don...@ni...>]

On 5/19/25 22:34, apnmbx-public--- via Tcl-Core wrote:
> Since we are tinkering with [package names], I'd like to point out that the
> term "available" is a little misleading, even in the current manpage.
> Currently, [package names] only returns names *currently* known to Tcl, not
> those available on disk. One has to do the completely non-intuitive
>
> catch {package require nosuchpackage}; # Force a full search along path
> package names
>
> to retrieve the packages available to be loaded.
>
> So the question is,
>
> - will [package names available] continue the same behavior of not actually
> searching the package paths ?

Short answer:
[package names available] should only report names of packages with ifneeded scripts
already recorded in Tcl's memory.  This command should not prompt a search.

If that makes "available" a misleading name, then it should change to something more accurate.
> - if it does not do a full search, can we add a [package names refresh] that
> will replace the (ugly imo) catch {} ?
> - if it does do a full search, [package names] as currently implemented is
> no longer the union of [package names loaded] and [package names available]

These ideas are digging deeper into the [package unknown] problems, and that is a much
deeper rabbit hole than we want to explore in this TIP I think.

For those who have forgotten, or never learned, the [package] command is customizable.

The [package unknown] command permits an app or an installation to replace Tcl's
default package finder with another one better suited to any local method of package
storage.

The default package finder is [tclPkgUnkown].  It is implemented in a way that if
you ask it to go find package "foo", along the way it may discover and record the
ifneeded scripts of packages "bar" and "baz" as well.  ***This behavior is not in the
[package unknown] contract***  Custom [package unknown] replacements need not
and probably will not have that "discover all the packages" functionality.

When you invoke the incantation [package require nosuch], you are not using any
[package] command feature.  You are using a side effect of [tclPkgUnknown].

If we need a "report all the findable packages" functionality to exploit, step one will
have to be to extend or replace the [package unknown] mechanism. That's a bigger
project that will have slower progress and better not joined to this fairly simple proposal.

If your curiosity is sparked by any of this, you might enjoy an ancient presentation
on the subject.

https://math.nist.gov/~DPorter/tcltk/oscon/

-- 
| Don Porter            Applied and Computational Mathematics Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

Re: [TCLCORE] New tip #722

[da S. P. J <pet...@fl...>]

It sounds like “package names” is misleading, but transitioning to a better name is problematic.

I am not sure it should automatically walk the file system implicitly, will that include vfs mounts? If it does, a way to refresh the list still needs to be available so that newly loaded vfs mounts, dynamically created packages (eg Speedtables), and packages installed after startup can be enumerated.

From: apnmbx-public--- via Tcl-Core <tcl...@li...>
Date: Monday, May 19, 2025 at 21:37
To: ch...@ch... <ch...@ch...>
Cc: 'Tcl Core' <tcl...@li...>
Subject: Re: [TCLCORE] New tip #722
Update done. (Could someone with the right privileges please give TIP repository write access to Christian?) Since we are tinkering with [package names], I'd like to point out that the term "available" is a little misleading, even in the current


Update done. (Could someone with the right privileges please give TIP

repository write access to Christian?)



Since we are tinkering with [package names], I'd like to point out that the

term "available" is a little misleading, even in the current manpage.

Currently, [package names] only returns names *currently* known to Tcl, not

those available on disk. One has to do the completely non-intuitive



catch {package require nosuchpackage}; # Force a full search along path

package names



to retrieve the packages available to be loaded.



So the question is,



- will [package names available] continue the same behavior of not actually

searching the package paths ?

- if it does not do a full search, can we add a [package names refresh] that

will replace the (ugly imo) catch {} ?

- if it does do a full search, [package names] as currently implemented is

no longer the union of [package names loaded] and [package names available]



I would actually prefer [package names] to do a full search though that

would be differ from current behavior. I'm not sure why it did not do that

in the first place, presumably performance, though I wonder if performance

is really an issue in situations where [package names] is used.



/Ashok



-----Original Message-----

From: Christian Werner <Chr...@t-...>

Good point, Don. So may I ask Ashok again to update the TIP entry to











_______________________________________________

Tcl-Core mailing list

Tcl...@li...

https://urldefense.com/v3/__https://lists.sourceforge.net/lists/listinfo/tcl-core__;!!MvWE!HhoyC6t3Z3QsMpJKI6zRG93Z-SlDggeyL8FzYh9mbmI_bY__OdNgKVlzyqV85ShKRFbc9ia1Q_otKhptMeOMyzoYZ2mxovQI6g$<https://urldefense.com/v3/__https:/lists.sourceforge.net/lists/listinfo/tcl-core__;!!MvWE!HhoyC6t3Z3QsMpJKI6zRG93Z-SlDggeyL8FzYh9mbmI_bY__OdNgKVlzyqV85ShKRFbc9ia1Q_otKhptMeOMyzoYZ2mxovQI6g$>

Re: [TCLCORE] New tip #722

[Schelte B. <tc...@tc...>]

On 20/05/2025 04:34, apnmbx-public--- via Tcl-Core wrote:
> Could someone with the right privileges please give TIP
> repository write access to Christian?

Done.


Schelte.

Re: [TCLCORE] Next bi-weekly meeting 19th May 2025 at 12:00 UTC

[<apn...@ya...>]

Brian,

 

Thanks for taking a look. I will commit after a few days in case anyone else takes a look and has objections.

 

Was originally targeting core-9-0-branch since no public API’s have changed but as Jan expressed reservations yesterday in terms of risk committing to a released minor version, will target 9.1a0 instead.

 

/Ashok

 

From: Brian Griffin <bri...@ea...> 



APN mentioned the new internal list representation proposal is ready for testing as per his May 15th email to TCLCORE. BrianG will review.

 

Hi Ashok,

 

I browsed through the implementation and it seems pretty much as I expected. It looks good to me, from a high level. I didn't comb through it looking for flaws.

 

Ship It!

 

-Brian





 

Re: [TCLCORE] Next bi-weekly meeting 19th May 2025 at 12:00 UTC

[Brian G. <bri...@ea...>]

See comments below.

On May 19, 2025, at 17:08, Steve Landers <st...@di...> wrote:

Here’s some notes from the meeting

Discussion TIP 718 (https://core.tcl-lang.org/tips/doc/trunk/tip/718.md)

  *   the issue is not all applications on Windows are able to move to utf-8
  *   this is generally recognised by those who care about encoding on Windows
  *   there was discussion on the relationship between TIP 718 and TIP 716 (https://core.tcl-lang.org/tips/doc/trunk/tip/716.md)
  *   do we delay Tcl 9.0.2 or delay TIPs 716/718
  *   it was recognised that breakage now is better than breakage later
  *   DGP leans towards delay so we can better understand the problem before making changes
  *   we don't want to go down a bad path
  *   options are 716 vs 718 vs status quo
  *   Ashok will email TCLCORE outlining the situation

Discussion re TIP 720 Updated Bytecode Opcodes (https://core.tcl-lang.org/tips/doc/trunk/tip/720.md)

  *   people familiar with BCE (ByteCode Engine) are comfortable
  *   Rolf raised concern about tclcompiler  and tbcload
  *   we need to find the "official" repository for these
  *   these need to be brought up to date with 9.0 then 9.1 + 720
  *   DGP flagged there may be opportunities for sponsored work given there are commercial organisations relying on these components

Documentation

  *   there is still strong desire to keep the documentation program moving forward, including markdown, nroff and html
  *   SteveL will shake the trees

APN mentioned the new internal list representation proposal is ready for testing as per his May 15th email to TCLCORE. BrianG will review.

Hi Ashok,

I browsed through the implementation and it seems pretty much as I expected. It looks good to me, from a high level. I didn't comb through it looking for flaws.

Ship It!

-Brian


DGP mentioned converting mpexpr to Tcl9 (https://core.tcl-lang.org/mpexpr/timeline).

RMax asked about getting a review of his new socket implementation [https://chiselapp.com/user/rmax/repository/sock/]

  *   he is especially interested in comments or suggestions on how to translate between symbolic names and their values. Schelte suggested that he may be able to use constant variables (tip #677). One possible limitation is that you can't have an array of constant variables.
  *   as a result of previous debates/pushback about UDP the implementation is not based on the channel abstraction but instead data is passed as a binary stringg
  *   while currently an extension, it is easily adaptable for the core at some future point. RMax is looking for help/guidance in adding a channel wrapper from some channel expert. Jan volunteered and Andreas (not present) is likely the most knowledgeable.

See you all on June 2 for the next meeting

-- Steve
On 19 May 2025 at 9:27 AM +0800, Steve Landers <st...@di...>, wrote:
On behalf of Harald (who is unavailable) I’d like to invite anyone interested to the bi-weekly Tcl/TK telco on Monday May 19th 2025 at 12:00 UTC via https://meet.jit.si/TclMonthlyMeetup

The bi-weekly meetings are designed to keep Tcl/Tk development moving forwards towards regular releases, allowing discussion and clarification of any issues. Agenda items for this meeting will include:

  *   the releases planned for this month - 9.0.2 and 9.1a0 (the first release of Tcl 9.1 features) as per TIP #713 https://core.tcl-lang.org/tips/doc/trunk/tip/713.md
  *   TIPs being finalised and voted upon
  *   documentation

plus anything else attendees might want to raise.

Thanks to all

-- Steve
_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core
_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] CFV warning: TIP #718: encoding compatibility for GDI/HammerDB/TPC/DB2

[<apn...@ya...>]

As promised in yesterday's meeting, I'll write up the current status w.r.t
the issues addressed by TIP's 716 and 718 but that's a separate post. This
is just a response to Jan's email below.

While Jan suggests separate TIP's, my belief is that TIP's in general should
be directed at a specific problem to be solved in its entirety as opposed to
split into fragmented TIP's that at the end of the day may or may not work
well together to completely address the issue.

In the case of 716, the issue is one of sharing data between processes or
libraries without a formal means of agreeing on the encoding. To fix this,

- UTF-8 encoding should not be forced onto libraries that do not support
UTF-8

- Since some external applications always use UTF-8 (git e.g.) while others
(Windows system command line programs like findstr, cacls etc.) use user
encoding, there needs to be a way for exec to handle both. Thus the
-encoding option is proposed.

- Since 9.0, UTF-8 is the default so that case is covered. For the Windows
system command line programs, there is currently (in 9.0) no way for Tcl
applications to know what encoding would be appropriate. Thus the [encoding
user] command to retrieve the user setting as used by those programs.

TIP 716 therefore covers all the above.

As an aside, the specification in TIP 716 is just over half a page so I am
not sure why it would be termed too big! The rest of the TIP is just
explanations and background material for folks who do not have prior
exposure to the discussion.

With respect to the -encoding option for exec not being worked out, that is
not the case. I was waiting for folks to express an opinion about allowing
it to be set via an env variable but since there have been no comments in
that regard, I will stick with the current implementation.

In any case, at the end of the day all the above is ancillary to the core
issue and most important difference between 716 and 718. That's for a follow
up post.


/Ashok

-----Original Message-----
From: Jan Nijtmans <jan...@gm...> 
Sent: Monday, May 19, 2025 1:59 PM
To: Tcl Core List <tcl...@li...>
Subject: [TCLCORE] CFV warning: TIP #718: encoding compatibility for
GDI/HammerDB/TPC/DB2

This is a CFV warning for TIP #718. for Tcl 9.0.2+:
    encoding compatibility for GDI/HammerDB/TPC/DB2
    <https://core.tcl-lang.org/tips/doc/trunk/tip/718.md>

This TIP is meant as a subset of TIP #716 (with a few modifications). In my
opinion TIP #716 is too big, it should have been split in - at least - 4
parts.
  1) bugfix, see <https://core.tcl-lang.org/tcl/tktview/8ffd8cabd1>
         this is a no-brainer, already committed.
  2) Remove UTF-8 from manifest. I don't agree with that, but I
      agree with providing a tclsh90c.exe with a different setting.
  3) New command "encoding user". Good idea, so I took it
      over in TIP #718 (slightly modified, the TIP explains why)
  4) -encoding option for "exec". Sounds OK to me, but TIP #716
      still contains too many questions about this feature. That
      tells me it's not sufficiently worked out yet.

Since "encoding user" is a good idea, we can already do something
useful now, in time for Tcl 9.0.2 (I hope). I did my best to describe
everything in the TIP text.

Regards,
         Jan Nijtmans


_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] New tip #722

[<apn...@ya...>]

Update done. (Could someone with the right privileges please give TIP
repository write access to Christian?)

Since we are tinkering with [package names], I'd like to point out that the
term "available" is a little misleading, even in the current manpage.
Currently, [package names] only returns names *currently* known to Tcl, not
those available on disk. One has to do the completely non-intuitive

catch {package require nosuchpackage}; # Force a full search along path
package names

to retrieve the packages available to be loaded.

So the question is,

- will [package names available] continue the same behavior of not actually
searching the package paths ?
- if it does not do a full search, can we add a [package names refresh] that
will replace the (ugly imo) catch {} ?
- if it does do a full search, [package names] as currently implemented is
no longer the union of [package names loaded] and [package names available]

I would actually prefer [package names] to do a full search though that
would be differ from current behavior. I'm not sure why it did not do that
in the first place, presumably performance, though I wonder if performance
is really an issue in situations where [package names] is used.

/Ashok

-----Original Message-----
From: Christian Werner <Chr...@t-...> 
Good point, Don. So may I ask Ashok again to update the TIP entry to

Re: [TCLCORE] Next bi-weekly meeting 19th May 2025 at 12:00 UTC

[Steve L. <st...@di...>]

Here’s some notes from the meeting

Discussion TIP 718 (https://core.tcl-lang.org/tips/doc/trunk/tip/718.md)

• the issue is not all applications on Windows are able to move to utf-8
• this is generally recognised by those who care about encoding on Windows
• there was discussion on the relationship between TIP 718 and TIP 716 (https://core.tcl-lang.org/tips/doc/trunk/tip/716.md)
• do we delay Tcl 9.0.2 or delay TIPs 716/718
• it was recognised that breakage now is better than breakage later
• DGP leans towards delay so we can better understand the problem before making changes
• we don't want to go down a bad path
• options are 716 vs 718 vs status quo
• Ashok will email TCLCORE outlining the situation

Discussion re TIP 720 Updated Bytecode Opcodes (https://core.tcl-lang.org/tips/doc/trunk/tip/720.md)

• people familiar with BCE (ByteCode Engine) are comfortable
• Rolf raised concern about tclcompiler  and tbcload
• we need to find the "official" repository for these
• these need to be brought up to date with 9.0 then 9.1 + 720
• DGP flagged there may be opportunities for sponsored work given there are commercial organisations relying on these components

Documentation

• there is still strong desire to keep the documentation program moving forward, including markdown, nroff and html
• SteveL will shake the trees

APN mentioned the new internal list representation proposal is ready for testing as per his May 15th email to TCLCORE. BrianG will review.

DGP mentioned converting mpexpr to Tcl9 (https://core.tcl-lang.org/mpexpr/timeline).

RMax asked about getting a review of his new socket implementation [https://chiselapp.com/user/rmax/repository/sock/]

• he is especially interested in comments or suggestions on how to translate between symbolic names and their values. Schelte suggested that he may be able to use constant variables (tip #677). One possible limitation is that you can't have an array of constant variables.
• as a result of previous debates/pushback about UDP the implementation is not based on the channel abstraction but instead data is passed as a binary stringg
• while currently an extension, it is easily adaptable for the core at some future point. RMax is looking for help/guidance in adding a channel wrapper from some channel expert. Jan volunteered and Andreas (not present) is likely the most knowledgeable.

See you all on June 2 for the next meeting

-- Steve
On 19 May 2025 at 9:27 AM +0800, Steve Landers <st...@di...>, wrote:
> On behalf of Harald (who is unavailable) I’d like to invite anyone interested to the bi-weekly Tcl/TK telco on Monday May 19th 2025 at 12:00 UTC via https://meet.jit.si/TclMonthlyMeetup
>
> The bi-weekly meetings are designed to keep Tcl/Tk development moving forwards towards regular releases, allowing discussion and clarification of any issues. Agenda items for this meeting will include:
>
> • the releases planned for this month - 9.0.2 and 9.1a0 (the first release of Tcl 9.1 features) as per TIP #713 https://core.tcl-lang.org/tips/doc/trunk/tip/713.md
> • TIPs being finalised and voted upon
> • documentation
>
> plus anything else attendees might want to raise.
>
> Thanks to all
>
> -- Steve
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] New tip #722

[Christian W. <Chr...@t-...>]

On 05/19/2025 05:24 PM, Donald G Porter via Tcl-Core wrote:
> On 5/19/25 11:10, Christian Werner wrote:
>> On 5/19/25 14:31, Christian Werner wrote:

>>  …
>> Massimo Manghi gave me his proposal "package names -loaded" which IMO is
>> a very good choice as it smoothly fits into the current implementation of
>> "package names". So the "-loaded" option simply modifies the if-condition
>> whose block appends to the result list. Streamlined.
>
> The existing command is [package names] with no arguments accepted.
>
> What do you think of adding two optional argument values:
>
> [package names provided] -- return names of packages provided in the current interp.
>
> [package names available] -- return names of packages where some "ifneeded" script has been found.
>
> and
>
> [package names] -- continue to return the union of the lists described above.
>
> This reuses the "provide" verbiage already part of [package], and avoids confusion
> with the "-loaded" term that better belongs with [load], etc.

Good point, Don. So may I ask Ashok again to update the TIP entry to

---8><---8><---

# TIP 722: Improve **package names** subcommand
     Author:         Christian Werner <und...@gm...>
     Tcl-Version:    9.1
     State:          Draft
     Type:           Project
     Created:        19-May-2025
     Keywords:       Tcl,package
-----

# Abstract

This TIP proposes to improve the **package names** subcommand such
that its result list can be filtered with respect to packages which are
provided (i.e. present) and packages which are not provided,
but for which a **package ifneeded** script is available.

# Rationale

Although the **package names** subcommand returns a list of all currently
provided and available packages there's no equivalent to retrieve only the
list of currently provided (i.e. present) packages. However, the latter can
be of use e.g. to build a software bill of materials of an application or
to help in generating a package dependency graph.

# Specification

For **package names** without any further arguments (current default
mode of operation) the result list is made up of package names which are
provided or for which a **package ifneeded** script is available.

For **package names available** the list contains only the packages
which are not provided (i.e. not present) but for which a **package ifneeded**
script is available.

For **package names provided** the list contains only the currently provided
(i.e. present) packages.

# Implementation

TBD.

# Copyright

This document has been placed in the public domain.

---8><---8><---

Thank you,
Christian

Re: [TCLCORE] New tip #722

[Donald G P. <don...@ni...>]

On 5/19/25 11:10, Christian Werner wrote:
> On 5/19/25 14:31, Christian Werner wrote:
>> On 05/19/2025 01:58 PM, Donald G Porter via Tcl-Core wrote:
>>
>>> Revising the behavior of an existing subcommand raises worries ( at
>>> least )
>>> about compatibility.
>>>
>>> Consider proposing a new subcommand instead.
>>
>> So "package names -present", i.e. a switch to an existing subcommand
>> isn't a nogo, too?
>> But from the natural language standpoint "package present" seemed the
>> most logical to me.
>> Then dear native speakers please find a good alternative for me.
>
> Massimo Manghi gave me his proposal "package names -loaded" which IMO is
> a very good choice as it smoothly fits into the current implementation of
> "package names". So the "-loaded" option simply modifies the if-condition
> whose block appends to the result list. Streamlined.

The existing command is [package names] with no arguments accepted.

What do you think of adding two optional argument values:

[package names provided] -- return names of packages provided in the current interp.

[package names available] -- return names of packages where some "ifneeded" script has been found.

and

[package names] -- continue to return the union of the lists described above.

This reuses the "provide" verbiage already part of [package], and avoids confusion
with the "-loaded" term that better belongs with [load], etc.

-- 
| Don Porter            Applied and Computational Mathematics Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|