tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] Tcl_Obj types and tkinter

[Jan N. <jan...@gm...>]

Op ma 1 jul 2024 om 15:49 schreef Marc Culler:
> Here are two simple ways to remove this unnecessary complication for tkinter as well as for other embedded Tcl interpreters:
>
> 1) Move the block of declarations reprinted at the bottom of this message from tclInt.h to tcl.h so that tkinter could include them without including internal Tcl headers.

Unfortunately, that wouldn't work with systems like Windows and AIX.
I will spare you the details .....

> 2) Register *all* of the core types declared in that block of declarations in TclInitObjSubsystem.
> (In Tcl 9 the missing ones are: "bignum", "boolean", "exprcode", "int", "index", and "ensembleCommand"; obviously int and boolean are very common as returned values of commands.)

Let's add one more possible solution:
3) Use Tcl_GetNumberFromObj()
(https://core.tcl-lang.org/tips/doc/trunk/tip/638.md)

> Is there any reason why neither of these simple remedies would be available to make Tk 9 more accessible to its most important clients?

The reason is that the only reliable way to know the type
is to parse the string. As long as that isn't done, the
objType field will be NULL, there is no way to tell
what the 'real' type is.

Example:
       Tcl_Obj *obj = new Tcl_NewString("542"):
When calling
       Tcl_GetIntFromObj()
this function will succeed, because the string clearly is an
integer. However, only after the Tcl_GetIntFromObj() call,
the objType field is set and will give the expected value.

The Tcl_GetNumberFromObj() function knows about
doubles, integers and bignums, so you can determine
the type with only a single function call. If the string
is already parsed, Tcl_GetNumberFromObj() won't
do it again.

Regarding the "boolean" 'type', it is only used for representing
string like "true" or "no". Tk never produces such booleans,
it always uses "0" or "1". I don't think it gives much
added values, but otherwise Tcl_GetBooleanFromObj()
is the only way to check for this type.

Hope this helps,
         Jan Nijtmans

Re: [TCLCORE] Tcl_Obj types and tkinter

[Harald O. <har...@el...>]

Am 01.07.2024 um 15:48 schrieb Marc Culler:
> Dear TCT,
> 
> I have been working on making tkinter work with Tk 9, which broke it.  
> The breakage seems to have been caused by a deliberate design choice.  
> While the choice may have been deliberate, I think it may also have been 
> arbitrary and that a different, less damaging, choice could have been 
> made without causing any issues for Tcl or Tk.
> 
> Probably everyone knows that tkinter is a Python module which embeds a 
> Tcl interpreter with the Tk package loaded. It provides Python objects 
> corresponding to Tk UI elements. The methods of those Python objects 
> call Tcl and Tk commands via the embedded interpreter. Of course the 
> result of such a command must be converted from a Tcl_Obj to a Python 
> object to make it useful in a Python program.  To do this, the tkinter 
> module must be able to determine the type of the Tcl_Obj.
> 
> The questionable design choice was that many of the core Tcl_Obj types 
> are not being registered in Tk 9.  Notably, the "int" type is not 
> registered.  This means that the public interface provided by 
> Tcl_GetObjType cannot be used to identify the type of an "int" Tcl_Obj 
> from its typePtr field.  One has to parse the string in the name field, 
> and that is costly.
> 
> Here are two simple ways to remove this unnecessary complication for 
> tkinter as well as for other embedded Tcl interpreters:
> 
> 1) Move the block of declarations reprinted at the bottom of this 
> message from tclInt.h to tcl.h so that tkinter could include them 
> without including internal Tcl headers.
> 2) Register *all* of the core types declared in that block of 
> declarations in TclInitObjSubsystem.
> (In Tcl 9 the missing ones are: "bignum", "boolean", "exprcode", "int", 
> "index", and "ensembleCommand"; obviously int and boolean are very 
> common as returned values of commands.)
> 
> Is there any reason why neither of these simple remedies would be 
> available to make Tk 9 more accessible to its most important clients?  
> (I say "most important" because anyone who has used a search engine to 
> find out something about Tk would come away from the experience 
> believing that everyone who uses Tk does so via its Python, Ruby or Perl 
> bindings.)
> 
> - Marc
> 
> *Declarations of "core types" from tclInt.h:*
> 
> /*
>   * Variables denoting the Tcl object types defined in the core.
>   */
> 
> MODULE_SCOPE const Tcl_ObjType tclBignumType;
> MODULE_SCOPE const Tcl_ObjType tclBooleanType;
> MODULE_SCOPE const Tcl_ObjType tclByteCodeType;
> MODULE_SCOPE const Tcl_ObjType tclDoubleType;
> MODULE_SCOPE const Tcl_ObjType tclExprCodeType;
> MODULE_SCOPE const Tcl_ObjType tclIntType;
> MODULE_SCOPE const Tcl_ObjType tclIndexType;
> MODULE_SCOPE const Tcl_ObjType tclListType;
> MODULE_SCOPE const Tcl_ObjType tclDictType;
> MODULE_SCOPE const Tcl_ObjType tclProcBodyType;
> MODULE_SCOPE const Tcl_ObjType tclStringType;
> MODULE_SCOPE const Tcl_ObjType tclEnsembleCmdType;
> MODULE_SCOPE const Tcl_ObjType tclRegexpType;
> MODULE_SCOPE Tcl_ObjType tclCmdNameType;
> 
> *The subset of these types which get registered in TclInitObjSystem:*
> 
>      Tcl_RegisterObjType(&tclDoubleType);
>      Tcl_RegisterObjType(&tclStringType);
>      Tcl_RegisterObjType(&tclListType);
>      Tcl_RegisterObjType(&tclDictType);
>      Tcl_RegisterObjType(&tclByteCodeType);
>      Tcl_RegisterObjType(&tclCmdNameType);
>      Tcl_RegisterObjType(&tclRegexpType);
>      Tcl_RegisterObjType(&tclProcBodyType);

Dear Marc,

thank you for this action, I really appreciate! I am sure, all TCL&Tk 
people are instantly ready to help you and the change was not to throw 
stones in your way.
We really thank you to interface TkInter and the TCL/Tk community and 
all the great work on Tk and the Macintosh.
I think, the only reason why this was done is, that your requirements 
are not known on this side.
When I co-organized the 2023 OpenACS/TCL/Tk conference and tried to put 
focus on Tk, I failed to get any information/participant/speaker from 
all the RubyTk/TkInter/PerlTk communities. I hope, we will get to know 
better of each other.
I am aware that partly, Tk still exists because there is TkInter, as the 
Tk community is very small.

About your ticket: would you please transfer your information to a TCL 
ticket? It is better to discuss there to not loose any information on 
the way....

About the conference: in two weeks, we have the great OpenACS/TCL/Tk 
conference and there is the Tk pannel with "New canvas", "New text 
widget", "New metawidget", Tk9.0 state of play and Tk pannel.
I will prepare a presentation for this. Here is the one from last year:
https://openacs.org/conf2023/info/download/file/state_of_tk_hao.pdf
If you have any information about TkInter, you are warmly welcome !
I may also add a page on the TCL state about your upper problem in the 
TCL pannel presentation:
https://openacs.org/conf2023/info/download/file/state_of_tcl_hao.pdf

Thank you for all,
Harald

[TCLCORE] Tcl_Obj types and tkinter

[Marc C. <mar...@gm...>]

Dear TCT,

I have been working on making tkinter work with Tk 9, which broke it.  The
breakage seems to have been caused by a deliberate design choice.  While
the choice may have been deliberate, I think it may also have been
arbitrary and that a different, less damaging, choice could have been made
without causing any issues for Tcl or Tk.

Probably everyone knows that tkinter is a Python module which embeds a Tcl
interpreter with the Tk package loaded. It provides Python objects
corresponding to Tk UI elements. The methods of those Python objects call
Tcl and Tk commands via the embedded interpreter. Of course the result of
such a command must be converted from a Tcl_Obj to a Python object to make
it useful in a Python program.  To do this, the tkinter module must be able
to determine the type of the Tcl_Obj.

The questionable design choice was that many of the core Tcl_Obj types are
not being registered in Tk 9.  Notably, the "int" type is not registered.
This means that the public interface provided by Tcl_GetObjType cannot be
used to identify the type of an "int" Tcl_Obj from its typePtr field.  One
has to parse the string in the name field, and that is costly.

Here are two simple ways to remove this unnecessary complication for
tkinter as well as for other embedded Tcl interpreters:

1) Move the block of declarations reprinted at the bottom of this message
from tclInt.h to tcl.h so that tkinter could include them without including
internal Tcl headers.
2) Register *all* of the core types declared in that block of declarations
in TclInitObjSubsystem.
(In Tcl 9 the missing ones are: "bignum", "boolean", "exprcode", "int",
"index", and "ensembleCommand"; obviously int and boolean are very common
as returned values of commands.)

Is there any reason why neither of these simple remedies would be available
to make Tk 9 more accessible to its most important clients?  (I say "most
important" because anyone who has used a search engine to find out
something about Tk would come away from the experience believing that
everyone who uses Tk does so via its Python, Ruby or Perl bindings.)

- Marc

*Declarations of "core types" from tclInt.h:*

/*
 * Variables denoting the Tcl object types defined in the core.
 */

MODULE_SCOPE const Tcl_ObjType tclBignumType;
MODULE_SCOPE const Tcl_ObjType tclBooleanType;
MODULE_SCOPE const Tcl_ObjType tclByteCodeType;
MODULE_SCOPE const Tcl_ObjType tclDoubleType;
MODULE_SCOPE const Tcl_ObjType tclExprCodeType;
MODULE_SCOPE const Tcl_ObjType tclIntType;
MODULE_SCOPE const Tcl_ObjType tclIndexType;
MODULE_SCOPE const Tcl_ObjType tclListType;
MODULE_SCOPE const Tcl_ObjType tclDictType;
MODULE_SCOPE const Tcl_ObjType tclProcBodyType;
MODULE_SCOPE const Tcl_ObjType tclStringType;
MODULE_SCOPE const Tcl_ObjType tclEnsembleCmdType;
MODULE_SCOPE const Tcl_ObjType tclRegexpType;
MODULE_SCOPE Tcl_ObjType tclCmdNameType;

*The subset of these types which get registered in TclInitObjSystem:*

    Tcl_RegisterObjType(&tclDoubleType);
    Tcl_RegisterObjType(&tclStringType);
    Tcl_RegisterObjType(&tclListType);
    Tcl_RegisterObjType(&tclDictType);
    Tcl_RegisterObjType(&tclByteCodeType);
    Tcl_RegisterObjType(&tclCmdNameType);
    Tcl_RegisterObjType(&tclRegexpType);
    Tcl_RegisterObjType(&tclProcBodyType);

Re: [TCLCORE] Migration tool for Tcl 9

[Harald O. <har...@el...>]

Hi Ashok,
impressive revolutionary work, I appreciate !
I tried the static checker on my code base and it gives good comments:
- finds variables in namespaces
- finds non-ascii encodings
I did not try the run-time stuff jet.

I would appreciate to have an option to check that check descends to 
subfolders, so a whole tree may be checked at once.

Thanks for all,
Harald

Am 01.07.2024 um 09:48 schrieb apnmbx-public--- via Tcl-Core:
> Sorry, forgot to mention – works best with recent trunk builds.
> 
> *From:*apnmbx-public--- via Tcl-Core <tcl...@li...>
> *Sent:* Monday, July 1, 2024 1:14 PM
> *To:* tcl...@li...
> *Subject:* [TCLCORE] Migration tool for Tcl 9
> 
> All,
> 
> I've been working on a tool to help with migration of Tcl 8 scripts to 
> Tcl 9. It includes a static checker based on Nagelfar as well as a 
> runtime shim that fixes up incompatibilities like i/o encoding errors, 
> tilde expansion, package requirements etc., allowing the scripts to be 
> loaded while emitting warnings.
> 
> Repository: https://github.com/apnadkarni/tcl9-migrate 
> <https://github.com/apnadkarni/tcl9-migrate>
> 
> Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md 
> <https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md>
> 
> Download: https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1 
> <https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1>
> 
> I'm looking for testers and feedback on both the tool as well as 
> documentation (explanation of the warnings). I'd appreciate testing with 
> your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above 
> repository.
> 
> Be warned it has (and always will have) false positives and false 
> negatives but the hope is it will nevertheless be useful in saving time.
> 
> Thanks
> 
> /Ashok

Re: [TCLCORE] Migration tool for Tcl 9

[<apn...@ya...>]

Sorry, forgot to mention - works best with recent trunk builds.

 

From: apnmbx-public--- via Tcl-Core <tcl...@li...> 
Sent: Monday, July 1, 2024 1:14 PM
To: tcl...@li...
Subject: [TCLCORE] Migration tool for Tcl 9

 

All,

 

I've been working on a tool to help with migration of Tcl 8 scripts to Tcl
9. It includes a static checker based on Nagelfar as well as a runtime shim
that fixes up incompatibilities like i/o encoding errors, tilde expansion,
package requirements etc., allowing the scripts to be loaded while emitting
warnings.

 

Repository: https://github.com/apnadkarni/tcl9-migrate

Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md

Download: https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1

 

I'm looking for testers and feedback on both the tool as well as
documentation (explanation of the warnings). I'd appreciate testing with
your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above
repository.

 

Be warned it has (and always will have) false positives and false negatives
but the hope is it will nevertheless be useful in saving time.

 

Thanks

 

/Ashok

[TCLCORE] Migration tool for Tcl 9

[<apn...@ya...>]

All,

 

I've been working on a tool to help with migration of Tcl 8 scripts to Tcl
9. It includes a static checker based on Nagelfar as well as a runtime shim
that fixes up incompatibilities like i/o encoding errors, tilde expansion,
package requirements etc., allowing the scripts to be loaded while emitting
warnings.

 

Repository: https://github.com/apnadkarni/tcl9-migrate

Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md

Download: https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1

 

I'm looking for testers and feedback on both the tool as well as
documentation (explanation of the warnings). I'd appreciate testing with
your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above
repository.

 

Be warned it has (and always will have) false positives and false negatives
but the hope is it will nevertheless be useful in saving time.

 

Thanks

 

/Ashok

Re: [TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

[Kevin W. <kw...@co...>]

<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><img width="1" height="1" src='https://fedbdhd.r.af.d.sendibt2.com/tr/op/c1RZpq_vRb7hxa1u7qfJg2oubNpAfW3Zb2c7oVLoTRJf9Ofges66JnOtNeEmhXpDs56LeTpnhNXrvq-fclMbMQ-nY2jLae92z_lRlJXOWORNZn5Z_jvfzZ1SNvnU51JI89XqmFtedHZrjYrpePkPeO8klSu-Tfr5O8RO1kkfz3J_9i9G3J89M1MTWRrsNyudLu0RDeXAlFUWylg6EeGT18zv8_Ce77S7' /><div dir="ltr"></div><div dir="ltr">I am persuaded by the arguments in favor of this TIP.&nbsp;</div><div dir="ltr"><br></div><div dir="ltr">TIP 699: YES</div><div dir="ltr"><br><blockquote type="cite">On Jun 27, 2024, at 2:41 PM, Marc Culler &lt;cul...@gm...&gt; wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div>TIP #699: YES</div><div><br></div><div>- Marc<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 27, 2024 at 3:59 AM Jan Nijtmans &lt;<a href="mailto:jan...@gm...">jan...@gm...</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Op di 25 jun 2024 om 13:30 schreef Jan Nijtmans:<br>
&gt; This is a CFV warning for TIP #699:<br>
&gt;&nbsp; &nbsp; &nbsp;Eliminate encoding alias "binary"; provide introspection for binary channels<br>
&gt;&nbsp; &nbsp; &nbsp;&lt;<a href="https://fedbdhd.r.af.d.sendibt2.com/tr/cl/QaMzoVQFCDpeBrTgfvH2ugtU2m8NNXTYpL-1NxM7dgzY9O-9af_eOjk0GxlqM7Q1OJ_O34eIMvkQ4ohFqoU9xXFJCWDXlIF3jVbFVPkHQsf0F12T-FuqXvX7cKAzbvlYeVvyMy7J3n-l5imcPWzL1Nv-qBs--Lr2BHjqlLDqvzd4u5q2wFmBN48tEjFaFcbRgPqW0Ohfzkzs3rzWBDaRzbdfrQMyEzu4FV5BKgOT2hhNV-V8TqW3cX3VQNUa_hyjoJLfggR88Zl9heJBzadNj0DjebQ5nn7hp8LHu6pMp0DUEw_E1A_-n2t8Xn7JhGMfPgoj" rel="noreferrer" target="_blank">https://core.tcl-lang.org/tips/doc/trunk/tip/699.md</a>&gt;<br>
<br>
I'm calling the votes on this TIP now. In order to<br>
gain time for the 9.0b3 release, it's only a few days!<br>
<br>
The voting period will finish Monday July 1,<br>
noon UTC, [clock format 1719835200].<br>
<br>
My vote:<br>
<br>
TIP #699: YES<br>
<br>
Regards,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Jan Nijtmans<br>
<br>
<br>
_______________________________________________<br>
Tcl-Core mailing list<br>
<a href="mailto:Tcl...@li..." target="_blank">Tcl...@li...</a><br>
<a href="https://fedbdhd.r.af.d.sendibt2.com/tr/cl/xbaQ6ODXK_x9D5Dt9TBVSd7zlb2bMhBRKcwxhv91xFmrLhQNHv_specm8k2JtrgN0O2h9BhrKZdwIjQXb28VXdUKJ5uwhpR_4sfEPKUkrOmoJ1oRskBcjdetKSVECTmjTPmZJcQiCbr_UrECzw6B2IO631maAV-5F23z2qEHl9LPxBjxpeK2COuj6VQrLrqhSY6YTriqi37vYKce9c0D9GwKiAV_45kB1GAmH3IVKqVav37WRZOBny1_DHfZwewzkW3HUTWT8cgKKXTS1Rzze4HbaQ05vtbAEUl2XATlFXyBHjwHxCEPhBVSY4Ne_kGUKQ" rel="noreferrer" target="_blank">https://lists.sourceforge.net/lists/listinfo/tcl-core</a><br>
</blockquote></div>
<span>_______________________________________________</span><br><span>Tcl-Core mailing list</span><br><span>Tcl...@li...</span><br><span>https://lists.sourceforge.net/lists/listinfo/tcl-core</span><br></div></blockquote></body></html>

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Kevin W. <kw...@co...>]

Would forcing utf-8 on Windows correct this issue?

> On Jun 30, 2024, at 11:39 AM, apnmbx-public--- via Tcl-Core <tcl...@li...> wrote:
> 
> 
> I realize everyone is tired of encoding issues but I feel this is important given the consequences ...
>  
> One of the changes in Tcl 9 is to add a manifest property on Windows which effectively sets [encoding system] to UTF-8 irrespective of the user's code page setting. This manifest only has effect on Windows versions Windows 10 Build 1903 and later. Earlier Windows systems ignore the setting and Tcl 9 will use the user's code page setting on those platforms.
>  
> This change, presumably made as part of TIP 587, has consequences serious enough to be reverted in my opinion.
>  
> Below "file" refers to files containing non-ASCII content and that it is read/written with the (default) system encoding. And "cannot be read" means either an encoding error is thrown or data is garbled depending on the combination of Tcl version, platform version and system encoding (!).
>  
> - A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice versa even on *on the same exact system* on modern (build 1903 or later) Windows systems. So for example, on US English (cp1252) systems "set fd [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in foo.txt that will raise an error in Tcl 9's readFile (or the longer open/read/close equivalent). This is not acceptable in my view.
>  
> - Along the same lines, a file written by a 9.x tclsh on pre-1903 Windows 10 cannot be read by the *same* 9.x tclsh on the *same* system after a *Windows* update. Also not acceptable.
>  
> - A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 9.x tclsh on Windows 10 1903 and vice versa even when the two share a code page. May be less serious but still undesirable.
>  
> - A third-party application (not using tclsh) that uses as its scripting language will not be able to read files written by tclsh and vice versa unless it also includes the utf-8 magic line in its manifest. This latter is unlikely as it would raise the same compatibility problems for them as for tclsh above. Also undesirable.
>  
> In my view, these are all serious compatibility problems.
>  
> The original motivation for the change in [encoding system] presumably came from this page - https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page. However, note the target audience is really applications that use the ANSI Api's and its usefulness is limited for applications (like Tcl) that use the wide character API's. Moreover, as stated on that page, if targeting platforms prior to Build 1903, "you must handle legacy code page detection and conversion as usual".
>  
> Given the above, I would propose reverting [encoding system] on Windows to go back to defaulting to the encoding as per the user's code page setting as in Tcl 8. Note this is not a code change, it is a change to the manifest resource in tclsh (and wish) to remove the ActiveCodePage property. The right place to move forward with UTF-8 as the default encoding belongs to the *applications* written in Tcl, not Tcl itself.
>  
> (Note: I am not proposing changing the encoding used by the source command)
>  
> What would be drawbacks of reverting this change? One could argue that it would reduce compatibility with other platforms like Unix which use UTF-8 as the system encoding. However, I would argue that
>  
> - this is no different from what's the case with 8.x today
>  
> - sharing of files between platforms really has to include explicit configuration of encoding as good practice and most cross-platform applications will do that
>  
> - the issues listed above for a single-platform, on the very same system even, are much more serious
>  
> Reverting would probably need to be TIP'ed but I wanted to get feedback before embarking on that path.
>  
> With regards to Unix, I do not have enough experience with encodings on Unix to know if similar issues would arise there. The [encoding system] in Tcl 9 has changed from iso8859-1 to utf-8. Someone should examine this closer.
>  
> Comments please. I’d prefer to avoid TIP overhead if someone can spot holes in the above.
>  
> /Ashok
>  
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

[Steve L. <st...@di...>]

I’m in two minds on this one.    On the one hand there is clearly an issue that needs addressing, and a major release is the time to do it. What I am not sure is whether this is the right solution and whether there has been enough time to flush out a better solution.  I do note the suggestion (on tkchat) to just change the behaviour of “encoding -binary” to match “translate -binary” and would have liked that to be fleshed out. And I node Ashok’s comment about the need for a binary encoding.  All this being said, I won’t oppose a concrete proposal in the absence of an alternative being presented.

TIP #699: PRESENT

-- Steve
On 27 Jun 2024 at 5:00 PM +0800, Jan Nijtmans <jan...@gm...>, wrote:
> Op di 25 jun 2024 om 13:30 schreef Jan Nijtmans:
> > This is a CFV warning for TIP #699:
> > Eliminate encoding alias "binary"; provide introspection for binary channels
> > <https://core.tcl-lang.org/tips/doc/trunk/tip/699.md>
>
> I'm calling the votes on this TIP now. In order to
> gain time for the 9.0b3 release, it's only a few days!
>
> The voting period will finish Monday July 1,
> noon UTC, [clock format 1719835200].

Re: [TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

[Jan N. <jan...@gm...>]

Op zo 30 jun 2024 om 20:29 schreef Rolf Ade:
> So, https://core.tcl-lang.org/tcl/info/49bdbf8c78767d51 was a bit
> premature, wasn't it?

Yes, it was. But done on purpose, gaining some time.
The only risk doing this is that the commit might need
to be reverted if enough NO votes were getting in.
That's not likely any more.

> TIP #699: PRESENT
>
> While I see the presure to get the next beta and finally the release out
> I feel a bit uncomfortable with the short discussion time (2 days) from
> CFV warning to CFV (and the short voting period) - for a case which
> apparently provokes strong reactions.
>
> I think discussion and listening to others (and not to assume that
> everything with value was already said and weighted righly prior to
> that) is important.

Thanks!    Your input is appreciated!

Hope this helps,
     Jan Nijtmans

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Jan N. <jan...@gm...>]

Op zo 30 jun 2024 om 21:24 schreef Christian Gollwitzer:
> Maybe the "missing piece" is a simple way to get to
> that encoding. In Tcl 9, does [encoding system] always return utf-8?
> Then there is no way to get to that information. Then we need [encoding
> native] or similar to be able to do

It shouldn't be too difficult to get the old code-page from the
registry, it can be found in the following registry key:
     HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage\ACP

Tcl_GetEncodingNameFromEnvironment(&buf) on Windows
just calls the function GetACP()

Implementing a "encoding system -native"  (or whatever this
command should be named) shouldn't be too difficult.

Hope this helps,
     Jan Nijtmans

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Donald P. <d.g...@co...>]

> On Jun 30, 2024, at 3:21 PM, Harald Oehlmann <har...@el...> wrote:
> What I think is the most criticel point is, that there is no way in TCL9 to find out, what the encoding was in TCL 8.6 on the same computer.

As you look into these matters, be sure to tend to

Tcl_GetEncodingNameFromEnvironment(&buf)

And anything similar to make a consistent whole.

https://www.tcl-lang.org/man/tcl8.6/TclLib/Encoding.htm

DGP

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Christian G. <aur...@gm...>]

Am 30.06.24 um 20:46 schrieb Rolf Ade:
> That said I would really like to stick with system encoding utf-8 for
> Tcl 9 on very platform.
>
>> The lesson: If you want to make your program portable between windows and UNIX
>> (and MacOS), and with both Tcl 8. and Tcl 9, always explicitly specify
>> the encoding.
>
> That sounds sensible to me.

I think we have the chance to change it now - after Tcl 9 it would
affect even more people and never get fixed. For other things it might
be debatable, but for this I'm pretty sure the only reasonable
assumption is default to utf-8 and any other choice would be one of
those dark corners & pitfalls.

But I can see Ashok's point - how to write / read a file in the Windows
default encoding? Maybe the "missing piece" is a simple way to get to
that encoding. In Tcl 9, does [encoding system] always return utf-8?
Then there is no way to get to that information. Then we need [encoding
native] or similar to be able to do

set fd [open something]
fconfigure $fd -encoding [encoding native]

           Christian

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Harald O. <har...@el...>]

Hi Ashok,
thank you for the analysis.
What I think is the most criticel point is, that there is no way in TCL9 
to find out, what the encoding was in TCL 8.6 on the same computer.

On my computer:
TCL 8.6.14: encoding system: cp 1252
TCL 9.0b2: encoding system: utf-8

But on TCL 9.0b2, there is no way to find out, what the TCL 8.6 encoding 
on the same computer is (e.g. find cp1252). So, it is even impossible to 
do an:
set f [open $p r]
fconfigure $f -encoding cp1252
as it is unknown, that cp1252 is the TCL8.6 system encoding.

The other consequence to my knowledge are environment variables and 
command line arguments, also the exec command.
All this gets IMHO incompatible and you can not do anything, as the 
encoding (of TCL 8.6) is not available on TCL 9.0.

The system encoding has on Windows much less consequence as on Linux, as 
most API's to the system use CESU-16 anyway.
But the problems are there.

I don't know if reverting is the right way to go. But it would be great 
to have the 8.6 encoding available in 9.0.

And of cause, "source" should use utf-8 anyway. And any open/socket 
should use an explicit encoding.
Having the system encoding set to UTF-8 would cause many lazy* programs 
do the right thing out of the box...
* "lazy" means, not setting the encoding explicitly.

A 2nd way would be, to assume utf-8 as default, if no encoding is 
specified...

Thank you for this critical discussion, great !
Harald


Am 30.06.2024 um 17:39 schrieb apnmbx-public--- via Tcl-Core:
> I realize everyone is tired of encoding issues but I feel this is 
> important given the consequences ...
> 
> One of the changes in Tcl 9 is to add a manifest property on Windows 
> which effectively sets [encoding system] to UTF-8 irrespective of the 
> user's code page setting. This manifest only has effect on Windows 
> versions Windows 10 Build 1903 and later. Earlier Windows systems ignore 
> the setting and Tcl 9 will use the user's code page setting on those 
> platforms.
> 
> This change, presumably made as part of TIP 587, has consequences 
> serious enough to be reverted in my opinion.
> 
> Below "file" refers to files containing non-ASCII content and that it is 
> read/written with the (default) system encoding. And "cannot be read" 
> means either an encoding error is thrown or data is garbled depending on 
> the combination of Tcl version, platform version and system encoding (!).
> 
> - A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice 
> versa even on *on the same exact system* on modern (build 1903 or later) 
> Windows systems. So for example, on US English (cp1252) systems "set fd 
> [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in 
> foo.txt that will raise an error in Tcl 9's readFile (or the longer 
> open/read/close equivalent). This is not acceptable in my view.
> 
> - Along the same lines, a file written by a 9.x tclsh on pre-1903 
> Windows 10 cannot be read by the **same** 9.x tclsh on the **same** 
> system after a **Windows** update. Also not acceptable.
> 
> - A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 
> 9.x tclsh on Windows 10 1903 and vice versa even when the two share a 
> code page. May be less serious but still undesirable.
> 
> - A third-party application (not using tclsh) that uses as its scripting 
> language will not be able to read files written by tclsh and vice versa 
> unless it also includes the utf-8 magic line in its manifest. This 
> latter is unlikely as it would raise the same compatibility problems for 
> them as for tclsh above. Also undesirable.
> 
> In my view, these are all serious compatibility problems.
> 
> The original motivation for the change in [encoding system] presumably 
> came from this page - 
> https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page <https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page>. However, note the target audience is really applications that use the ANSI Api's and its usefulness is limited for applications (like Tcl) that use the wide character API's. Moreover, as stated on that page, if targeting platforms prior to Build 1903, "you must handle legacy code page detection and conversion as usual".
> 
> *Given the above, I would propose reverting [encoding system] on Windows 
> to go back to defaulting to the encoding as per the user's code page 
> setting as in Tcl 8.* Note this is not a code change, it is a change to 
> the manifest resource in tclsh (and wish) to remove the ActiveCodePage 
> property. The right place to move forward with UTF-8 as the default 
> encoding belongs to the *applications* written in Tcl, not Tcl itself.
> 
> (Note: I am not proposing changing the encoding used by the source command)
> 
> What would be drawbacks of reverting this change? One could argue that 
> it would reduce compatibility with other platforms like Unix which use 
> UTF-8 as the system encoding. However, I would argue that
> 
> - this is no different from what's the case with 8.x today
> 
> - sharing of files between platforms really has to include explicit 
> configuration of encoding as good practice and most cross-platform 
> applications will do that
> 
> - the issues listed above for a single-platform, on the very same system 
> even, are much more serious
> 
> Reverting would probably need to be TIP'ed but I wanted to get feedback 
> before embarking on that path.
> 
> With regards to Unix, I do not have enough experience with encodings on 
> Unix to know if similar issues would arise there. The [encoding system] 
> in Tcl 9 has changed from iso8859-1 to utf-8. Someone should examine 
> this closer.
> 
> Comments please. I’d prefer to avoid TIP overhead if someone can spot 
> holes in the above.
> 
> /Ashok

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Rolf A. <tcl...@po...>]

Jan Nijtmans <jan...@pu...>
writes:

> Op zo 30 jun 2024 om 17:39 schreef Ashok:
>> - A file written by a 8.x tclsh cannot be read by a 9.x tclsh and
>> vice versa even on *on the same exact system* on modern (build 1903 or
>> later) Windows systems. So for example, on US English (cp1252) systems
>> "set fd [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will
>> result in foo.txt that will raise an error in Tcl 9's readFile (or the
>> longer open/read/close equivalent). This is not acceptable in my view.
>
> See: <https://learn.microsoft.com/en-us/lifecycle/announcements/windows-10-1903-end-of-servicing>
> Windows 10 build 1903 already was end-of-service since december 8, 2020.

This is not a convincing argument in my book. The vendor has his own
reasons for such decisions (as declaring end of service), we should have
our own. One could be: how many of our users are affected by this and
how hard are they hit. Though, I have only a gut feeling about that and
no hard numbers.

> We should also consider WSL (Windows System for Linux). Currently, a
> file written
> by a 8.x tclsh cannot be read by a 8.x tclsh running on WSL. To me
> that's a far more
> problem than the problem you are describing. If we ever want to make an
> incompatible change, now's the time.

This gut feeling from above say that there are lesser WSL user than User
of an older Windows. But I surely may be way wrong with this guess.

That said I would really like to stick with system encoding utf-8 for
Tcl 9 on very platform. 

> The lesson: If you want to make your program portable between windows and UNIX
> (and MacOS), and with both Tcl 8. and Tcl 9, always explicitly specify
> the encoding.

That sounds sensible to me.

rolf

Re: [TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

[Rolf A. <tcl...@po...>]

Jan Nijtmans writes:
> Op di 25 jun 2024 om 13:30 schreef Jan Nijtmans:
>> This is a CFV warning for TIP #699:
>>     Eliminate encoding alias "binary"; provide introspection for binary channels
>>     <https://core.tcl-lang.org/tips/doc/trunk/tip/699.md>
>
> I'm calling the votes on this TIP now. In order to
> gain time for the 9.0b3 release, it's only a few days!
>
> The voting period will finish Monday July 1,
> noon UTC, [clock format 1719835200].

So, https://core.tcl-lang.org/tcl/info/49bdbf8c78767d51 was a bit
premature, wasn't it?

My vote:

TIP #699: PRESENT

While I see the presure to get the next beta and finally the release out
I feel a bit uncomfortable with the short discussion time (2 days) from
CFV warning to CFV (and the short voting period) - for a case which
apparently provokes strong reactions.

I think discussion and listening to others (and not to assume that
everything with value was already said and weighted righly prior to
that) is important.

rolf


>
> My vote:
>
> TIP #699: YES
>
> Regards,
>          Jan Nijtmans

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Karl K. <ka...@ac...>]

Regarding Linux: iso8859 has been obsolete for a very long time. Red Hat has had utf-8 locales as default for 22 years (!). UTF-8 has been around since 1992. iso-8859-1 doesn’t work for most people and TCL has supported Unicode since 8.1 . Linux also has never suffered from this Windows-nonsense where utf-8 was not a regular codepage and translations required extra work.


  *   Karl

From: apnmbx-public--- via Tcl-Core <tcl...@li...>
Sent: Sunday, June 30, 2024 8:39 AM
To: tcl...@li...
Subject: [TCLCORE] Propose reverting [encoding system] on Windows

I realize everyone is tired of encoding issues but I feel this is important given the consequences ... One of the changes in Tcl 9 is to add a manifest property on Windows which effectively sets [encoding system] to UTF-8 irrespective of the user's code page setting. This manifes ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

<https://usb.report.cybergraph.mimecast.com/alert-details/?dep=YSLa8ErmT3%2FMUy7IPAuYcA%3D%3DadbPGZOk65S%2BQ0aTkBtU1CNqqgjA94deaoswcUgHiRRGMt1J3Ln6fFKLI%2FW1%2FzeU57SoG52Tc87O0o15WbyzHY4Z0IBlmnK85RFHLCGBcORlJcFKCu0t%2Bap5Raf2GxYY9jQinfYol8JHI5laRgL5jgKpQ2KHVZgS%2BaZJZtqlaf2ncENgUs3VDRPqFeT40Txf33kA0Kujdt9EWf1ny1ELqUgjHZBJWtvlloGjUk8TmQqRlfMp0gSXD7INjenP0wDZM4wB8BKr58tPPVqRmza3SuAcs6SqHZMUWhRTl5sLs7RQFOD4CrbsIni4t2xSTBnKwQjtzKBFyh7kfc9Xrgb5Rpbg%2BROYaYBoL9UIRDVs4PxCcmGiOawjFYdn9tMCwjNKGiUoHaij1v0J3vIhTHigFzCe%2Fu5lR3i3ehbh4x4Fpg92%2BCEGD0jUZkoomC64dX2U7dx4Qn22u4YqH%2F%2FbgOUBEmeWv4sgUbyzW1m2A%2BM1uSXDClV9N26X%2Fl4Grk5zY8%2Fm6SI5XM652Gr9Vdmjlgdqjz%2F5PRLBlhdptYcrMsaewAQZ0zUJE%2BaX%2Blgh2ljDJjvFTVhSiOKCOGUq9O8MxavMAqi52VDGd3BVA0gDqsmbmaWRnIJS5vTElLMTwu4DGiK2HzVOmM3xFWMUhukgGm1B5sQHxgo9xFfjoMRnE5iaz4c0df9kDnW2f%2BGpgcxJsAXh4NtHYGqep44dC7NgHX9G1vxIWgXK7PkRDOs8QvInNZPz1T3DXrgvdl%2FGxb0Jkx8p4ceWL6LTcHEZkcz%2Bus9h7Q%3D%3D>
I realize everyone is tired of encoding issues but I feel this is important given the consequences ...

One of the changes in Tcl 9 is to add a manifest property on Windows which effectively sets [encoding system] to UTF-8 irrespective of the user's code page setting. This manifest only has effect on Windows versions Windows 10 Build 1903 and later. Earlier Windows systems ignore the setting and Tcl 9 will use the user's code page setting on those platforms.

This change, presumably made as part of TIP 587, has consequences serious enough to be reverted in my opinion.

Below "file" refers to files containing non-ASCII content and that it is read/written with the (default) system encoding. And "cannot be read" means either an encoding error is thrown or data is garbled depending on the combination of Tcl version, platform version and system encoding (!).

- A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice versa even on *on the same exact system* on modern (build 1903 or later) Windows systems. So for example, on US English (cp1252) systems "set fd [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in foo.txt that will raise an error in Tcl 9's readFile (or the longer open/read/close equivalent). This is not acceptable in my view.

- Along the same lines, a file written by a 9.x tclsh on pre-1903 Windows 10 cannot be read by the *same* 9.x tclsh on the *same* system after a *Windows* update. Also not acceptable.

- A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 9.x tclsh on Windows 10 1903 and vice versa even when the two share a code page. May be less serious but still undesirable.

- A third-party application (not using tclsh) that uses as its scripting language will not be able to read files written by tclsh and vice versa unless it also includes the utf-8 magic line in its manifest. This latter is unlikely as it would raise the same compatibility problems for them as for tclsh above. Also undesirable.

In my view, these are all serious compatibility problems.

The original motivation for the change in [encoding system] presumably came from this page - https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page<https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page>. However, note the target audience is really applications that use the ANSI Api's and its usefulness is limited for applications (like Tcl) that use the wide character API's. Moreover, as stated on that page, if targeting platforms prior to Build 1903, "you must handle legacy code page detection and conversion as usual".

Given the above, I would propose reverting [encoding system] on Windows to go back to defaulting to the encoding as per the user's code page setting as in Tcl 8. Note this is not a code change, it is a change to the manifest resource in tclsh (and wish) to remove the ActiveCodePage property. The right place to move forward with UTF-8 as the default encoding belongs to the *applications* written in Tcl, not Tcl itself.

(Note: I am not proposing changing the encoding used by the source command)

What would be drawbacks of reverting this change? One could argue that it would reduce compatibility with other platforms like Unix which use UTF-8 as the system encoding. However, I would argue that

- this is no different from what's the case with 8.x today

- sharing of files between platforms really has to include explicit configuration of encoding as good practice and most cross-platform applications will do that

- the issues listed above for a single-platform, on the very same system even, are much more serious

Reverting would probably need to be TIP'ed but I wanted to get feedback before embarking on that path.

With regards to Unix, I do not have enough experience with encodings on Unix to know if similar issues would arise there. The [encoding system] in Tcl 9 has changed from iso8859-1 to utf-8. Someone should examine this closer.

Comments please. I’d prefer to avoid TIP overhead if someone can spot holes in the above.

/Ashok

Re: [TCLCORE] Propose reverting [encoding system] on Windows

[Jan N. <jan...@gm...>]

Op zo 30 jun 2024 om 17:39 schreef Ashok:
> - A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice versa even on *on the same exact system* on modern (build 1903 or later) Windows systems. So for example, on US English (cp1252) systems "set fd [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in foo.txt that will raise an error in Tcl 9's readFile (or the longer open/read/close equivalent). This is not acceptable in my view.

See: <https://learn.microsoft.com/en-us/lifecycle/announcements/windows-10-1903-end-of-servicing>
Windows 10 build 1903 already was end-of-service since december 8, 2020.

We should also consider WSL (Windows System for Linux). Currently, a
file written
by a 8.x tclsh cannot be read by a 8.x tclsh running on WSL. To me
that's a far more
problem than the problem you are describing. If we ever want to make an
incompatible change, now's the time.

The lesson: If you want to make your program portable between windows and UNIX
(and MacOS), and with both Tcl 8. and Tcl 9, always explicitly specify
the encoding.
All Tcl packages already were modified to add "-encoding utf-8", so
doing just that.
Other packages should do the same. Then - gradually - the world
switches to utf-8
and for systems new enough (any currently supported Windows 10+/UNIX/MacOS).

Hope this helps,
    Jan Nijtmans

[TCLCORE] Propose reverting [encoding system] on Windows

[<apn...@ya...>]

I realize everyone is tired of encoding issues but I feel this is important
given the consequences ...

 

One of the changes in Tcl 9 is to add a manifest property on Windows which
effectively sets [encoding system] to UTF-8 irrespective of the user's code
page setting. This manifest only has effect on Windows versions Windows 10
Build 1903 and later. Earlier Windows systems ignore the setting and Tcl 9
will use the user's code page setting on those platforms.

 

This change, presumably made as part of TIP 587, has consequences serious
enough to be reverted in my opinion.

 

Below "file" refers to files containing non-ASCII content and that it is
read/written with the (default) system encoding. And "cannot be read" means
either an encoding error is thrown or data is garbled depending on the
combination of Tcl version, platform version and system encoding (!).

 

- A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice versa
even on *on the same exact system* on modern (build 1903 or later) Windows
systems. So for example, on US English (cp1252) systems "set fd [open
foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in foo.txt that
will raise an error in Tcl 9's readFile (or the longer open/read/close
equivalent). This is not acceptable in my view.

 

- Along the same lines, a file written by a 9.x tclsh on pre-1903 Windows 10
cannot be read by the *same* 9.x tclsh on the *same* system after a
*Windows* update. Also not acceptable.

 

- A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 9.x
tclsh on Windows 10 1903 and vice versa even when the two share a code page.
May be less serious but still undesirable.

 

- A third-party application (not using tclsh) that uses as its scripting
language will not be able to read files written by tclsh and vice versa
unless it also includes the utf-8 magic line in its manifest. This latter is
unlikely as it would raise the same compatibility problems for them as for
tclsh above. Also undesirable.

 

In my view, these are all serious compatibility problems.

 

The original motivation for the change in [encoding system] presumably came
from this page -
https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-c
ode-page. However, note the target audience is really applications that use
the ANSI Api's and its usefulness is limited for applications (like Tcl)
that use the wide character API's. Moreover, as stated on that page, if
targeting platforms prior to Build 1903, "you must handle legacy code page
detection and conversion as usual".

 

Given the above, I would propose reverting [encoding system] on Windows to
go back to defaulting to the encoding as per the user's code page setting as
in Tcl 8. Note this is not a code change, it is a change to the manifest
resource in tclsh (and wish) to remove the ActiveCodePage property. The
right place to move forward with UTF-8 as the default encoding belongs to
the *applications* written in Tcl, not Tcl itself.

 

(Note: I am not proposing changing the encoding used by the source command)

 

What would be drawbacks of reverting this change? One could argue that it
would reduce compatibility with other platforms like Unix which use UTF-8 as
the system encoding. However, I would argue that

 

- this is no different from what's the case with 8.x today

 

- sharing of files between platforms really has to include explicit
configuration of encoding as good practice and most cross-platform
applications will do that

 

- the issues listed above for a single-platform, on the very same system
even, are much more serious

 

Reverting would probably need to be TIP'ed but I wanted to get feedback
before embarking on that path.

 

With regards to Unix, I do not have enough experience with encodings on Unix
to know if similar issues would arise there. The [encoding system] in Tcl 9
has changed from iso8859-1 to utf-8. Someone should examine this closer.

 

Comments please. I'd prefer to avoid TIP overhead if someone can spot holes
in the above.

 

/Ashok

 

Re: [TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

[Brian G. <bri...@ea...>]

TIP #699: PRESENT

I can’t vote yes, and won’t vote no. I feel that this TIP exchanges one confusing definition with another confusing definition.  It’s only marginally better and not what it (c|sh)ould be. 

-Brian

> On Jun 27, 2024, at 02:59, Jan Nijtmans <jan...@gm...> wrote:
> 
> Op di 25 jun 2024 om 13:30 schreef Jan Nijtmans:
>> This is a CFV warning for TIP #699:
>>    Eliminate encoding alias "binary"; provide introspection for binary channels
>>    <https://core.tcl-lang.org/tips/doc/trunk/tip/699.md>
> 
> I'm calling the votes on this TIP now. In order to
> gain time for the 9.0b3 release, it's only a few days!
> 
> The voting period will finish Monday July 1,
> noon UTC, [clock format 1719835200].
> 
> My vote:
> 
> TIP #699: YES
> 
> Regards,
>         Jan Nijtmans
> 
> 
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
> 

Re: [TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

[<apn...@ya...>]

TIP 699: YES

There is a tiny inconsequential oversight in the TIP

 "Remark: the "-translation binary" will set the "-encoding" to "iso8859-1" and "-eofchar" to "lf""

in that -eofchar is set to {}, not "lf" and -translation is set to "lf".

I vacillated between PRESENT and YES, the former because of the same concerns voiced earlier by Brian and Sergey. As they commented, "binary" I/O where the low order 8 bits of a Unicode code point is passed on as is, conceptually differs from an encoding transform even when that transform currently does that same exact operation. In particular, as I think Brian was pointing out, encodings standards can change, as iso8859-1 itself has as per my reading of its history. I would have preferred an explicit "binary" encoding to be added to Tcl but that ship I think has long sailed.

At the end of the day though, I felt the TIP presents the best possible path forward in the present circumstances.

/Ashok


-----Original Message-----
From: Harald Oehlmann <har...@el...> 
Sent: Thursday, June 27, 2024 11:41 PM
To: tcl...@li...
Subject: Re: [TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

Vote: TIP 699: yes

I can understand, that it feels a) "surprisingly unclear" and b) hard 
for backward compatibility.

But TCL 9.0 is made to clarify things. This is clarified. "Binary 
encoding" = "iso8859-1". We should not make compromises here.
And for backward compatibility, there is "open $f b" or "fconfigure 
-translation binary". Both were always the better choice. Only for the 
rare care, where the translation or eof-char should not be in the binary 
position, there is a change (e.g. specify "-encoding iso8859-1" instead 
"-encoding binary".

Thank you for the work,
Harald


Am 27.06.2024 um 10:58 schrieb Jan Nijtmans:
> Op di 25 jun 2024 om 13:30 schreef Jan Nijtmans:
>> This is a CFV warning for TIP #699:
>>      Eliminate encoding alias "binary"; provide introspection for binary channels
>>      <https://core.tcl-lang.org/tips/doc/trunk/tip/699.md>
> 
> I'm calling the votes on this TIP now. In order to
> gain time for the 9.0b3 release, it's only a few days!
> 
> The voting period will finish Monday July 1,
> noon UTC, [clock format 1719835200].
> 
> My vote:
> 
> TIP #699: YES
> 
> Regards,
>           Jan Nijtmans
> 
> 
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

-- 
ELMICRON Dr. Harald Oehlmann GmbH
Koesener Str. 85
06618 NAUMBURG - Germany
Phone: +49 3445 781120
Direct: +49 3445 781127
www.Elmicron.de
German legal references:
Geschaeftsfuehrer: Dr. Harald Oehlmann
UST Nr. / VAT ID No.: DE206105272
HRB 212803 Stendal

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Colin M. <col...@ya...>]

Hello all, here is a follow-up on this issue.

I consulted Upstash.com support, I will attach a transcript of our 
messages.  On 20th June they added a reply which I did not see until now 
because it did not get emailed to me.  Part of this says:

Normally server doesn't buffer response unless a pipeline is detected. A 
pipeline is when a client sends multiple consequent requests without 
waiting response of earlier requests.

This makes me suspect that the Redis server wrongly gets into its 
pipeline mode (https://redis.io/docs/latest/develop/use/pipelining/) and 
so buffers its replies.  However I'm not going to spend more time 
investigating this myself because I'm now using redis-cloud.com which 
does not require TLS and works fine with Retcl.

Colin.

On 05/06/2024 14:22, Pietro Cerutti wrote:
> Hi,
>
> I and Colin have been investigating an issue when using the TLS 
> extension (https://core.tcl-lang.org/tcltls) to interact with a Redis 
> server using retcl (https://code.ptrcrt.ch/retcl/).
>
> I have a small reproducer here, which I have trimmed down to a 
> minimum.  It doesn't include any retcl code, just TLS sockets:
> https://gist.github.com/gahr/a9258f0f9cdd20ec0479a409e0db49fd
>
> The protocol is line-oriented, with \r\n as terminators.
>
> The setup is as follows:
>
> 1. we connect to a Redis instance using [tls::socket]
> 2. we [chan configure $sock -blocking 0 -translation binary]
> 3. we send a command string, and we [chan flush $sock]
> 4. we get a read event on the $sock
> 5. we [read $sock] in the event handler and expect to get a response
>
> Point 5. works fine against a local Redis instance w/o TLS.
>
> When we turn TLS on, we see weird things:
>
> 1. we get a number of read events where [read $sock] returns ""
> 2. sometimes we get meaningful read events afterwards, where a [read 
> $sock] returns the expected response
> 3. sometimes we don't get any meaningful read events after the empty ones
>
> Under the gist, I have included two comments with sample runs against 
> my local Redis server and against an intance on upstash.com. You can 
> see the randomness of the empty / meaningful responses.
>
> I can provide the output when tcltls is compiled with 
> -DTCLEXT_TCLTLS_DEBUG, if anyone wants to take a look.
>
> Any pointers would be appreciated.
>

Re: [TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

[Marc C. <cul...@gm...>]

TIP #699: YES

- Marc

On Thu, Jun 27, 2024 at 3:59 AM Jan Nijtmans <jan...@gm...> wrote:

> Op di 25 jun 2024 om 13:30 schreef Jan Nijtmans:
> > This is a CFV warning for TIP #699:
> >     Eliminate encoding alias "binary"; provide introspection for binary
> channels
> >     <https://core.tcl-lang.org/tips/doc/trunk/tip/699.md>
>
> I'm calling the votes on this TIP now. In order to
> gain time for the 9.0b3 release, it's only a few days!
>
> The voting period will finish Monday July 1,
> noon UTC, [clock format 1719835200].
>
> My vote:
>
> TIP #699: YES
>
> Regards,
>          Jan Nijtmans
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

[Harald O. <har...@el...>]

Vote: TIP 699: yes

I can understand, that it feels a) "surprisingly unclear" and b) hard 
for backward compatibility.

But TCL 9.0 is made to clarify things. This is clarified. "Binary 
encoding" = "iso8859-1". We should not make compromises here.
And for backward compatibility, there is "open $f b" or "fconfigure 
-translation binary". Both were always the better choice. Only for the 
rare care, where the translation or eof-char should not be in the binary 
position, there is a change (e.g. specify "-encoding iso8859-1" instead 
"-encoding binary".

Thank you for the work,
Harald


Am 27.06.2024 um 10:58 schrieb Jan Nijtmans:
> Op di 25 jun 2024 om 13:30 schreef Jan Nijtmans:
>> This is a CFV warning for TIP #699:
>>      Eliminate encoding alias "binary"; provide introspection for binary channels
>>      <https://core.tcl-lang.org/tips/doc/trunk/tip/699.md>
> 
> I'm calling the votes on this TIP now. In order to
> gain time for the 9.0b3 release, it's only a few days!
> 
> The voting period will finish Monday July 1,
> noon UTC, [clock format 1719835200].
> 
> My vote:
> 
> TIP #699: YES
> 
> Regards,
>           Jan Nijtmans
> 
> 
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

-- 
ELMICRON Dr. Harald Oehlmann GmbH
Koesener Str. 85
06618 NAUMBURG - Germany
Phone: +49 3445 781120
Direct: +49 3445 781127
www.Elmicron.de
German legal references:
Geschaeftsfuehrer: Dr. Harald Oehlmann
UST Nr. / VAT ID No.: DE206105272
HRB 212803 Stendal

[TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

[Jan N. <jan...@gm...>]

Op di 25 jun 2024 om 13:30 schreef Jan Nijtmans:
> This is a CFV warning for TIP #699:
>     Eliminate encoding alias "binary"; provide introspection for binary channels
>     <https://core.tcl-lang.org/tips/doc/trunk/tip/699.md>

I'm calling the votes on this TIP now. In order to
gain time for the 9.0b3 release, it's only a few days!

The voting period will finish Monday July 1,
noon UTC, [clock format 1719835200].

My vote:

TIP #699: YES

Regards,
         Jan Nijtmans