tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] [MACTCL] Major update to the macOS port of Tk

[Uwe K. <Uwe...@gm...>]

Hello Marc,
why not have one CALayer per widget?

That way one could easily implement OpenGL and Metal widgets (CAOpenGLLayer, and CAMetalLayer)

In the one CALayer-per-toplevel scenario an OpenGL or Metal widget would need to first render to texture, then copy the texture to a rectangle in the toplevel's CGImage, and finally the toplevel's CALayer would have the system composit the CGImage onscreen. That seems wasteful in a situation where speed matters most...

I work with a UIKit implementation for macOS called Chameleon which uses CALayer to implement UIView on AppKit (this was long before Apple started Catalyst). It works great and UIView can be thought of as the equivalent to a tk widget. That's why I believe one CALayer per tk widget is the way to go...

Also, this would open the route to tcl/tk on iPadOS and visionOS (if the new tk drops HIToolbox).


best regards,
Uwe


> On 3. Jul 2024, at 18:12, Marc Culler <mar...@gm...> wrote:
> 
> 
> Tk Timeline watchers may have noticed a long chain of commits to the cgimage_with_crossing branch over the last month.  This announcement is to explain what has been going on there.  In brief, that branch contains a major change to how drawing works in the macOS port of Tk.  This change makes drawing in the macOS port work in a way which is much closer to how it works in the other platforms, and much closer to what the generic Tk code expects. The result is better performance and increased stability.
> 
> The work is based on an idea of Christopher Chavez's.  He realized that Apple's Appkit design allowed for an alternative drawing strategy, and wrote a proof-of-concept implementation.  I had created a branch containing his implementation a couple of years ago.  Recently I merged that branch with Tk 9 and then spent a few weeks ironing out wrinkles.  Nicolas Bats, Torsten Berg, Steve Landers, and Kevin Walzer kindly offered to help me by testing the new implementation on their apps, some of which provide very challenging environments for Tk.  The result of this work is in the cgimage-with-crossing branch.  (The branch also includes a fix for ticket [22349fc78a] which is about <Enter> and <Leave> events - hence the reference to crossing in the branch tag.)
> 
> The five of us are confident that this is a significant improvement. Some highlights are:
>     * Better performance - CPU-intensive apps use about 20% fewer CPU cycles
>     * Better conformance - Most runs of the regression test suite on macOS 14 show no failures.  This is not close to being the case for the trunk branch.
>     * Fewer graphical artifacts (and work continues on those that remain.)
>     * Fewer crashes (none that we know of at the moment.)
> 
> Given that the branch has been tested on large and complex apps - all of which are working well - this is a major step forwards and we’d like to see it tested more widely.  To that end, although we are late in the release cycle we are comfortable recommending that this be merged into trunk and made part of beta3.  That’s the only way we can guarantee wider testing. Of course, if any issues crop up they will be addressed in a timely manner.  If you are a macOS tkchat user please download and try https://www.codebykevin.com/TkChat_Setup_1.5.2.dmg   If you want to try the new macOS Tk with your own code and are comfortable building it yourself please checkout and build the Tk cgimage_with_crossing branch
> 
> The rest of this message gives some more details about the changes.
> 
> The main difference is that the new drawing strategy does away with the asynchronous drawing via the drawRect method.  A macOS app which uses drawRect is supposed to do all of its drawing within the drawRect method, which can only be called by the NSApplication object.  The app can request that drawRect be called by setting the viewNeedsDisplay flag, but can not call drawRect.  The original design of the macOS Aqua port only partially complied with this strategy.  In fact it was possible to obtain a valid graphics context for drawing to the backing layer of a window's contentView even outside of the drawRect method.  The original port would do this, e.g. in a widget display proc which is being run as an idle task.  But it would also draw in the drawRect method by first generating expose events for all widgets that need updates, then processing those expose events to create idle tasks, then processing all of those idle tasks immediately.
> 
> Obviously this scheme was not what Tk expected.  Nor was it what Apple expected.  And Apple put an abrupqt end to half of it with the release of macOS Mojave, which made it impossible to obtain a valid graphics context outside of the drawRect method. When Tk was first built on Mojave, it produced nothing but totally black windows.  Eventually this was worked around by arranging that any drawing operation which failed due to an invalid graphics context would be rescheduled and also to modify drawRect to make it (attempt to) run all of those rescheduled idle tasks.
> 
> With the new macOS code it is once again true that Tk always has a valid graphics context available and hence can draw at any time.  It draws into a CGBitmapImageContext which serves as the backing store for a Tk Window.  Instead of calling drawRect, it is configured with Apple's other option which is to call updateLayer instead.  In the new setup updateLayer installs the CGImage as the CALayer of the ContentView of the NSWindow which hosts the Tk toplevel, causing the contents of the image to appear in the window.  It also creates a new CGImage containing a copy of the window in which subsequent drawing operations take place.  This allows drawing operations to take place at the expected time and in the expected order.  That makes for better stability and less unpredictability.  In addition, it removes the overhead that arises from attempting a drawing operation, having it fail due to an invalid graphics context, rescheduling and rerunning the operation.  This accounts for the reduced CPU usage.
> 
> - Marc
> _______________________________________________
> Tcl-mac mailing list
> tc...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-mac

[TCLCORE] Tcl_ObjType public surface

[Poor Y. <org...@po...>]

Speaking of the long march to correct the mistakes of the past, it would
also be good not to repeat the mistakes of the past.  One of the big
mistakes of the past was exposing the details of the Tcl_Obj structure
in tcl.h.  Don Porter of others have worked for years to move in the
direction of correcting that mistake.  Now, Tcl is posed to repeat that
mistake in the release of version 9 by expanding the public surface of 
the
Tcl_ObjType structure.  It would be wise to correct that before the
release of Tcl 9, and instead provide a functional interface for
registering the implementations of functions that are now available for
operating on a Tcl_Obj having a certain type.

-- 
Yorick

Re: [TCLCORE] CFV warning: TIP #699: Eliminate encoding alias "binary"...

[Poor Y. <org...@po...>]

On 2024-06-26 00:07, Brian Griffin wrote:
[SNIP]
> 
> When dealing with strings, in a language where everything is a string,
> this is perfectly reasonable.
> However, not all data are strings.  When there is a need to process
> and pass non-string data, there needs to be a way to manage this in
> Tcl. The existing (8.6) "binary" implementation more-or-less works
> when needed. Binary data is not strings, so we have to stop discussing
> it within the context of strings.

At the implementation level lists are not strings, dictionaries are not
strings and other values are not strings.  Your argument that binary
data are not strings is true at the implementation level, but it is
false at the script level.  The fundamental abstraction of Tcl is that
everything is a string, and that means that binary data are strings.
You spoke earlier about leaking concepts into the script level that
should remain implementation details.  Many Tcl programmers have the
false impression "binary" is is some sort of shady hack that "allows
binary data to pass through the script level unscathed."  I had that
impression for a few years when I picked up Tcl.  The truth is that it's
not a shady hack, but an elegant design:  At the script level the
difference between "binary" and "string" data melts away because the
string abstraction naturally accommodates binary data.

You argued that removing "binary" as an encoding presents a leak of an
implementation detail to the script level.  That's backwards.  The fake
"binary" encoding is the leak of the implementation detail that
sometimes a string can be represented as an array of single-byte octets.
Even if the same string is represented as an array of 4-byte code
points, it's still a sequence of bytes for all command procedures that
interpret their arguments as sequences of bytes, i.e. as "binary"
values.  The internal type of a "binary" string value can be anything at
all, and command procedures that interpret their data as bytes will
produce exactly the same results from them.  There is an appropriate
place to introduce interpretations of strings, and individuals
procedures like [list] and [dict] are the right place.  A fake encoding
that implies a value type other than string is the wrong place.

At the script level we should absolutely be discussing binary data in
the context of strings, because strings are all we have, and when we
consistently drive that point home throughout the documentation,
programmers can learn to truly understand Tcl.  Understanding Tcl is a
surprisingly long journey given the simplicity of the language.  One
thing that makes the journey longer is bad practice that panders to
misunderstanding in the name of "being helpful".  Tcl has a long track
record of being unhelpfully "helpful", and it's been a long march to
correct the mistakes of the past.  It's worth it to correct them.

-- 
Yorick

Re: [TCLCORE] [MACTCL] Major update to the macOS port of Tk

[Brian G. <bri...@ea...>]

I did try this a few weeks ago, on a couple simple applications I have. I was shocked at the startup time! Very impressive improvement, even obvious on simple (smallish) code.
I also noticed that a bug I had went away. The code has a binding for the selection event in a ttk::treeview that performed some value updates in other widgets. When using the up/down arrow keys, the first key press works fine, but subsequent key presses wouldn’t. And not always. Sometimes it worked for a couple presses before stopping.
Now it works flawlessly.

Good job!!
-Brian
(from mobile device)

On Jul 3, 2024, at 09:12, Marc Culler <mar...@gm...> wrote:


Tk Timeline watchers may have noticed a long chain of commits to the cgimage_with_crossing branch over the last month.  This announcement is to explain what has been going on there.  In brief, that branch contains a major change to how drawing works in the macOS port of Tk.  This change makes drawing in the macOS port work in a way which is much closer to how it works in the other platforms, and much closer to what the generic Tk code expects. The result is better performance and increased stability.

The work is based on an idea of Christopher Chavez's.  He realized that Apple's Appkit design allowed for an alternative drawing strategy, and wrote a proof-of-concept implementation.  I had created a branch containing his implementation a couple of years ago.  Recently I merged that branch with Tk 9 and then spent a few weeks ironing out wrinkles.  Nicolas Bats, Torsten Berg, Steve Landers, and Kevin Walzer kindly offered to help me by testing the new implementation on their apps, some of which provide very challenging environments for Tk.  The result of this work is in the cgimage-with-crossing branch.  (The branch also includes a fix for ticket [22349fc78a] which is about <Enter> and <Leave> events - hence the reference to crossing in the branch tag.)

The five of us are confident that this is a significant improvement. Some highlights are:
    * Better performance - CPU-intensive apps use about 20% fewer CPU cycles
    * Better conformance - Most runs of the regression test suite on macOS 14 show no failures.  This is not close to being the case for the trunk branch.
    * Fewer graphical artifacts (and work continues on those that remain.)
    * Fewer crashes (none that we know of at the moment.)

Given that the branch has been tested on large and complex apps - all of which are working well - this is a major step forwards and we’d like to see it tested more widely.  To that end, although we are late in the release cycle we are comfortable recommending that this be merged into trunk and made part of beta3.  That’s the only way we can guarantee wider testing. Of course, if any issues crop up they will be addressed in a timely manner.  If you are a macOS tkchat user please download and try https://www.codebykevin.com/TkChat_Setup_1.5.2.dmg   If you want to try the new macOS Tk with your own code and are comfortable building it yourself please checkout and build the Tk cgimage_with_crossing branch

The rest of this message gives some more details about the changes.

The main difference is that the new drawing strategy does away with the asynchronous drawing via the drawRect method.  A macOS app which uses drawRect is supposed to do all of its drawing within the drawRect method, which can only be called by the NSApplication object.  The app can request that drawRect be called by setting the viewNeedsDisplay flag, but can not call drawRect.  The original design of the macOS Aqua port only partially complied with this strategy.  In fact it was possible to obtain a valid graphics context for drawing to the backing layer of a window's contentView even outside of the drawRect method.  The original port would do this, e.g. in a widget display proc which is being run as an idle task.  But it would also draw in the drawRect method by first generating expose events for all widgets that need updates, then processing those expose events to create idle tasks, then processing all of those idle tasks immediately.

Obviously this scheme was not what Tk expected.  Nor was it what Apple expected.  And Apple put an abrupqt end to half of it with the release of macOS Mojave, which made it impossible to obtain a valid graphics context outside of the drawRect method. When Tk was first built on Mojave, it produced nothing but totally black windows.  Eventually this was worked around by arranging that any drawing operation which failed due to an invalid graphics context would be rescheduled and also to modify drawRect to make it (attempt to) run all of those rescheduled idle tasks.

With the new macOS code it is once again true that Tk always has a valid graphics context available and hence can draw at any time.  It draws into a CGBitmapImageContext which serves as the backing store for a Tk Window.  Instead of calling drawRect, it is configured with Apple's other option which is to call updateLayer instead.  In the new setup updateLayer installs the CGImage as the CALayer of the ContentView of the NSWindow which hosts the Tk toplevel, causing the contents of the image to appear in the window.  It also creates a new CGImage containing a copy of the window in which subsequent drawing operations take place.  This allows drawing operations to take place at the expected time and in the expected order.  That makes for better stability and less unpredictability.  In addition, it removes the overhead that arises from attempting a drawing operation, having it fail due to an invalid graphics context, rescheduling and rerunning the operation.  This accounts for the reduced CPU usage.

- Marc
_______________________________________________
Tcl-mac mailing list
tc...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-mac

[TCLCORE] Major update to the macOS port of Tk

[Marc C. <mar...@gm...>]

Tk Timeline watchers may have noticed a long chain of commits to the
cgimage_with_crossing branch over the last month.  This announcement is to
explain what has been going on there.  In brief, that branch contains a
major change to how drawing works in the macOS port of Tk.  This change
makes drawing in the macOS port work in a way which is much closer to how
it works in the other platforms, and much closer to what the generic Tk
code expects. The result is better performance and increased stability.

The work is based on an idea of Christopher Chavez's.  He realized that
Apple's Appkit design allowed for an alternative drawing strategy, and
wrote a proof-of-concept implementation.  I had created a branch containing
his implementation a couple of years ago.  Recently I merged that branch
with Tk 9 and then spent a few weeks ironing out wrinkles.  Nicolas Bats,
Torsten Berg, Steve Landers, and Kevin Walzer kindly offered to help me by
testing the new implementation on their apps, some of which provide very
challenging environments for Tk.  The result of this work is in the
cgimage-with-crossing branch.  (The branch also includes a fix for ticket
[22349fc78a] which is about <Enter> and <Leave> events - hence the
reference to crossing in the branch tag.)

The five of us are confident that this is a significant improvement. Some
highlights are:
    * Better performance - CPU-intensive apps use about 20% fewer CPU cycles
    * Better conformance - Most runs of the regression test suite on macOS
14 show no failures.  This is not close to being the case for the trunk
branch.
    * Fewer graphical artifacts (and work continues on those that remain.)
    * Fewer crashes (none that we know of at the moment.)

Given that the branch has been tested on large and complex apps - all of
which are working well - this is a major step forwards and we’d like to see
it tested more widely.  To that end, although we are late in the release
cycle we are comfortable recommending that this be merged into trunk and
made part of beta3.  That’s the only way we can guarantee wider testing. Of
course, if any issues crop up they will be addressed in a timely manner.
If you are a macOS tkchat user please download and try
https://www.codebykevin.com/TkChat_Setup_1.5.2.dmg   If you want to try the
new macOS Tk with your own code and are comfortable building it yourself
please checkout and build the Tk cgimage_with_crossing branch

The rest of this message gives some more details about the changes.

The main difference is that the new drawing strategy does away with the
asynchronous drawing via the drawRect method.  A macOS app which uses
drawRect is supposed to do all of its drawing within the drawRect method,
which can only be called by the NSApplication object.  The app can request
that drawRect be called by setting the viewNeedsDisplay flag, but can not
call drawRect.  The original design of the macOS Aqua port only partially
complied with this strategy.  In fact it was possible to obtain a valid
graphics context for drawing to the backing layer of a window's contentView
even outside of the drawRect method.  The original port would do this, e.g.
in a widget display proc which is being run as an idle task.  But it would
also draw in the drawRect method by first generating expose events for all
widgets that need updates, then processing those expose events to create
idle tasks, then processing all of those idle tasks immediately.

Obviously this scheme was not what Tk expected.  Nor was it what Apple
expected.  And Apple put an abrupqt end to half of it with the release of
macOS Mojave, which made it impossible to obtain a valid graphics context
outside of the drawRect method. When Tk was first built on Mojave, it
produced nothing but totally black windows.  Eventually this was worked
around by arranging that any drawing operation which failed due to an
invalid graphics context would be rescheduled and also to modify drawRect
to make it (attempt to) run all of those rescheduled idle tasks.

With the new macOS code it is once again true that Tk always has a valid
graphics context available and hence can draw at any time.  It draws into a
CGBitmapImageContext which serves as the backing store for a Tk Window.
Instead of calling drawRect, it is configured with Apple's other option
which is to call updateLayer instead.  In the new setup updateLayer
installs the CGImage as the CALayer of the ContentView of the NSWindow
which hosts the Tk toplevel, causing the contents of the image to appear in
the window.  It also creates a new CGImage containing a copy of the window
in which subsequent drawing operations take place.  This allows drawing
operations to take place at the expected time and in the expected order.
That makes for better stability and less unpredictability.  In addition, it
removes the overhead that arises from attempting a drawing operation,
having it fail due to an invalid graphics context, rescheduling and
rerunning the operation.  This accounts for the reduced CPU usage.

- Marc

Re: [TCLCORE] [External] Propose reverting [encoding system] on Windows

[<apn...@ya...>]

There has been no change to Windows line ending conventions in Tcl 9.

 

From: Da Silva, Peter J Collins <pet...@fl...> 
Sent: Wednesday, July 3, 2024 3:10 AM
To: apn...@ya...; tcl...@li...
Subject: Re: [External] [TCLCORE] Propose reverting [encoding system] on
Windows

 

As a UNIX user, I normally assume that text files created on Windows will
require careful handling in any case, because of the Windows line ending
convention. Is this no longer an issue?

 

From: apnmbx-public--- via Tcl-Core <tcl...@li...
<mailto:tcl...@li...> >
Date: Sunday, June 30, 2024 at 10:40
To: tcl...@li... <mailto:tcl...@li...>
<tcl...@li... <mailto:tcl...@li...> >
Subject: [External] [TCLCORE] Propose reverting [encoding system] on Windows

I realize everyone is tired of encoding issues but I feel this is important
given the consequences ...

 

One of the changes in Tcl 9 is to add a manifest property on Windows which
effectively sets [encoding system] to UTF-8 irrespective of the user's code
page setting. This manifest only has effect on Windows versions Windows 10
Build 1903 and later. Earlier Windows systems ignore the setting and Tcl 9
will use the user's code page setting on those platforms.

 

This change, presumably made as part of TIP 587, has consequences serious
enough to be reverted in my opinion.

 

Below "file" refers to files containing non-ASCII content and that it is
read/written with the (default) system encoding. And "cannot be read" means
either an encoding error is thrown or data is garbled depending on the
combination of Tcl version, platform version and system encoding (!).

 

- A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice versa
even on *on the same exact system* on modern (build 1903 or later) Windows
systems. So for example, on US English (cp1252) systems "set fd [open
foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in foo.txt that
will raise an error in Tcl 9's readFile (or the longer open/read/close
equivalent). This is not acceptable in my view.

 

- Along the same lines, a file written by a 9.x tclsh on pre-1903 Windows 10
cannot be read by the *same* 9.x tclsh on the *same* system after a
*Windows* update. Also not acceptable.

 

- A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 9.x
tclsh on Windows 10 1903 and vice versa even when the two share a code page.
May be less serious but still undesirable.

 

- A third-party application (not using tclsh) that uses as its scripting
language will not be able to read files written by tclsh and vice versa
unless it also includes the utf-8 magic line in its manifest. This latter is
unlikely as it would raise the same compatibility problems for them as for
tclsh above. Also undesirable.

 

In my view, these are all serious compatibility problems.

 

The original motivation for the change in [encoding system] presumably came
from this page -
https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-c
ode-page
<https://urldefense.com/v3/__https:/learn.microsoft.com/en-us/windows/apps/d
esign/globalizing/use-utf8-code-page__;!!MvWE!BNNxzazI5SkKt4FKj0Ynj6hmy3L4VF
K5gj1EEHXm_emSGvfDXkbLcXauqr8n-w21KEmegYp4wSdLfgeUTtiF_-GzrmiBypytdg$> .
However, note the target audience is really applications that use the ANSI
Api's and its usefulness is limited for applications (like Tcl) that use the
wide character API's. Moreover, as stated on that page, if targeting
platforms prior to Build 1903, "you must handle legacy code page detection
and conversion as usual".

 

Given the above, I would propose reverting [encoding system] on Windows to
go back to defaulting to the encoding as per the user's code page setting as
in Tcl 8. Note this is not a code change, it is a change to the manifest
resource in tclsh (and wish) to remove the ActiveCodePage property. The
right place to move forward with UTF-8 as the default encoding belongs to
the *applications* written in Tcl, not Tcl itself.

 

(Note: I am not proposing changing the encoding used by the source command)

 

What would be drawbacks of reverting this change? One could argue that it
would reduce compatibility with other platforms like Unix which use UTF-8 as
the system encoding. However, I would argue that

 

- this is no different from what's the case with 8.x today

 

- sharing of files between platforms really has to include explicit
configuration of encoding as good practice and most cross-platform
applications will do that

 

- the issues listed above for a single-platform, on the very same system
even, are much more serious

 

Reverting would probably need to be TIP'ed but I wanted to get feedback
before embarking on that path.

 

With regards to Unix, I do not have enough experience with encodings on Unix
to know if similar issues would arise there. The [encoding system] in Tcl 9
has changed from iso8859-1 to utf-8. Someone should examine this closer.

 

Comments please. I'd prefer to avoid TIP overhead if someone can spot holes
in the above.

 

/Ashok

 

Re: [TCLCORE] Tcl_Obj types and tkinter

[Marc C. <cul...@gm...>]

Hi Jan,

The Python _tkinter extension module definitely needs to be able to convert
any Tcl_Obj to some sort of Python object.  The reason is that tkinter
allows running any Tcl command in an embedded Tcl interpreter.  The Python
class tkinter.Tk encapsulates a Tcl interpreter which has loaded the Tk
package.  The interpreter has a call method which can be passed a sequence
of strings to be run as a Tcl command.

Here is an example of how you can generate a Tcl bignum using Python's
tkinter package, which then converts it to a Python integer:

$ python3
Python 3.11.8 (v3.11.8:db85d51d3e, Feb  6 2024, 18:02:37) [Clang 13.0.0
(clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tkinter
>>> interp = tkinter.Tk()
>>> x = interp.call('expr', '11111111111111111111', '*',
'22222222222222222222')
>>> print(x)
246913580246913580241975308641975308642
>>> type(x)
<class 'int'>

- Marc

On Tue, Jul 2, 2024 at 7:26 PM Marc Culler <cul...@gm...> wrote:

> On Tue, Jul 2, 2024 at 3:13 AM Jan Nijtmans <jan...@gm...>
> wrote:
>
>> Let me explain the reason for the removal of "boolean" and "int".
>
>
> Thank you!
>
>
>> This means that code which uses the "int" or "boolean"
>> types need to be inspected: code which worked for 8.x
>> won't necessarily behave the same in 9.0.
>>
>
> That is understandable, although I don't see what it has to do with
> whether registration was a good idea or not.
>
> As far as _tkinter is concerned (based on what I can see  from reading the
> code - I had no hand in writing it) here is the issue.  The _tkinter module
> allows running Tk commands in an embedded interpreter.  The result is a
> pointer to a Tcl_Obj of some type.  Several of the possible types can be
> converted into comparable Python objects: this is true for int, string,
> list, dict, bignum, and boolean at least.  For each such type there is a
> function in _tkinter.c which converts that Tcl_Obj type to the
> corresponding Python type.  (You are saying that those functions will need
> to be rewritten.  That is extremely useful to know, but unrelated to how
> the registration is used in _tkinter.c.)  There are also types which do not
> convert naturally.  There is a function in _tkinter.c to convert such a
> Tcl_Obj to a Python object which basically just records the type name and
> is of no particular use in Python, although it can be converted to a
> string.  In addition to these utility functions there is a function FromObj
> which converts the result of any Tcl command to a Python object by calling
> the appropriate converter.  The FromObj just needs to know what the type of
> the TclObj is so it can call the right converter.  The converter is
> responsible for knowing enough about the internal structure of the Tcl_Obj
> to construct  its python object.
>
> Here is how the current _tkinter code makes that decision.  It caches a
> pointer to each of the Tcl_ObjType structs used by a type for which it has
> a converter.  When it receives a Tcl_Obj it checks the typePtr to the
> cached values and if it gets a match it calls the corresponding converter.
> You have pointed out that a Tcl_Obj of a given type (as determined by its
> type name) might have a NULL typePtr, meaning it is uninitialized.  But
> Python has no concept of an uninitialized objects.  So if _tkinter receives
> a Tcl_Obj with a NULL typePtr it has to treat it as an unknown type, even
> though it has a well-defined type name.  It cannot be converted to anything.
>
> The way that the authors of _tkinter were using Tcl_GetObjType was simply
> to initialize the list of cached pointers to Tcl_ObjType structs.  The code
> had already been patched to deal with unregistered types by parsing the
> type name and updating the cache the first time that it sees a Tcl_Obj of a
> recognized name with a non-NULL typePtr.  Of course they could have chosen
> to parse the type name for every Tcl_Obj received.  But I am pretty sure
> they thought (correctly) that doing that was going to be unnecessarily slow.
>
> Here is a fourth workaround option which could be added to my list.  How
> about if each Tcl_Obj had a 32-bit (or even 8-bit) integer field which is
> an identifier for the the type of the Tcl_Obj.  There would be a unique
> value for each type defined by the core and one other value meaning that
> the type is registered by an extension.  Then it would be possible to very
> quickly decide what the type of a Tcl_Obj is.  Those identifiers could be
> conveniently used in a switch statement.  One could check whether an object
> has been initialized by checking if its typePtr is NULL.  It would be quick
> to determine when a call to Tcl_GetObjType is needed.
>
>
>> Let's see. I doubt the usefulness of "boolean", "bignum",
>> "exprcode" and "ensembleCommand" for tkInter. Tk never
>> returns a Tcl_Obj of the "boolean" and the "bignum" type, it
>> uses the "int" type for that (since pixels never exceed the
>> wideint range). "exprcode" and "ensembleCommand"
>> were not exported in Tcl 8.6, why do you want them now?
>>
>
> I don't want types that could never appear.  It is just that I saw no
> reason why some core types should be registered and others not.  The
> tkinter module is not the only consumer.  It is simpler, conceptually and
> programmatically, to simply register all of the core types.  Also, I am
> pretty surprised to hear that boolean and bignum types could never appear
> as the result of a Tcl command processed by tkinter, given that _tkinter.c
> includes code for converting those types to Python types.  Are you sure
> about that?
>
> - Marc
>
>
>>
>> The "bignum" implementation changed between 8.6
>> and 8.7/9.0 too: the libtommath library in 8.6 was
>> always compiled for 32-bit. In 8.7 it's compiled for
>> 64-bit on 64-bit platforms. So, accessing the internal
>> fields won't be the same as in Tcl 8.6!
>>
>> That leaves "int" and "index". I can imagine those are
>> useful for optimization purposes in tkInter. If
>> Tcl_GetNumberFromObj (which doesn't handle "index")
>> is not a useful function for what tkInter needs it for,
>> I'm OK with adding those two back in Tcl 9.0.
>>
>> So:
>> 1) Marc, can you live with the compromise of adding
>> "int" and "index" only? IMHO "boolean" and
>> "bignum" are not useful for special-casing in tkInter.
>> For "exprcode" and "ensembleCommand" I don't
>> see a use-case at all.
>> 2) What would be needed for Tcl_GetNumberFromObj()
>> to make it a useful alternative for use in tkInter? Would
>> adding "index" and "boolean" (two additional enum
>> values) help there?
>>
>> I won't create a quick TIP for adding "index" and
>> "boolean" to Tcl_GetNumberFromObj(). Maybe for 9.1.
>>
>> Hope this helps,
>>        Jan Nijtmans
>>
>>
>> _______________________________________________
>> Tcl-Core mailing list
>> Tcl...@li...
>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>>
>

[TCLCORE] Fwd: Tcl_Obj types and tkinter

[Marc C. <cul...@gm...>]

On Tue, Jul 2, 2024 at 3:13 AM Jan Nijtmans <jan...@gm...> wrote:

> Let me explain the reason for the removal of "boolean" and "int".


Thank you!


> This means that code which uses the "int" or "boolean"
> types need to be inspected: code which worked for 8.x
> won't necessarily behave the same in 9.0.
>

That is understandable, although I don't see what it has to do with whether
registration was a good idea or not.

As far as _tkinter is concerned (based on what I can see  from reading the
code - I had no hand in writing it) here is the issue.  The _tkinter module
allows running Tk commands in an embedded interpreter.  The result is a
pointer to a Tcl_Obj of some type.  Several of the possible types can be
converted into comparable Python objects: this is true for int, string,
list, dict, bignum, and boolean at least.  For each such type there is a
function in _tkinter.c which converts that Tcl_Obj type to the
corresponding Python type.  (You are saying that those functions will need
to be rewritten.  That is extremely useful to know, but unrelated to how
the registration is used in _tkinter.c.)  There are also types which do not
convert naturally.  There is a function in _tkinter.c to convert such a
Tcl_Obj to a Python object which basically just records the type name and
is of no particular use in Python, although it can be converted to a
string.  In addition to these utility functions there is a function FromObj
which converts the result of any Tcl command to a Python object by calling
the appropriate converter.  The FromObj just needs to know what the type of
the TclObj is so it can call the right converter.  The converter is
responsible for knowing enough about the internal structure of the Tcl_Obj
to construct  its python object.

Here is how the current _tkinter code makes that decision.  It caches a
pointer to each of the Tcl_ObjType structs used by a type for which it has
a converter.  When it receives a Tcl_Obj it checks the typePtr to the
cached values and if it gets a match it calls the corresponding converter.
You have pointed out that a Tcl_Obj of a given type (as determined by its
type name) might have a NULL typePtr, meaning it is uninitialized.  But
Python has no concept of an uninitialized objects.  So if _tkinter receives
a Tcl_Obj with a NULL typePtr it has to treat it as an unknown type, even
though it has a well-defined type name.  It cannot be converted to anything.

The way that the authors of _tkinter were using Tcl_GetObjType was simply
to initialize the list of cached pointers to Tcl_ObjType structs.  The code
had already been patched to deal with unregistered types by parsing the
type name and updating the cache the first time that it sees a Tcl_Obj of a
recognized name with a non-NULL typePtr.  Of course they could have chosen
to parse the type name for every Tcl_Obj received.  But I am pretty sure
they thought (correctly) that doing that was going to be unnecessarily slow.

Here is a fourth workaround option which could be added to my list.  How
about if each Tcl_Obj had a 32-bit (or even 8-bit) integer field which is
an identifier for the the type of the Tcl_Obj.  There would be a unique
value for each type defined by the core and one other value meaning that
the type is registered by an extension.  Then it would be possible to very
quickly decide what the type of a Tcl_Obj is.  Those identifiers could be
conveniently used in a switch statement.  One could check whether an object
has been initialized by checking if its typePtr is NULL.  It would be quick
to determine when a call to Tcl_GetObjType is needed.


> Let's see. I doubt the usefulness of "boolean", "bignum",
> "exprcode" and "ensembleCommand" for tkInter. Tk never
> returns a Tcl_Obj of the "boolean" and the "bignum" type, it
> uses the "int" type for that (since pixels never exceed the
> wideint range). "exprcode" and "ensembleCommand"
> were not exported in Tcl 8.6, why do you want them now?
>

I don't want types that could never appear.  It is just that I saw no
reason why some core types should be registered and others not.  The
tkinter module is not the only consumer.  It is simpler, conceptually and
programmatically, to simply register all of the core types.  Also, I am
pretty surprised to hear that boolean and bignum types could never appear
as the result of a Tcl command processed by tkinter, given that _tkinter.c
includes code for converting those types to Python types.  Are you sure
about that?

- Marc


>
> The "bignum" implementation changed between 8.6
> and 8.7/9.0 too: the libtommath library in 8.6 was
> always compiled for 32-bit. In 8.7 it's compiled for
> 64-bit on 64-bit platforms. So, accessing the internal
> fields won't be the same as in Tcl 8.6!
>
> That leaves "int" and "index". I can imagine those are
> useful for optimization purposes in tkInter. If
> Tcl_GetNumberFromObj (which doesn't handle "index")
> is not a useful function for what tkInter needs it for,
> I'm OK with adding those two back in Tcl 9.0.
>
> So:
> 1) Marc, can you live with the compromise of adding
> "int" and "index" only? IMHO "boolean" and
> "bignum" are not useful for special-casing in tkInter.
> For "exprcode" and "ensembleCommand" I don't
> see a use-case at all.
> 2) What would be needed for Tcl_GetNumberFromObj()
> to make it a useful alternative for use in tkInter? Would
> adding "index" and "boolean" (two additional enum
> values) help there?
>
> I won't create a quick TIP for adding "index" and
> "boolean" to Tcl_GetNumberFromObj(). Maybe for 9.1.
>
> Hope this helps,
>        Jan Nijtmans
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] [External] Propose reverting [encoding system] on Windows

[simon <si...@wh...>]

It's more than just line endings. Output from Powershell commands redirected to a text file results in UTF-16 (with LE BOM) encoding. It can bet set to UTF-8 but that's not the default.Simon
-------- Original message --------From: "Da Silva, Peter J                            Collins" <pet...@fl...> Date: 02/07/2024  23:05  (GMT+00:00) To: apn...@ya..., tcl...@li... Subject: Re: [TCLCORE] [External] Propose reverting [encoding system] on Windows 

As a UNIX user, I normally assume that text files created on Windows will require careful handling in any case, because of the Windows line ending convention. Is this no longer an issue?
 



From:
apnmbx-public--- via Tcl-Core <tcl...@li...>
Date: Sunday, June 30, 2024 at 10:40
To: tcl...@li... <tcl...@li...>
Subject: [External] [TCLCORE] Propose reverting [encoding system] on Windows

I realize everyone is tired of encoding issues but I feel this is important given the consequences ...
 
One of the changes in Tcl 9 is to add a manifest property on Windows which effectively sets [encoding system] to UTF-8 irrespective of the user's code page setting. This manifest
 only has effect on Windows versions Windows 10 Build 1903 and later. Earlier Windows systems ignore the setting and Tcl 9 will use the user's code page setting on those platforms.
 
This change, presumably made as part of TIP 587, has consequences serious enough to be reverted in my opinion.
 
Below "file" refers to files containing non-ASCII content and that it is read/written with the (default) system encoding. And "cannot be read" means either an encoding error is
 thrown or data is garbled depending on the combination of Tcl version, platform version and system encoding (!).
 
- A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice versa even on *on the same exact system* on modern (build 1903 or later) Windows systems. So for example,
 on US English (cp1252) systems "set fd [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in foo.txt that will raise an error in Tcl 9's readFile (or the longer open/read/close equivalent). This is not acceptable in my view.
 
- Along the same lines, a file written by a 9.x tclsh on pre-1903 Windows 10 cannot be read by the *same* 9.x tclsh on the *same* system after a *Windows*
 update. Also not acceptable.
 
- A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 9.x tclsh on Windows 10 1903 and vice versa even when the two share a code page. May be less serious but still
 undesirable.
 
- A third-party application (not using tclsh) that uses as its scripting language will not be able to read files written by tclsh and vice versa unless it also includes the utf-8
 magic line in its manifest. This latter is unlikely as it would raise the same compatibility problems for them as for tclsh above. Also undesirable.
 
In my view, these are all serious compatibility problems.
 
The original motivation for the change in [encoding system] presumably came from this page -

https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page. However, note the target audience is really applications that use the ANSI Api's and its usefulness is limited for applications (like Tcl) that use the wide character
 API's. Moreover, as stated on that page, if targeting platforms prior to Build 1903, "you must handle legacy code page detection and conversion as usual".
 
Given the above, I would propose reverting [encoding system] on Windows to go back to defaulting to the encoding as per the user's code page setting as in Tcl 8.
 Note this is not a code change, it is a change to the manifest resource in tclsh (and wish) to remove the ActiveCodePage property. The right place to move forward with UTF-8 as the default encoding belongs to the *applications* written in Tcl, not Tcl itself.
 
(Note: I am not proposing changing the encoding used by the source command)
 
What would be drawbacks of reverting this change? One could argue that it would reduce compatibility with other platforms like Unix which use UTF-8 as the system encoding. However,
 I would argue that
 
- this is no different from what's the case with 8.x today
 
- sharing of files between platforms really has to include explicit configuration of encoding as good practice and most cross-platform applications will do that
 
- the issues listed above for a single-platform, on the very same system even, are much more serious
 
Reverting would probably need to be TIP'ed but I wanted to get feedback before embarking on that path.
 
With regards to Unix, I do not have enough experience with encodings on Unix to know if similar issues would arise there. The [encoding system] in Tcl 9 has changed from iso8859-1
 to utf-8. Someone should examine this closer.
 
Comments please. I’d prefer to avoid TIP overhead if someone can spot holes in the above.
 
/Ashok
 

Re: [TCLCORE] [External] Propose reverting [encoding system] on Windows

[Da S. P. J C. <pet...@fl...>]

Oh joy. WHY Microsoft?

In that case I would definitely not consider compatibility between UNIX and Windows text files a strong argument.

From: simon <si...@wh...>
Date: Tuesday, July 2, 2024 at 17:54
To: Da Silva, Peter J Collins <pet...@fl...>, apn...@ya... <apn...@ya...>, tcl...@li... <tcl...@li...>
Subject: Re: [TCLCORE] [External] Propose reverting [encoding system] on Windows
It's more than just line endings. Output from Powershell commands redirected to a text file results in UTF-16 (with LE BOM) encoding. It can bet set to UTF-8 but that's not the default.

Simon

-------- Original message --------
From: "Da Silva, Peter J Collins" <pet...@fl...>
Date: 02/07/2024 23:05 (GMT+00:00)
To: apn...@ya..., tcl...@li...
Subject: Re: [TCLCORE] [External] Propose reverting [encoding system] on Windows

As a UNIX user, I normally assume that text files created on Windows will require careful handling in any case, because of the Windows line ending convention. Is this no longer an issue?

From: apnmbx-public--- via Tcl-Core <tcl...@li...>
Date: Sunday, June 30, 2024 at 10:40
To: tcl...@li... <tcl...@li...>
Subject: [External] [TCLCORE] Propose reverting [encoding system] on Windows
I realize everyone is tired of encoding issues but I feel this is important given the consequences ...

One of the changes in Tcl 9 is to add a manifest property on Windows which effectively sets [encoding system] to UTF-8 irrespective of the user's code page setting. This manifest only has effect on Windows versions Windows 10 Build 1903 and later. Earlier Windows systems ignore the setting and Tcl 9 will use the user's code page setting on those platforms.

This change, presumably made as part of TIP 587, has consequences serious enough to be reverted in my opinion.

Below "file" refers to files containing non-ASCII content and that it is read/written with the (default) system encoding. And "cannot be read" means either an encoding error is thrown or data is garbled depending on the combination of Tcl version, platform version and system encoding (!).

- A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice versa even on *on the same exact system* on modern (build 1903 or later) Windows systems. So for example, on US English (cp1252) systems "set fd [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in foo.txt that will raise an error in Tcl 9's readFile (or the longer open/read/close equivalent). This is not acceptable in my view.

- Along the same lines, a file written by a 9.x tclsh on pre-1903 Windows 10 cannot be read by the *same* 9.x tclsh on the *same* system after a *Windows* update. Also not acceptable.

- A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 9.x tclsh on Windows 10 1903 and vice versa even when the two share a code page. May be less serious but still undesirable.

- A third-party application (not using tclsh) that uses as its scripting language will not be able to read files written by tclsh and vice versa unless it also includes the utf-8 magic line in its manifest. This latter is unlikely as it would raise the same compatibility problems for them as for tclsh above. Also undesirable.

In my view, these are all serious compatibility problems.

The original motivation for the change in [encoding system] presumably came from this page - https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page<https://urldefense.com/v3/__https:/learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page__;!!MvWE!BNNxzazI5SkKt4FKj0Ynj6hmy3L4VFK5gj1EEHXm_emSGvfDXkbLcXauqr8n-w21KEmegYp4wSdLfgeUTtiF_-GzrmiBypytdg$>. However, note the target audience is really applications that use the ANSI Api's and its usefulness is limited for applications (like Tcl) that use the wide character API's. Moreover, as stated on that page, if targeting platforms prior to Build 1903, "you must handle legacy code page detection and conversion as usual".

Given the above, I would propose reverting [encoding system] on Windows to go back to defaulting to the encoding as per the user's code page setting as in Tcl 8. Note this is not a code change, it is a change to the manifest resource in tclsh (and wish) to remove the ActiveCodePage property. The right place to move forward with UTF-8 as the default encoding belongs to the *applications* written in Tcl, not Tcl itself.

(Note: I am not proposing changing the encoding used by the source command)

What would be drawbacks of reverting this change? One could argue that it would reduce compatibility with other platforms like Unix which use UTF-8 as the system encoding. However, I would argue that

- this is no different from what's the case with 8.x today

- sharing of files between platforms really has to include explicit configuration of encoding as good practice and most cross-platform applications will do that

- the issues listed above for a single-platform, on the very same system even, are much more serious

Reverting would probably need to be TIP'ed but I wanted to get feedback before embarking on that path.

With regards to Unix, I do not have enough experience with encodings on Unix to know if similar issues would arise there. The [encoding system] in Tcl 9 has changed from iso8859-1 to utf-8. Someone should examine this closer.

Comments please. I’d prefer to avoid TIP overhead if someone can spot holes in the above.

/Ashok

Re: [TCLCORE] [External] Re: Migration tool for Tcl 9

[Da S. P. J C. <pet...@fl...>]

Find ... -name ‘*.tcl’ -print0 | xargs -0 tclsh migrate.tcl check

From: apnmbx-public--- via Tcl-Core <tcl...@li...>
Date: Monday, July 1, 2024 at 21:40
To: 'Harald Oehlmann' <har...@el...>, tcl...@li... <tcl...@li...>
Subject: [External] Re: [TCLCORE] Migration tool for Tcl 9
For now, to check a tree, you can pass multiple glob arguments, e.g.

Tclsh migrate.tcl check *.tcl */*.tcl */*/*.tcl

Etc. Not ideal but not sure when I'll have time for feature enhancements (as opposed to bug fixes)

/Ashok

-----Original Message-----
From: Harald Oehlmann <har...@el...>
Sent: Monday, July 1, 2024 1:37 PM
To: tcl...@li...
Subject: Re: [TCLCORE] Migration tool for Tcl 9

Hi Ashok,
impressive revolutionary work, I appreciate !
I tried the static checker on my code base and it gives good comments:
- finds variables in namespaces
- finds non-ascii encodings
I did not try the run-time stuff jet.

I would appreciate to have an option to check that check descends to
subfolders, so a whole tree may be checked at once.

Thanks for all,
Harald

Am 01.07.2024 um 09:48 schrieb apnmbx-public--- via Tcl-Core:
> Sorry, forgot to mention – works best with recent trunk builds.
>
> *From:*apnmbx-public--- via Tcl-Core <tcl...@li...>
> *Sent:* Monday, July 1, 2024 1:14 PM
> *To:* tcl...@li...
> *Subject:* [TCLCORE] Migration tool for Tcl 9
>
> All,
>
> I've been working on a tool to help with migration of Tcl 8 scripts to
> Tcl 9. It includes a static checker based on Nagelfar as well as a
> runtime shim that fixes up incompatibilities like i/o encoding errors,
> tilde expansion, package requirements etc., allowing the scripts to be
> loaded while emitting warnings.
>
> Repository: https://urldefense.com/v3/__https://github.com/apnadkarni/tcl9-migrate__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04RzJeWbcqA$<https://urldefense.com/v3/__https:/github.com/apnadkarni/tcl9-migrate__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04RzJeWbcqA$>
> <https://urldefense.com/v3/__https://github.com/apnadkarni/tcl9-migrate__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04RzJeWbcqA$ >
>
> Docs: https://urldefense.com/v3/__https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04RxeiRMC4Q$<https://urldefense.com/v3/__https:/github.com/apnadkarni/tcl9-migrate/blob/main/README.md__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04RxeiRMC4Q$>
> <https://urldefense.com/v3/__https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04RxeiRMC4Q$ >
>
> Download: https://urldefense.com/v3/__https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04Rx6v5z-ZQ$<https://urldefense.com/v3/__https:/github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04Rx6v5z-ZQ$>
> <https://urldefense.com/v3/__https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04Rx6v5z-ZQ$ >
>
> I'm looking for testers and feedback on both the tool as well as
> documentation (explanation of the warnings). I'd appreciate testing with
> your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above
> repository.
>
> Be warned it has (and always will have) false positives and false
> negatives but the hope is it will nevertheless be useful in saving time.
>
> Thanks
>
> /Ashok




_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://urldefense.com/v3/__https://lists.sourceforge.net/lists/listinfo/tcl-core__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04Rypp-Eb3w$<https://urldefense.com/v3/__https:/lists.sourceforge.net/lists/listinfo/tcl-core__;!!MvWE!Bz40c4s9by3nynL634mfP329mx89fCXIWwmEtu3sldpaoxVWwpTOwfJdzUTKqSaGjBv8DUe3D2uP7PI5yp_k4Fc04Rypp-Eb3w$>

Re: [TCLCORE] [External] Propose reverting [encoding system] on Windows

[Da S. P. J C. <pet...@fl...>]

As a UNIX user, I normally assume that text files created on Windows will require careful handling in any case, because of the Windows line ending convention. Is this no longer an issue?

From: apnmbx-public--- via Tcl-Core <tcl...@li...>
Date: Sunday, June 30, 2024 at 10:40
To: tcl...@li... <tcl...@li...>
Subject: [External] [TCLCORE] Propose reverting [encoding system] on Windows
I realize everyone is tired of encoding issues but I feel this is important given the consequences ...

One of the changes in Tcl 9 is to add a manifest property on Windows which effectively sets [encoding system] to UTF-8 irrespective of the user's code page setting. This manifest only has effect on Windows versions Windows 10 Build 1903 and later. Earlier Windows systems ignore the setting and Tcl 9 will use the user's code page setting on those platforms.

This change, presumably made as part of TIP 587, has consequences serious enough to be reverted in my opinion.

Below "file" refers to files containing non-ASCII content and that it is read/written with the (default) system encoding. And "cannot be read" means either an encoding error is thrown or data is garbled depending on the combination of Tcl version, platform version and system encoding (!).

- A file written by a 8.x tclsh cannot be read by a 9.x tclsh and vice versa even on *on the same exact system* on modern (build 1903 or later) Windows systems. So for example, on US English (cp1252) systems "set fd [open foo.txt w]; puts $fd \xa9; close $fd" in Tcl 8 will result in foo.txt that will raise an error in Tcl 9's readFile (or the longer open/read/close equivalent). This is not acceptable in my view.

- Along the same lines, a file written by a 9.x tclsh on pre-1903 Windows 10 cannot be read by the *same* 9.x tclsh on the *same* system after a *Windows* update. Also not acceptable.

- A file written by a 9.x tclsh on Windows 7 or 8 cannot be read by a 9.x tclsh on Windows 10 1903 and vice versa even when the two share a code page. May be less serious but still undesirable.

- A third-party application (not using tclsh) that uses as its scripting language will not be able to read files written by tclsh and vice versa unless it also includes the utf-8 magic line in its manifest. This latter is unlikely as it would raise the same compatibility problems for them as for tclsh above. Also undesirable.

In my view, these are all serious compatibility problems.

The original motivation for the change in [encoding system] presumably came from this page - https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page<https://urldefense.com/v3/__https:/learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page__;!!MvWE!BNNxzazI5SkKt4FKj0Ynj6hmy3L4VFK5gj1EEHXm_emSGvfDXkbLcXauqr8n-w21KEmegYp4wSdLfgeUTtiF_-GzrmiBypytdg$>. However, note the target audience is really applications that use the ANSI Api's and its usefulness is limited for applications (like Tcl) that use the wide character API's. Moreover, as stated on that page, if targeting platforms prior to Build 1903, "you must handle legacy code page detection and conversion as usual".

Given the above, I would propose reverting [encoding system] on Windows to go back to defaulting to the encoding as per the user's code page setting as in Tcl 8. Note this is not a code change, it is a change to the manifest resource in tclsh (and wish) to remove the ActiveCodePage property. The right place to move forward with UTF-8 as the default encoding belongs to the *applications* written in Tcl, not Tcl itself.

(Note: I am not proposing changing the encoding used by the source command)

What would be drawbacks of reverting this change? One could argue that it would reduce compatibility with other platforms like Unix which use UTF-8 as the system encoding. However, I would argue that

- this is no different from what's the case with 8.x today

- sharing of files between platforms really has to include explicit configuration of encoding as good practice and most cross-platform applications will do that

- the issues listed above for a single-platform, on the very same system even, are much more serious

Reverting would probably need to be TIP'ed but I wanted to get feedback before embarking on that path.

With regards to Unix, I do not have enough experience with encodings on Unix to know if similar issues would arise there. The [encoding system] in Tcl 9 has changed from iso8859-1 to utf-8. Someone should examine this closer.

Comments please. I’d prefer to avoid TIP overhead if someone can spot holes in the above.

/Ashok

Re: [TCLCORE] Tcl_Obj types and tkinter

[Jan N. <jan...@gm...>]

Op ma 1 jul 2024 om 15:49 schreef Marc Culler:
> The questionable design choice was that many of the core Tcl_Obj types are not being registered in Tk 9.  Notably, the "int" type is not registered.  This means that the public interface provided by Tcl_GetObjType cannot be used to identify the type of an "int" Tcl_Obj from its typePtr field.  One has to parse the string in the name field, and that is costly.

Let me explain the reason for the removal of "boolean" and "int". It
also shows that
the registration of types was (IMHO) a questionable design choice to
start with ..... ;-)

The "boolean" and "int" types originally stored their value in
internalRep->longValue.
In Tcl 9, this longValue is not used any more, "int" and "boolean" now
store their value
in internalRep->wideValue. That means, the contract changed. In Tcl 8.x you
can do:
       if (obj->typePtr == cached.intType) {
                return obj->internalRep->longValue;
       }
and it returns the right value. In Tcl 9.0, you have to change "longValue"
to "wideValue" (for booleans too). The compiler won't detect the difference!
You might say, I'm not doing that ..... but this:
       if (obj->typePtr == cached.intType) {
                Tcl_GetLongFromObj(interp, obj, &value);
                return value;
       }
But that will not survive either: In Tcl 9.0 (on 32-bit platforms),
Tcl_GetLongFromObj() might return TCL_ERROR if
internalRep->wideValue < LONG_MIN or > LONG_MAX.

This means that code which uses the "int" or "boolean"
types need to be inspected: code which worked for 8.x
won't necessarily behave the same in 9.0.

> 2) Register *all* of the core types declared in that block of declarations in TclInitObjSubsystem.
> (In Tcl 9 the missing ones are: "bignum", "boolean", "exprcode", "int", "index", and "ensembleCommand"; obviously int and boolean are very common as returned values of commands.)

Let's see. I doubt the usefulness of "boolean", "bignum",
"exprcode" and "ensembleCommand" for tkInter. Tk never
returns a Tcl_Obj of the "boolean" and the "bignum" type, it
uses the "int" type for that (since pixels never exceed the
wideint range). "exprcode" and "ensembleCommand"
were not exported in Tcl 8.6, why do you want them now?

The "bignum" implementation changed between 8.6
and 8.7/9.0 too: the libtommath library in 8.6 was
always compiled for 32-bit. In 8.7 it's compiled for
64-bit on 64-bit platforms. So, accessing the internal
fields won't be the same as in Tcl 8.6!

That leaves "int" and "index". I can imagine those are
useful for optimization purposes in tkInter. If
Tcl_GetNumberFromObj (which doesn't handle "index")
is not a useful function for what tkInter needs it for,
I'm OK with adding those two back in Tcl 9.0.

So:
1) Marc, can you live with the compromise of adding
"int" and "index" only? IMHO "boolean" and
"bignum" are not useful for special-casing in tkInter.
For "exprcode" and "ensembleCommand" I don't
see a use-case at all.
2) What would be needed for Tcl_GetNumberFromObj()
to make it a useful alternative for use in tkInter? Would
adding "index" and "boolean" (two additional enum
values) help there?

I won't create a quick TIP for adding "index" and
"boolean" to Tcl_GetNumberFromObj(). Maybe for 9.1.

Hope this helps,
       Jan Nijtmans

[TCLCORE] Critical Phase for TCL/Tk 9

[Harald O. <har...@el...>]

Dear TCL/Tk team,

we are approaching TCL/Tk 9.0 again, similar to last year.
And again, critical issues arise, as more people test TCL 9.
This is great on one side and an issue on the other side.
TCL 9 development now took around 7 years.
Within those years, a lot of decisions were made and discussed, which 
are critical and show-up again now.
- Windows encoding system
- encoding binary
- internal TCL_OBJ reform and access functions (TkInter&Sergey)
- MAC API
To speed-up, the following happened:
- postpone TCL/Tk 8.7
- Start Tk 9.0

The main line why you want TCL/Tk 9:
- large memory objects and file support
- expr {08+09}
- emojis
- Tk visual reform and scaling
- buildin starkits

I personally have no idea how to proceed. IMHO, we need a release of 9.0 
soon. To my perception, any new development is blocked here.

Don Porter will not release if there are major questions. And last run, 
it took one year to clarify the encoding issue...

I can understand that major wizards are disappointed for any 
compatibility change which causes useless work with no gain.
The discussion culture is often rough - computer nerds are no social 
workers like my wife ;-). I feel personally blessed and tend to get quiet.

So, I thank anybody for testing and being in the boat. Please keep on. 
We will find solutions !

Thank you for all,
Harald

Re: [TCLCORE] Migration tool for Tcl 9

[Steve L. <st...@di...>]

On *nix you could do

	find . -name '*.tcl' -type f -print | xargs tclsh /path/to/migrate.tcl check

-- Steve
On 2 Jul 2024 at 10:40 AM +0800, apnmbx-public--- via Tcl-Core <tcl...@li...>, wrote:
> For now, to check a tree, you can pass multiple glob arguments, e.g.
>
> Tclsh migrate.tcl check *.tcl */*.tcl */*/*.tcl
>
> Etc. Not ideal but not sure when I'll have time for feature enhancements (as opposed to bug fixes)
>
> /Ashok
>
> -----Original Message-----
> From: Harald Oehlmann <har...@el...>
> Sent: Monday, July 1, 2024 1:37 PM
> To: tcl...@li...
> Subject: Re: [TCLCORE] Migration tool for Tcl 9
>
> Hi Ashok,
> impressive revolutionary work, I appreciate !
> I tried the static checker on my code base and it gives good comments:
> - finds variables in namespaces
> - finds non-ascii encodings
> I did not try the run-time stuff jet.
>
> I would appreciate to have an option to check that check descends to
> subfolders, so a whole tree may be checked at once.
>
> Thanks for all,
> Harald
>
> Am 01.07.2024 um 09:48 schrieb apnmbx-public--- via Tcl-Core:
> > Sorry, forgot to mention – works best with recent trunk builds.
> >
> > *From:*apnmbx-public--- via Tcl-Core <tcl...@li...>
> > *Sent:* Monday, July 1, 2024 1:14 PM
> > *To:* tcl...@li...
> > *Subject:* [TCLCORE] Migration tool for Tcl 9
> >
> > All,
> >
> > I've been working on a tool to help with migration of Tcl 8 scripts to
> > Tcl 9. It includes a static checker based on Nagelfar as well as a
> > runtime shim that fixes up incompatibilities like i/o encoding errors,
> > tilde expansion, package requirements etc., allowing the scripts to be
> > loaded while emitting warnings.
> >
> > Repository: https://github.com/apnadkarni/tcl9-migrate
> > <https://github.com/apnadkarni/tcl9-migrate>
> >
> > Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md
> > <https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md>
> >
> > Download: https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1
> > <https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1>
> >
> > I'm looking for testers and feedback on both the tool as well as
> > documentation (explanation of the warnings). I'd appreciate testing with
> > your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above
> > repository.
> >
> > Be warned it has (and always will have) false positives and false
> > negatives but the hope is it will nevertheless be useful in saving time.
> >
> > Thanks
> >
> > /Ashok
>
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Migration tool for Tcl 9

[<apn...@ya...>]

Oops, I mean false positive, not false negative 😊 It should not flag a warning.

 

From: apnmbx-public--- via Tcl-Core <tcl...@li...> 
Sent: Tuesday, July 2, 2024 8:01 AM
To: 'Paul Obermeier' <pa...@po...>; tcl...@li...
Subject: Re: [TCLCORE] Migration tool for Tcl 9

 

Yes, that would seem to be a false negative. Does not seem to account for reference to array elements when the array variable is declared. Should be fixable..I think.

 

Thanks for the feedback

 

From: Paul Obermeier <pa...@po... <mailto:pa...@po...> > 
Sent: Tuesday, July 2, 2024 3:14 AM
To: tcl...@li... <mailto:tcl...@li...> 
Subject: Re: [TCLCORE] Migration tool for Tcl 9

 

For the following simple test script:
namespace eval poImg {
    variable DrawPos

    set DrawPos(x) 0
    set DrawPos(y) 0
}

I get the following warnings from tcl9-migrate:
Line   4: W Variable "DrawPos(x)" possibly set without a variable or global declaration within namespace ::poImg. [UNDECLARED]
Line   5: W Variable "DrawPos(y)" possibly set without a variable or global declaration within namespace ::poImg. [UNDECLARED]

Is this a false negative?

Paul

Am 01.07.2024 um 22:42 schrieb Paul Obermeier:

Hi Ashok,

thanks for this great tool.

I found several Tcl9 related issues (esp. [RELATIVENS] and [UNDECLARED]) in my Tcl sources,
which are otherwise difficult to find.
The runtime checker works, too. 

Best regards,
Paul

Am 01.07.2024 um 09:44 schrieb apnmbx-public--- via Tcl-Core:

All,

 

I've been working on a tool to help with migration of Tcl 8 scripts to Tcl 9. It includes a static checker based on Nagelfar as well as a runtime shim that fixes up incompatibilities like i/o encoding errors, tilde expansion, package requirements etc., allowing the scripts to be loaded while emitting warnings.

 

Repository: https://github.com/apnadkarni/tcl9-migrate

Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md

Download: https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1

 

I'm looking for testers and feedback on both the tool as well as documentation (explanation of the warnings). I'd appreciate testing with your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above repository.

 

Be warned it has (and always will have) false positives and false negatives but the hope is it will nevertheless be useful in saving time.

 

Thanks

 

/Ashok





_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core






_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core

 

Re: [TCLCORE] Migration tool for Tcl 9

[<apn...@ya...>]

For now, to check a tree, you can pass multiple glob arguments, e.g. 

Tclsh migrate.tcl check *.tcl */*.tcl */*/*.tcl

Etc. Not ideal but not sure when I'll have time for feature enhancements (as opposed to bug fixes)

/Ashok

-----Original Message-----
From: Harald Oehlmann <har...@el...> 
Sent: Monday, July 1, 2024 1:37 PM
To: tcl...@li...
Subject: Re: [TCLCORE] Migration tool for Tcl 9

Hi Ashok,
impressive revolutionary work, I appreciate !
I tried the static checker on my code base and it gives good comments:
- finds variables in namespaces
- finds non-ascii encodings
I did not try the run-time stuff jet.

I would appreciate to have an option to check that check descends to 
subfolders, so a whole tree may be checked at once.

Thanks for all,
Harald

Am 01.07.2024 um 09:48 schrieb apnmbx-public--- via Tcl-Core:
> Sorry, forgot to mention – works best with recent trunk builds.
> 
> *From:*apnmbx-public--- via Tcl-Core <tcl...@li...>
> *Sent:* Monday, July 1, 2024 1:14 PM
> *To:* tcl...@li...
> *Subject:* [TCLCORE] Migration tool for Tcl 9
> 
> All,
> 
> I've been working on a tool to help with migration of Tcl 8 scripts to 
> Tcl 9. It includes a static checker based on Nagelfar as well as a 
> runtime shim that fixes up incompatibilities like i/o encoding errors, 
> tilde expansion, package requirements etc., allowing the scripts to be 
> loaded while emitting warnings.
> 
> Repository: https://github.com/apnadkarni/tcl9-migrate 
> <https://github.com/apnadkarni/tcl9-migrate>
> 
> Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md 
> <https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md>
> 
> Download: https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1 
> <https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1>
> 
> I'm looking for testers and feedback on both the tool as well as 
> documentation (explanation of the warnings). I'd appreciate testing with 
> your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above 
> repository.
> 
> Be warned it has (and always will have) false positives and false 
> negatives but the hope is it will nevertheless be useful in saving time.
> 
> Thanks
> 
> /Ashok

Re: [TCLCORE] Migration tool for Tcl 9

[<apn...@ya...>]

Yes, that would seem to be a false negative. Does not seem to account for reference to array elements when the array variable is declared. Should be fixable..I think.

 

Thanks for the feedback

 

From: Paul Obermeier <pa...@po...> 
Sent: Tuesday, July 2, 2024 3:14 AM
To: tcl...@li...
Subject: Re: [TCLCORE] Migration tool for Tcl 9

 

For the following simple test script:
namespace eval poImg {
    variable DrawPos

    set DrawPos(x) 0
    set DrawPos(y) 0
}

I get the following warnings from tcl9-migrate:
Line   4: W Variable "DrawPos(x)" possibly set without a variable or global declaration within namespace ::poImg. [UNDECLARED]
Line   5: W Variable "DrawPos(y)" possibly set without a variable or global declaration within namespace ::poImg. [UNDECLARED]

Is this a false negative?

Paul

Am 01.07.2024 um 22:42 schrieb Paul Obermeier:

Hi Ashok,

thanks for this great tool.

I found several Tcl9 related issues (esp. [RELATIVENS] and [UNDECLARED]) in my Tcl sources,
which are otherwise difficult to find.
The runtime checker works, too. 

Best regards,
Paul

Am 01.07.2024 um 09:44 schrieb apnmbx-public--- via Tcl-Core:

All,

 

I've been working on a tool to help with migration of Tcl 8 scripts to Tcl 9. It includes a static checker based on Nagelfar as well as a runtime shim that fixes up incompatibilities like i/o encoding errors, tilde expansion, package requirements etc., allowing the scripts to be loaded while emitting warnings.

 

Repository: https://github.com/apnadkarni/tcl9-migrate

Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md

Download: https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1

 

I'm looking for testers and feedback on both the tool as well as documentation (explanation of the warnings). I'd appreciate testing with your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above repository.

 

Be warned it has (and always will have) false positives and false negatives but the hope is it will nevertheless be useful in saving time.

 

Thanks

 

/Ashok






_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core







_______________________________________________
Tcl-Core mailing list
Tcl...@li... <mailto:Tcl...@li...> 
https://lists.sourceforge.net/lists/listinfo/tcl-core

 

Re: [TCLCORE] Migration tool for Tcl 9

[Paul O. <pa...@po...>]

For the following simple test script:
namespace eval poImg {
     variable DrawPos

     set DrawPos(x) 0
     set DrawPos(y) 0
}

I get the following warnings from tcl9-migrate:
Line   4: W Variable "DrawPos(x)" possibly set without a variable or global declaration within namespace ::poImg. [UNDECLARED]
Line   5: W Variable "DrawPos(y)" possibly set without a variable or global declaration within namespace ::poImg. [UNDECLARED]

Is this a false negative?

Paul

Am 01.07.2024 um 22:42 schrieb Paul Obermeier:
> Hi Ashok,
>
> thanks for this great tool.
>
> I found several Tcl9 related issues (esp. [RELATIVENS] and [UNDECLARED]) in my Tcl sources,
> which are otherwise difficult to find.
> The runtime checker works, too.
>
> Best regards,
> Paul
>
> Am 01.07.2024 um 09:44 schrieb apnmbx-public--- via Tcl-Core:
>>
>> All,
>>
>> I've been working on a tool to help with migration of Tcl 8 scripts to Tcl 9. It includes a static checker based on Nagelfar as well as a runtime shim that fixes up incompatibilities like i/o encoding errors, tilde expansion, package requirements etc., allowing the scripts to be loaded while emitting warnings.
>>
>> Repository: https://github.com/apnadkarni/tcl9-migrate
>>
>> Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md
>>
>> Download: https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1
>>
>> I'm looking for testers and feedback on both the tool as well as documentation (explanation of the warnings). I'd appreciate testing with your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above repository.
>>
>> Be warned it has (and always will have) false positives and false negatives but the hope is it will nevertheless be useful in saving time.
>>
>> Thanks
>>
>> /Ashok
>>
>>
>>
>> _______________________________________________
>> Tcl-Core mailing list
>> Tcl...@li...
>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] CFV TIP #699: Eliminate encoding alias "binary"...

[Jan N. <jan...@gm...>]

Op do 27 jun 2024 om 10:58 schreef Jan Nijtmans:
> The voting period will finish Monday July 1,
> noon UTC, [clock format 1719835200].

The voting period ended, here is the result:

Vote-Summary:   Accepted 5/0/3
Votes-For:      AN, HO, JN, KW, MC
Votes-Against:  None
Votes-Present:  BG, SL, RA

This means the TIP is accepted, and now
available in core-8-branch ("chan isbinary"
function only) and trunk.

Thanks, all, for voting!
      Jan Nijtmans

Re: [TCLCORE] Migration tool for Tcl 9

[Paul O. <pa...@po...>]

Hi Ashok,

thanks for this great tool.

I found several Tcl9 related issues (esp. [RELATIVENS] and [UNDECLARED]) in my Tcl sources,
which are otherwise difficult to find.
The runtime checker works, too.

Best regards,
Paul

Am 01.07.2024 um 09:44 schrieb apnmbx-public--- via Tcl-Core:
>
> All,
>
> I've been working on a tool to help with migration of Tcl 8 scripts to Tcl 9. It includes a static checker based on Nagelfar as well as a runtime shim that fixes up incompatibilities like i/o encoding errors, tilde expansion, package requirements etc., allowing the scripts to be loaded while emitting warnings.
>
> Repository: https://github.com/apnadkarni/tcl9-migrate
>
> Docs: https://github.com/apnadkarni/tcl9-migrate/blob/main/README.md
>
> Download: https://github.com/apnadkarni/tcl9-migrate/releases/tag/v0.1
>
> I'm looking for testers and feedback on both the tool as well as documentation (explanation of the warnings). I'd appreciate testing with your Tcl 8 (or ported to Tcl 9) code bases. Please log tickets in above repository.
>
> Be warned it has (and always will have) false positives and false negatives but the hope is it will nevertheless be useful in saving time.
>
> Thanks
>
> /Ashok
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

[TCLCORE] Tklib 0.8 Release Candidate

[Andreas K. <and...@gm...>]

Hello all.

Working on the 0.8 release of Tklib a release candidate is now available at:

  https://core.tcl-lang.org/tklib/zip/tklib-0.8rc.zip?uuid=tklib-0-8-rc
  https://core.tcl-lang.org/tklib/tarball/tklib-0.8rc.tar.gz?uuid=tklib-0-8-rc

More details about the release are at

  https://core.tcl-lang.org/tklib/doc/tklib-0-8-rc/support/releases/history/REA
DME-0.8.md
  https://core.tcl-lang.org/tklib/doc/tklib-0-8-rc/support/releases/history/REA
DME-0.8.txt

The release work happens on the
[tklib-0-8-rc](https://core.tcl-lang.org/tklib/timeline?r=tklib-0-8-rc)
branch of the
[tklib](https://core.tcl-lang.org/tklib)
repository.

While work was done to make Tklib compatible with Tcl 9 the recently released
migration checker was not applied yet.

-- 
Happy Tcling,
	Andreas Kupries <and...@gm...>
			<https://core.tcl-lang.org/akupries/>
			<https://akupries.tclers.tk/>
	Developer @	SUSE Software Solutions Germany GmbH
-------------------------------------------------------------------------------

Re: [TCLCORE] Tcl_Obj types and tkinter

[Harald O. <har...@el...>]

Am 01.07.2024 um 17:13 schrieb Marc Culler:
>     The Tcl_GetNumberFromObj() function knows about
>     doubles, integers and bignums, so you can determine
>     the type with only a single function call. If the string
>     is already parsed, Tcl_GetNumberFromObj() won't
>     do it again.
> 
> 
> Unfortunately, that particular efficiency is not so useful, since the 
> conversion from Tcl_Obj to Python object only happens once anyway.  It 
> seems possible that the authors of tkinter felt that comparing pointers 
> and having a generic fallback if the typePtr is NULL would be much 
> faster then calling Tcl_GetXXXXFromObj for all possible object types.  
> Also, the current tkinter code would work fine with no changes if all of 
> the core object types were registered.

Marc,
thank you for the great discussion. I hope, we will get some answers.
May I comment the efficiency? This efficiency is also valid, if there is 
only one call. But that is not your problem.
You may also add to the ticket cited by Sergey if you think, it is 
appropriate.

Take care and thank you for all,
Harald

Re: [TCLCORE] Tcl_Obj types and tkinter

[Dipl. I. S. G. B. <se...@us...>]

 

01.07.2024 16:43, Jan Nijtmans wrote: 

> Op ma 1 jul 2024 om 15:49 schreef Marc Culler:
> 
>> Here are two simple ways to remove this unnecessary complication for tkinter as well as for other embedded Tcl interpreters: 1) Move the block of declarations reprinted at the bottom of this message from tclInt.h to tcl.h so that tkinter could include them without including internal Tcl headers.
> 
> Unfortunately, that wouldn't work with systems like Windows and AIX.
> I will spare you the details .....
> 
>> 2) Register *all* of the core types declared in that block of declarations in TclInitObjSubsystem. (In Tcl 9 the missing ones are: "bignum", "boolean", "exprcode", "int", "index", and "ensembleCommand"; obviously int and boolean are very common as returned values of commands.)
> 
> Let's add one more possible solution:
> 3) Use Tcl_GetNumberFromObj()
> (https://core.tcl-lang.org/tips/doc/trunk/tip/638.md [1])

I already added few thoughts about that in [b5bd08df8d61b4ea] [2].
Also an explanation why Tcl_GetNumberFromObj is not really an
alternative.
The removal of registration for types is definitely a conceptional error
(not always it is possible to use tclInt.h and the trick with create
object, obtain type, delete object is a workaround in my opinion and can
be used only if there is no another possibility). 

>> Is there any reason why neither of these simple remedies would be available to make Tk 9 more accessible to its most important clients?
> 
> The reason is that the only reliable way to know the type
> is to parse the string. As long as that isn't done, the
> objType field will be NULL, there is no way to tell
> what the 'real' type is.

No, there are several ways to know that. For instance an object with
type Int, is ALWAYS an integer... emphasis on KNOWN TYPE, so I
intentionally not consider the situation that the type is unknown or
something else (string).
Often it is enough to know positive case (object is already an int), to
do or vice versa not to do something. 

Regards,
Sergey. 

Links:
------
[1] https://core.tcl-lang.org/tips/doc/trunk/tip/638.md
[2] https://core.tcl-lang.org/tcl/info/b5bd08df8d61b4ea

Re: [TCLCORE] Tcl_Obj types and tkinter

[Marc C. <cul...@gm...>]

On Mon, Jul 1, 2024 at 9:44 AM Jan Nijtmans <jan...@gm...> wrote:

> Unfortunately, that wouldn't work with systems like Windows and AIX.
> I will spare you the details .....
>

OK. Thanks.  Let's forget about that.


> Let's add one more possible solution:
> 3) Use Tcl_GetNumberFromObj()
> (https://core.tcl-lang.org/tips/doc/trunk/tip/638.md)
>

I will test that in tkinter.  You list it as option 3 but, if I understand
you correctly, this option is different from 1 and 2 because it is entirely
on the client side.  Correct?)

The reason is that the only reliable way to know the type
> is to parse the string. As long as that isn't done, the
> objType field will be NULL, there is no way to tell
> what the 'real' type is.
>

The way that tkinter currently handles a NULL objType field is it returns a
Python object which is a wrapper for a generic Tcl_Obj.
The wrapper object can always be converted to a string, because it has the
__str__ method.  However, it cannot be used as a Python object of the
expected type.  For example, the following code generates an exception
because the size of a font is no longer a number if tkinter is built
against Tk 9:
>>> import tkinter
>>> root = tkinter.Tk()
>>> from tkinter import font
>>> f = font.nametofont('TkDefaultFont')
>>> abs(f['size']) #### This raises an exception

That is the paradigm behind most of the breakage with tkinter and Tk 9.  It
was not an issue with Tk 8 because the "int" type was registered, and the
tkinter code would cache its address by calling Tcl_GetObjType and then use
the cached address to identify an "int" TclObj by comparing its typePtr
with the cached value.

The Tcl_GetNumberFromObj() function knows about
> doubles, integers and bignums, so you can determine
> the type with only a single function call. If the string
> is already parsed, Tcl_GetNumberFromObj() won't
> do it again.
>

Unfortunately, that particular efficiency is not so useful, since the
conversion from Tcl_Obj to Python object only happens once anyway.  It
seems possible that the authors of tkinter felt that comparing pointers and
having a generic fallback if the typePtr is NULL would be much faster then
calling Tcl_GetXXXXFromObj for all possible object types.  Also, the
current tkinter code would work fine with no changes if all of the core
object types were registered.

So registering all of the core types would seem to be the most effective
remedy from many different points of view.  Let me now rephrase my question:

Is there any reason not to register all of the core object types, or at
least all of the numerical core object types?

While I wait for the answer I will open a ticket and create a bugfix branch
that makes that change.

- Marc