tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] New tip #722

[Schelte B. <tc...@tc...>]

On 20/05/2025 04:34, apnmbx-public--- via Tcl-Core wrote:
> Could someone with the right privileges please give TIP
> repository write access to Christian?

Done.


Schelte.

Re: [TCLCORE] Next bi-weekly meeting 19th May 2025 at 12:00 UTC

[<apn...@ya...>]

Brian,

 

Thanks for taking a look. I will commit after a few days in case anyone else takes a look and has objections.

 

Was originally targeting core-9-0-branch since no public API’s have changed but as Jan expressed reservations yesterday in terms of risk committing to a released minor version, will target 9.1a0 instead.

 

/Ashok

 

From: Brian Griffin <bri...@ea...> 



APN mentioned the new internal list representation proposal is ready for testing as per his May 15th email to TCLCORE. BrianG will review.

 

Hi Ashok,

 

I browsed through the implementation and it seems pretty much as I expected. It looks good to me, from a high level. I didn't comb through it looking for flaws.

 

Ship It!

 

-Brian





 

Re: [TCLCORE] Next bi-weekly meeting 19th May 2025 at 12:00 UTC

[Brian G. <bri...@ea...>]

See comments below.

On May 19, 2025, at 17:08, Steve Landers <st...@di...> wrote:

Here’s some notes from the meeting

Discussion TIP 718 (https://core.tcl-lang.org/tips/doc/trunk/tip/718.md)

  *   the issue is not all applications on Windows are able to move to utf-8
  *   this is generally recognised by those who care about encoding on Windows
  *   there was discussion on the relationship between TIP 718 and TIP 716 (https://core.tcl-lang.org/tips/doc/trunk/tip/716.md)
  *   do we delay Tcl 9.0.2 or delay TIPs 716/718
  *   it was recognised that breakage now is better than breakage later
  *   DGP leans towards delay so we can better understand the problem before making changes
  *   we don't want to go down a bad path
  *   options are 716 vs 718 vs status quo
  *   Ashok will email TCLCORE outlining the situation

Discussion re TIP 720 Updated Bytecode Opcodes (https://core.tcl-lang.org/tips/doc/trunk/tip/720.md)

  *   people familiar with BCE (ByteCode Engine) are comfortable
  *   Rolf raised concern about tclcompiler  and tbcload
  *   we need to find the "official" repository for these
  *   these need to be brought up to date with 9.0 then 9.1 + 720
  *   DGP flagged there may be opportunities for sponsored work given there are commercial organisations relying on these components

Documentation

  *   there is still strong desire to keep the documentation program moving forward, including markdown, nroff and html
  *   SteveL will shake the trees

APN mentioned the new internal list representation proposal is ready for testing as per his May 15th email to TCLCORE. BrianG will review.

Hi Ashok,

I browsed through the implementation and it seems pretty much as I expected. It looks good to me, from a high level. I didn't comb through it looking for flaws.

Ship It!

-Brian


DGP mentioned converting mpexpr to Tcl9 (https://core.tcl-lang.org/mpexpr/timeline).

RMax asked about getting a review of his new socket implementation [https://chiselapp.com/user/rmax/repository/sock/]

  *   he is especially interested in comments or suggestions on how to translate between symbolic names and their values. Schelte suggested that he may be able to use constant variables (tip #677). One possible limitation is that you can't have an array of constant variables.
  *   as a result of previous debates/pushback about UDP the implementation is not based on the channel abstraction but instead data is passed as a binary stringg
  *   while currently an extension, it is easily adaptable for the core at some future point. RMax is looking for help/guidance in adding a channel wrapper from some channel expert. Jan volunteered and Andreas (not present) is likely the most knowledgeable.

See you all on June 2 for the next meeting

-- Steve
On 19 May 2025 at 9:27 AM +0800, Steve Landers <st...@di...>, wrote:
On behalf of Harald (who is unavailable) I’d like to invite anyone interested to the bi-weekly Tcl/TK telco on Monday May 19th 2025 at 12:00 UTC via https://meet.jit.si/TclMonthlyMeetup

The bi-weekly meetings are designed to keep Tcl/Tk development moving forwards towards regular releases, allowing discussion and clarification of any issues. Agenda items for this meeting will include:

  *   the releases planned for this month - 9.0.2 and 9.1a0 (the first release of Tcl 9.1 features) as per TIP #713 https://core.tcl-lang.org/tips/doc/trunk/tip/713.md
  *   TIPs being finalised and voted upon
  *   documentation

plus anything else attendees might want to raise.

Thanks to all

-- Steve
_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core
_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] CFV warning: TIP #718: encoding compatibility for GDI/HammerDB/TPC/DB2

[<apn...@ya...>]

As promised in yesterday's meeting, I'll write up the current status w.r.t
the issues addressed by TIP's 716 and 718 but that's a separate post. This
is just a response to Jan's email below.

While Jan suggests separate TIP's, my belief is that TIP's in general should
be directed at a specific problem to be solved in its entirety as opposed to
split into fragmented TIP's that at the end of the day may or may not work
well together to completely address the issue.

In the case of 716, the issue is one of sharing data between processes or
libraries without a formal means of agreeing on the encoding. To fix this,

- UTF-8 encoding should not be forced onto libraries that do not support
UTF-8

- Since some external applications always use UTF-8 (git e.g.) while others
(Windows system command line programs like findstr, cacls etc.) use user
encoding, there needs to be a way for exec to handle both. Thus the
-encoding option is proposed.

- Since 9.0, UTF-8 is the default so that case is covered. For the Windows
system command line programs, there is currently (in 9.0) no way for Tcl
applications to know what encoding would be appropriate. Thus the [encoding
user] command to retrieve the user setting as used by those programs.

TIP 716 therefore covers all the above.

As an aside, the specification in TIP 716 is just over half a page so I am
not sure why it would be termed too big! The rest of the TIP is just
explanations and background material for folks who do not have prior
exposure to the discussion.

With respect to the -encoding option for exec not being worked out, that is
not the case. I was waiting for folks to express an opinion about allowing
it to be set via an env variable but since there have been no comments in
that regard, I will stick with the current implementation.

In any case, at the end of the day all the above is ancillary to the core
issue and most important difference between 716 and 718. That's for a follow
up post.


/Ashok

-----Original Message-----
From: Jan Nijtmans <jan...@gm...> 
Sent: Monday, May 19, 2025 1:59 PM
To: Tcl Core List <tcl...@li...>
Subject: [TCLCORE] CFV warning: TIP #718: encoding compatibility for
GDI/HammerDB/TPC/DB2

This is a CFV warning for TIP #718. for Tcl 9.0.2+:
    encoding compatibility for GDI/HammerDB/TPC/DB2
    <https://core.tcl-lang.org/tips/doc/trunk/tip/718.md>

This TIP is meant as a subset of TIP #716 (with a few modifications). In my
opinion TIP #716 is too big, it should have been split in - at least - 4
parts.
  1) bugfix, see <https://core.tcl-lang.org/tcl/tktview/8ffd8cabd1>
         this is a no-brainer, already committed.
  2) Remove UTF-8 from manifest. I don't agree with that, but I
      agree with providing a tclsh90c.exe with a different setting.
  3) New command "encoding user". Good idea, so I took it
      over in TIP #718 (slightly modified, the TIP explains why)
  4) -encoding option for "exec". Sounds OK to me, but TIP #716
      still contains too many questions about this feature. That
      tells me it's not sufficiently worked out yet.

Since "encoding user" is a good idea, we can already do something
useful now, in time for Tcl 9.0.2 (I hope). I did my best to describe
everything in the TIP text.

Regards,
         Jan Nijtmans


_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] New tip #722

[<apn...@ya...>]

Update done. (Could someone with the right privileges please give TIP
repository write access to Christian?)

Since we are tinkering with [package names], I'd like to point out that the
term "available" is a little misleading, even in the current manpage.
Currently, [package names] only returns names *currently* known to Tcl, not
those available on disk. One has to do the completely non-intuitive

catch {package require nosuchpackage}; # Force a full search along path
package names

to retrieve the packages available to be loaded.

So the question is,

- will [package names available] continue the same behavior of not actually
searching the package paths ?
- if it does not do a full search, can we add a [package names refresh] that
will replace the (ugly imo) catch {} ?
- if it does do a full search, [package names] as currently implemented is
no longer the union of [package names loaded] and [package names available]

I would actually prefer [package names] to do a full search though that
would be differ from current behavior. I'm not sure why it did not do that
in the first place, presumably performance, though I wonder if performance
is really an issue in situations where [package names] is used.

/Ashok

-----Original Message-----
From: Christian Werner <Chr...@t-...> 
Good point, Don. So may I ask Ashok again to update the TIP entry to

Re: [TCLCORE] Next bi-weekly meeting 19th May 2025 at 12:00 UTC

[Steve L. <st...@di...>]

Here’s some notes from the meeting

Discussion TIP 718 (https://core.tcl-lang.org/tips/doc/trunk/tip/718.md)

• the issue is not all applications on Windows are able to move to utf-8
• this is generally recognised by those who care about encoding on Windows
• there was discussion on the relationship between TIP 718 and TIP 716 (https://core.tcl-lang.org/tips/doc/trunk/tip/716.md)
• do we delay Tcl 9.0.2 or delay TIPs 716/718
• it was recognised that breakage now is better than breakage later
• DGP leans towards delay so we can better understand the problem before making changes
• we don't want to go down a bad path
• options are 716 vs 718 vs status quo
• Ashok will email TCLCORE outlining the situation

Discussion re TIP 720 Updated Bytecode Opcodes (https://core.tcl-lang.org/tips/doc/trunk/tip/720.md)

• people familiar with BCE (ByteCode Engine) are comfortable
• Rolf raised concern about tclcompiler  and tbcload
• we need to find the "official" repository for these
• these need to be brought up to date with 9.0 then 9.1 + 720
• DGP flagged there may be opportunities for sponsored work given there are commercial organisations relying on these components

Documentation

• there is still strong desire to keep the documentation program moving forward, including markdown, nroff and html
• SteveL will shake the trees

APN mentioned the new internal list representation proposal is ready for testing as per his May 15th email to TCLCORE. BrianG will review.

DGP mentioned converting mpexpr to Tcl9 (https://core.tcl-lang.org/mpexpr/timeline).

RMax asked about getting a review of his new socket implementation [https://chiselapp.com/user/rmax/repository/sock/]

• he is especially interested in comments or suggestions on how to translate between symbolic names and their values. Schelte suggested that he may be able to use constant variables (tip #677). One possible limitation is that you can't have an array of constant variables.
• as a result of previous debates/pushback about UDP the implementation is not based on the channel abstraction but instead data is passed as a binary stringg
• while currently an extension, it is easily adaptable for the core at some future point. RMax is looking for help/guidance in adding a channel wrapper from some channel expert. Jan volunteered and Andreas (not present) is likely the most knowledgeable.

See you all on June 2 for the next meeting

-- Steve
On 19 May 2025 at 9:27 AM +0800, Steve Landers <st...@di...>, wrote:
> On behalf of Harald (who is unavailable) I’d like to invite anyone interested to the bi-weekly Tcl/TK telco on Monday May 19th 2025 at 12:00 UTC via https://meet.jit.si/TclMonthlyMeetup
>
> The bi-weekly meetings are designed to keep Tcl/Tk development moving forwards towards regular releases, allowing discussion and clarification of any issues. Agenda items for this meeting will include:
>
> • the releases planned for this month - 9.0.2 and 9.1a0 (the first release of Tcl 9.1 features) as per TIP #713 https://core.tcl-lang.org/tips/doc/trunk/tip/713.md
> • TIPs being finalised and voted upon
> • documentation
>
> plus anything else attendees might want to raise.
>
> Thanks to all
>
> -- Steve
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] New tip #722

[Christian W. <Chr...@t-...>]

On 05/19/2025 05:24 PM, Donald G Porter via Tcl-Core wrote:
> On 5/19/25 11:10, Christian Werner wrote:
>> On 5/19/25 14:31, Christian Werner wrote:

>>  …
>> Massimo Manghi gave me his proposal "package names -loaded" which IMO is
>> a very good choice as it smoothly fits into the current implementation of
>> "package names". So the "-loaded" option simply modifies the if-condition
>> whose block appends to the result list. Streamlined.
>
> The existing command is [package names] with no arguments accepted.
>
> What do you think of adding two optional argument values:
>
> [package names provided] -- return names of packages provided in the current interp.
>
> [package names available] -- return names of packages where some "ifneeded" script has been found.
>
> and
>
> [package names] -- continue to return the union of the lists described above.
>
> This reuses the "provide" verbiage already part of [package], and avoids confusion
> with the "-loaded" term that better belongs with [load], etc.

Good point, Don. So may I ask Ashok again to update the TIP entry to

---8><---8><---

# TIP 722: Improve **package names** subcommand
     Author:         Christian Werner <und...@gm...>
     Tcl-Version:    9.1
     State:          Draft
     Type:           Project
     Created:        19-May-2025
     Keywords:       Tcl,package
-----

# Abstract

This TIP proposes to improve the **package names** subcommand such
that its result list can be filtered with respect to packages which are
provided (i.e. present) and packages which are not provided,
but for which a **package ifneeded** script is available.

# Rationale

Although the **package names** subcommand returns a list of all currently
provided and available packages there's no equivalent to retrieve only the
list of currently provided (i.e. present) packages. However, the latter can
be of use e.g. to build a software bill of materials of an application or
to help in generating a package dependency graph.

# Specification

For **package names** without any further arguments (current default
mode of operation) the result list is made up of package names which are
provided or for which a **package ifneeded** script is available.

For **package names available** the list contains only the packages
which are not provided (i.e. not present) but for which a **package ifneeded**
script is available.

For **package names provided** the list contains only the currently provided
(i.e. present) packages.

# Implementation

TBD.

# Copyright

This document has been placed in the public domain.

---8><---8><---

Thank you,
Christian

Re: [TCLCORE] New tip #722

[Donald G P. <don...@ni...>]

On 5/19/25 11:10, Christian Werner wrote:
> On 5/19/25 14:31, Christian Werner wrote:
>> On 05/19/2025 01:58 PM, Donald G Porter via Tcl-Core wrote:
>>
>>> Revising the behavior of an existing subcommand raises worries ( at
>>> least )
>>> about compatibility.
>>>
>>> Consider proposing a new subcommand instead.
>>
>> So "package names -present", i.e. a switch to an existing subcommand
>> isn't a nogo, too?
>> But from the natural language standpoint "package present" seemed the
>> most logical to me.
>> Then dear native speakers please find a good alternative for me.
>
> Massimo Manghi gave me his proposal "package names -loaded" which IMO is
> a very good choice as it smoothly fits into the current implementation of
> "package names". So the "-loaded" option simply modifies the if-condition
> whose block appends to the result list. Streamlined.

The existing command is [package names] with no arguments accepted.

What do you think of adding two optional argument values:

[package names provided] -- return names of packages provided in the current interp.

[package names available] -- return names of packages where some "ifneeded" script has been found.

and

[package names] -- continue to return the union of the lists described above.

This reuses the "provide" verbiage already part of [package], and avoids confusion
with the "-loaded" term that better belongs with [load], etc.

-- 
| Don Porter            Applied and Computational Mathematics Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

Re: [TCLCORE] New tip #722

[Christian W. <Chr...@t-...>]

On 5/19/25 14:31, Christian Werner wrote:
> On 05/19/2025 01:58 PM, Donald G Porter via Tcl-Core wrote:
>
>> Revising the behavior of an existing subcommand raises worries ( at
>> least )
>> about compatibility.
>>
>> Consider proposing a new subcommand instead.
>
> So "package names -present", i.e. a switch to an existing subcommand
> isn't a nogo, too?
> But from the natural language standpoint "package present" seemed the
> most logical to me.
> Then dear native speakers please find a good alternative for me.

Massimo Manghi gave me his proposal "package names -loaded" which IMO is
a very good choice as it smoothly fits into the current implementation of
"package names". So the "-loaded" option simply modifies the if-condition
whose block appends to the result list. Streamlined.

BR,
Christian

Re: [TCLCORE] New tip #722

[Christian W. <Chr...@t-...>]

On 05/19/2025 01:58 PM, Donald G Porter via Tcl-Core wrote:

> Revising the behavior of an existing subcommand raises worries ( at least )
> about compatibility.
>
> Consider proposing a new subcommand instead.

So "package names -present", i.e. a switch to an existing subcommand isn't a nogo, too?
But from the natural language standpoint "package present" seemed the most logical to me.
Then dear native speakers please find a good alternative for me.

Thank you,
Christian

Re: [TCLCORE] New tip #722

[Christian W. <Chr...@t-...>]

On 05/19/2025 01:56 PM, Donald G Porter via Tcl-Core wrote:

> Tcl 8.6 is not open for new (sub)commands or features.

Then 9.1, as Ashok wisely put in the TIP index page.

BR,
Christian

Re: [TCLCORE] New tip #722

[Donald G P. <don...@ni...>]

Revising the behavior of an existing subcommand raises worries ( at least )
about compatibility.

Consider proposing a new subcommand instead.

On 5/19/25 01:40, Christian Werner wrote:
> Hello all,
>
> (posted again in case I've sent it with the wrong account)
>
> please could some tip fossil maintainer add the following tip.
>
> ---8><---8><---
>
> # TIP 722 Improve **package present** subcommand
>         Author:         Christian Werner <und...@gm...>
>         State:          Draft
>         Type:           Project
>         Created:        19-May-2025
>         Keywords:       Tcl,package
> -----
>
> # Abstract
>
> This TIP proposes to improve the **package present** subcommand such
> that it can be used to return a list of all currently loaded packages.
>
> # Rationale
>
> Although there is a **package names** subcommand which returns a list of
> all currently available packages there's no equivalent to retrieve the
> list of currently loaded packages. However, the latter can be of use
> e.g. to build a software bill of materials of an application or to help
> in generating a package dependency graph.
>
> # Specification
>
> For **package present** without any further arguments (which currently
> generates an error message) the implementation of **package names** is
> borrowed but its selection clause while iterating over the package hash
> table is changed to add package names to the result list for which a
> version is known (which means that the package has been loaded).
>
> ** Implementation
>
> The code block to add to the current **package present** implementation is:
>
> '''
> if (objc < 3) {
>     Tcl_Obj *resultObj;
>
>     TclNewObj(resultObj);
>     tablePtr = &iPtr->packageTable;
>     for (hPtr = Tcl_FirstHashEntry(tablePtr, &search); hPtr != NULL;
>              hPtr = Tcl_NextHashEntry(&search)) {
>         pkgPtr = Tcl_GetHashValue(hPtr);
>         if (pkgPtr->version != NULL) {
>             Tcl_ListObjAppendElement(NULL, resultObj, Tcl_NewStringObj(
>                     Tcl_GetHashKey(tablePtr, hPtr), -1));
>         }
>     }
>     Tcl_SetObjResult(interp, resultObj);
>     break;
> }
> '''
>
> The test code in ```package.test``` must be changed for test case 8.11 to:
>
> ```
> test package-8.11 {Tcl_PackageCmd procedure, "present" option} -body {
>     package present
> } -match glob -result *
> ```
>
> # Copyright
>
> This document has been placed in the public domain.
>
> ---8><---8><---
>
> Thank you,
> Christian
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core


-- 
| Don Porter            Applied and Computational Mathematics Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

Re: [TCLCORE] CFV warning: TIP #718: encoding compatibility for GDI/HammerDB/TPC/DB2

[Jan N. <jan...@gm...>]

Op ma 19 mei 2025 om 13:05 schreef Donald Porter:
>
> I’m not immersed in this.  I don’t know what is involved.  Have not read the TIP.  I’d like to keep it that way and let those who are more familiar sort out the details.
>
> That said, I see mention of adding new public features to Tcl 9.0.2.  This is making me care.
>
> New features in a patch release is contrary to usual practice.  Is there really a compelling need?
> A need I can comprehend without expertise in Windows encoding practices?
> Or am I misunderstanding what is proposed?

Indeed, adding an API in a patch release should be done with care. It won't
be the first time, TIP #701 dit it as well (with my agreement!):
      <https://core.tcl-lang.org/tips/doc/trunk/tip/701.md>

I think - in this case - the necessary precautions are taken, but it's
open for discussion. If found not acceptable, it can be made a
MODULE_SCOPE function (which will be exposed in 9.1)
as well.

Hope this helps,
       Jan Nijtmans

Re: [TCLCORE] New tip #722

[Donald G P. <don...@ni...>]

Tcl 8.6 is not open for new (sub)commands or features.

On 5/19/25 02:57, Christian Werner wrote:
> On 05/19/2025 08:31 AM, apnmbx-public--- via Tcl-Core wrote:
>> Done. Note tip does not mention target version.
>
> Thank you, Ashok.
>
> Target version is 8.6 and above since the proposal should work for all.
>
> Thanks again,
> Christian
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core


-- 
| Don Porter            Applied and Computational Mathematics Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

Re: [TCLCORE] CFV warning: TIP #718: encoding compatibility for GDI/HammerDB/TPC/DB2

[Donald P. <d.g...@co...>]

I’m not immersed in this.  I don’t know what is involved.  Have not read the TIP.  I’d like to keep it that way and let those who are more familiar sort out the details.

That said, I see mention of adding new public features to Tcl 9.0.2.  This is making me care.

New features in a patch release is contrary to usual practice.  Is there really a compelling need?
A need I can comprehend without expertise in Windows encoding practices?
Or am I misunderstanding what is proposed?

DGP

> On May 19, 2025, at 4:28 AM, Jan Nijtmans <jan...@gm...> wrote:
> 
> This is a CFV warning for TIP #718. for Tcl 9.0.2+:
>    encoding compatibility for GDI/HammerDB/TPC/DB2
>    <https://core.tcl-lang.org/tips/doc/trunk/tip/718.md>
> 
> This TIP is meant as a subset of TIP #716 (with a few modifications). In my
> opinion TIP #716 is too big, it should have been split in - at least - 4
> parts.
>  1) bugfix, see <https://core.tcl-lang.org/tcl/tktview/8ffd8cabd1>
>         this is a no-brainer, already committed.
>  2) Remove UTF-8 from manifest. I don't agree with that, but I
>      agree with providing a tclsh90c.exe with a different setting.
>  3) New command "encoding user". Good idea, so I took it
>      over in TIP #718 (slightly modified, the TIP explains why)
>  4) -encoding option for "exec". Sounds OK to me, but TIP #716
>      still contains too many questions about this feature. That
>      tells me it's not sufficiently worked out yet.
> 
> Since "encoding user" is a good idea, we can already do something
> useful now, in time for Tcl 9.0.2 (I hope). I did my best to describe
> everything in the TIP text.

[TCLCORE] CFV warning: TIP #718: encoding compatibility for GDI/HammerDB/TPC/DB2

[Jan N. <jan...@gm...>]

This is a CFV warning for TIP #718. for Tcl 9.0.2+:
    encoding compatibility for GDI/HammerDB/TPC/DB2
    <https://core.tcl-lang.org/tips/doc/trunk/tip/718.md>

This TIP is meant as a subset of TIP #716 (with a few modifications). In my
opinion TIP #716 is too big, it should have been split in - at least - 4
parts.
  1) bugfix, see <https://core.tcl-lang.org/tcl/tktview/8ffd8cabd1>
         this is a no-brainer, already committed.
  2) Remove UTF-8 from manifest. I don't agree with that, but I
      agree with providing a tclsh90c.exe with a different setting.
  3) New command "encoding user". Good idea, so I took it
      over in TIP #718 (slightly modified, the TIP explains why)
  4) -encoding option for "exec". Sounds OK to me, but TIP #716
      still contains too many questions about this feature. That
      tells me it's not sufficiently worked out yet.

Since "encoding user" is a good idea, we can already do something
useful now, in time for Tcl 9.0.2 (I hope). I did my best to describe
everything in the TIP text.

Regards,
         Jan Nijtmans

Re: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)

[Donal F. <don...@ma...>]

Regarding the IDEOGRAPHICSPACE bug, IsEmptyToken does not appear in any other branch than no-variable-width-instruction-issue so must have been introduced there. And fossil blame shows you as the author 😊 Perhaps you might be confusing the IsEmptyToken function hosting the bug with Tcl_IsEmpty? It’s all trivial to fix as long as we’re all on the same page as to what constitutes whitespace.
Fixed it by using the tclCharTypeTable notion of space-ness from the parser. And I've transformed your demonstration into a test case: dict-22.24 😄

Thanks. (We have a potential problem with the different notions of spaces, but not fixing that today.)

The problems with signed/unsigned are most likely to show up for 32-bit systems in the handling of negative indices for variables (which currently signal that the variable is not a local variable, and always use -1 to signal that). The sensitivity occurs in various places in tclVar.c. For 64-bit correctness, we NEED those types to be signed inside TEBC (or a world of pain with address arithmetic ensues) and yet they're 32-bit unsigned values in the bytecode itself (and have been for some time). The patch makes that all work on 64-bit systems; on 32-bit, I'm guessing we need to limit the range of LVT entries slightly (by making -1 special) and make sure that we never just check if the index is negative in the places that care.

Not a high priority fix given that it needs 2**31 distinct local variables to hit, and that's likely to hit script length limits first; I can't see how to approach that in a script whose length is less than 2**32 bytes (and thus be prevented from going further in other ways).

________________________________
From: apn...@ya... <apn...@ya...> on behalf of apn...@ya... <apn...@ya...>
Sent: Monday, May 19, 2025 03:53
To: Donal Fellows <don...@ma...>
Cc: 'Tcl Core' <tcl...@li...>
Subject: RE: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)


Donal,



Thanks for the detailed explanations.



Regarding the IDEOGRAPHICSPACE bug, IsEmptyToken does not appear in any other branch than no-variable-width-instruction-issue so must have been introduced there. And fossil blame shows you as the author 😊 Perhaps you might be confusing the IsEmptyToken function hosting the bug with Tcl_IsEmpty? It’s all trivial to fix as long as we’re all on the same page as to what constitutes whitespace.



One other comment, not directly pertaining to 720 but related, regarding the other branch no-var-width-plus-opnd-types fixing TEBCResume types. I think the use of Tcl_Size fixes type mismatches for 64-bit archs but not 32-bit. For example, fragments like



Tcl_Size numArgs;

...

numArgs = TclGetUInt4AtPtr(pc + 1);

objResultPtr = OBJ_AT_DEPTH(numArgs);



will work on 64-bit arch but run into the same unsigned->signed offset issues on 32-bit architectures. Again, to reiterate, not that it currently matters because of the script and argument length limits of INT_MAX currently in place but will probably matter with TIP 626 approval.



/Ashok



From: Donal Fellows <don...@ma...>
Sent: Monday, May 19, 2025 2:46 AM
To: apn...@ya...; 'Tcl Core' <tcl...@li...>
Subject: Re: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)



Re the compilation errors, I've no idea how they ended up in there, but they're fixed now. If there's still MSVC errors, please let me know what the error message is; I'm having problems triggering it here. (I might just make that code path depend on recent enough standard C if I can't figure out any other alternative; this will be fixed prior to merging.)



Re the change to use IsEmptyToken, it's the intention that an empty body results in nothing happening, and that if IDEOGRAPHIC SPACE is now considered a space for the purpose of that function, then the code is now behaving correctly. [dict with] is very happy to delegate that decision. If IsEmptyToken is reporting tokens as empty when they could be compiled to code with side effects, that's definitely a bug... but not with the callers of IsEmptyToken. I'm not going to probe that further; I didn't write IsEmptyToken, but I need something that says "this token definitely doesn't generate code and so definitely doesn't generate errors".



1.      Literal sharing is one of those things that I think we may have been doing a bit too much of, and which has caused unnecessary representation churn. OTOH, I'm not strongly invested in either sharing or not sharing those objects for the most part. My usage is on the basis of it being a convenient API; I have a Tcl_Obj and I want to arrange for it (or something remarkably similar) to get pushed onto the stack by the bytecode engine at that point.

2.      I believe I checked the cases where I switched the code over to bounce the refcount; they're supposed to be places where the called code may take a reference or may not (typically on error) but definitely won't be decrementing the reference count. (It's the functions with an internal Tcl_DecrRefCount that you need to watch out for.)

3.      The switch to use Tcl_ListObjReplace was indeed deliberate. It's so that appending a zero-length list works correctly, and that's necessary for the short lappend-expansion branch to work (which makes [lappend] do the right thing with {*}). Previously, [lappend foo] was an ugly special case precisely because the opcodes worked differently from the non-compiled [lappend]. So. Many. Edge. Cases.

4.      Remove the clause, run append.test and appendComp.test (I forget which one) and see something very strange with some tests. I think it might be append-7.4 that triggers it, but I'm not sure as I've not poked at it for a few weeks (that have been very busy ones for me). The code before this branch (https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=856,860 [core.tcl-lang.org]<https://urldefense.com/v3/__https://core.tcl-lang.org/tcl/file?name=generic*tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=856,860__;Lw!!PDiH4ENfjr2_Jw!FWs6btZouNOYM2NfgC9t7A6ACWzn7Bp_74tMAKU_Li-I53dfEFH7SSQQaCcN5NEWTMzfmE7m2rXuq8pVn2cfQbtgXEVAIqLHjk65$>) is also odd in this area; in a sane world, the first condition there would be be numWords<2 and the second would be without looking at the envPtr->procPtr at all (at that point; it should just be a concern for picking how to access a variable, such as in https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=897,909 [core.tcl-lang.org]<https://urldefense.com/v3/__https://core.tcl-lang.org/tcl/file?name=generic*tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=897,909__;Lw!!PDiH4ENfjr2_Jw!FWs6btZouNOYM2NfgC9t7A6ACWzn7Bp_74tMAKU_Li-I53dfEFH7SSQQaCcN5NEWTMzfmE7m2rXuq8pVn2cfQbtgXEVAIirKdEBB$> and https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=923,935 [core.tcl-lang.org]<https://urldefense.com/v3/__https://core.tcl-lang.org/tcl/file?name=generic*tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=923,935__;Lw!!PDiH4ENfjr2_Jw!FWs6btZouNOYM2NfgC9t7A6ACWzn7Bp_74tMAKU_Li-I53dfEFH7SSQQaCcN5NEWTMzfmE7m2rXuq8pVn2cfQbtgXEVAIroTFAQm$>).

5.      I'm generally supportive; the deprecated opcodes should be fairly easy to identify and remove with things as they stand. We do not generate any of them at this point, but there's still code that knows how to work with them (and execute them, of course). If backward compatibility is truly undesired, we can also remove the binding of the remaining opcodes to their current values. But technically out of scope right now. 😄



Donal.



________________________________

From: apn...@ya...<mailto:apn...@ya...> <apn...@ya...<mailto:apn...@ya...>> on behalf of apn...@ya...<mailto:apn...@ya...> <apn...@ya...<mailto:apn...@ya...>>
Sent: Saturday, May 17, 2025 17:37
To: Donal Fellows <don...@ma...<mailto:don...@ma...>>; 'Tcl Core' <tcl...@li...<mailto:tcl...@li...>>
Subject: RE: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)



Ok, having read the main components in tclComp*.c and tclExecute.c, I think I'm good to go on TIP 720. The code generation is much more readable making future development easier.



Even the minor beef I had with the INT_MAX/UINT_MAX limits is not an issue currently as two separate INT_MAX limits, one on the length of a script and the other on the number of arguments, make the point moot. The (INT_MAX..UINT_MAX] case will never actually occur afaict. At least until TIP 626 at which point it becomes Jan's problem :-)



There are some compilation errors still on Windows as mentioned earlier and also on Unix with --enable-symbols=all (works without that).



One little buglet, not actually related to TIP 720 but related to the changes in [dict with] arising from modification to using Unicode aware IsEmptyToken. Didn't check if [dict update] also had the same issue.



Tcl 9.0.1

% info patch

9.0.1

% writeFile space.tcl "proc \u3000 {} {puts IDEOGRAPHICSPACE}\n; set d \[dict create a 0];dict with d {\u3000}"

% source space.tcl

IDEOGRAPHICSPACE



TIP-720

% info patch

9.1a0

% writeFile space.tcl "proc \u3000 {} {puts IDEOGRAPHICSPACE}\n; set d \[dict create a 0];dict with d {\u3000}"

% source space.tcl

% (no output as the dict with script block is treated as an empty script)



Some more questions, as much for my own edification, as anything else…



(1) Code sequences of the form



Tcl_Size len;

const char *bytes = TclGetStringFromObj(objPtr, &len);

PushLiteral(envPtr, bytes, len)



have been replaced by



PUSH_OBJ(objPtr)



Whereas PushLiteral resolves to TclRegisterLiteral, PUSH_OBJ resolves to TclAddLiteralObj. The former adds the value to the global literal array, but the latter does not. Is this difference intentional? Since the literal is going to be in the local array anyways, doesn't it make sense to have it globally available as well for memory sharing as in 9.0.0?



(2) I also see sequences like



TclNewObj(objPtr);

Tcl_IncrRefCount(objPtr);

SomeFunction(...objPtr...);

Tcl_DecrRefCount(objPtr);



replaced by



TclNewObj(objPtr);

SomeFunction(...objPtr...);

Tcl_BounceRefCount(objPtr);



but the semantics of the above sequences are not the same as the latter does not guarantee objPtr is still accessible on return from SomeFunction. I have not looked into every such "SomeFunction" but presumably, they do not manipulate (e.g. incr+decr) the ref count of objPtr. Still, the original sequence seems safer unless all SomeFunction’s in this context are known to be safe in this respect.



(3) In the TEBCResume function, the under label lappendListPtr:, the call to TclListObjAppendElements has been replaced by Tcl_ListObjReplace. Why so? The latter is more general and therefore has to do a little more work to append. Clearly the change is intentional so I am curious as to the reason.



(4) There is the following comment somewhere in the code which I think you just added:



    /*

     * The weird cluster of bugs around INST_LAPPEND_STK without a LVT ought

     * to be sorted out. INST_LAPPEND_LIST_STK does the right thing.

     */"

What does this cluster of bugs refer to ? Are they recorded somewhere? I've worked on lappend and have not seen or noticed any. This is just my curiosity as to what I might have missed.



(5) As an RFE, not necessarily part of TIP 720, I would be tempted to expunge the deprecated opcodes from TEBC as well (not just disabled via macros). It would significantly reduce and simplify TEBCresume. I understand the potential incompatibility with respect to tbcload and possible the Tcl debugger/prodebug. However, I do not see any need to maintain 8.6 tbcload compat given we don't even maintain 8.6 compat at script or C API level. This is the time to eliminate that code (before people depend on tbcloaded 9.0 applications) else we would have to wait for 10.0. (As an aside, does tbcload work with Tcl 9? If not, is anyone even working on it?)



In summary, all looks good as far as I am concerned once the compiler errors with --enable-symbols=all are fixed.



/Ashok







From: Donal Fellows <don...@ma...<mailto:don...@ma...>>
Sent: Friday, May 9, 2025 5:47 PM
To: Tcl Core <tcl...@li...<mailto:tcl...@li...>>
Subject: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)



As promised, I've done a TIP for the updates to the bytecode compiler. It's largely there to document the notable features of a substantial changeset.



https://core.tcl-lang.org/tips/doc/trunk/tip/720.md [core.tcl-lang.org]<https://urldefense.com/v3/__https:/core.tcl-lang.org/tips/doc/trunk/tip/720.md__;!!PDiH4ENfjr2_Jw!ALVTJ_a_6mdCvhJUS8w77mC9_5peSjdi7rDWwxWMs-J5kkF5N-WFyVR3RhRASZXkoY05xtNSr7c8mGrrzoP0wC2pvNCkBDRi034Z$>



It covers the essentials; of particular note is that no true public API is affected. That said, tcl::unsupported::assemble is affected, but via enhancement (it gets new opcodes), and tdkcompiler/tbcload will definitely need more work due to the new auxiliary structure type (for a numeric jump table).



Given the lack of public surface changes - if you're not messing with compilation guts, you shouldn't need to care - I'm going to call a vote on this TIP immediately.



Votes to here by [clock format 1747393200] (Fri May 16 12:00:00 BST 2025) in the usual form.



TIP 720: YES



Donal.

Re: [TCLCORE] New tip #722

[Christian W. <Chr...@t-...>]

On 05/19/2025 08:31 AM, apnmbx-public--- via Tcl-Core wrote:
> Done. Note tip does not mention target version.

Thank you, Ashok.

Target version is 8.6 and above since the proposal should work for all.

Thanks again,
Christian

Re: [TCLCORE] New tip #722

[<apn...@ya...>]

Done. Note tip does not mention target version. 

Yahoo Mail: Search, organise, conquer 
 
  On Mon, 19 May 2025 at 11:11 am, Christian Werner<Chr...@t-...> wrote:   Hello all,

(posted again in case I've sent it with the wrong account)

please could some tip fossil maintainer add the following tip.

---8><---8><---

# TIP 722 Improve **package present** subcommand
        Author:        Christian Werner <und...@gm...>
        State:          Draft
        Type:          Project
        Created:        19-May-2025
        Keywords:      Tcl,package
-----

# Abstract

This TIP proposes to improve the **package present** subcommand such
that it can be used to return a list of all currently loaded packages.

# Rationale

Although there is a **package names** subcommand which returns a list of
all currently available packages there's no equivalent to retrieve the
list of currently loaded packages. However, the latter can be of use
e.g. to build a software bill of materials of an application or to help
in generating a package dependency graph.

# Specification

For **package present** without any further arguments (which currently
generates an error message) the implementation of **package names** is
borrowed but its selection clause while iterating over the package hash
table is changed to add package names to the result list for which a
version is known (which means that the package has been loaded).

** Implementation

The code block to add to the current **package present** implementation is:

'''
if (objc < 3) {
    Tcl_Obj *resultObj;

    TclNewObj(resultObj);
    tablePtr = &iPtr->packageTable;
    for (hPtr = Tcl_FirstHashEntry(tablePtr, &search); hPtr != NULL;
              hPtr = Tcl_NextHashEntry(&search)) {
        pkgPtr = Tcl_GetHashValue(hPtr);
        if (pkgPtr->version != NULL) {
            Tcl_ListObjAppendElement(NULL, resultObj, Tcl_NewStringObj(
                    Tcl_GetHashKey(tablePtr, hPtr), -1));
        }
    }
    Tcl_SetObjResult(interp, resultObj);
    break;
}
'''

The test code in ```package.test``` must be changed for test case 8.11 to:

```
test package-8.11 {Tcl_PackageCmd procedure, "present" option} -body {
    package present
} -match glob -result *
```

# Copyright

This document has been placed in the public domain.

---8><---8><---

Thank you,
Christian



_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core
  

[TCLCORE] New tip #722

[Christian W. <Chr...@t-...>]

Hello all,

(posted again in case I've sent it with the wrong account)

please could some tip fossil maintainer add the following tip.

---8><---8><---

# TIP 722 Improve **package present** subcommand
         Author:         Christian Werner <und...@gm...>
         State:          Draft
         Type:           Project
         Created:        19-May-2025
         Keywords:       Tcl,package
-----

# Abstract

This TIP proposes to improve the **package present** subcommand such
that it can be used to return a list of all currently loaded packages.

# Rationale

Although there is a **package names** subcommand which returns a list of
all currently available packages there's no equivalent to retrieve the
list of currently loaded packages. However, the latter can be of use
e.g. to build a software bill of materials of an application or to help
in generating a package dependency graph.

# Specification

For **package present** without any further arguments (which currently
generates an error message) the implementation of **package names** is
borrowed but its selection clause while iterating over the package hash
table is changed to add package names to the result list for which a
version is known (which means that the package has been loaded).

** Implementation

The code block to add to the current **package present** implementation is:

'''
if (objc < 3) {
     Tcl_Obj *resultObj;

     TclNewObj(resultObj);
     tablePtr = &iPtr->packageTable;
     for (hPtr = Tcl_FirstHashEntry(tablePtr, &search); hPtr != NULL;
              hPtr = Tcl_NextHashEntry(&search)) {
         pkgPtr = Tcl_GetHashValue(hPtr);
         if (pkgPtr->version != NULL) {
             Tcl_ListObjAppendElement(NULL, resultObj, Tcl_NewStringObj(
                     Tcl_GetHashKey(tablePtr, hPtr), -1));
         }
     }
     Tcl_SetObjResult(interp, resultObj);
     break;
}
'''

The test code in ```package.test``` must be changed for test case 8.11 to:

```
test package-8.11 {Tcl_PackageCmd procedure, "present" option} -body {
     package present
} -match glob -result *
```

# Copyright

This document has been placed in the public domain.

---8><---8><---

Thank you,
Christian

[TCLCORE] New tip #722

[Christian W. <und...@gm...>]

Hello all,

please could some tip fossil maintainer add the following tip.

---8><---8><---

# TIP 722 Improve **package present** subcommand
         Author:         Christian Werner <und...@gm...>
         State:          Draft
         Type:           Project
         Created:        19-May-2025
         Keywords:       Tcl,package
-----

# Abstract

This TIP proposes to improve the **package present** subcommand such 
that it can be used to return a list of all currently loaded packages.

# Rationale

Although there is a **package names** subcommand which returns a list of 
all currently available packages there's no equivalent to retrieve the 
list of currently loaded packages. However, the latter can be of use 
e.g. to build a software bill of materials of an application or to help 
in generating a package dependency graph.

# Specification

For **package present** without any further arguments (which currently 
generates an error message) the implementation of **package names** is 
borrowed but its selection clause while iterating over the package hash 
table is changed to add package names to the result list for which a 
version is known (which means that the package has been loaded).

** Implementation

The code block to add to the current **package present** implementation is:

'''
if (objc < 3) {
     Tcl_Obj *resultObj;

     TclNewObj(resultObj);
     tablePtr = &iPtr->packageTable;
     for (hPtr = Tcl_FirstHashEntry(tablePtr, &search); hPtr != NULL;
              hPtr = Tcl_NextHashEntry(&search)) {
         pkgPtr = Tcl_GetHashValue(hPtr);
         if (pkgPtr->version != NULL) {
             Tcl_ListObjAppendElement(NULL, resultObj, Tcl_NewStringObj(
                     Tcl_GetHashKey(tablePtr, hPtr), -1));
         }
     }
     Tcl_SetObjResult(interp, resultObj);
     break;
}
'''

The test code in ```package.test``` must be changed for test case 8.11 to:

```
test package-8.11 {Tcl_PackageCmd procedure, "present" option} -body {
     package present
} -match glob -result *
```

# Copyright

This document has been placed in the public domain.

---8><---8><---

Thank you,

Christian

Re: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)

[Steve L. <st...@di...>]

I’m conflicted on this one.   The intent is good and (as Donal said in his original email) it arguable doesn’t need a TIP. On the other hand I don’t understand the details of the implementation and don’t have the time nor energy to come up to speed on the bytecode engine.  Having said all of that, I have followed the discussion with interest and wouldn’t stand in the way.

So do I vote PRESENT or YES?    If this was a user-visible or breaking change I’d make it PRESENT but on this occasion I’ll make it ....

TIP #720:  YES

-- Steve
On 19 May 2025 at 10:55 AM +0800, apnmbx-public--- via Tcl-Core <tcl...@li...>, wrote:
> Donal,
>
> Thanks for the detailed explanations.
>
> Regarding the IDEOGRAPHICSPACE bug, IsEmptyToken does not appear in any other branch than no-variable-width-instruction-issue so must have been introduced there. And fossil blame shows you as the author 😊 Perhaps you might be confusing the IsEmptyToken function hosting the bug with Tcl_IsEmpty? It’s all trivial to fix as long as we’re all on the same page as to what constitutes whitespace.
>
> One other comment, not directly pertaining to 720 but related, regarding the other branch no-var-width-plus-opnd-types fixing TEBCResume types. I think the use of Tcl_Size fixes type mismatches for 64-bit archs but not 32-bit. For example, fragments like
>
> Tcl_Size numArgs;
> ...
> numArgs = TclGetUInt4AtPtr(pc + 1);
> objResultPtr = OBJ_AT_DEPTH(numArgs);
>
> will work on 64-bit arch but run into the same unsigned->signed offset issues on 32-bit architectures. Again, to reiterate, not that it currently matters because of the script and argument length limits of INT_MAX currently in place but will probably matter with TIP 626 approval.
>
> /Ashok
>
> From: Donal Fellows <don...@ma...>
> Sent: Monday, May 19, 2025 2:46 AM
> To: apn...@ya...; 'Tcl Core' <tcl...@li...>
> Subject: Re: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)
>
> Re the compilation errors, I've no idea how they ended up in there, but they're fixed now. If there's still MSVC errors, please let me know what the error message is; I'm having problems triggering it here. (I might just make that code path depend on recent enough standard C if I can't figure out any other alternative; this will be fixed prior to merging.)
>
> Re the change to use IsEmptyToken, it's the intention that an empty body results in nothing happening, and that if IDEOGRAPHIC SPACE is now considered a space for the purpose of that function, then the code is now behaving correctly. [dict with] is very happy to delegate that decision. If IsEmptyToken is reporting tokens as empty when they could be compiled to code with side effects, that's definitely a bug... but not with the callers of IsEmptyToken. I'm not going to probe that further; I didn't write IsEmptyToken, but I need something that says "this token definitely doesn't generate code and so definitely doesn't generate errors".
>
> Literal sharing is one of those things that I think we may have been doing a bit too much of, and which has caused unnecessary representation churn. OTOH, I'm not strongly invested in either sharing or not sharing those objects for the most part. My usage is on the basis of it being a convenient API; I have a Tcl_Obj and I want to arrange for it (or something remarkably similar) to get pushed onto the stack by the bytecode engine at that point.
> I believe I checked the cases where I switched the code over to bounce the refcount; they're supposed to be places where the called code may take a reference or may not (typically on error) but definitely won't be decrementing the reference count. (It's the functions with an internal Tcl_DecrRefCount that you need to watch out for.)
> The switch to use Tcl_ListObjReplace was indeed deliberate. It's so that appending a zero-length list works correctly, and that's necessary for the short lappend-expansion branch to work (which makes [lappend] do the right thing with {*}). Previously, [lappend foo] was an ugly special case precisely because the opcodes worked differently from the non-compiled [lappend]. So. Many. Edge. Cases.
> Remove the clause, run append.test and appendComp.test (I forget which one) and see something very strange with some tests. I think it might be append-7.4 that triggers it, but I'm not sure as I've not poked at it for a few weeks (that have been very busy ones for me). The code before this branch (https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=856,860) is also odd in this area; in a sane world, the first condition there would be be numWords<2 and the second would be without looking at the envPtr->procPtr at all (at that point; it should just be a concern for picking how to access a variable, such as in https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=897,909 and https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=923,935).
> I'm generally supportive; the deprecated opcodes should be fairly easy to identify and remove with things as they stand. We do not generate any of them at this point, but there's still code that knows how to work with them (and execute them, of course). If backward compatibility is truly undesired, we can also remove the binding of the remaining opcodes to their current values. But technically out of scope right now. 😄
>
> Donal.
>
> From: apn...@ya... <apn...@ya...> on behalf of apn...@ya... <apn...@ya...>
> Sent: Saturday, May 17, 2025 17:37
> To: Donal Fellows <don...@ma...>; 'Tcl Core' <tcl...@li...>
> Subject: RE: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)
>
> Ok, having read the main components in tclComp*.c and tclExecute.c, I think I'm good to go on TIP 720. The code generation is much more readable making future development easier.
>
> Even the minor beef I had with the INT_MAX/UINT_MAX limits is not an issue currently as two separate INT_MAX limits, one on the length of a script and the other on the number of arguments, make the point moot. The (INT_MAX..UINT_MAX] case will never actually occur afaict. At least until TIP 626 at which point it becomes Jan's problem :-)
>
> There are some compilation errors still on Windows as mentioned earlier and also on Unix with --enable-symbols=all (works without that).
>
> One little buglet, not actually related to TIP 720 but related to the changes in [dict with] arising from modification to using Unicode aware IsEmptyToken. Didn't check if [dict update] also had the same issue.
>
> Tcl 9.0.1
> % info patch
> 9.0.1
> % writeFile space.tcl "proc \u3000 {} {puts IDEOGRAPHICSPACE}\n; set d \[dict create a 0];dict with d {\u3000}"
> % source space.tcl
> IDEOGRAPHICSPACE
>
> TIP-720
> % info patch
> 9.1a0
> % writeFile space.tcl "proc \u3000 {} {puts IDEOGRAPHICSPACE}\n; set d \[dict create a 0];dict with d {\u3000}"
> % source space.tcl
> % (no output as the dict with script block is treated as an empty script)
>
> Some more questions, as much for my own edification, as anything else…
>
> (1) Code sequences of the form
>
> Tcl_Size len;
> const char *bytes = TclGetStringFromObj(objPtr, &len);
> PushLiteral(envPtr, bytes, len)
>
> have been replaced by
>
> PUSH_OBJ(objPtr)
>
> Whereas PushLiteral resolves to TclRegisterLiteral, PUSH_OBJ resolves to TclAddLiteralObj. The former adds the value to the global literal array, but the latter does not. Is this difference intentional? Since the literal is going to be in the local array anyways, doesn't it make sense to have it globally available as well for memory sharing as in 9.0.0?
>
> (2) I also see sequences like
>
> TclNewObj(objPtr);
> Tcl_IncrRefCount(objPtr);
> SomeFunction(...objPtr...);
> Tcl_DecrRefCount(objPtr);
>
> replaced by
>
> TclNewObj(objPtr);
> SomeFunction(...objPtr...);
> Tcl_BounceRefCount(objPtr);
>
> but the semantics of the above sequences are not the same as the latter does not guarantee objPtr is still accessible on return from SomeFunction. I have not looked into every such "SomeFunction" but presumably, they do not manipulate (e.g. incr+decr) the ref count of objPtr. Still, the original sequence seems safer unless all SomeFunction’s in this context are known to be safe in this respect.
>
> (3) In the TEBCResume function, the under label lappendListPtr:, the call to TclListObjAppendElements has been replaced by Tcl_ListObjReplace. Why so? The latter is more general and therefore has to do a little more work to append. Clearly the change is intentional so I am curious as to the reason.
>
> (4) There is the following comment somewhere in the code which I think you just added:
>
>     /*
>      * The weird cluster of bugs around INST_LAPPEND_STK without a LVT ought
>      * to be sorted out. INST_LAPPEND_LIST_STK does the right thing.
>      */"
> What does this cluster of bugs refer to ? Are they recorded somewhere? I've worked on lappend and have not seen or noticed any. This is just my curiosity as to what I might have missed.
>
> (5) As an RFE, not necessarily part of TIP 720, I would be tempted to expunge the deprecated opcodes from TEBC as well (not just disabled via macros). It would significantly reduce and simplify TEBCresume. I understand the potential incompatibility with respect to tbcload and possible the Tcl debugger/prodebug. However, I do not see any need to maintain 8.6 tbcload compat given we don't even maintain 8.6 compat at script or C API level. This is the time to eliminate that code (before people depend on tbcloaded 9.0 applications) else we would have to wait for 10.0. (As an aside, does tbcload work with Tcl 9? If not, is anyone even working on it?)
>
> In summary, all looks good as far as I am concerned once the compiler errors with --enable-symbols=all are fixed.
>
> /Ashok
>
>
>
> From: Donal Fellows <don...@ma...>
> Sent: Friday, May 9, 2025 5:47 PM
> To: Tcl Core <tcl...@li...>
> Subject: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)
>
> As promised, I've done a TIP for the updates to the bytecode compiler. It's largely there to document the notable features of a substantial changeset.
>
> https://core.tcl-lang.org/tips/doc/trunk/tip/720.md [core.tcl-lang.org]
>
> It covers the essentials; of particular note is that no true public API is affected. That said, tcl::unsupported::assemble is affected, but via enhancement (it gets new opcodes), and tdkcompiler/tbcload will definitely need more work due to the new auxiliary structure type (for a numeric jump table).
>
> Given the lack of public surface changes - if you're not messing with compilation guts, you shouldn't need to care - I'm going to call a vote on this TIP immediately.
>
> Votes to here by [clock format 1747393200] (Fri May 16 12:00:00 BST 2025) in the usual form.
>
> TIP 720: YES
>
> Donal.
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)

[<apn...@ya...>]

Donal,

 

Thanks for the detailed explanations.

 

Regarding the IDEOGRAPHICSPACE bug, IsEmptyToken does not appear in any other branch than no-variable-width-instruction-issue so must have been introduced there. And fossil blame shows you as the author 😊 Perhaps you might be confusing the IsEmptyToken function hosting the bug with Tcl_IsEmpty? It’s all trivial to fix as long as we’re all on the same page as to what constitutes whitespace.

 

One other comment, not directly pertaining to 720 but related, regarding the other branch no-var-width-plus-opnd-types fixing TEBCResume types. I think the use of Tcl_Size fixes type mismatches for 64-bit archs but not 32-bit. For example, fragments like

 

Tcl_Size numArgs;

...

numArgs = TclGetUInt4AtPtr(pc + 1);

objResultPtr = OBJ_AT_DEPTH(numArgs);

 

will work on 64-bit arch but run into the same unsigned->signed offset issues on 32-bit architectures. Again, to reiterate, not that it currently matters because of the script and argument length limits of INT_MAX currently in place but will probably matter with TIP 626 approval.

 

/Ashok

 

From: Donal Fellows <don...@ma...> 
Sent: Monday, May 19, 2025 2:46 AM
To: apn...@ya...; 'Tcl Core' <tcl...@li...>
Subject: Re: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)

 

Re the compilation errors, I've no idea how they ended up in there, but they're fixed now. If there's still MSVC errors, please let me know what the error message is; I'm having problems triggering it here. (I might just make that code path depend on recent enough standard C if I can't figure out any other alternative; this will be fixed prior to merging.)

 

Re the change to use IsEmptyToken, it's the intention that an empty body results in nothing happening, and that if IDEOGRAPHIC SPACE is now considered a space for the purpose of that function, then the code is now behaving correctly. [dict with] is very happy to delegate that decision. If IsEmptyToken is reporting tokens as empty when they could be compiled to code with side effects, that's definitely a bug... but not with the callers of IsEmptyToken. I'm not going to probe that further; I didn't write IsEmptyToken, but I need something that says "this token definitely doesn't generate code and so definitely doesn't generate errors".

 

1.      Literal sharing is one of those things that I think we may have been doing a bit too much of, and which has caused unnecessary representation churn. OTOH, I'm not strongly invested in either sharing or not sharing those objects for the most part. My usage is on the basis of it being a convenient API; I have a Tcl_Obj and I want to arrange for it (or something remarkably similar) to get pushed onto the stack by the bytecode engine at that point.

2.      I believe I checked the cases where I switched the code over to bounce the refcount; they're supposed to be places where the called code may take a reference or may not (typically on error) but definitely won't be decrementing the reference count. (It's the functions with an internal Tcl_DecrRefCount that you need to watch out for.)

3.      The switch to use Tcl_ListObjReplace was indeed deliberate. It's so that appending a zero-length list works correctly, and that's necessary for the short lappend-expansion branch to work (which makes [lappend] do the right thing with {*}). Previously, [lappend foo] was an ugly special case precisely because the opcodes worked differently from the non-compiled [lappend]. So. Many. Edge. Cases.

4.      Remove the clause, run append.test and appendComp.test (I forget which one) and see something very strange with some tests. I think it might be append-7.4 that triggers it, but I'm not sure as I've not poked at it for a few weeks (that have been very busy ones for me). The code before this branch (https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c <https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=856,860> &ci=22313fe35a5e2956&ln=856,860) is also odd in this area; in a sane world, the first condition there would be be numWords<2 and the second would be without looking at the envPtr->procPtr at all (at that point; it should just be a concern for picking how to access a variable, such as in https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c <https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=897,909> &ci=22313fe35a5e2956&ln=897,909 and https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c <https://core.tcl-lang.org/tcl/file?name=generic/tclCompCmdsGR.c&ci=22313fe35a5e2956&ln=923,935> &ci=22313fe35a5e2956&ln=923,935).

5.      I'm generally supportive; the deprecated opcodes should be fairly easy to identify and remove with things as they stand. We do not generate any of them at this point, but there's still code that knows how to work with them (and execute them, of course). If backward compatibility is truly undesired, we can also remove the binding of the remaining opcodes to their current values. But technically out of scope right now. 😄

 

Donal.

 

  _____  

From: apn...@ya... <mailto:apn...@ya...>  <apn...@ya... <mailto:apn...@ya...> > on behalf of apn...@ya... <mailto:apn...@ya...>  <apn...@ya... <mailto:apn...@ya...> >
Sent: Saturday, May 17, 2025 17:37
To: Donal Fellows <don...@ma... <mailto:don...@ma...> >; 'Tcl Core' <tcl...@li... <mailto:tcl...@li...> >
Subject: RE: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY) 

 

Ok, having read the main components in tclComp*.c and tclExecute.c, I think I'm good to go on TIP 720. The code generation is much more readable making future development easier.

 

Even the minor beef I had with the INT_MAX/UINT_MAX limits is not an issue currently as two separate INT_MAX limits, one on the length of a script and the other on the number of arguments, make the point moot. The (INT_MAX..UINT_MAX] case will never actually occur afaict. At least until TIP 626 at which point it becomes Jan's problem :-)

 

There are some compilation errors still on Windows as mentioned earlier and also on Unix with --enable-symbols=all (works without that).

 

One little buglet, not actually related to TIP 720 but related to the changes in [dict with] arising from modification to using Unicode aware IsEmptyToken. Didn't check if [dict update] also had the same issue.

 

Tcl 9.0.1

% info patch

9.0.1

% writeFile space.tcl "proc \u3000 {} {puts IDEOGRAPHICSPACE}\n; set d \[dict create a 0];dict with d {\u3000}"

% source space.tcl

IDEOGRAPHICSPACE

 

TIP-720

% info patch

9.1a0

% writeFile space.tcl "proc \u3000 {} {puts IDEOGRAPHICSPACE}\n; set d \[dict create a 0];dict with d {\u3000}"

% source space.tcl

% (no output as the dict with script block is treated as an empty script)

 

Some more questions, as much for my own edification, as anything else…

 

(1) Code sequences of the form

 

Tcl_Size len;

const char *bytes = TclGetStringFromObj(objPtr, &len);

PushLiteral(envPtr, bytes, len)

 

have been replaced by

 

PUSH_OBJ(objPtr)

 

Whereas PushLiteral resolves to TclRegisterLiteral, PUSH_OBJ resolves to TclAddLiteralObj. The former adds the value to the global literal array, but the latter does not. Is this difference intentional? Since the literal is going to be in the local array anyways, doesn't it make sense to have it globally available as well for memory sharing as in 9.0.0?

 

(2) I also see sequences like

 

TclNewObj(objPtr);

Tcl_IncrRefCount(objPtr);

SomeFunction(...objPtr...);

Tcl_DecrRefCount(objPtr);

 

replaced by

 

TclNewObj(objPtr);

SomeFunction(...objPtr...);

Tcl_BounceRefCount(objPtr);

 

but the semantics of the above sequences are not the same as the latter does not guarantee objPtr is still accessible on return from SomeFunction. I have not looked into every such "SomeFunction" but presumably, they do not manipulate (e.g. incr+decr) the ref count of objPtr. Still, the original sequence seems safer unless all SomeFunction’s in this context are known to be safe in this respect.

 

(3) In the TEBCResume function, the under label lappendListPtr:, the call to TclListObjAppendElements has been replaced by Tcl_ListObjReplace. Why so? The latter is more general and therefore has to do a little more work to append. Clearly the change is intentional so I am curious as to the reason.

 

(4) There is the following comment somewhere in the code which I think you just added:

 

    /*

     * The weird cluster of bugs around INST_LAPPEND_STK without a LVT ought

     * to be sorted out. INST_LAPPEND_LIST_STK does the right thing.

     */"

What does this cluster of bugs refer to ? Are they recorded somewhere? I've worked on lappend and have not seen or noticed any. This is just my curiosity as to what I might have missed.

 

(5) As an RFE, not necessarily part of TIP 720, I would be tempted to expunge the deprecated opcodes from TEBC as well (not just disabled via macros). It would significantly reduce and simplify TEBCresume. I understand the potential incompatibility with respect to tbcload and possible the Tcl debugger/prodebug. However, I do not see any need to maintain 8.6 tbcload compat given we don't even maintain 8.6 compat at script or C API level. This is the time to eliminate that code (before people depend on tbcloaded 9.0 applications) else we would have to wait for 10.0. (As an aside, does tbcload work with Tcl 9? If not, is anyone even working on it?)

 

In summary, all looks good as far as I am concerned once the compiler errors with --enable-symbols=all are fixed.

 

/Ashok

 

 

 

From: Donal Fellows <don...@ma... <mailto:don...@ma...> >
Sent: Friday, May 9, 2025 5:47 PM
To: Tcl Core <tcl...@li... <mailto:tcl...@li...> >
Subject: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)

 

As promised, I've done a TIP for the updates to the bytecode compiler. It's largely there to document the notable features of a substantial changeset.

 

https://core.tcl-lang.org/tips/doc/trunk/tip/720.md [core.tcl-lang.org] <https://urldefense.com/v3/__https:/core.tcl-lang.org/tips/doc/trunk/tip/720.md__;!!PDiH4ENfjr2_Jw!ALVTJ_a_6mdCvhJUS8w77mC9_5peSjdi7rDWwxWMs-J5kkF5N-WFyVR3RhRASZXkoY05xtNSr7c8mGrrzoP0wC2pvNCkBDRi034Z$> 

 

It covers the essentials; of particular note is that no true public API is affected. That said, tcl::unsupported::assemble is affected, but via enhancement (it gets new opcodes), and tdkcompiler/tbcload will definitely need more work due to the new auxiliary structure type (for a numeric jump table).

 

Given the lack of public surface changes - if you're not messing with compilation guts, you shouldn't need to care - I'm going to call a vote on this TIP immediately.

 

Votes to here by [clock format 1747393200] (Fri May 16 12:00:00 BST 2025) in the usual form.

 

TIP 720: YES

 

Donal.

 

 

[TCLCORE] Next bi-weekly meeting 19th May 2025 at 12:00 UTC

[Steve L. <st...@di...>]

On behalf of Harald (who is unavailable) I’d like to invite anyone interested to the bi-weekly Tcl/TK telco on Monday May 19th 2025 at 12:00 UTC via https://meet.jit.si/TclMonthlyMeetup

The bi-weekly meetings are designed to keep Tcl/Tk development moving forwards towards regular releases, allowing discussion and clarification of any issues. Agenda items for this meeting will include:

• the releases planned for this month - 9.0.2 and 9.1a0 (the first release of Tcl 9.1 features) as per TIP #713 https://core.tcl-lang.org/tips/doc/trunk/tip/713.md
• TIPs being finalised and voted upon
• documentation

plus anything else attendees might want to raise.

Thanks to all

-- Steve

Re: [TCLCORE] TIP 720: Updated Tcl Bytecode Opcodes CFV (NOTE: NO DELAY)

[Donal F. <don...@ma...>]

apn...@ya...
2025-05-17 17:37
Re the compilation errors, I've no idea how they ended up in there, but they're fixed now.
Ugh, they were mainly caused by me not thinking too much when copying existing bits and pieces to make those TRACE calls. I'd tried the --symbols=all builds before, but that must have been prior to writing INST_DICT_PUT and INST_DICT_REMOVE. Removing the extra double quotes made things work as intended.

Donal.