tcl-core — Tcl Core Team discussions about Tcl development

[TCLCORE] Biweekly telco on 3rd of November at 12:00 UTC (TIP 561 added)

[Harald O. <har...@el...>]

@Eric: Thanks for TIP 561, added as Top 3
@Donal: great that you will make it. If you want to reorder the topics 
due to time constraints, just drop me an E-Mail.

Dear Tcl/Tk team,

please feel invited to the next Tcl/Tk beweekly telco:
3rd of November at 12:00 UTC
At: https://meet.jit.si/TclMonthlyMeetup

Caution: in US and Europe, day saving time ended. So it is one hour 
earlier your time compared to last meeting.

Agenda proposal:

Top 1) Release calender (TIP 713)
- 9.0.3: October (this week)
- 9.1a1: November (4 weeks left)
Top 2) TIP 672: $(1+1)
Top 3) TIP 561: Console command for Linux/Unix
Top 4) TIP 733: accessability (test status)
Top 5) TIP 732: TCL library path (discussion)
TOP 6) TIP 615: string is index
TOP 7) TIP 735: lfilter: "dict filter" for lists, but expr instead eval
Top 8) TIP 721: Tcl_AttemptGetString
Top 9) TIP 715: supported build systems
Top 10) TIP 734 nested mutex (go or not)
    Discussion with Ashok
Top 11) European conference
Top 12) AOB
Top 13) Next meeting:
       17th of November 12:00 UTC.

Other dates: (sorry, Harald can not attend to any of them, travelling a 
lot in November)
4th of November German speaking user meeting 19:00 UTC
11th of November: Monthly TCL meetup 19:00UTC

Thank you for all,
Harald

Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

[EricT <tw...@gm...>]

Hi Florent,

Thanks for thinking through the bridge token approach - it's clever!
However, I think you're right that the path forward is TCL_TOKEN_SUB_EXPR
with new compiler code that doesn't assume buffer continuity.

The challenge with bridge tokens is that every piece of code doing pointer
arithmetic would need to be taught to recognize and skip over them - that
could be dozens of places throughout the codebase. The "new token type with
new code" approach is cleaner because the new code paths don't inherit the
old assumptions.

I look forward to seeing what you discover as you explore the
TCL_TOKEN_SUB_EXPR approach. Understanding the compiler deeply will be
valuable for getting this right.

Regarding the lseq naked expression issue - I was involved in the TIP
process that led to lseq, though Brian Griffin did the implementation. The
naked expression feature allows things like lseq 1 to $x*2-1 (note: still
needs the $ for variable substitution). I'm not certain if that's what they
were referring to, but I understand the bytecode compiler concerns about
commands invoking expr recursively.

This is actually one reason I proposed $(...) in the first place - to
provide expression substitution at the language level so that commands like
lseq wouldn't need to handle naked expressions themselves. Let the
parser/compiler handle it once, properly, rather than having individual
commands invoke expr recursively.

Additionally, the naked expression feature in lseq has other issues that
may lead to it being deprecated, which further argues for a language-level
solution.

Best regards,
Eric


On Fri, Oct 31, 2025 at 8:18 AM Donal Fellows <
don...@ma...> wrote:

>
> Someone said (and I've lost track of who; thanks, Outlook!):
> ​
> lrange ... X*2-1 (end-1)/2
> ​
>
> Please don't. That makes producing nice bytecode really awkward. That we
> did something like that with *lseq* is bad enough; the bytecode engine
> has to recursively call itself to handle that and that's just terrible.
> (Fortunately, it's not used so much.)
>
> Donal.
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

[TCLCORE] TIP 561 - Working console.tcl implementation with channel transforms

[EricT <tw...@gm...>]

Hello Tcl Core Team,

I've successfully resolved the Unicode handling issues with the channel
transform
implementation for TIP 561 (Console command for Linux/Unix).

The key fix was recognizing that channel transforms receive UTF-8 encoded
bytes,
which need to be decoded back to Unicode strings before passing to the Tk
console
widget:

    set data [encoding convertfrom utf-8 $data]

This eliminates the need for the pre-8.6 puts wrapper method, providing a
clean,
API-based solution using TIP#230 channel transforms.

The working implementation is available at:

https://github.com/rocketship88/tcl-console-unix

Since I don't have permissions to update the TIP or its branch, could
someone
kindly review the code and update console.tcl in the TIP 561 repository if
it
looks acceptable?

The implementation has been tested on Pop!_OS Linux with proper Unicode
display,
console cleanup, and output restoration to the terminal.

Thank you,
Eric

Re: [TCLCORE] Biweekly telco on 3rd of November at 12:00 UTC

[Donal F. <don...@ma...>]

I hope to make this one even if I can't stay to the bitter end (because of family and other meetings); I remembered to put it in my diary this time...

Donal.


________________________________
From: Harald Oehlmann <har...@el...>
Sent: Friday, October 31, 2025 16:37
To: Tcl Core List <tcl...@li...>
Subject: [TCLCORE] Biweekly telco on 3rd of November at 12:00 UTC

please feel invited to the next Tcl/Tk beweekly telco:
3rd of November at 12:00 UTC
At: https://meet.jit.si/TclMonthlyMeetup

[TCLCORE] Biweekly telco on 3rd of November at 12:00 UTC

[Harald O. <har...@el...>]

Dear Tcl/Tk team,

please feel invited to the next Tcl/Tk beweekly telco:
3rd of November at 12:00 UTC
At: https://meet.jit.si/TclMonthlyMeetup

Caution: in US and Europe, day saving time ended. So it is one hour 
earlier your time compared to last meeting.

Agenda proposal:

Top 1) Release calender (TIP 713)
- 9.0.3: October (this week)
- 9.1a1: November (4 weeks left)
Top 2) TIP 672: $(1+1)
Top 3) TIP 733: accessability (test status)
Top 4) TIP 732: TCL library path (discussion)
TOP 5) TIP 615: string is index
TOP 6) TIP 735: lfilter: "dict filter" for lists, but expr instead eval
Top 7) TIP 721: Tcl_AttemptGetString
Top 8) TIP 715: supported build systems
Top 9) TIP 734 nested mutex (go or not)
   Discussion with Ashok
Top 10) European conference
Top 11) AOB
Top 12) Next meeting:
      17th of November 12:00 UTC.

Other dates: (sorry, Harald can not attend to any of them, travelling a 
lot in November)
4th of November German speaking user meeting 19:00 UTC
11th of November: Monthly TCL meetup 19:00UTC

Thank you for all,
Harald

Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

[Donal F. <don...@ma...>]

Someone said (and I've lost track of who; thanks, Outlook!):
​
lrange ... X*2-1 (end-1)/2
​

Please don't. That makes producing nice bytecode really awkward. That we did something like that with lseq is bad enough; the bytecode engine has to recursively call itself to handle that and that's just terrible. (Fortunately, it's not used so much.)

Donal.

Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

[da S. P. J <pet...@fl...>]

> Even $((...)) and [(...)] do scratch a bit on the border line towards "perlishness", but if I see ... $=$X*2-1 ...

I have been convinced by this discussion the whole thing is galloping perlism. :(

From: Andreas Leitgeb <av...@lo...>
Date: Thursday, October 30, 2025 at 09:44
To: tcl...@li... <tcl...@li...>
Subject: Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

CAUTION - EXTERNAL EMAIL:
This message originated from outside of your organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


EricT <tw...@gm...> wrote:
> You left out the one that I actually like the best, the $=(...) option.

Well, I don't share that preference.
Short answer is: it looks too "perlish" for my taste.

Even $((...)) and [(...)] do scratch a bit on the border line
towards "perlishness", but if I see  ... $=$X*2-1 ...
(with or without parentheses) I immediately wonder what equation
or assignment this would be supposed to be.  The "=" just doesn't
fit at all to the proposed use - imho. And I also can't think of
any other character that would fit better.

> % set X 2; set Y 625 ; lrange [lseq 0 to 50]    $=$X*2-1    end-$=int( sqrt( $Y ) )

I wish, that the [= X*2-1] approach could be incorporated directly
into commands taking indices, and that they'd define values for "end"
and other anchors.  That would eliminate the need for "=" and allow:
   lrange ... X*2-1  (end-1)/2
   lrange ... [getSomeIndex] (start+end)/2    (assuming new anchor "start")
Only thing left to define is, how to separate anchornames from local
variables and variable/array names from function-names like min,max.



_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://urldefense.us/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_tcl-2Dcore&d=DwICAg&c=MASr1KIcYm9UGIT-jfIzwQg1YBeAkaJoBtxV_4o83uQ&r=BRyGRggIJd8TmKOhvEmGElFuDuCl3O5mT8opva3f-Uc&m=dFOY6M3wFlTTn_sZeS_S2x37oqkxVAyKbumWdj9PUzT1p1DYv_qdt2xKtXV2Qc-X&s=HOhpe7jGf8XZyMmIg2wPpiatP3ugJpxoSmuNpG6d3gs&e=

Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

[Florent M. <flo...@gm...>]

Hi Eric,

I just noticed a problem, while you explained it, and very clearly, imho.

Your explanation points well a whole category of coding logic : "pointer 
arithmetic on the parsed buffer"

How many times is this logic used ? In what files ? For which purposes 
? As you showed, now we know that it is used at least in "Line Number 
Tracking" and "array indice compilation". This could be investigate 
further. Maybe those occurences are rare enough to be resolved ?

I must admit, I was interested by the "synthetic string" approach. It's 
a kind of "Subst", in the parsing period of compilation, it make me 
think about "Macro". Maybe it could be usefull sometimes.

So, let me just give an idea on it : Could a "TCL_TOKEN_SYNTHETIC" token 
be created, to account for the gap between the parsed buffer and the 
synthetic one ?

--------------------------------------------------------------

Tcl_token *syntheticToken;

syntheticToken -> start = parseBufferLocation; // the location in the 
parsed buffer where the synthethic string has began

syntheticToken -> size = (syntheticStringLocation 
- parseBufferLocation); // bridge : go forward over the gap;

syntheticToken -> type = TCL_TOKEN_SYNTHETIC;

syntheticToken -> numComponent = 2;

Tcl_token *commandToken;

commandToken -> start = syntheticStringLocation; // the location of the 
synthetic buffer

commandToken -> size = syntheticStringLength; // the length of the 
synthetic buffer

commandToken -> type = TCL_TOKEN_Command;

commandToken -> numComponent = 0;

Tcl_token *syntheticToken;

syntheticToken -> start = parseBufferLocation; // the location in the 
parsed buffer where the synthethic string has end

syntheticToken -> size = (parseBufferLocation - 
syntheticStringLocation); // bridge : how to go backward over the gap;

syntheticToken -> type = TCL_TOKEN_SYNTHETIC;

syntheticToken -> numComponent = 0;

------------------------------------------

Of course, I doubt this can resolve all issues, since I imagine that the 
parse informations aren't kept forever. Those remaining issues would 
have to be resolved into the line number tracking function...

So, as you said, the simplest road is to use a new TCL_TOKEN.

Myself, I used TCL_TOKEN_SUB_EXPR into parse_Expr, because I wanted the 
syntax to be nestable, and since this TOKEN seems to be used nowhere 
anymore (supposely an historical result of TCL evolution). I may be 
wrong on this assumption.

I even tried to use it (quickly) in Parse_Token, instead of using a 
Command Token that point to the synthetic string. But I couldn't get it 
working. As I had no time to investigate why, so I had reverted my code 
back to the synthetic solution.

Since you discover that synthetic string is not a viable solution, and 
explain clearly the reason why, I think I will have to insist more on 
this abvorted try : I will detect the ranges of "expression substitution 
syntax" into Parse_Token, but mark these ranges as a TCL_TOKEN_SUB_EXPR, 
then I will have to modifiy all the compilers parts that are concerned 
to evaluate it as expression, till it works. But I will have to study 
deeply this compiler (even maybe the execution part), what is not 
possible yet.

Anyway, thank you for those great insights you gave about the TCL 
internal. It will help a lot.

Best regards

Florent

Le 30/10/2025 à 17:02, EricT a écrit :
> Subject: Array subscript limitation in TIP 672 prototype - architectural issue
>
> I've discovered a fundamental limitation in the TIP 672 prototype implementation that affects array subscripts. I want to explain the issue clearly and acknowledgeFlorent's observation about this.
>
> The Problem:
>
> The synthetic string approach (transforming $(expr) into [expr {expr}] at parse time) works well for simple command arguments, but breaks for array subscripts like:
>
> set x($(1+2)) 1    # Crashes during compilation
>
> Root Cause - The Pointer Arithmetic Problem:
>
> Tcl's parser is buffer-based - all tokens must point into the same contiguous source buffer. The compiler and other systems use pointer arithmetic between tokens to calculate sizes and positions.
>
> This has caused two problems with the synthetic string approach:
>
>
> Problem 1: Line Number Tracking
>
> Early in development, I encountered issues withTclLogCommandInfo() which scans from the script start to the command location counting newlines. The original code was:
>
> for (p = script; p != command; p++) {
>      if (*p == '\n') {
>          iPtr->errorLine++;
>      }
> }
>
> When command pointed to the synthetic buffer and script pointed to the original source, p would never equal command (different memory regions), so the loop would scan forever through garbage memory until it crashed.
>
> I patched this by adding a null terminator check and pointer ordering (mentioned in the comments for this routine):
>
>          // Only scan if command appears to be within script's memory region
>          if (command >= script) {
>              for (p = script; p != command && *p != '\0'; p++) {
>                  if (*p == '\n') {
>                      iPtr->errorLine++;
>                  }
>              }
>          }
>          // If command < script, something is wrong - just use line 1
>
> This prevented crashes by stopping at the null byte, but meant we never computed accurate line numbers for errors in synthetic strings - the scan would stop early when hitting the end of the original source, producing incorrect line counts.
>
>
> Problem 2: Array Subscript Compilation (Florent's Discovery)
>
> Florent quickly identified that array subscripts would be problematic. Investigation confirms this is the same underlying issue - the compiler does pointer arithmetic between adjacent tokens:
>
> // FromTclPushVarName intclCompile.c:
> 	    name =varTokenPtr[1].start;
> 	nameLen = p -varTokenPtr[1].start;
> 	elName = p + 1;
> 	remainingLen = (varTokenPtr[2].start - p) - 1;
> 	elNameLen = (varTokenPtr[n].start-p) +varTokenPtr[n].size - 1;
>
> For set x($(1+2)) 1, the tokens are (for example during debug):
>
> Token 3: TEXT "x("        > points to ORIGINAL source (0x281f7eb0184)
> Token 4: COMMAND "[expr]" > points to SYNTHETIC buffer (0x281f7ec3070)
> Token 5: TEXT ")"         > points to ORIGINAL source (0x281f7eb018d)
>
> When the compiler calculatesvarTokenPtr[2].start - p (synthetic address minus original address), it produces garbage values - I've seen 2+ million byte sizes - leading to crashes inmemcpy.
>
> The Architectural Challenge:
>
> I don't know if pointer arithmetic between tokens can be eliminated throughoutTcl's codebase - it may be a fundamental assumption. This is likely whyFlorent's approach of a new token type is a more sound approach:
>
> - Creates a proper token type handled by new code
> - New code paths don't make assumptions about token address relationships
> - Works withTcl's architecture rather than trying to work around it
>
> The Pattern:
>
> Both issues stem from the same root: trying to do stream-based macro expansion (synthetic string replacement) in a buffer-based parser that assumes all tokens share a common address space for pointer arithmetic.
>
> Why This Matters:
>
> In a stream-based parser (likelex/yacc), you can expand macros by pushing characters back onto the input stream.Tcl's buffer-based architecture doesn't allow this - tokens are pointer/length pairs into a fixed buffer, and pointer arithmetic assumes all tokens share the same base address. Back in 2018 this limitation led to the synthetic approach. And our implementation of (...) didn't permit use as subscripts, so this never was tested until now. Thanks Florent.
>
> The Prototype's Value:
>
> The TIP 672 prototype successfully:
> - Proved expression shorthand is implementable
> - Moved past 20 years of syntaxbikeshedding  
> - Generated concrete discussion and alternatives
> - DemonstratedJimTcl compatibility is achievable
> - Has 251 passing test cases for the parsing logic
>
> However, even with this extensive testing, array subscripts fell through the cracks. This reveals how the prototype focused on proving the concept for simple command arguments, where the synthetic string approach works, but didn't expose the architectural incompatibility with contexts requiring token pointer arithmetic.
>
> Going Forward:
>
> The synthetic string approach may be fundamentally incompatible with contexts requiring token pointer arithmetic. This suggests either:
>
> 1. Modifying the expression parser (Florent's approach) to handle the syntaxnatively
> 2. Finding a way to keep all tokens in the original buffer (unclear how)
> 3. Some other architectural approach I haven't thought of
>
> I want to be transparent about this limitation. The prototype served its purpose in proving viability, but production implementation needs deeper integration withTcl's parser architecture.
>
> Thoughts?
>
> Best regards,
> Eric
>
>
> On Thu, Oct 30, 2025 at 7:43 AM Andreas Leitgeb <av...@lo...> wrote:
>
>     EricT <tw...@gm...> wrote:
>     > You left out the one that I actually like the best, the $=(...)
>     option.
>
>     Well, I don't share that preference.
>     Short answer is: it looks too "perlish" for my taste.
>
>     Even $((...)) and [(...)] do scratch a bit on the border line
>     towards "perlishness", but if I see  ... $=$X*2-1 ...
>     (with or without parentheses) I immediately wonder what equation
>     or assignment this would be supposed to be.  The "=" just doesn't
>     fit at all to the proposed use - imho. And I also can't think of
>     any other character that would fit better.
>
>     > % set X 2; set Y 625 ; lrange [lseq 0 to 50]    $=$X*2-1  
>     end-$=int( sqrt( $Y ) )
>
>     I wish, that the [= X*2-1] approach could be incorporated directly
>     into commands taking indices, and that they'd define values for "end"
>     and other anchors.  That would eliminate the need for "=" and allow:
>        lrange ... X*2-1  (end-1)/2
>        lrange ... [getSomeIndex] (start+end)/2    (assuming new anchor
>     "start")
>     Only thing left to define is, how to separate anchornames from local
>     variables and variable/array names from function-names like min,max.
>
>
>
>     _______________________________________________
>     Tcl-Core mailing list
>     Tcl...@li...
>     https://lists.sourceforge.net/lists/listinfo/tcl-core
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Tk project skip_entire_test_files: invitation for final review

[Erik L. <el...@xs...>]

On 30/10/2025 20:00, Erik Leunissen via Tcl-Core wrote:
> 
> The primary goal of the project is to simplify the initialization of 
> test files.
> 

Uhmmm ...that should be:

The primary goal of the project is to prevent the execution of test commands
for tests that are skipped because their test constraints aren't satisfied.

Erik Leunissen.

[TCLCORE] Tk project skip_entire_test_files: invitation for final review

[Erik L. <el...@xs...>]

L.S.

The Tk project "skip_entire_test_files" is ready for a final review,
a review leading up to a merge into the target branches trunk and 
core-9-0 branch.

The primary goal of the project is to simplify the initialization of 
test files.

A summary of the project's results is appended to this message.

I'm welcoming comments from maintainers, TCT-members and other knowledgeable
developers frequenting this mailing list, especially those who are concerned
with Tk. The project's ticket holds further useful information for 
reviewers:

     https://core.tcl-lang.org/tk/tktview/83eed90f93

I'm asking that you direct your response to this mailing list's thread, 
or to
the project's ticket before 9-nov-2025 24:00 UTC.

Of course, I'm prepared to provide clarification where needed.

Thanks for your attention,
Erik Leunissen.
--

Summary of project results
--------------------------
* The following table specifies which test files are skipped entirely, 
based on
   which test constraints (if they aren't satisfied):

	-------------------------------------------------------------------------------
	Constraint	Test files skipped entirely
	-------------------------------------------------------------------------------
	testutils	testutils.test
	unix		unixButton.test unixEmbed.test unixFont.test unixMenu.test \
				unixSelect.test unixWm.test
	vista		ttk/vsapi.test
	win		winButton.test winClipboard.test winDialog.test winFont.test \
				winMenu.test winMsgbox.test winSend.test winWm.test
	-------------------------------------------------------------------------------
* The following table shows the impact of test file skipping for each 
major platform,
   based on the standard test suite invocation "make test" (no options). 
It indicates
   the count of test files skipped, and the count of test commands 
skipped as a
   consequence.

	-------------------------------------------------------
	Platform	# testfiles	# test commands skipped
			  skipped	  by consequence
	-------------------------------------------------------
	Linux/x11	    10		377 (of 10646 = 3.5 %)
	Windows		    7		560 (of 10593 = 5.3 %)
	macOS/aqua	    10		378 (of 10654 = 3.5 %)
	-------------------------------------------------------
* Checks on the counts of skipped test commands revealed two tests with
   an incorrect test constraint, and one test placed in the wrong test file.
   These issues were corrected.
* The relevant test file have received a note in the NOTES section, 
mentioning
   the circumstances that induce its skipping.
--

Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

[EricT <tw...@gm...>]

Andreas:

Others have commented that $= makes one think, oh, this is going to
replace the $ with the expression on the right side. (See comments by
Rich on comp.lang.tcl)

The problem I have with expr is two fold. The security hole and the
hard on the eyes with too much text surrounding the expression. That's
why I like the idea of getting rid of the extra ()'s when they're not
needed.

Well, the prototype has served it's purpose. But it's not the
solution. Here's why.

Subject: Array subscript limitation in TIP 672 prototype - architectural issue

I've discovered a fundamental limitation in the TIP 672 prototype
implementation that affects array subscripts. I want to explain the
issue clearly and acknowledge Florent's observation about this.

The Problem:

The synthetic string approach (transforming $(expr) into [expr {expr}]
at parse time) works well for simple command arguments, but breaks for
array subscripts like:

set x($(1+2)) 1    # Crashes during compilation

Root Cause - The Pointer Arithmetic Problem:
Tcl's parser is buffer-based - all tokens must point into the same
contiguous source buffer. The compiler and other systems use pointer
arithmetic between tokens to calculate sizes and positions.

This has caused two problems with the synthetic string approach:


Problem 1: Line Number Tracking

Early in development, I encountered issues with TclLogCommandInfo()
which scans from the script start to the command location counting
newlines. The original code was:

for (p = script; p != command; p++) {
    if (*p == '\n') {
        iPtr->errorLine++;
    }
}

When command pointed to the synthetic buffer and script pointed to the
original source, p would never equal command (different memory
regions), so the loop would scan forever through garbage memory until
it crashed.

I patched this by adding a null terminator check and pointer ordering
(mentioned in the comments for this routine):

        // Only scan if command appears to be within script's memory region
        if (command >= script) {
            for (p = script; p != command && *p != '\0'; p++) {
                if (*p == '\n') {
                    iPtr->errorLine++;
                }
            }
        }
        // If command < script, something is wrong - just use line 1

This prevented crashes by stopping at the null byte, but meant we
never computed accurate line numbers for errors in synthetic strings -
the scan would stop early when hitting the end of the original source,
producing incorrect line counts.


Problem 2: Array Subscript Compilation (Florent's Discovery)
Florent quickly identified that array subscripts would be problematic.
Investigation confirms this is the same underlying issue - the
compiler does pointer arithmetic between adjacent tokens:

// From TclPushVarName in tclCompile.c:
	    name = varTokenPtr[1].start;
	    nameLen = p - varTokenPtr[1].start;
	    elName = p + 1;
	    remainingLen = (varTokenPtr[2].start - p) - 1;
	    elNameLen = (varTokenPtr[n].start-p) + varTokenPtr[n].size - 1;

For set x($(1+2)) 1, the tokens are (for example during debug):

Token 3: TEXT "x("        > points to ORIGINAL source (0x281f7eb0184)
Token 4: COMMAND "[expr]" > points to SYNTHETIC buffer (0x281f7ec3070)
Token 5: TEXT ")"         > points to ORIGINAL source (0x281f7eb018d)

When the compiler calculates varTokenPtr[2].start - p (synthetic
address minus original address), it produces garbage values - I've
seen 2+ million byte sizes - leading to crashes in memcpy.

The Architectural Challenge:

I don't know if pointer arithmetic between tokens can be eliminated
throughout Tcl's codebase - it may be a fundamental assumption. This
is likely why Florent's approach of a new token type is a more sound
approach:

- Creates a proper token type handled by new code
- New code paths don't make assumptions about token address relationships
- Works with Tcl's architecture rather than trying to work around it

The Pattern:

Both issues stem from the same root: trying to do stream-based macro
expansion (synthetic string replacement) in a buffer-based parser that
assumes all tokens share a common address space for pointer
arithmetic.

Why This Matters:

In a stream-based parser (like lex/yacc), you can expand macros by
pushing characters back onto the input stream. Tcl's buffer-based
architecture doesn't allow this - tokens are pointer/length pairs into
a fixed buffer, and pointer arithmetic assumes all tokens share the
same base address. Back in 2018 this limitation led to the synthetic
approach. And our implementation of (...) didn't permit use as
subscripts, so this never was tested until now. Thanks Florent.

The Prototype's Value:

The TIP 672 prototype successfully:
- Proved expression shorthand is implementable
- Moved past 20 years of syntax bikeshedding
- Generated concrete discussion and alternatives
- Demonstrated JimTcl compatibility is achievable
- Has 251 passing test cases for the parsing logic

However, even with this extensive testing, array subscripts fell
through the cracks. This reveals how the prototype focused on proving
the concept for simple command arguments, where the synthetic string
approach works, but didn't expose the architectural incompatibility
with contexts requiring token pointer arithmetic.

Going Forward:

The synthetic string approach may be fundamentally incompatible with
contexts requiring token pointer arithmetic. This suggests either:

1. Modifying the expression parser (Florent's approach) to handle the
syntax natively
2. Finding a way to keep all tokens in the original buffer (unclear how)
3. Some other architectural approach I haven't thought of

I want to be transparent about this limitation. The prototype served
its purpose in proving viability, but production implementation needs
deeper integration with Tcl's parser architecture.

Thoughts?

Best regards,
Eric



On Thu, Oct 30, 2025 at 7:43 AM Andreas Leitgeb <av...@lo...> wrote:

> EricT <tw...@gm...> wrote:
> > You left out the one that I actually like the best, the $=(...) option.
>
> Well, I don't share that preference.
> Short answer is: it looks too "perlish" for my taste.
>
> Even $((...)) and [(...)] do scratch a bit on the border line
> towards "perlishness", but if I see  ... $=$X*2-1 ...
> (with or without parentheses) I immediately wonder what equation
> or assignment this would be supposed to be.  The "=" just doesn't
> fit at all to the proposed use - imho. And I also can't think of
> any other character that would fit better.
>
> > % set X 2; set Y 625 ; lrange [lseq 0 to 50]    $=$X*2-1    end-$=int(
> sqrt( $Y ) )
>
> I wish, that the [= X*2-1] approach could be incorporated directly
> into commands taking indices, and that they'd define values for "end"
> and other anchors.  That would eliminate the need for "=" and allow:
>    lrange ... X*2-1  (end-1)/2
>    lrange ... [getSomeIndex] (start+end)/2    (assuming new anchor "start")
> Only thing left to define is, how to separate anchornames from local
> variables and variable/array names from function-names like min,max.
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

[Andreas L. <av...@lo...>]

EricT <tw...@gm...> wrote:
> You left out the one that I actually like the best, the $=(...) option.

Well, I don't share that preference.
Short answer is: it looks too "perlish" for my taste.

Even $((...)) and [(...)] do scratch a bit on the border line 
towards "perlishness", but if I see  ... $=$X*2-1 ...
(with or without parentheses) I immediately wonder what equation
or assignment this would be supposed to be.  The "=" just doesn't
fit at all to the proposed use - imho. And I also can't think of
any other character that would fit better.

> % set X 2; set Y 625 ; lrange [lseq 0 to 50]    $=$X*2-1    end-$=int( sqrt( $Y ) )

I wish, that the [= X*2-1] approach could be incorporated directly
into commands taking indices, and that they'd define values for "end"
and other anchors.  That would eliminate the need for "=" and allow:
   lrange ... X*2-1  (end-1)/2
   lrange ... [getSomeIndex] (start+end)/2    (assuming new anchor "start")
Only thing left to define is, how to separate anchornames from local
variables and variable/array names from function-names like min,max.

Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

[Andreas L. <av...@lo...>]

Jan Nijtmans <jan...@gm...> opened my eyes:
> Op do 30 okt 2025 om 02:38 schreef Andreas Leitgeb:
> > If some code contains:    ... [subst -nocommands $userString]
> > then the subst shall never ever cause any command to be called,
> > no matter how it appears in the $userString.
> Are you sure?:
> % set userstring {$a([puts stdout "Command executed"])}
> $a([puts stdout "Command executed"])
> % subst -nocommand $userstring
> Command executed
> can't read "a()": no such variable
> %

Damn.

> From the documentation:
>     Note that the substitution of one kind can include substitution of
>     other kinds. For example, even when the \fB\-novariables\fR option
>     is specified, command substitution is performed without restriction.
>     This means that any variable substitution necessary to complete the
>     command substitution will still take place. Likewise, any command
>     substitution necessary to complete a variable substitution will
>     take place, even when \fB\-nocommands\fR is specified.

I wasn't aware of that...  gotta check my codebase for such 
occurrances and see how I could rewrite them to fix this worm
hole.

I know I used subst recently with both -nocommands and -novars - to
just let it do backslash-treatment.  Does this also have a pathway to
command-subst, or will it have one, if expr turns into a "separate"
thing?

> What we can do is remove the "-noexpression" option, only
> providing the "-expression" option. So, make it off by default.

As I learned, excluding certain substitutions turns out to be a broken
design in the first place. Requiring "-expression" to be explicitly
turned on (rather than off) would probably at least save the current
semantics of [subst -nocommands -novariables $userString], so it causes
least pain among the expr-sensitive alternatives offered so far.

Of course, it is highly confusing to future users of subst, that
some types need excluding, while others need including, so it isn't
really "good design", either.

My lesson for myself is: avoid [subst] altogether.
I might miss the next dicussion about next "new thing", and
suddenly subst even with all previously known -no* options
turns essentially into eval...

Last question: if we get syntactic sugar for expr, is it
technically really not possible to still treat the new syntax
as if it were [expr {...}] for [subst] ?

Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

[Jan N. <jan...@gm...>]

Op do 30 okt 2025 om 02:38 schreef Andreas Leitgeb:
> If some code contains:    ... [subst -nocommands $userString]
> then the subst shall never ever cause any command to be called,
> no matter how it appears in the $userString.

Are you sure?:
% set userstring {$a([puts stdout "Command executed"])}
$a([puts stdout "Command executed"])
% subst -nocommand $userstring
Command executed
can't read "a()": no such variable
%

>From the documentation:
    Note that the substitution of one kind can include substitution of
    other kinds. For example, even when the \fB\-novariables\fR option
    is specified, command substitution is performed without restriction.
    This means that any variable substitution necessary to complete the
    command substitution will still take place. Likewise, any command
    substitution necessary to complete a variable substitution will
    take place, even when \fB\-nocommands\fR is specified.

What we can do is remove the "-noexpression" option, only
providing the "-expression" option. So, make it off by default.

Hope this helps,
      Jan Nijtmans

Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

[EricT <tw...@gm...>]

Hi, Andreas

Jan has already implemented the changes for subst, they're in the tip-672
branch. I don't use the command much so I can't say for certain that it
solves the problem.

That was a nice summary of the current choices. You left out the one that I
actually like the best, the $=(...) option. It only breaks very rare cases
where someone wants a literal $= and can easily modify it to \$=
anyway, and likely already has by not realizing it wasn't needed.

But there's one little twist I've been playing with, that can drop the
outer ()'s as long as you don't need any whitespace outside ()'s, and you
can always add them if you do:

% set X 2; set Y 625 ; lrange [lseq 0 to 50]    $=$X*2-1    end-$=int(
sqrt( $Y ) )
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Sorta like index expressions with regards to whitespace. Not too much of a
change.

On Wed, Oct 29, 2025 at 6:38 PM Andreas Leitgeb <av...@lo...> wrote:

> Jan Nijtmans <jan...@gm...> wrote:
> > 3) The "subst" command (and the Tcl_SubstObj() function) need
> >     new options/flags to handle the new "expression" substitution,
> >     separate to "variable" substitution and "command" substitution.
> >     That solves the discussion, whether this substitution belongs
> >     to "variable" or "command" substitution. I think it is neither ;-)
>
> This is very dangerous, imho:
>
> If some code contains:    ... [subst -nocommands $userString]
> then the subst shall never ever cause any command to be called,
> no matter how it appears in the $userString.
>
> E.g. if $userString is {foo bar [exec snafu] } then the command "snafu"
> just should not be executed...  ("snafu" might just as well be "rm -rf /")
>
> Now, if expr-subst gets considered a "different" thing, then code like
> the one above might end up executing snafu, if $userString is just slightly
> modified to {foo bar $(([exec snafu]))}, which definitely is a severe
> breach of "subst"'s promises. Existing code shouldn't be forced to add a
> new option to all "subst -nocommands" invocations to not become vulnerable.
>
> PS: some more opinions on this topic:
>
>   - I like empty-named arrays and use them quite often, so I'd
>     consider $(...) redefinition for expr out of way until maybe Tcl 10.
>
>   - $((...)) is fine by me, and I'm proud of having (a while back) started
>     a discussion that led to disallowing certain characters in literal
> array
>     indices.  My motivation back then was paving the way for assignment
> (even
>     to array elements) within expr - without ambiguity with
> builtin-functions.
>
>   - [(...)] looks good to me, too - I think the compatibility issues are
>     rather theoretic. If brackets are necessary, to allow subst -nocommands
>     to do its job correctly, then this would be my preference.
>
>   - [= ...] with a modified expr-language using barewords as variable
>     names would probably solve many (if not most) of the cases, where
>     [expr {...}] really "hurts" for its verbosity.
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] Fwd: TIP 672 Implementation Complete - Ready for Sponsorship

[Andreas L. <av...@lo...>]

Jan Nijtmans <jan...@gm...> wrote:
> 3) The "subst" command (and the Tcl_SubstObj() function) need
>     new options/flags to handle the new "expression" substitution,
>     separate to "variable" substitution and "command" substitution.
>     That solves the discussion, whether this substitution belongs
>     to "variable" or "command" substitution. I think it is neither ;-)

This is very dangerous, imho:

If some code contains:    ... [subst -nocommands $userString]
then the subst shall never ever cause any command to be called,
no matter how it appears in the $userString.

E.g. if $userString is {foo bar [exec snafu] } then the command "snafu"
just should not be executed...  ("snafu" might just as well be "rm -rf /")

Now, if expr-subst gets considered a "different" thing, then code like
the one above might end up executing snafu, if $userString is just slightly
modified to {foo bar $(([exec snafu]))}, which definitely is a severe
breach of "subst"'s promises. Existing code shouldn't be forced to add a
new option to all "subst -nocommands" invocations to not become vulnerable.

PS: some more opinions on this topic:

  - I like empty-named arrays and use them quite often, so I'd
    consider $(...) redefinition for expr out of way until maybe Tcl 10.

  - $((...)) is fine by me, and I'm proud of having (a while back) started
    a discussion that led to disallowing certain characters in literal array
    indices.  My motivation back then was paving the way for assignment (even
    to array elements) within expr - without ambiguity with builtin-functions.

  - [(...)] looks good to me, too - I think the compatibility issues are
    rather theoretic. If brackets are necessary, to allow subst -nocommands
    to do its job correctly, then this would be my preference.

  - [= ...] with a modified expr-language using barewords as variable
    names would probably solve many (if not most) of the cases, where
    [expr {...}] really "hurts" for its verbosity.

Re: [TCLCORE] Variation on the Expr Shorthand

[Florent M. <flo...@gm...>]

Hi Eric.
Thanks for your comment.
I can see in effect some tests in my code are weak. I dont have enough time
now to perfect it. At least, i wanted to produce a prototype to check if
the idea is feasible.  i wanted to create a cleaner github implementation
later, beginning 2026. The actual code is full of printf.

Your comments on the subject are really interesting. I will keep it in mind.

I notice also that (in my try) it doesn't working in array index.
% set x([(1+1)]) 1
Won't set x(2). I dont understand why. It should have gone throught
parseToken, it did, but not only.

I've test only with tclsh. Sadly, i will be too busy next month to
investigate it.

About your parsing routine : i supposed it WOULD have problem with quote in
brace or brace in quote, but this guess was not founded on factuel test. I
may be wrong.

Nevertheless, using the expr parser seems more natural to me.

My first interest was with {=} prefix, till i understood it was limited to
a word. That's why i shift to try an [(  ... )] syntax.

Of course, [( .... )] will break any code that has defined a proc ( args
{...}.

Myself, i always prefer to give to a command a name that correspond to its
semantic. So, i never gave the name "(" to any proc. Who can advice it's a
good name to give ? But it's a matter of taste...

Nesting subexpression :

Writing it is of course totally unsane. Why someone would write [( 3 * [(2
+ 1)] )] instead of [(3*(2+1))] ?

I can be only accidental. Either due to a programmer who modified an
existing code, in a very long expression, or due to a machine who generated
it.

Support those accidents seems important to me.

Whatever, to add this possibility has implied less than 10 lines of code in
this prototype.

Best regards,
Florent

Le mer. 29 oct. 2025, 20:18, EricT <tw...@gm...> a écrit :

> Hi Florent,
>
> Thank you for exploring alternative approaches to expression shorthand syntax! It's valuable to have different perspectives, and I appreciate the work you've put into your [(..)] prototype.
>
> A Safety Note on Buffer Access:
>
> I noticed a potential issue in your ParseTokens code:
>
>     } else if (src[0] == '[' && src[1] == '(') {
>
> This should check numBytes before accessing src[1]:
>
>     } else if (numBytes > 1 && src[0] == '[' && src[1] == '(') {
>
> ParseTokens is very sensitive to numBytes management. If numBytes goes negative or you read past the buffer, you can encounter:
>
> - Infinite loops (the while (numBytes && ...) continues with negative values)
> - Reading garbage memory past null terminators (especially in proc bodies)
> - Undefined behavior or crashes
>
> I spent several hours debugging exactly this issue in mode 3 $((...)) when the second closing paren was missing at the end of a proc body. The symptoms were subtle until numBytes reached -4000 in a tight loop allocating tokens each pass.
>
> Compatibility Consideration:
>
> As Peter noted, [( has the same class of compatibility issue that $() faces - both break rare but valid existing syntax. Worth considering in the design.
>
> A Question:
>
> I'm curious - in earlier discussions, I thought you were exploring [{...}] syntax. What led you to shift to [(..)]? The brace version seemed interesting as an alternative approach.
>
> Quote-in-Braces Issue:
>
> You mentioned "a quote inside braces would create an error" as a flaw in the basic parsing. Could you provide a specific test case that demonstrates this? In my testing with $=:
>
> % set foo $=([string length {text with "}])
> 11
>
> This works correctly, since the paren-matching algorithm treats braces as literal characters within expressions. Which of the three TIP 672 modes $( vs $= vs $(()) did you test? I'd like to understand what edge case you've identified so I can verify and address it if needed.
>
>
> Nested Expression Substitutions:
>
> I noticed your nested example. I'm curious whether nesting is a requirement of your implementation, or if you're demonstrating it as a feature? With expression shorthand, nesting typically isn't necessary since regular parentheses already provide grouping within expressions.
>
> One of the design goals for TIP 672 was to keep the syntax simple by not requiring nestability - the existing expression syntax handles grouping naturally. Is there a use case where nestable expression substitutions provide an advantage I'm missing?
>
> Thanks again for contributing to this discussion!
>
> Best regards,
> Eric
>
>
>
> On Wed, Oct 29, 2025 at 7:45 AM Florent Merlet <flo...@gm...>
> wrote:
>
>> Hi dear Tcl community,
>>
>> An Expr shorthand syntax has been a long time demand between us.
>>
>> Those discussions always focus on the syntax aspect of the subject :
>>
>>    - Like in bash $(...) or $((...))
>>    - Through an alias [= ...]
>>    - A new command (vexpr or let)
>>    - A word prefix {=}
>>    - ...
>>
>> A lot of TIPs exists on that matter. Numerous discussions occurs, which
>> never ended to get a consensus.
>>
>> That's because the look of this shorthand is a matter of taste. Everybody
>> has his own taste. Some people like fish when it's cooked in water, some
>> people like it when it's fried. Some people even don't like fish at all !
>> Every taste is in the nature.
>>
>> Everybody can agree that Tcl is a big and complex machinery, that must be
>> handled with care. So maybe the problem must be taken the other way round :
>>
>>    - Shall we deduce the Tcl C source code machinery from a new syntax,
>>    we had previously decided (the one doesn't make consensus)
>>    - Or shall we deduce the new syntax from the Tcl C source code
>>    machinery, as it exists ?
>>
>> My opinion is that it's better to deduce the syntax from the Tcl C source
>> code, rather than to deduce the Tcl source C code from the syntax .
>>
>> TIP 672 is hacking the variable substitution. To do this, it has to make
>> a very basic parsing of the expression to estimate its length. It has to
>> transmute a TCL_VARIABLE Token into a TCL_COMMAND token. It then use a call
>> to Tcl_ParseCommand on a synthetic string to check errors.
>>
>> This very basic parsing will make it buggy. For instance, a shorthand
>> expression can't be nested in another one. A quote inside braces would
>> create an error. To make this parsing strong, we would have to reinvent all
>> the expression parsing from scratch.
>>
>> But shall we create a new parsing expression routine for this shorthand
>> ?  No, there exist already an expression parsing machinery, that can handle
>> words between Quotes or Braces and can handle Nested Commands, exactly how
>> the Expr command do.
>>
>> « Deduce the shorthand syntax from the Tcl C source Code » imply to find
>> a syntax which allows us to use the existing machinery.
>>
>> That's what I'm trying now :
>>
>> As Expr is a command in Tcl, it seems logical to me to implement the
>> shorthand syntax in the Command branch "[" of parseToken procedure. That's
>> what I choosed.
>>
>> The second step is to parse the expression, so to go through
>> Tcl_ParseExpr routine, then to the ParseExpr routine. The difficulty here
>> is to get the end of the substitution script in the ParseExpr routine. If I
>> don't want to disturb parseExpr too much, it's better to choose, as
>> character which ends the expression script, a character that
>> is significant for this parser, so the main task of detecting it is already
>> done, but can be adapted gently.
>>
>> Maybe I could have used any of those operators : '+', '=', '-', '*', '(',
>> ')', '|',...etc. But I choosed to use ')' : infix language needs
>> parenthesis.
>>
>> That is how I defined the end of the expression substitution script to be
>> ")]". By symetry, I defined the beginning of the substitution script to be
>> "[(".
>>
>> Here is the genesis of my proposal of "[( ...)]" as a shorthand.
>>
>> To make it work, I had to used the same clever hacking than Eric Taylor :
>> create a synthethic string and parse it as a command.
>>
>> At the end, the [(...)] is working as expected (so far I've tested). Here
>> are the main changes I have done to accomplish it :
>>
>> In file Tcl_Parse.c : in function parseTokens,  I add a new branch in the
>> test
>>
>> ----------------------------------------
>>
>>  ... } else if (src[0] == '[' && src[1] == '(') {
>>
>> ///////////////////////////////////////////////////////////////////////
>>         /* Expression substition context */
>>         // to do : noSubstExpr
>>         Tcl_Parse *exprParsePtr;
>>         exprParsePtr =(Tcl_Parse *)TclStackAlloc(parsePtr->interp,
>> sizeof(Tcl_Parse));
>>
>>         src++;     // src == '['
>>         numBytes --;
>>         // Use it only to know the length of the expression, and store it
>> into exprParsePtr->commandSize
>>         Tcl_ParseExpr(parsePtr->interp, src, numBytes, exprParsePtr);
>>
>>         src++;  // src == '('
>>         numBytes --;
>>
>>         // Here is the famous hack of Eric Taylor
>>         Tcl_Size syntheticLen = exprParsePtr->commandSize + 9;  // "[expr
>> {" + expr + "}]"
>>
>>         char *synthetic = (char *)Tcl_Alloc(syntheticLen + 1);
>>
>>         memcpy(synthetic, "[expr {", 7);
>>         memcpy(synthetic + 7, src, exprParsePtr->commandSize);
>>
>>         memcpy(synthetic + 7 + exprParsePtr->commandSize, "}]", 3);
>>         synthetic[syntheticLen] = '\0';
>>         // Maybe a Tcl_Obj could be of use for memory management ?
>>
>>
>>         Tcl_Obj *exprObjCommand =
>> Tcl_NewStringObj(synthetic,syntheticLen);
>>
>>         src+=exprParsePtr->commandSize+2;
>>         numBytes-=exprParsePtr->commandSize+2;
>>
>>         TclStackFree(parsePtr->interp, exprParsePtr);
>>
>>         tokenPtr->type = TCL_TOKEN_COMMAND;
>>         tokenPtr->start = Tcl_GetStringFromObj(exprObjCommand, NULL);
>>         tokenPtr->size = syntheticLen;
>>         parsePtr->numTokens++;
>>
>>         continue;
>>
>> } else if (*src == '[') {...
>>
>> ---------------------------------------
>>
>> To detect the end and transfer the size of the parsed expression I had to
>> modify :
>>
>> 1° the Tcl_ParseExpr function :
>>
>> ... if (code == TCL_OK) {
>>     if(start[-1] == '[' && start[0] == '(' ) {
>>         // Expression Substitution Context : just transfer the size
>> information to the caller
>>              parsePtr->commandSize =exprParsePtr->commandSize;
>>     } else {
>>         TclParseInit(interp, start, numBytes, parsePtr);
>>         ConvertTreeToTokens(start, numBytes,
>>         opTree, exprParsePtr->tokenPtr, parsePtr);
>>     } ...
>>
>> 2° the ParseExpr fonction
>>
>> int nb_paren=0;
>> int substExpressionContext=0;
>>
>> if(start[-1] == '[' && start[0] == '(' ) {
>>     substExpressionContext=1;
>>
>>     // Expression substitution
>>     start++;  //skip the open parenthesis '(' : it's part of the
>> expression substitution syntax
>>     numBytes--;
>> }
>>
>> ...
>>
>> case UNARY:
>>
>>         //////////////////////////////////
>>
>>         if (substExpressionContext == 1) {
>>
>>             // Beyond binary operators, there is Open paren, count it
>>
>>            if (start[0]== '(') {
>>
>>                 // Count the open parenthesis in this context
>>
>>                  nb_paren++;
>>            }
>>         }
>>
>> case BINARY: {
>>         ...
>>         if (substExpressionContext == 1) {
>>
>>             // Beyond binary operators, there is closed Paren, count it.
>>
>>             if (start[0] == ')') {
>>                 nb_paren--;
>>                 if (nb_paren == -1 && start[1] ==']') {
>>                 //// End of expression
>>                     parsePtr->commandSize = originalLength - numBytes - 1;
>>                     numBytes=0;
>>                     continue; // and exit the loop, since numbytes == 0 ;)
>>                  }
>>              }
>>         }
>>
>> ----------------------------------------
>>
>> I add also make it nestable, ie : set x [(1 + [(2+3)] )]
>>
>> in the function Parse_Expr :
>>
>> case SCRIPT : {
>>
>> ...
>>
>>     if (start[1] == '(') {
>>
>>             // an open braket followed by an open paren is denoting the
>> expression shorthand
>>
>>             tokenPtr->type = TCL_TOKEN_SUB_EXPR;
>>         } else {
>>             tokenPtr->type = TCL_TOKEN_COMMAND;
>>         }
>>
>> ...
>>
>> In the function TclCompileTokens (file tclCompile.c), I add :
>>
>> case TCL_TOKEN_SUB_EXPR :
>>         envPtr->line += adjust;
>>         TclCompileExpr(interp,  tokenPtr->start+1, tokenPtr->size-2,
>> envPtr, 0);
>>         envPtr->line -= adjust;
>>         numObjsToConcat++;
>>
>>         break;
>>
>> ---------------------
>>
>> Then, I can write :
>>
>> % set x [(1+1)]
>>
>> 2
>>
>> % set y [($x + [(1 + 1)] )]
>>
>> 4
>>
>> % set z [($y + [($x * [(1+1)] )] )]
>>
>> 8
>>
>> -----------------------------
>>
>> Surely there is corner cases that this prototype doesn't resolve. More
>> investigations are needed and it should be extensively tested, but this
>> prove that the [(...)] expression shorthand is possible at little cost.
>> Maybe even the TCL_TOKEN_SUB_EXPR Token could be used instead of creating a
>> synthetic string. I may investigate this las option later...
>>
>> Florent
>>
>>
>> _______________________________________________
>> Tcl-Core mailing list
>> Tcl...@li...
>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>>
>

Re: [TCLCORE] Variation on the Expr Shorthand

[EricT <tw...@gm...>]

Subject: Variation on the Expr Shorthand - Attribution

Hi Florent,

Regarding your comment "the famous hack of Eric Taylor" - I should clarify
for the historical record that the synthetic string technique originated
from Heinrich Martin's work in 2018. Heinrich and I collaborated on a
similar expression shorthand proposal using (...) syntax, and he pioneered
this parsing approach. TIP 672 builds on that foundation.

Credit where credit is due!

Best, Eric


On Wed, Oct 29, 2025 at 12:17 PM EricT <tw...@gm...> wrote:

> Hi Florent,
>
> Thank you for exploring alternative approaches to expression shorthand syntax! It's valuable to have different perspectives, and I appreciate the work you've put into your [(..)] prototype.
>
> A Safety Note on Buffer Access:
>
> I noticed a potential issue in your ParseTokens code:
>
>     } else if (src[0] == '[' && src[1] == '(') {
>
> This should check numBytes before accessing src[1]:
>
>     } else if (numBytes > 1 && src[0] == '[' && src[1] == '(') {
>
> ParseTokens is very sensitive to numBytes management. If numBytes goes negative or you read past the buffer, you can encounter:
>
> - Infinite loops (the while (numBytes && ...) continues with negative values)
> - Reading garbage memory past null terminators (especially in proc bodies)
> - Undefined behavior or crashes
>
> I spent several hours debugging exactly this issue in mode 3 $((...)) when the second closing paren was missing at the end of a proc body. The symptoms were subtle until numBytes reached -4000 in a tight loop allocating tokens each pass.
>
> Compatibility Consideration:
>
> As Peter noted, [( has the same class of compatibility issue that $() faces - both break rare but valid existing syntax. Worth considering in the design.
>
> A Question:
>
> I'm curious - in earlier discussions, I thought you were exploring [{...}] syntax. What led you to shift to [(..)]? The brace version seemed interesting as an alternative approach.
>
> Quote-in-Braces Issue:
>
> You mentioned "a quote inside braces would create an error" as a flaw in the basic parsing. Could you provide a specific test case that demonstrates this? In my testing with $=:
>
> % set foo $=([string length {text with "}])
> 11
>
> This works correctly, since the paren-matching algorithm treats braces as literal characters within expressions. Which of the three TIP 672 modes $( vs $= vs $(()) did you test? I'd like to understand what edge case you've identified so I can verify and address it if needed.
>
>
> Nested Expression Substitutions:
>
> I noticed your nested example. I'm curious whether nesting is a requirement of your implementation, or if you're demonstrating it as a feature? With expression shorthand, nesting typically isn't necessary since regular parentheses already provide grouping within expressions.
>
> One of the design goals for TIP 672 was to keep the syntax simple by not requiring nestability - the existing expression syntax handles grouping naturally. Is there a use case where nestable expression substitutions provide an advantage I'm missing?
>
> Thanks again for contributing to this discussion!
>
> Best regards,
> Eric
>
>
>
> On Wed, Oct 29, 2025 at 7:45 AM Florent Merlet <flo...@gm...>
> wrote:
>
>> Hi dear Tcl community,
>>
>> An Expr shorthand syntax has been a long time demand between us.
>>
>> Those discussions always focus on the syntax aspect of the subject :
>>
>>    - Like in bash $(...) or $((...))
>>    - Through an alias [= ...]
>>    - A new command (vexpr or let)
>>    - A word prefix {=}
>>    - ...
>>
>> A lot of TIPs exists on that matter. Numerous discussions occurs, which
>> never ended to get a consensus.
>>
>> That's because the look of this shorthand is a matter of taste. Everybody
>> has his own taste. Some people like fish when it's cooked in water, some
>> people like it when it's fried. Some people even don't like fish at all !
>> Every taste is in the nature.
>>
>> Everybody can agree that Tcl is a big and complex machinery, that must be
>> handled with care. So maybe the problem must be taken the other way round :
>>
>>    - Shall we deduce the Tcl C source code machinery from a new syntax,
>>    we had previously decided (the one doesn't make consensus)
>>    - Or shall we deduce the new syntax from the Tcl C source code
>>    machinery, as it exists ?
>>
>> My opinion is that it's better to deduce the syntax from the Tcl C source
>> code, rather than to deduce the Tcl source C code from the syntax .
>>
>> TIP 672 is hacking the variable substitution. To do this, it has to make
>> a very basic parsing of the expression to estimate its length. It has to
>> transmute a TCL_VARIABLE Token into a TCL_COMMAND token. It then use a call
>> to Tcl_ParseCommand on a synthetic string to check errors.
>>
>> This very basic parsing will make it buggy. For instance, a shorthand
>> expression can't be nested in another one. A quote inside braces would
>> create an error. To make this parsing strong, we would have to reinvent all
>> the expression parsing from scratch.
>>
>> But shall we create a new parsing expression routine for this shorthand
>> ?  No, there exist already an expression parsing machinery, that can handle
>> words between Quotes or Braces and can handle Nested Commands, exactly how
>> the Expr command do.
>>
>> « Deduce the shorthand syntax from the Tcl C source Code » imply to find
>> a syntax which allows us to use the existing machinery.
>>
>> That's what I'm trying now :
>>
>> As Expr is a command in Tcl, it seems logical to me to implement the
>> shorthand syntax in the Command branch "[" of parseToken procedure. That's
>> what I choosed.
>>
>> The second step is to parse the expression, so to go through
>> Tcl_ParseExpr routine, then to the ParseExpr routine. The difficulty here
>> is to get the end of the substitution script in the ParseExpr routine. If I
>> don't want to disturb parseExpr too much, it's better to choose, as
>> character which ends the expression script, a character that
>> is significant for this parser, so the main task of detecting it is already
>> done, but can be adapted gently.
>>
>> Maybe I could have used any of those operators : '+', '=', '-', '*', '(',
>> ')', '|',...etc. But I choosed to use ')' : infix language needs
>> parenthesis.
>>
>> That is how I defined the end of the expression substitution script to be
>> ")]". By symetry, I defined the beginning of the substitution script to be
>> "[(".
>>
>> Here is the genesis of my proposal of "[( ...)]" as a shorthand.
>>
>> To make it work, I had to used the same clever hacking than Eric Taylor :
>> create a synthethic string and parse it as a command.
>>
>> At the end, the [(...)] is working as expected (so far I've tested). Here
>> are the main changes I have done to accomplish it :
>>
>> In file Tcl_Parse.c : in function parseTokens,  I add a new branch in the
>> test
>>
>> ----------------------------------------
>>
>>  ... } else if (src[0] == '[' && src[1] == '(') {
>>
>> ///////////////////////////////////////////////////////////////////////
>>         /* Expression substition context */
>>         // to do : noSubstExpr
>>         Tcl_Parse *exprParsePtr;
>>         exprParsePtr =(Tcl_Parse *)TclStackAlloc(parsePtr->interp,
>> sizeof(Tcl_Parse));
>>
>>         src++;     // src == '['
>>         numBytes --;
>>         // Use it only to know the length of the expression, and store it
>> into exprParsePtr->commandSize
>>         Tcl_ParseExpr(parsePtr->interp, src, numBytes, exprParsePtr);
>>
>>         src++;  // src == '('
>>         numBytes --;
>>
>>         // Here is the famous hack of Eric Taylor
>>         Tcl_Size syntheticLen = exprParsePtr->commandSize + 9;  // "[expr
>> {" + expr + "}]"
>>
>>         char *synthetic = (char *)Tcl_Alloc(syntheticLen + 1);
>>
>>         memcpy(synthetic, "[expr {", 7);
>>         memcpy(synthetic + 7, src, exprParsePtr->commandSize);
>>
>>         memcpy(synthetic + 7 + exprParsePtr->commandSize, "}]", 3);
>>         synthetic[syntheticLen] = '\0';
>>         // Maybe a Tcl_Obj could be of use for memory management ?
>>
>>
>>         Tcl_Obj *exprObjCommand =
>> Tcl_NewStringObj(synthetic,syntheticLen);
>>
>>         src+=exprParsePtr->commandSize+2;
>>         numBytes-=exprParsePtr->commandSize+2;
>>
>>         TclStackFree(parsePtr->interp, exprParsePtr);
>>
>>         tokenPtr->type = TCL_TOKEN_COMMAND;
>>         tokenPtr->start = Tcl_GetStringFromObj(exprObjCommand, NULL);
>>         tokenPtr->size = syntheticLen;
>>         parsePtr->numTokens++;
>>
>>         continue;
>>
>> } else if (*src == '[') {...
>>
>> ---------------------------------------
>>
>> To detect the end and transfer the size of the parsed expression I had to
>> modify :
>>
>> 1° the Tcl_ParseExpr function :
>>
>> ... if (code == TCL_OK) {
>>     if(start[-1] == '[' && start[0] == '(' ) {
>>         // Expression Substitution Context : just transfer the size
>> information to the caller
>>              parsePtr->commandSize =exprParsePtr->commandSize;
>>     } else {
>>         TclParseInit(interp, start, numBytes, parsePtr);
>>         ConvertTreeToTokens(start, numBytes,
>>         opTree, exprParsePtr->tokenPtr, parsePtr);
>>     } ...
>>
>> 2° the ParseExpr fonction
>>
>> int nb_paren=0;
>> int substExpressionContext=0;
>>
>> if(start[-1] == '[' && start[0] == '(' ) {
>>     substExpressionContext=1;
>>
>>     // Expression substitution
>>     start++;  //skip the open parenthesis '(' : it's part of the
>> expression substitution syntax
>>     numBytes--;
>> }
>>
>> ...
>>
>> case UNARY:
>>
>>         //////////////////////////////////
>>
>>         if (substExpressionContext == 1) {
>>
>>             // Beyond binary operators, there is Open paren, count it
>>
>>            if (start[0]== '(') {
>>
>>                 // Count the open parenthesis in this context
>>
>>                  nb_paren++;
>>            }
>>         }
>>
>> case BINARY: {
>>         ...
>>         if (substExpressionContext == 1) {
>>
>>             // Beyond binary operators, there is closed Paren, count it.
>>
>>             if (start[0] == ')') {
>>                 nb_paren--;
>>                 if (nb_paren == -1 && start[1] ==']') {
>>                 //// End of expression
>>                     parsePtr->commandSize = originalLength - numBytes - 1;
>>                     numBytes=0;
>>                     continue; // and exit the loop, since numbytes == 0 ;)
>>                  }
>>              }
>>         }
>>
>> ----------------------------------------
>>
>> I add also make it nestable, ie : set x [(1 + [(2+3)] )]
>>
>> in the function Parse_Expr :
>>
>> case SCRIPT : {
>>
>> ...
>>
>>     if (start[1] == '(') {
>>
>>             // an open braket followed by an open paren is denoting the
>> expression shorthand
>>
>>             tokenPtr->type = TCL_TOKEN_SUB_EXPR;
>>         } else {
>>             tokenPtr->type = TCL_TOKEN_COMMAND;
>>         }
>>
>> ...
>>
>> In the function TclCompileTokens (file tclCompile.c), I add :
>>
>> case TCL_TOKEN_SUB_EXPR :
>>         envPtr->line += adjust;
>>         TclCompileExpr(interp,  tokenPtr->start+1, tokenPtr->size-2,
>> envPtr, 0);
>>         envPtr->line -= adjust;
>>         numObjsToConcat++;
>>
>>         break;
>>
>> ---------------------
>>
>> Then, I can write :
>>
>> % set x [(1+1)]
>>
>> 2
>>
>> % set y [($x + [(1 + 1)] )]
>>
>> 4
>>
>> % set z [($y + [($x * [(1+1)] )] )]
>>
>> 8
>>
>> -----------------------------
>>
>> Surely there is corner cases that this prototype doesn't resolve. More
>> investigations are needed and it should be extensively tested, but this
>> prove that the [(...)] expression shorthand is possible at little cost.
>> Maybe even the TCL_TOKEN_SUB_EXPR Token could be used instead of creating a
>> synthetic string. I may investigate this las option later...
>>
>> Florent
>>
>>
>> _______________________________________________
>> Tcl-Core mailing list
>> Tcl...@li...
>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>>
>

Re: [TCLCORE] Variation on the Expr Shorthand

[EricT <tw...@gm...>]

Hi Florent,

Thank you for exploring alternative approaches to expression shorthand
syntax! It's valuable to have different perspectives, and I appreciate
the work you've put into your [(..)] prototype.

A Safety Note on Buffer Access:

I noticed a potential issue in your ParseTokens code:

    } else if (src[0] == '[' && src[1] == '(') {

This should check numBytes before accessing src[1]:

    } else if (numBytes > 1 && src[0] == '[' && src[1] == '(') {

ParseTokens is very sensitive to numBytes management. If numBytes goes
negative or you read past the buffer, you can encounter:

- Infinite loops (the while (numBytes && ...) continues with negative values)
- Reading garbage memory past null terminators (especially in proc bodies)
- Undefined behavior or crashes

I spent several hours debugging exactly this issue in mode 3 $((...))
when the second closing paren was missing at the end of a proc body.
The symptoms were subtle until numBytes reached -4000 in a tight loop
allocating tokens each pass.

Compatibility Consideration:

As Peter noted, [( has the same class of compatibility issue that $()
faces - both break rare but valid existing syntax. Worth considering
in the design.

A Question:

I'm curious - in earlier discussions, I thought you were exploring
[{...}] syntax. What led you to shift to [(..)]? The brace version
seemed interesting as an alternative approach.

Quote-in-Braces Issue:

You mentioned "a quote inside braces would create an error" as a flaw
in the basic parsing. Could you provide a specific test case that
demonstrates this? In my testing with $=:

% set foo $=([string length {text with "}])
11

This works correctly, since the paren-matching algorithm treats braces
as literal characters within expressions. Which of the three TIP 672
modes $( vs $= vs $(()) did you test? I'd like to understand what edge
case you've identified so I can verify and address it if needed.


Nested Expression Substitutions:

I noticed your nested example. I'm curious whether nesting is a
requirement of your implementation, or if you're demonstrating it as a
feature? With expression shorthand, nesting typically isn't necessary
since regular parentheses already provide grouping within expressions.

One of the design goals for TIP 672 was to keep the syntax simple by
not requiring nestability - the existing expression syntax handles
grouping naturally. Is there a use case where nestable expression
substitutions provide an advantage I'm missing?

Thanks again for contributing to this discussion!

Best regards,
Eric



On Wed, Oct 29, 2025 at 7:45 AM Florent Merlet <flo...@gm...>
wrote:

> Hi dear Tcl community,
>
> An Expr shorthand syntax has been a long time demand between us.
>
> Those discussions always focus on the syntax aspect of the subject :
>
>    - Like in bash $(...) or $((...))
>    - Through an alias [= ...]
>    - A new command (vexpr or let)
>    - A word prefix {=}
>    - ...
>
> A lot of TIPs exists on that matter. Numerous discussions occurs, which
> never ended to get a consensus.
>
> That's because the look of this shorthand is a matter of taste. Everybody
> has his own taste. Some people like fish when it's cooked in water, some
> people like it when it's fried. Some people even don't like fish at all !
> Every taste is in the nature.
>
> Everybody can agree that Tcl is a big and complex machinery, that must be
> handled with care. So maybe the problem must be taken the other way round :
>
>    - Shall we deduce the Tcl C source code machinery from a new syntax,
>    we had previously decided (the one doesn't make consensus)
>    - Or shall we deduce the new syntax from the Tcl C source code
>    machinery, as it exists ?
>
> My opinion is that it's better to deduce the syntax from the Tcl C source
> code, rather than to deduce the Tcl source C code from the syntax .
>
> TIP 672 is hacking the variable substitution. To do this, it has to make a
> very basic parsing of the expression to estimate its length. It has to
> transmute a TCL_VARIABLE Token into a TCL_COMMAND token. It then use a call
> to Tcl_ParseCommand on a synthetic string to check errors.
>
> This very basic parsing will make it buggy. For instance, a shorthand
> expression can't be nested in another one. A quote inside braces would
> create an error. To make this parsing strong, we would have to reinvent all
> the expression parsing from scratch.
>
> But shall we create a new parsing expression routine for this shorthand
> ?  No, there exist already an expression parsing machinery, that can handle
> words between Quotes or Braces and can handle Nested Commands, exactly how
> the Expr command do.
>
> « Deduce the shorthand syntax from the Tcl C source Code » imply to find a
> syntax which allows us to use the existing machinery.
>
> That's what I'm trying now :
>
> As Expr is a command in Tcl, it seems logical to me to implement the
> shorthand syntax in the Command branch "[" of parseToken procedure. That's
> what I choosed.
>
> The second step is to parse the expression, so to go through Tcl_ParseExpr
> routine, then to the ParseExpr routine. The difficulty here is to get the
> end of the substitution script in the ParseExpr routine. If I don't want to
> disturb parseExpr too much, it's better to choose, as character which ends
> the expression script, a character that is significant for this parser, so
> the main task of detecting it is already done, but can be adapted gently.
>
> Maybe I could have used any of those operators : '+', '=', '-', '*', '(',
> ')', '|',...etc. But I choosed to use ')' : infix language needs
> parenthesis.
>
> That is how I defined the end of the expression substitution script to be
> ")]". By symetry, I defined the beginning of the substitution script to be
> "[(".
>
> Here is the genesis of my proposal of "[( ...)]" as a shorthand.
>
> To make it work, I had to used the same clever hacking than Eric Taylor :
> create a synthethic string and parse it as a command.
>
> At the end, the [(...)] is working as expected (so far I've tested). Here
> are the main changes I have done to accomplish it :
>
> In file Tcl_Parse.c : in function parseTokens,  I add a new branch in the
> test
>
> ----------------------------------------
>
>  ... } else if (src[0] == '[' && src[1] == '(') {
>
> ///////////////////////////////////////////////////////////////////////
>         /* Expression substition context */
>         // to do : noSubstExpr
>         Tcl_Parse *exprParsePtr;
>         exprParsePtr =(Tcl_Parse *)TclStackAlloc(parsePtr->interp,
> sizeof(Tcl_Parse));
>
>         src++;     // src == '['
>         numBytes --;
>         // Use it only to know the length of the expression, and store it
> into exprParsePtr->commandSize
>         Tcl_ParseExpr(parsePtr->interp, src, numBytes, exprParsePtr);
>
>         src++;  // src == '('
>         numBytes --;
>
>         // Here is the famous hack of Eric Taylor
>         Tcl_Size syntheticLen = exprParsePtr->commandSize + 9;  // "[expr
> {" + expr + "}]"
>
>         char *synthetic = (char *)Tcl_Alloc(syntheticLen + 1);
>
>         memcpy(synthetic, "[expr {", 7);
>         memcpy(synthetic + 7, src, exprParsePtr->commandSize);
>
>         memcpy(synthetic + 7 + exprParsePtr->commandSize, "}]", 3);
>         synthetic[syntheticLen] = '\0';
>         // Maybe a Tcl_Obj could be of use for memory management ?
>
>
>         Tcl_Obj *exprObjCommand = Tcl_NewStringObj(synthetic,syntheticLen);
>
>         src+=exprParsePtr->commandSize+2;
>         numBytes-=exprParsePtr->commandSize+2;
>
>         TclStackFree(parsePtr->interp, exprParsePtr);
>
>         tokenPtr->type = TCL_TOKEN_COMMAND;
>         tokenPtr->start = Tcl_GetStringFromObj(exprObjCommand, NULL);
>         tokenPtr->size = syntheticLen;
>         parsePtr->numTokens++;
>
>         continue;
>
> } else if (*src == '[') {...
>
> ---------------------------------------
>
> To detect the end and transfer the size of the parsed expression I had to
> modify :
>
> 1° the Tcl_ParseExpr function :
>
> ... if (code == TCL_OK) {
>     if(start[-1] == '[' && start[0] == '(' ) {
>         // Expression Substitution Context : just transfer the size
> information to the caller
>              parsePtr->commandSize =exprParsePtr->commandSize;
>     } else {
>         TclParseInit(interp, start, numBytes, parsePtr);
>         ConvertTreeToTokens(start, numBytes,
>         opTree, exprParsePtr->tokenPtr, parsePtr);
>     } ...
>
> 2° the ParseExpr fonction
>
> int nb_paren=0;
> int substExpressionContext=0;
>
> if(start[-1] == '[' && start[0] == '(' ) {
>     substExpressionContext=1;
>
>     // Expression substitution
>     start++;  //skip the open parenthesis '(' : it's part of the
> expression substitution syntax
>     numBytes--;
> }
>
> ...
>
> case UNARY:
>
>         //////////////////////////////////
>
>         if (substExpressionContext == 1) {
>
>             // Beyond binary operators, there is Open paren, count it
>
>            if (start[0]== '(') {
>
>                 // Count the open parenthesis in this context
>
>                  nb_paren++;
>            }
>         }
>
> case BINARY: {
>         ...
>         if (substExpressionContext == 1) {
>
>             // Beyond binary operators, there is closed Paren, count it.
>
>             if (start[0] == ')') {
>                 nb_paren--;
>                 if (nb_paren == -1 && start[1] ==']') {
>                 //// End of expression
>                     parsePtr->commandSize = originalLength - numBytes - 1;
>                     numBytes=0;
>                     continue; // and exit the loop, since numbytes == 0 ;)
>                  }
>              }
>         }
>
> ----------------------------------------
>
> I add also make it nestable, ie : set x [(1 + [(2+3)] )]
>
> in the function Parse_Expr :
>
> case SCRIPT : {
>
> ...
>
>     if (start[1] == '(') {
>
>             // an open braket followed by an open paren is denoting the
> expression shorthand
>
>             tokenPtr->type = TCL_TOKEN_SUB_EXPR;
>         } else {
>             tokenPtr->type = TCL_TOKEN_COMMAND;
>         }
>
> ...
>
> In the function TclCompileTokens (file tclCompile.c), I add :
>
> case TCL_TOKEN_SUB_EXPR :
>         envPtr->line += adjust;
>         TclCompileExpr(interp,  tokenPtr->start+1, tokenPtr->size-2,
> envPtr, 0);
>         envPtr->line -= adjust;
>         numObjsToConcat++;
>
>         break;
>
> ---------------------
>
> Then, I can write :
>
> % set x [(1+1)]
>
> 2
>
> % set y [($x + [(1 + 1)] )]
>
> 4
>
> % set z [($y + [($x * [(1+1)] )] )]
>
> 8
>
> -----------------------------
>
> Surely there is corner cases that this prototype doesn't resolve. More
> investigations are needed and it should be extensively tested, but this
> prove that the [(...)] expression shorthand is possible at little cost.
> Maybe even the TCL_TOKEN_SUB_EXPR Token could be used instead of creating a
> synthetic string. I may investigate this las option later...
>
> Florent
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] Variation on the Expr Shorthand

[da S. P. J <pet...@fl...>]

That has the same problem of breaking existing syntax.

    % proc ( {args} { puts $args }
    % ( hello
    hello


From: Florent Merlet <flo...@gm...>
Date: Wednesday, October 29, 2025 at 09:45
To: tcl...@li... <tcl...@li...>
Subject: [TCLCORE] Variation on the Expr Shorthand

CAUTION - EXTERNAL EMAIL:
This message originated from outside of your organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


Hi dear Tcl community,

An Expr shorthand syntax has been a long time demand between us.

Those discussions always focus on the syntax aspect of the subject :

  *   Like in bash $(...) or $((...))
  *   Through an alias [= ...]
  *   A new command (vexpr or let)
  *   A word prefix {=}
  *   ...

A lot of TIPs exists on that matter. Numerous discussions occurs, which never ended to get a consensus.

That's because the look of this shorthand is a matter of taste. Everybody has his own taste. Some people like fish when it's cooked in water, some people like it when it's fried. Some people even don't like fish at all ! Every taste is in the nature.

Everybody can agree that Tcl is a big and complex machinery, that must be handled with care. So maybe the problem must be taken the other way round :

  *   Shall we deduce the Tcl C source code machinery from a new syntax, we had previously decided (the one doesn't make consensus)
  *   Or shall we deduce the new syntax from the Tcl C source code machinery, as it exists ?

My opinion is that it's better to deduce the syntax from the Tcl C source code, rather than to deduce the Tcl source C code from the syntax .

TIP 672 is hacking the variable substitution. To do this, it has to make a very basic parsing of the expression to estimate its length. It has to transmute a TCL_VARIABLE Token into a TCL_COMMAND token. It then use a call to Tcl_ParseCommand on a synthetic string to check errors.

This very basic parsing will make it buggy. For instance, a shorthand expression can't be nested in another one. A quote inside braces would create an error. To make this parsing strong, we would have to reinvent all the expression parsing from scratch.

But shall we create a new parsing expression routine for this shorthand ?  No, there exist already an expression parsing machinery, that can handle words between Quotes or Braces and can handle Nested Commands, exactly how the Expr command do.

« Deduce the shorthand syntax from the Tcl C source Code » imply to find a syntax which allows us to use the existing machinery.

That's what I'm trying now :

As Expr is a command in Tcl, it seems logical to me to implement the shorthand syntax in the Command branch "[" of parseToken procedure. That's what I choosed.

The second step is to parse the expression, so to go through Tcl_ParseExpr routine, then to the ParseExpr routine. The difficulty here is to get the end of the substitution script in the ParseExpr routine. If I don't want to disturb parseExpr too much, it's better to choose, as character which ends the expression script, a character that is significant for this parser, so the main task of detecting it is already done, but can be adapted gently.

Maybe I could have used any of those operators : '+', '=', '-', '*', '(', ')', '|',...etc. But I choosed to use ')' : infix language needs parenthesis.

That is how I defined the end of the expression substitution script to be ")]". By symetry, I defined the beginning of the substitution script to be "[(".

Here is the genesis of my proposal of "[( ...)]" as a shorthand.

To make it work, I had to used the same clever hacking than Eric Taylor : create a synthethic string and parse it as a command.

At the end, the [(...)] is working as expected (so far I've tested). Here are the main changes I have done to accomplish it :

In file Tcl_Parse.c : in function parseTokens,  I add a new branch in the test

----------------------------------------

 ... } else if (src[0] == '[' && src[1] == '(') {
        ///////////////////////////////////////////////////////////////////////
        /* Expression substition context */
        // to do : noSubstExpr
        Tcl_Parse *exprParsePtr;
        exprParsePtr =(Tcl_Parse *)TclStackAlloc(parsePtr->interp, sizeof(Tcl_Parse));

        src++;     // src == '['
        numBytes --;
        // Use it only to know the length of the expression, and store it into exprParsePtr->commandSize
        Tcl_ParseExpr(parsePtr->interp, src, numBytes, exprParsePtr);

        src++;  // src == '('
        numBytes --;

        // Here is the famous hack of Eric Taylor
        Tcl_Size syntheticLen = exprParsePtr->commandSize + 9;  // "[expr {" + expr + "}]"

        char *synthetic = (char *)Tcl_Alloc(syntheticLen + 1);

        memcpy(synthetic, "[expr {", 7);
        memcpy(synthetic + 7, src, exprParsePtr->commandSize);

        memcpy(synthetic + 7 + exprParsePtr->commandSize, "}]", 3);
        synthetic[syntheticLen] = '\0';
        // Maybe a Tcl_Obj could be of use for memory management ?

        Tcl_Obj *exprObjCommand = Tcl_NewStringObj(synthetic,syntheticLen);

        src+=exprParsePtr->commandSize+2;
        numBytes-=exprParsePtr->commandSize+2;

        TclStackFree(parsePtr->interp, exprParsePtr);

        tokenPtr->type = TCL_TOKEN_COMMAND;
        tokenPtr->start = Tcl_GetStringFromObj(exprObjCommand, NULL);
        tokenPtr->size = syntheticLen;
        parsePtr->numTokens++;

        continue;

} else if (*src == '[') {...

---------------------------------------

To detect the end and transfer the size of the parsed expression I had to modify :

1° the Tcl_ParseExpr function :

... if (code == TCL_OK) {
    if(start[-1] == '[' && start[0] == '(' ) {
        // Expression Substitution Context : just transfer the size information to the caller
             parsePtr->commandSize =exprParsePtr->commandSize;
    } else {
        TclParseInit(interp, start, numBytes, parsePtr);
        ConvertTreeToTokens(start, numBytes,
        opTree, exprParsePtr->tokenPtr, parsePtr);
    } ...

2° the ParseExpr fonction

int nb_paren=0;
int substExpressionContext=0;

if(start[-1] == '[' && start[0] == '(' ) {
    substExpressionContext=1;

    // Expression substitution
    start++;  //skip the open parenthesis '(' : it's part of the expression substitution syntax
    numBytes--;
}

...

case UNARY:

        //////////////////////////////////

        if (substExpressionContext == 1) {

            // Beyond binary operators, there is Open paren, count it

           if (start[0]== '(') {

                // Count the open parenthesis in this context

                 nb_paren++;
           }
        }

case BINARY: {
        ...
        if (substExpressionContext == 1) {

            // Beyond binary operators, there is closed Paren, count it.

            if (start[0] == ')') {
                nb_paren--;
                if (nb_paren == -1 && start[1] ==']') {
                //// End of expression
                    parsePtr->commandSize = originalLength - numBytes - 1;
                    numBytes=0;
                    continue; // and exit the loop, since numbytes == 0 ;)
                 }
             }
        }

----------------------------------------

I add also make it nestable, ie : set x [(1 + [(2+3)] )]

in the function Parse_Expr :

case SCRIPT : {

...

    if (start[1] == '(') {

            // an open braket followed by an open paren is denoting the expression shorthand

            tokenPtr->type = TCL_TOKEN_SUB_EXPR;
        } else {
            tokenPtr->type = TCL_TOKEN_COMMAND;
        }

...

In the function TclCompileTokens (file tclCompile.c), I add :

case TCL_TOKEN_SUB_EXPR :
        envPtr->line += adjust;
        TclCompileExpr(interp,  tokenPtr->start+1, tokenPtr->size-2, envPtr, 0);
        envPtr->line -= adjust;
        numObjsToConcat++;

        break;

---------------------

Then, I can write :

% set x [(1+1)]

2

% set y [($x + [(1 + 1)] )]

4

% set z [($y + [($x * [(1+1)] )] )]

8

-----------------------------

Surely there is corner cases that this prototype doesn't resolve. More investigations are needed and it should be extensively tested, but this prove that the [(...)] expression shorthand is possible at little cost. Maybe even the TCL_TOKEN_SUB_EXPR Token could be used instead of creating a synthetic string. I may investigate this las option later...

Florent

[TCLCORE] Variation on the Expr Shorthand

[Florent M. <flo...@gm...>]

Hi dear Tcl community,

An Expr shorthand syntax has been a long time demand between us.

Those discussions always focus on the syntax aspect of the subject :

  * Like in bash $(...) or $((...))
  * Through an alias [= ...]
  * A new command (vexpr or let)
  * A word prefix {=}
  * ...

A lot of TIPs exists on that matter. Numerous discussions occurs, which 
never ended to get a consensus.

That's because the look of this shorthand is a matter of taste. 
Everybody has his own taste. Some people like fish when it's cooked in 
water, some people like it when it's fried. Some people even don't like 
fish at all ! Every taste is in the nature.

Everybody can agree that Tcl is a big and complex machinery, that must 
be handled with care. So maybe the problem must be taken the other way 
round :

  * Shall we deduce the Tcl C source code machinery from a new syntax,
    we had previously decided (the one doesn't make consensus)
  * Or shall we deduce the new syntax from the Tcl C source code
    machinery, as it exists ?

My opinion is that it's better to deduce the syntax from the Tcl C 
source code, rather than to deduce the Tcl source C code from the syntax .

TIP 672 is hacking the variable substitution. To do this, it has to make 
a very basic parsing of the expression to estimate its length. It has to 
transmute a TCL_VARIABLE Token into a TCL_COMMAND token. It then use a 
call to Tcl_ParseCommand on a synthetic string to check errors.

This very basic parsing will make it buggy. For instance, a shorthand 
expression can't be nested in another one. A quote inside braces would 
create an error. To make this parsing strong, we would have to reinvent 
all the expression parsing from scratch.

But shall we create a new parsing expression routine for this shorthand 
?  No, there exist already an expression parsing machinery, that can 
handle words between Quotes or Braces and can handle Nested Commands, 
exactly how the Expr command do.

« Deduce the shorthand syntax from the Tcl C source Code » imply to find 
a syntax which allows us to use the existing machinery.

That's what I'm trying now :

As Expr is a command in Tcl, it seems logical to me to implement the 
shorthand syntax in the Command branch "[" of parseToken procedure. 
That's what I choosed.

The second step is to parse the expression, so to go through 
Tcl_ParseExpr routine, then to the ParseExpr routine. The difficulty 
here is to get the end of the substitution script in the ParseExpr 
routine. If I don't want to disturb parseExpr too much, it's better to 
choose, as character which ends the expression script, a character that 
is significant for this parser, so the main task of detecting it is 
already done, but can be adapted gently.

Maybe I could have used any of those operators : '+', '=', '-', '*', 
'(', ')', '|',...etc. But I choosed to use ')' : infix language needs 
parenthesis.

That is how I defined the end of the expression substitution script to 
be ")]". By symetry, I defined the beginning of the substitution script 
to be "[(".

Here is the genesis of my proposal of "[( ...)]" as a shorthand.

To make it work, I had to used the same clever hacking than Eric Taylor 
: create a synthethic string and parse it as a command.

At the end, the [(...)] is working as expected (so far I've tested). 
Here are the main changes I have done to accomplish it :

In file Tcl_Parse.c : in function parseTokens,  I add a new branch in 
the test

----------------------------------------

  ... } else if (src[0] == '[' && src[1] == '(') {
///////////////////////////////////////////////////////////////////////
         /* Expression substition context */
         // to do : noSubstExpr
         Tcl_Parse *exprParsePtr;
         exprParsePtr =(Tcl_Parse *)TclStackAlloc(parsePtr->interp, 
sizeof(Tcl_Parse));

         src++;     // src == '['
         numBytes --;
         // Use it only to know the length of the expression, and store 
it into exprParsePtr->commandSize
         Tcl_ParseExpr(parsePtr->interp, src, numBytes, exprParsePtr);

         src++;  // src == '('
         numBytes --;

         // Here is the famous hack of Eric Taylor
         Tcl_Size syntheticLen = exprParsePtr->commandSize + 9; // 
"[expr {" + expr + "}]"

         char *synthetic = (char *)Tcl_Alloc(syntheticLen + 1);

         memcpy(synthetic, "[expr {", 7);
         memcpy(synthetic + 7, src, exprParsePtr->commandSize);

         memcpy(synthetic + 7 + exprParsePtr->commandSize, "}]", 3);
         synthetic[syntheticLen] = '\0';
         // Maybe a Tcl_Obj could be of use for memory management ?

         Tcl_Obj *exprObjCommand = Tcl_NewStringObj(synthetic,syntheticLen);

         src+=exprParsePtr->commandSize+2;
         numBytes-=exprParsePtr->commandSize+2;

         TclStackFree(parsePtr->interp, exprParsePtr);

         tokenPtr->type = TCL_TOKEN_COMMAND;
         tokenPtr->start = Tcl_GetStringFromObj(exprObjCommand, NULL);
         tokenPtr->size = syntheticLen;
         parsePtr->numTokens++;

         continue;

} else if (*src == '[') {...

---------------------------------------

To detect the end and transfer the size of the parsed expression I had 
to modify :

1° the Tcl_ParseExpr function :

... if (code == TCL_OK) {
     if(start[-1] == '[' && start[0] == '(' ) {
         // Expression Substitution Context : just transfer the size 
information to the caller
              parsePtr->commandSize =exprParsePtr->commandSize;
     } else {
         TclParseInit(interp, start, numBytes, parsePtr);
         ConvertTreeToTokens(start, numBytes,
         opTree, exprParsePtr->tokenPtr, parsePtr);
     } ...

2° the ParseExpr fonction

int nb_paren=0;
int substExpressionContext=0;

if(start[-1] == '[' && start[0] == '(' ) {
     substExpressionContext=1;

     // Expression substitution
     start++;  //skip the open parenthesis '(' : it's part of the 
expression substitution syntax
     numBytes--;
}

...

case UNARY:

         //////////////////////////////////

         if (substExpressionContext == 1) {

             // Beyond binary operators, there is Open paren, count it

            if (start[0]== '(') {

                 // Count the open parenthesis in this context

                  nb_paren++;
            }
         }

case BINARY: {
         ...
         if (substExpressionContext == 1) {

             // Beyond binary operators, there is closed Paren, count it.

             if (start[0] == ')') {
                 nb_paren--;
                 if (nb_paren == -1 && start[1] ==']') {
                 //// End of expression
                     parsePtr->commandSize = originalLength - numBytes - 1;
                     numBytes=0;
                     continue; // and exit the loop, since numbytes == 0 ;)
                  }
              }
         }

----------------------------------------

I add also make it nestable, ie : set x [(1 + [(2+3)] )]

in the function Parse_Expr :

case SCRIPT : {

...

     if (start[1] == '(') {

             // an open braket followed by an open paren is denoting the 
expression shorthand

             tokenPtr->type = TCL_TOKEN_SUB_EXPR;
         } else {
             tokenPtr->type = TCL_TOKEN_COMMAND;
         }

...

In the function TclCompileTokens (file tclCompile.c), I add :

case TCL_TOKEN_SUB_EXPR :
         envPtr->line += adjust;
         TclCompileExpr(interp,  tokenPtr->start+1, tokenPtr->size-2, 
envPtr, 0);
         envPtr->line -= adjust;
         numObjsToConcat++;

         break;

---------------------

Then, I can write :

% set x [(1+1)]

2

% set y [($x + [(1 + 1)] )]

4

% set z [($y + [($x * [(1+1)] )] )]

8

-----------------------------

Surely there is corner cases that this prototype doesn't resolve. More 
investigations are needed and it should be extensively tested, but this 
prove that the [(...)] expression shorthand is possible at little cost. 
Maybe even the TCL_TOKEN_SUB_EXPR Token could be used instead of 
creating a synthetic string. I may investigate this las option later...

Florent

Re: [TCLCORE] Backport of TIP 729

[Harald O. <har...@el...>]

Am 28.10.2025 um 22:31 schrieb Emiliano:
> Hi all.
> 
> FYI, I've withdrawn my own tkcargo package and instead backported the
> implementation of TIP 729 (aka [tk attribtable]), along with tests and
> documentation. It is avaiable here:
> 
> https://chiselapp.com/user/egavilan/repository/attribtable
> 
> To use it with both 8.X and 9.0 simply use [package require attribtable].
> It was tested on linux against 9.0 and 8.6.14
> 
> Regards
> 

Great !
Perhaps, you may add this information as remark to the TIP?
Backport stuff is always great.

Thanks,
Harald

[TCLCORE] Backport of TIP 729

[Emiliano <emi...@gm...>]

Hi all.

FYI, I've withdrawn my own tkcargo package and instead backported the
implementation of TIP 729 (aka [tk attribtable]), along with tests and
documentation. It is avaiable here:

https://chiselapp.com/user/egavilan/repository/attribtable

To use it with both 8.X and 9.0 simply use [package require attribtable].
It was tested on linux against 9.0 and 8.6.14

Regards

-- 
Emiliano

Re: [TCLCORE] Mousewheel for entry

[Harald O. <har...@el...>]

Am 28.10.2025 um 11:51 schrieb Zaumseil René via Tcl-Core:
> Hello
> 
> I would propose the following additional bindings.
> 
> These bindings allow display of hidden parts of entries.
> 
> bind Entry <MouseWheel> {tk::MouseWheel %W x %D -40.0 units}
> 
> bind TEntry <MouseWheel> {tk::MouseWheel %W x %D -40.0 units}
> 
> Can this be done without a tip?
Yes, I think so. But a ticket would be great.

Thanks for all,
Harald

[TCLCORE] Mousewheel for entry

[Zaumseil R. <RZa...@kk...>]

Hello

I would propose the following additional bindings.
These bindings allow display of hidden parts of entries.

bind Entry <MouseWheel> {tk::MouseWheel %W x %D -40.0 units}
bind TEntry <MouseWheel> {tk::MouseWheel %W x %D -40.0 units}

Can this be done without a tip?


Regards
rene