Menu
▾
▴
Re: [TCLCORE] [=] for concise expressions (was Re: TIP 672 Implementation Complete - Ready for Sponsorship)
|
From: Colin M. <col...@ya...> - 2025-11-04 14:57:11
|
Also relying on an "unsupported" facility is fine for a prototype but
risky for production code.
On 04/11/2025 14:51, Colin Macleod via Tcl-Core wrote:
>
> Hi Eric,
>
> That's very neat!
>
> Yes, a pure Tcl version could go into TclLib. I still think it may be
> worth trying a C implementation though. The work-around that's needed
> for array references [= 2* $a(b)] would defeat the caching, so it
> would be good to speed up the parsing if possible. Also I think your
> caching may be equivalent to doing byte-compilation, in which case it
> may make sense to use the framework which already exists for that.
>
> Colin.
>
> On 04/11/2025 01:18, EricT wrote:
>> that is:
>>
>> if {[info exist ::cache($exp)]} {
>> tailcall ::tcl::unsupported::assemble $::cache($exp)
>> }
>>
>> (hate gmail!)
>>
>>
>> On Mon, Nov 3, 2025 at 5:17 PM EricT <tw...@gm...> wrote:
>>
>> and silly of me, it should be:
>> if {[info exist ::cache($exp)]} {
>> tailcall ::tcl::unsupported::assemble $::cache($exp)
>> }
>>
>>
>> On Mon, Nov 3, 2025 at 4:50 PM EricT <tw...@gm...> wrote:
>>
>> With a debug line back in plus the tailcall:
>>
>> proc = args {
>> set exp [join $args]
>> if { [info exist ::cache($exp)] } {
>> return [tailcall ::tcl::unsupported::assemble
>> $::cache($exp)]
>> }
>> set tokens [tokenise $exp]
>> deb1 "TOKENS = '$tokens'"
>> set code [compile $tokens]
>> deb1 "GENERATED CODE:\n$code\n"
>> puts "exp= |$exp| code= |$code| ifexist: [info exist
>> ::cache($exp)]"
>> set ::cache($exp) $code
>> uplevel [list ::tcl::unsupported::assemble $code]
>> }
>>
>> % set a 5
>> 5
>> % set b 10
>> 10
>> % = a + b
>> exp= |a + b| code= |push a; loadStk; push b; loadStk; add; |
>> ifexist: 0
>> 15
>> % = a + b
>> 15
>>
>> % time {= a + b} 1000
>> 1.73 microseconds per iteration
>>
>>
>> Faster still!
>>
>> I thought the uplevel was needed to be able to get the local
>> variables, seems not.
>>
>> % proc foo arg {set a 5; set b 10; set c [= a+b+arg]}
>> % foo 5
>> exp= |a+b+arg| code= |push a; loadStk; push b; loadStk; add;
>> push arg; loadStk; add; | ifexist: 0
>> 20
>> % foo 5
>> 20
>>
>> % proc foo arg {global xxx; set a 5; set b 10; set c [=
>> a+b+arg+xxx]}
>>
>> % set xxx 100
>> 100
>> % foo 200
>> 315
>> % time {foo 200} 10000
>> 2.1775 microseconds per iteration
>>
>> % parray cache
>> cache(a + b) = push a; loadStk; push b; loadStk; add;
>> cache(a+b+arg) = push a; loadStk; push b; loadStk; add;
>> push arg; loadStk; add;
>> cache(a+b+arg+xxx) = push a; loadStk; push b; loadStk; add;
>> push arg; loadStk; add; push xxx; loadStk; add;
>>
>>
>> Very Impressive, great job Colin! Great catch Don!
>>
>> Eric
>>
>>
>>
>>
>> On Mon, Nov 3, 2025 at 4:22 PM Donald Porter via Tcl-Core
>> <tcl...@li...> wrote:
>>
>> Check what effect replacing [uplevel] with [tailcall] has.
>>
>>> On Nov 3, 2025, at 7:13 PM, EricT <tw...@gm...> wrote:
>>>
>>> Subject: Your bytecode expression evaluator - impressive
>>> results with caching!
>>>
>>> Hey Colin:
>>>
>>> I took a look at your bytecode-based expression
>>> evaluator and was intrigued by the approach. I made a
>>> small modification to add caching and the results are
>>> really impressive. Here's what I changed:
>>>
>>> proc = args {
>>> set exp [join $args]
>>> if {[info exist ::cache($exp)]} {
>>> return [uplevel [list
>>> ::tcl::unsupported::assemble $::cache($exp)]]
>>> }
>>> set tokens [tokenise $exp]
>>> deb1 "TOKENS = '$tokens'"
>>> set code [compile $tokens]
>>> deb1 "GENERATED CODE:\n$code\n"
>>> set ::cache($exp) $code
>>> uplevel [list ::tcl::unsupported::assemble $code]
>>> }
>>>
>>> The cache is just a simple array lookup - one line to
>>> store, one line to retrieve. But the performance impact
>>> is huge:
>>>
>>> Performance Tests
>>>
>>> Without caching
>>> % time {= 1 + 2} 1000
>>> 24.937 microseconds per iteration
>>>
>>> With caching
>>> % time {= 1 + 2} 1000
>>> 1.8 microseconds per iteration
>>>
>>> That's a 13x speedup! The tokenize and parse steps were
>>> eating about 92% of the execution time.
>>>
>>> The Real Magic: Bare Variables + Caching
>>>
>>> What really impressed me is how well your bare variable
>>> feature synergizes with caching:
>>>
>>> % set a 5
>>> 5
>>> % set b 6
>>> 6
>>> % = a + b
>>> 11
>>> % time {= a + b} 1000
>>> 2.079 microseconds per iteration
>>>
>>> Now change the variable values
>>> % set a 10
>>> 10
>>> % = a + b
>>> 16
>>> % time {= a + b} 1000
>>> 2.188 microseconds per iteration
>>>
>>> The cache entry stays valid even when the variable
>>> values change! Why? Because the bytecode stores variable
>>> names, not values:
>>>
>>> push a; loadStk; push b; loadStk; add;
>>>
>>> The loadStk instruction does runtime lookup, so:
>>> - Cache key is stable: "a + b"
>>> - Works for any values of a and b
>>> - One cache entry handles all value combinations
>>>
>>> Compare this to if we used $-substitution:
>>>
>>> = $a + $b # With a=5, b=6 becomes "5 + 6"
>>> = $a + $b # With a=10, b=6 becomes "10 + 6" -
>>> different cache key!
>>>
>>> Every value change would create a new cache entry or
>>> worse, a cache miss.
>>>
>>> Comparison to Other Approaches
>>>
>>> Tcl's expr: about 0.40 microseconds
>>> Direct C evaluator: about 0.53 microseconds
>>> Your cached approach: about 1.80 microseconds
>>> Your uncached approach: about 24.9 microseconds
>>>
>>> With caching, you're only 3-4x slower than a direct C
>>> evaluator.
>>>
>>>
>>> My Assessment
>>>
>>> Your design is excellent. The bare variable feature
>>> isn't just syntax sugar - it's essential for good cache
>>> performance. The synergy between:
>>>
>>> 1. Bare variables leading to stable cache keys
>>> 2. Runtime lookup keeping cache hot
>>> 3. Simple caching providing dramatic speedup
>>>
>>> makes this really elegant.
>>>
>>> My recommendation: Keep it in Tcl! The implementation is
>>> clean, performance is excellent (1.8 microseconds is
>>> plenty fast), and converting to C would add significant
>>> complexity for minimal gain (maybe getting to about 1.0
>>> microseconds).
>>>
>>> The Tcl prototype with caching is actually the right
>>> solution here. Sometimes the prototype IS the product!
>>>
>>> Excellent work on this. The bytecode approach really
>>> shines with caching enabled.
>>>
>>> On Sun, Nov 2, 2025 at 10:14 AM Colin Macleod via
>>> Tcl-Core <tcl...@li...> wrote:
>>>
>>> Hi again,
>>>
>>> I've now made a slightly more serious prototype, see
>>> https://cmacleod.me.uk/tcl/expr_ng
>>>
>>> This is a modified version of the prototype I wrote
>>> for tip 676. It's still in Tcl, but doesn't use
>>> `expr`. It tokenises and parses the input, then
>>> generates TAL bytecode and uses
>>> ::tcl::unsupported::assemble to run that. A few
>>> examples:
>>>
>>> (bin) 100 % set a [= 3.0/4]
>>> 0.75
>>> (bin) 101 % set b [= sin(a*10)]
>>> 0.9379999767747389
>>> (bin) 102 % set c [= (b-a)*100]
>>> 18.79999767747389
>>> (bin) 103 % namespace eval nn {set d [= 10**3]}
>>> 1000
>>> (bin) 104 % set e [= a?nn::d:b]
>>> 1000
>>> (bin) 105 % = {3 + [pwd]}
>>> Calc: expected start of expression but found '[pwd]'
>>> (bin) 106 % = {3 + $q}
>>> Calc: expected start of expression but found '$q'
>>> (bin) 107 % = sin (12)
>>> -0.5365729180004349
>>>
>>> (bin) 108 % array set rr {one 1 two 2 three 3}
>>> (bin) 110 % = a * rr(two)
>>> Calc: expected operator but found '('
>>> (bin) 111 % = a * $rr(two)
>>> 1.5
>>>
>>> - You can use $ to get an array value substituted
>>> before the `=` code sees the expression.
>>>
>>> (bin) 112 % string repeat ! [= nn::d / 15]
>>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>
>>> Colin.
>>>
>>> On 02/11/2025 09:04, Donal Fellows wrote:
>>>> Doing the job properly would definitely involve
>>>> changing the expression parser, with my suggested
>>>> fix being to turn all bare words not otherwise
>>>> recognised as constants or in positions that look
>>>> like function calls (it's a parser with some
>>>> lookahead) into simple variable reads (NB: C
>>>> resolves such ambiguities within itself
>>>> differently, but that's one of the nastiest parts
>>>> of the language). We would need to retain $ support
>>>> for resolving ambiguity (e.g., array reads vs
>>>> function calls; you can't safely inspect the
>>>> interpreter to resolve it at the time of compiling
>>>> the expression due to traces and unknown handlers)
>>>> as well as compatibility, but that's doable as it
>>>> is a change only in cases that are currently errors.
>>>>
>>>> Adding assignment is quite a bit trickier, as that
>>>> needs a new major syntax class to describe the left
>>>> side of the assignment. I suggest omitting that
>>>> from consideration at this stage.
>>>>
>>>> Donal.
>>>>
>>>> -------- Original message --------
>>>> From: Colin Macleod via Tcl-Core
>>>> <tcl...@li...>
>>>> <mailto:tcl...@li...>
>>>> Date: 02/11/2025 08:13 (GMT+00:00)
>>>> To: Pietro Cerutti <ga...@ga...>
>>>> <mailto:ga...@ga...>
>>>> Cc: tcl...@li..., av...@lo...
>>>> Subject: Re: [TCLCORE] Fwd: TIP 672 Implementation
>>>> Complete - Ready for Sponsorship
>>>>
>>>> Indeed, this toy implementation doesn't handle that:
>>>>
>>>> % = sin (12)
>>>> can't read "sin": no such variable
>>>>
>>>> I'm not sure that's serious, but it could be fixed
>>>> in a C implementation.
>>>>
>>> _______________________________________________
>>> Tcl-Core mailing list
>>> Tcl...@li...
>>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>>>
>>> _______________________________________________
>>> Tcl-Core mailing list
>>> Tcl...@li...
>>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>>
>> _______________________________________________
>> Tcl-Core mailing list
>> Tcl...@li...
>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>>
>>
>>
>> _______________________________________________
>> Tcl-Core mailing list
>> Tcl...@li...
>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core |
