tcl-quadcode — Development of the tclquadcode Tcl Native Compilation System

[tclquadcode] Type of 'initException'

[Kevin K. <kev...@gm...>]

I implemented last night (on the 'inline' branch) the 'initException'
sequence that Donal suggested, and found that it worked quite well
indeed (once I got rid of a misplaced close-brace that took forever to
track down). Right now, it's implemented only for 'invokeExpanded'
with wrong numbers of arguments, but I can do a fast follow-on for
'invoke'.

As I look at it, however, I am realizing that the approach I'm using
(generate terrible code and let optimization clean it up) is not
threading 'jumpMaybe' correctly, because the available type
information doesn't inform it that the result of the 'initException'
always fails. This winds up having a cascading effect in code quality,
I'm afraid.

Would it cause code generation tremendous heartburn if 'initException'
were to return a simple FAIL (rather than a FAIL STRING) if its return
code argument is {literal 1}? That would make life much easier for the
optimizer. I can generate just the 'initException', and code in a
subsequent pass
(https://core.tcl.tk/tclquadcode/artifact?ln=491-519&name=78060ce0e981c3c7)
will tidy up the 'jumpMaybe' and detect that the following 'jump' is
unreachable.

If this isn't feasible, I can still deal with it, by rewriting the
entire code burst

    moveToCallFrame cf2 cr1 {literal name1} {var value11} ...
    invoke res1 cf2 proc args...
    retrieveResult res2 res1
    extractCallFrame cf3 res1
    moveFromCallFrame {var value12} cf3 {literal name1}
        ...
    jumpMaybe {bb fail} res2
    jump {bb ok}

rather than just the [invoke], but that's rather more work.

(We're also most likely going to have to come up with a type calculus
for break, continue, return, but that's not today's problem.)

Re: [tclquadcode] 'invoke' with wrong mumber of arguments

[Donal K. F. <don...@ma...>]

On 08/01/2018 19:28, Kevin Kenny wrote:
> I'd really like to have a way to emit code for the "wrong # args' case 
> without trying to call a proc, since wherever the arg check happens, it 
> won't be inside the quadcode, giving me problems if an invocation with 
> possible wrong args happens and I want to inline the target procedure. 
> Moreover, I think that the current behaviour is likely a time bomb - I'm 
> sure that the specialized procedure with the wrong arg list is useless - 
> and it may be sitting there waiting to tickle other bugs, since it 
> likely also contains incorrect code.

Exceptions are thrown by preparing them with initException and jumping 
to the relevant exception handler. The initException configures the 
exception within the Tcl interpreter, and produces a FAIL that can be 
discarded (it's useful when dealing with code that only *may* throw an 
exception, such as when compiling the finally clause of a [try]).

The arguments that are useful here are:

   initException TGT MSG OPT CODE LVL

MSG is the error message. OPT is the options dictionary *without* the 
-code or -level options, and the -errorinfo option will be initialised 
from the message if it is absent. CODE is the Tcl return code, and LVL 
is the level. For wrong#args, you'd use:

   initException {temp TGTID} \
       {literal {wrong # args: should be ...}} \
       {literal {-errorcode {TCL WRONGARGS}}} \
       {literal 1} \
       {literal 0}

You also need to get the @debug-related pseudoopcodes right because the 
information they contain is used to fill in the error information. So 
long as this throw happens when the right one is current, it will all 
work out right. (If not, it's a bug that I must fix; I think I chased 
all those bits down.) By right one, I mean that if we know the number of 
arguments that an invoke *should* take, we can replace the invoke with a 
version that checks the count and throws, otherwise doing a real invoke; 
substituting the plain invoke opcode like that will put the throw in the 
correct place; it's still part of the expansion of what was an 
INST_INVOKE_STK1, INST_INVOKE_STK4 or INST_INVOKE_EXPANDED in the 
original bytecode, and of the same Tcl command implementation.

The result type of the initException above is FAIL STRING because the 
type of the first input is STRING, but we know because we're not using 0 
0 that this is a definite failure so we don't need to use a jumpMaybe to 
test. (All that side of things gets important with non-constant 
arguments, which can't be analysed so easily.)

The exception handling is just the usual stuff; the jumpMaybe opcode 
does nothing complicated.

Donal.

[tclquadcode] 'invoke' with wrong mumber of arguments

[Kevin K. <kev...@gm...>]

Over the weekend, I managed to do the promised refactoring of
'invokeExpanded' handling (which was needed to deal with procedures that
were invoked with {*}, might have the wrong number of arguments, and might
[upvar] into the current callframe, to avoid trying to unify return types
with and without CALLFRAME).

It mostly works.

For no apparent reason, in the case where there is a procedure invocation
that is known statically to have the wrong number of arguments.

What the code appears to do in such a case is to specialize the procedure
with an incorrect arg list. There may be a procedure called p with
arguments {a b c}, but apparently, if p is invoked with no args, there's a
specialization to p {}.  I have Absolutely No Idea how that makes it
through. I suspect that in the cases where I've seen such a beast, it gets
tidied up later because of unreachable code, but I'm by no means positive.

I can easily arrange for callers not to call a proc with too few or too
many args. If I'm calling a compiled proc with 'invokeExpanded,' I already
do that by translating correct 'invokeExpanded' to 'invoke', leaving only
wrong-number-of-args cases

I'd really like to have a way to emit code for the "wrong # args' case
without trying to call a proc, since wherever the arg check happens, it
won't be inside the quadcode, giving me problems if an invocation with
possible wrong args happens and I want to inline the target procedure.
Moreover, I think that the current behaviour is likely a time bomb - I'm
sure that the specialized procedure with the wrong arg list is useless -
and it may be sitting there waiting to tickle other bugs, since it likely
also contains incorrect code.

Current code is in the 'inline' branch.
https://core.tcl.tk/tclquadcode/artifact?ln=1121-1129&name=5edd4283b25da012
are test cases that once failed and now succeed.

With this refactoring done, I can move on to trying to inline procedures
that return CALLFRAME FAIL SOMETHING.

Re: [tclquadcode] Procedure inlining

[Donal K. F. <don...@ma...>]

On 02/01/2018 16:17, Donal K. Fellows wrote:
> Yes. There will need to be some sort of widening on the normal exit
> path, and the error exit path will need something a little smarter than
> what is there right now. I've added a "procLeave" opcode on the exit
> path that does the on-procedure-exit exception transformation (things
> like converting an unscoped [break] into an error) while binding in the
> name of the procedure; that's on trunk because it works independently of
> everything else. But now we need to make the return types match up right
> so that "returnException" can be removed entirely.
> 
> Thinking about it, it'd also make "procLeave" produce a FAIL instead of
> the current INT; the real return code will end up packed within the
> FAIL. That can then be widened to the correct result type, and the
> widening code will grow a bit more hair.

This is now done. Every Tcl procedure that has a possibility of an 
erroring exit gets a codeburst at the end of its quadcode that massages 
the current result code into a FAIL (that's one of the things that the 
‘procLeave’ opcode now does, as well as some evil stuff to sort out the 
error stack, etc.) and returns that (which is widened as appropriate) 
and that now appears to work. As that's the only path out of a procedure 
that produces a FAIL, it should be easy to handle the routing of error 
paths after inlining so that they don't need to go through widening at 
all. But the ‘procLeave’ is not eliminatable with the current class of 
analysis. (LLVM might be able to do a better job; it can see further 
under the covers of error handling.)

The metadata stuff is still broken. But I fixed a couple of dopey errors 
in the [dict update] implementation instead (wrong issued code, wrong 
narrowing of ‘arrayExists’); they were cancelling out in a lot of test 
cases, but that didn't make either of them right.

Donal.

Re: [tclquadcode] Procedure inlining

[Donal K. F. <don...@ma...>]

On 01/01/2018 06:19, Kevin Kenny wrote:
> That sounds rather like what I had in mind. One caveat:
> 'return' at present appears to take a FOO and cause the procedure to
> return a Just FOO. A cursory reading of the code suggests that it won't
> work with a FAIL without some tweaking. (Which would be a good thing
> anyway, if it leads to simplification elsewhere.)

Yes. There will need to be some sort of widening on the normal exit
path, and the error exit path will need something a little smarter than
what is there right now. I've added a "procLeave" opcode on the exit
path that does the on-procedure-exit exception transformation (things
like converting an unscoped [break] into an error) while binding in the
name of the procedure; that's on trunk because it works independently of
everything else. But now we need to make the return types match up right
so that "returnException" can be removed entirely.

Thinking about it, it'd also make "procLeave" produce a FAIL instead of
the current INT; the real return code will end up packed within the
FAIL. That can then be widened to the correct result type, and the
widening code will grow a bit more hair.

Then it will be just a matter of fixing the metadata...

Donal.

Re: [tclquadcode] Procedure inlining

[Kevin K. <kev...@gm...>]

Best wishes for a safe, healthy and prosperous 2018!
(Did you know that 2018! has exactly 555 8's in its decimal representation?)

On Sun, Dec 31, 2017 at 9:00 PM, Donal K. Fellows <
don...@ma...> wrote:

> On 30/12/2017 21:48, Kevin Kenny wrote:
>
>> First, and most important by far, is that I can't inline a procedure that
>> might return failure. I've simply not found a way to do it with the
>> existing quadcode instruction repertoire. I can see the 'returnException'
>> instruction in the procedure to be inlined, but I don't see a good way to
>> construct the object that the procedure would have returned. (Ideally,
>> including the stack trace that would have appeared, for which I have all
>> necessary information, I think.) This is blocking all but about a dozen of
>> the potential opportunities for inlining.
>>
>
> Yes, that's important and will require a new opcode. The key things it
> will need to do will be to release the stack frame (I presume we have a
> plan to create new stack frames too?) and process the procedure exit
> sequence. That opcode will need to know what procedure it is actually in
> (currently, 'returnException' uses [namespace tail $cmd]).
>

I'd really like to keep the callframe management separate, because I
need to do the error information even when the inlined procedure has a
suppressed callframe. There is a fair subset of procedures that don't
actually need a frame.


> We might wish to split our current 'returnException', which would
> coincidentally allow the error exit sequence to use 'return'.
>

That sounds rather like what I had in mind. One caveat:
'return' at present appears to take a FOO and cause the procedure to
return a Just FOO. A cursory reading of the code suggests that it won't
work with a FAIL without some tweaking. (Which would be a good thing
anyway, if it leads to simplification elsewhere.)

There will also need to be some review of 'param' usage. That opcode's
> implementation accesses the bytecode definition dict in some cases (with
> a warning about "default injection" if it does so; I think we don't spit
> those out at the moment so we might be OK though I don't know how
> thorough our testing is).


In inlined procedures, 'param's get replaced with copies. That part's
working well already. When I'm calling any compiled procedure, default
injection isn't needed; I supply defaults on caller side. And even
'invokeExpanded' will get it right (except that I'm wrestling with a
bug at the moment - it turns out that [upvar] in an 'invokeExpanded'
fails rather nastily. I never contemplated having CALLFRAME FAIL FOO
pass through a phi - and I don't *want* to contemplate it, so I have
to refactor how I'm handling 'invokeExpanded'.


> A second limitation is that I don't seem to have any way in the
>> @debug quadcodes to indicate that I've brought in code from a
>> different source file. That will mess up stack traces, compile-time
>> messages and so on, if a procedure in one file inlines a procedure
>> defined in a different file.
>>
>
> There will need to be something for the shift in source file
> (spitballing: @debug-file {} {literal /path/to/file}) and something for
> the shift in procedure. (The order of these things is likely to be
> important.)
>
> I'm hoping that Donal will be able to help me out.
>>
>
> Of course. But it might take quite a bit of tinkering to get the debug
> info exactly correct.


Sure. Let me know how I can help you help me! (By the way,
I don't think that TIP 86 information for a [proc] inside a
[namespace eval] is making it all the way through. Another
mystery to track down.)


> I suspect that in this specific case, LLVM is smart enough to inline
>> the 'impure' procedure itself and generates approximately the same
>> code.
>>
>
> Reading the (rather long) output of
>
>   tclsh8.6 demos/perftest/tester.tcl -just impure -asm 1
>
> indicates that this indeed the case. Indeed, it goes further in some
> cases and inlines the result into the thunk function that provides the
> actual Tcl command. (What's curious is that it doesn't when we use an
> external optimiser. I guess the llvmtcl internal one has looser limits
> on that sort of thing.)


ISTR that inlining in the default clang configuration is deferred until
link-time optimization. If we're not looking at output from the LLVM
linker, we may not see that stuff.

I have enough to go a little farther before I get stuck. I'd like to
have non-TCL_OK returns working before tackling the harder
parts, which means that the first thing I need is the separated
'returnException' operation. I can generate it easily enough,
including passing through the '@debug-file', '@debug-line'
and '@debug-script' values that are current, so you don't
have to look them up. (It's easier for me to look them up, they
follow the dominator tree.) Let me know what you need the
operands to be. (They should not ordinarily, I think, need the
callframe, since we want this part to work in proc's with omitted
frames.)

I suspect you've done more thinking than I have about how you
want to handle push and pop of callframes. I am imagining
that a fair amound of callframe work can turn static if we know
that we're in an invocation of proc B from proc A, and that
the inlining is safe. There's also no need to worry about
recursion at that stage. By that point, every connected component
of more than one node in the call graph has had its head
node marked 'never inline'.

That sort of thing is what allows me to make inlining work now.
The code is copied inline, 'return' is replaced with a copy
and a jump to the procedure exit, 'param' is replaced with
a copy from the arg to 'invoke', and all variables, temps, basic
blocks and split markers are renumbered. That turns
out to be a subtle beauty of SSA. I can have a variable 'x'
in the caller, and a variable 'x' in the callee, and they
simply will have different instance numbers and data
flows, and it all works out. OK, preserving SSA is a little
hairy when doing this sort of code surgery, but I've
gotten good at that.

Is there any reason I shouldn't merge what I have so far
into trunk? All tests pass, except for some new ones that
are still commented out. It simply inlines a lot less than
it otherwise might.

Re: [tclquadcode] Procedure inlining

[Donal K. F. <don...@ma...>]

On 30/12/2017 21:48, Kevin Kenny wrote:
> First, and most important by far, is that I can't inline a procedure 
> that might return failure. I've simply not found a way to do it with the 
> existing quadcode instruction repertoire. I can see the 
> 'returnException' instruction in the procedure to be inlined, but I 
> don't see a good way to construct the object that the procedure would 
> have returned. (Ideally, including the stack trace that would have 
> appeared, for which I have all necessary information, I think.) This is 
> blocking all but about a dozen of the potential opportunities for inlining.

Yes, that's important and will require a new opcode. The key things it
will need to do will be to release the stack frame (I presume we have a
plan to create new stack frames too?) and process the procedure exit
sequence. That opcode will need to know what procedure it is actually in
(currently, 'returnException' uses [namespace tail $cmd]).

We might wish to split our current 'returnException', which would
coincidentally allow the error exit sequence to use 'return'.

There will also need to be some review of 'param' usage. That opcode's
implementation accesses the bytecode definition dict in some cases (with
a warning about "default injection" if it does so; I think we don't spit
those out at the moment so we might be OK though I don't know how
thorough our testing is).

> A second limitation is that I don't seem to have any way in the
> @debug quadcodes to indicate that I've brought in code from a
> different source file. That will mess up stack traces, compile-time
> messages and so on, if a procedure in one file inlines a procedure
> defined in a different file.

There will need to be something for the shift in source file
(spitballing: @debug-file {} {literal /path/to/file}) and something for
the shift in procedure. (The order of these things is likely to be
important.)

> I'm hoping that Donal will be able to help me out.

Of course. But it might take quite a bit of tinkering to get the debug
info exactly correct.

> I suspect that in this specific case, LLVM is smart enough to inline
> the 'impure' procedure itself and generates approximately the same
> code.

Reading the (rather long) output of

   tclsh8.6 demos/perftest/tester.tcl -just impure -asm 1

indicates that this indeed the case. Indeed, it goes further in some
cases and inlines the result into the thunk function that provides the
actual Tcl command. (What's curious is that it doesn't when we use an
external optimiser. I guess the llvmtcl internal one has looser limits
on that sort of thing.)

Donal.

[tclquadcode] Procedure inlining

[Kevin K. <kev...@gm...>]

I have an extremely rough first cut at an implementation of procedure
inlining, on the 'inline' branch. While the eventual objective is to make
procedures that do [uplevel] behave as macros, in order to make it possible
to try to compile them, it's not nearly there yet.

A couple of the limitations seem to be fairly silly, and I suspect that
they'll be straightforward to fix.

First, and most important by far, is that I can't inline a procedure that
might return failure. I've simply not found a way to do it with the
existing quadcode instruction repertoire. I can see the 'returnException'
instruction in the procedure to be inlined, but I don't see a good way to
construct the object that the procedure would have returned. (Ideally,
including the stack trace that would have appeared, for which I have all
necessary information, I think.) This is blocking all but about a dozen of
the potential opportunities for inlining.

A second limitation is that I don't seem to have any way in the @debug
quadcodes to indicate that I've brought in code from a different source
file. That will mess up stack traces, compile-time messages and so on, if a
procedure in one file inlines a procedure defined in a different file.

Beyond these issues, the work on these issues also exposed a couple of bugs:
    https://core.tcl.tk/tclquadcode/tktview?name=58e3e71962
    https://core.tcl.tk/tclquadcode/tktview?name=4f8bd5f5b2

Once these are settled, the next big step will be to allow invocation of
procs that use the callframe - and this is a necessary prerequisite before
we can actually address [uplevel]. What's going to be interesting there is
that we will now have multiple callframe objects; one for the inline
procedure and one for the procedure that embeds it. I can identify readily
the points at which the embedded callframe must be stacked and unstacked,
and it will be easy to bind moveToCallFrame and moveFromCallFrame (and
friends) to the appropriate frame. But I really don't understand callframe
management well enough to implement this without a lot of study.

I'm hoping that Donal will be able to help me out.

For what it's worth, I'm really pleased with the result of embedding
'impure' into 'impure-caller'. It begins to look really, really
minimalistic; I'd expect that it could easily be as fast as C.

Procedure: ::impure-caller:
0: entry {} {literal {}}
1: @debug-line {} {literal 1259}
2: @debug-script {} {literal {impure 10 10000 10}}
3: @debug-line {} {literal 1229}
4: @debug-script {} {literal {set x 0}}
5: @debug-line {} {literal 1230}
6: @debug-script {} {literal {set i $a}}
7: @debug-script {} {literal {for {set i $a} {$i < $b} {incr i $c} {
        set x [expr {$x + $i}]
    }}}
8: {widenTo 60 INT} {var x 8} {literal 0}
9: copy {temp phi 9} {literal 10}
10: phi {var x 10} {var x 8} {pc 9} {var x 25} {pc 33}
11: phi {var i 11} {temp phi 9} {pc 9} {var i 30} {pc 33}
12: lt {temp 12 12} {var i 11} {literal 10000}
13: jumpTrue {pc 19} {temp 12 12}
14: free {} {temp 12 12}
15: free {} {var i 11}
16: @debug-line {} {literal 1233}
17: @debug-script {} {literal {return $x}}
18: return {} Nothing {var x 10}
19: free {} {temp 12 12}
20: @debug-line {} {literal 1231}
21: @debug-script {} {literal {expr {$x + $i}}}
22: add {temp 12 22} {var x 10} {var i 11}
23: free {} {var x 10}
24: @debug-script {} {literal {set x [expr {$x + $i}]}}
25: copy {var x 25} {temp 12 22}
26: free {} {temp 12 22}
27: @debug-line {} {literal 1230}
28: @debug-script {} {literal {for {set i $a} {$i < $b} {incr i $c} {
        set x [expr {$x + $i}]
    }}}
29: @debug-script {} {literal {incr i $c}}
30: add {var i 30} {var i 11} {literal 10}
31: free {} {var i 11}
32: @debug-script {} {literal {for {set i $a} {$i < $b} {incr i $c} {
        set x [expr {$x + $i}]
    }}}
33: jump {pc 10}

Unfortunately, this didn't yield a substantial performance gain, at least
on my machine. I suspect that in this specific case, LLVM is smart enough
to inline the 'impure' procedure itself and generates approximately the
same code. That said, the real purpose of this exercise is to comvert
proc's into macros in order to enable at least the simple uses of
[uplevel]. which is surely going to go beyond LLVM's powers of analysis.

Kevin

Re: [tclquadcode] Arrays

[Donal K. F. <don...@ma...>]

On 03/12/2017 09:16, Donal K. Fellows wrote:
> I'm pleased to say that my branch with making Tcl arrays be first-class 
> objects seems to be working. It's not finished yet (there's a suspicious 
> number of "FIXME"s still in the code) and is neither cleaned up nor 
> documented, but is passing our current tests.

I've now merged this to trunk; we can now have arrays in local variables 
that are modified from standard Tcl code that we call without crippling 
performance penalties or strongly divergent semantics.

Donal.

[tclquadcode] Arrays

[Donal K. F. <don...@ma...>]

I'm pleased to say that my branch with making Tcl arrays be first-class 
objects seems to be working. It's not finished yet (there's a suspicious 
number of "FIXME"s still in the code) and is neither cleaned up nor 
documented, but is passing our current tests. (I've not yet done a 
performance comparison with trunk.) Better yet, as the arrays are real 
Tcl arrays when in the callframe — and the cost of moving them in and 
out isn't silly like it would be with dicts (i.e., it is shuffling 
pointers instead of iterating over the whole dict/array) — it will 
support [upvar] and access from standard Tcl commands much better.

At the quadcode level, this is all done through the introduction of a 
new type bit, ARRAY, which indicates that the value is an array. When 
mixed with a STRING, it means that the value is either an array or a 
string. It also mixes correctly with flags like NEXIST, FAIL or CALLFRAME.

To go with this, there's a slew of new quadcodes, including:

   arrayExists — is the value an array?
   throwIfNotArray — throw if value doesn't support array ops
   throwNotArray — throw, value definitely known to not support array ops
   throwIfArray — throw if value does support array ops
   throwIsArray — throw, value definitely known to support array ops
   extractScalar — get the non-array component
   extractArray — get the array component

   initArray — create a new array value
   initArrayIfNotExists — create array if arg was NEXIST, else pass

   arrayElementExists - is element present in array
   arrayGet — read element from array (yields NEXIST STRING)
   arraySet — write element to array, yielding updated ARRAY
   arrayUnset — remove element from array, yielding updated ARRAY

I may have missed some: this is ongoing work after all. :-)

Higher-level operations like incrementing, lappending, etc. are 
described by composing these other basic operations. The pattern of 
handling conditional throws and so on is modelled after the way we 
tackle NEXISTs.


At the implementation level, an ARRAY is a reference type. It is 
implemented as a pointer to a struct of type ARRAYIMPL, which contains 
three elements: the pointer to the TclVarHashTable (standard Tcl 
internal type), a reference count (I hypothesise that this never gets 
larger than 2 and then only momentarily) and a "provenance" bit, that 
gets set when the TclVarHashTable is linked from a callframe and so is 
not safe to delete directly. As we can't put arrays inside arrays and 
always give array variables names, the reference count handling is 
trivial in practice and operations that transform an array can actually 
just update it and return it.

There's also an ARRAYSTRING which is a simple tagged "union" of STRING 
and ARRAY.


I also had to add another new quadcode, setReturnCode, to get some of 
our error handling right (it's issued where an INST_BEGIN_CATCH4 would 
be). It's got no result at all, and its sole side effect is to reset the 
current Tcl result code to zero. Without it, I was getting weird effects 
from [try/finally] in a loop with caught failures. I'll backport that 
stuff to trunk along with a few fixes to exception handling implementations.

Donal.

[tclquadcode] llvmtcl 3.8 released

[Donal K. F. <don...@ma...>]

I'm very happy to announce that I've just done the release of version 
3.8 of llvmtcl, which is the layer binding to LLVM that allows 
tclquadcode to issue code. The main extra feature that it has mainly 
adds the ability to build on top of LLVM 5.0, while still supporting 
earlier versions (I've tested back to LLVM 3.8). You can get the source 
code from the release page on github.

    https://github.com/dkfellows/llvmtcl/releases

The trunk of tclquadcode has been adapted to support this release, which 
is almost but not 100% backward compatible. (LLVM altered the definition 
of some of its intrinsics slightly in LLVM 5, and that simply can't be 
concealed if your code happens to use those intrinsics.)

Donal.

[tclquadcode] LLVM 5

[Donal K. F. <don...@ma...>]

This is a heads-up because of a change (by others) that I can't neatly 
conceal.

I've added support for LLVM 5 to llvmtcl, but one of the changes that 
introduces is to the signature of some of the intrinsics that we use 
(one of the arguments to llvm.lifetime.start and llvm.lifetime.end ends 
up changing to be of variable type instead of fixed type) and that's not 
something that llvmtcl can conceal from the script level. This means 
that you'll need to update to use the changes I made today to the trunk 
of tclquadcode *if you are using LLVM 5*. Older LLVMs don't need the 
changes, of course, but should continue to work fine. Once you've got a 
build of llvmtcl against LLVM 5, you'll need to work with tclquadcode 
that knows how to pass the correct type arguments in when selecting 
those intrinsics.

Naturally, the change that caused this was not announced and the fact 
that the signature of these intrinsics has changed is not listed in the 
LLVM documentation. (I guess I ought to file a bug report with them, but 
I hate using bugzilla and they make user registration annoying too.)

On the plus side, I've added in sufficient checking that the crashes 
that getting those type arguments wrong causes (because LLVM reports 
most errors by doing the equivalent of Tcl_Panic, and only gives error 
messages with those in debug mode) should now be a thing of the past, 
and that works — and has been tested — with LLVMs back at least as far 
as 3.8 (I don't have anything older on this system at the moment).

Donal.

Re: [tclquadcode] New stuff for array accesses

[Donal K. F. <don...@ma...>]

On 12/11/2017 20:06, Kevin Kenny wrote:
> OK, I'm kind of with you on this. I think that there are, however, a
> significant fraction of cases where we can detect that an array
> doesn't escape from our compiled code and still use the (much faster)
> dict implementation. I'm not sure I'm ready to give it up, partly
> because so far, whenever we have heavy variable access through the
> callframe, we wind up with test cases that are actually slower than
> the interpreted code. (Extra boxing/unboxing steps, I think...)

I don't think it will make too much difference; the ARRAY type I'm 
thinking of is not much more complex than a DICT in the first place, 
except that it doesn't have the conversion to/from STRING. A consequence 
is that I'll need to introduce a 'throwIfArray' and 'throwIfNotArray' to 
make the types work. Still at the deeply experimental stage.

Donal.

Re: [tclquadcode] New stuff for array accesses

[Kevin K. <kev...@gm...>]

On Sun, Nov 12, 2017 at 6:37 AM, Donal K. Fellows
<don...@ma...> wrote:
> *SUMMARY:* I think I need to change arrays to use a new internal type that
> isn't a STRING, but which can be put in or removed from a Tcl_Var
> efficiently. I'll call such a thing an ARRAY for the moment.

OK, I'm kind of with you on this. I think that there are, however, a
significant fraction of cases where we can detect that an array
doesn't escape from our compiled code and still use the (much faster)
dict implementation. I'm not sure I'm ready to give it up, partly
because so far, whenever we have heavy variable access through the
callframe, we wind up with test cases that are actually slower than
the interpreted code. (Extra boxing/unboxing steps, I think...)

>> You appear to have missed a few places that have to be adapted to the
>> new codes:
>>
>> callframe.tcl (callFrameMovesAfter):
>>
>>      Operations that return FAIL have to be called out here.  Ignore
>>      the 'default' case - the mention of 'directSet' and friends there
>>      is erroneous. (Moreover, I've found a bug there, I think. Will
>>      fix,)
>
>
> FWIW, if an operation produces a FAIL then surely that's something that can
> be known from the type and not from being explicitly identified? That's
> assuming that the rule is as simple as that, of course. ;-)

The problem here is that the callFrameMotion pass runs before SSA, and hence
necessarily before type analysis. What it does is to insert moveToCallFrame
and moveFromCallFrame instructions before and after any instruction that
potentially updates a linked variable. At that point, we don't even know the
variable name (it's still in a temporary derived from the Tcl stack, we
haven't done any copy propagation yet!), don't have any type information, and
for 'invoke' and friends, don't even know what's being invoked.

Given the difficulties of adding code to SSA form after the fact, it's easiest
to simply insert these moves everywhere and then optmize away the ones that
aren't relevant. There is a twist: we don't allow FAIL to hit stored
variables, so the instructions actually bracket the possibly-failing
instruction and the associated'extractMaybe'.  ('extractCallframe' is handled
this way, too.)

[tclquadcode] Mapping the Code (was Re: New stuff for array accesses)

[Donal K. F. <don...@ma...>]

On 12/11/2017 11:37, Donal K. Fellows wrote:
> Also, thanks for the list of places I need to beware of. That's very 
> useful.

Thinking about this, it occurs to me that we could do with some crude 
maps of the code that says where to look for key things so that we can 
find our ways around each other's code. Yes, these things will change 
over time, but even an obsolete map is better than _terra incognita_.

Here's what's what in the code generator. As of 11/11/2017.

Front-End Code
--------------
jit.tcl
    Front-end drivers to connect the pieces together. (Probably badly
    misnamed; just not sure what to call it.)
config.tcl
    Configuration code for the code generator.

Quadcode Translation
--------------------
compile.tcl
    Takes quadcode and issues LLVM IR. The core of the code generator.

Binding Layer
-------------
build.tcl
    The interface between the code issuer and the back end implementation
    functions. (Glue. Lots of long-winded glue. See also the automatic
    type widener in struct.tcl.)
tycon.tcl
    Maps implementation types and constants.

Tcl Runtime Interface
---------------------
tclapi.tcl
    The binding to Tcl's API functions. (Exactly what is generated
    depends on whether we're building in stubs mode or not, but the rest
    of the code is entirely unaware.)
macros.tcl
    Versions of macros in Tcl's API. (Macros are complex to convert
    across, as we have exactly zero access to the C preprocessor.)
thunk.tcl
    The code to generate bindings of the functions we create to Tcl
    commands. (Without this, calling generated code from user code would
    be impossible.)

Inlineable Functions Implementing Quadcodes
-------------------------------------------
stdlib.tcl
    Definitions of functions that implement quadcodes. (These are all
    designed to be inlined.)
mathlib.tcl
    Math related functions (extracted from stdlib.tcl for reasons of file
    length).
varframe.tcl
    Tcl variable and callframe related functions (extracted from
    stdlib.tcl for reasons of file length).

LLVM API Binding
----------------
llvmbuilder.tcl
    Code to *safely* issue LLVM opcodes. (This is the part that I'm
    considering migrating into C or C++ first.)
debug.tcl
    Code to connect to the LLVM debugging metadata system.
struct.tcl
    Definitions of structural entities such as modules/code-units,
    functions and basic blocks. (Also includes definitions of all Tcl
    structures and the type widening logic used in build.tcl; those bits
    probably need to move elsewhere)


In terms of classes, there are these:

llvmEntity — support methods (logging, warnings, closures)
   TclCompiler — code generator core
     TclInterproceduralCompiler — miscellaneous wrappers
   BuildSupport — definitions of how to write an inlineable function
     LLVMBuilder — base instruction issuer that integrates with debugging
                   support layer
       Builder — adds the autowidening type system (through parsing of
                 method names)
     ThunkBuilder — interfaces to Tcl runtime
   Module — LLVM major code unit
   Function — LLVM function
   Block — LLVM basic block
   Debugging — LLVM debugging metadata generator

I'm not at all convinced that I've got this all right yet. Some parts 
are definitely good ideas, but others creak at the seams.

Donal.

Re: [tclquadcode] New stuff for array accesses

[Donal K. F. <don...@ma...>]

*SUMMARY:* I think I need to change arrays to use a new internal type 
that isn't a STRING, but which can be put in or removed from a Tcl_Var 
efficiently. I'll call such a thing an ARRAY for the moment.


On 11/11/2017 16:52, Kevin Kenny wrote:
> You added the correct result type to types.tcl - that's good.

I did the places that I knew about or that raised there heads above the 
parapet with a warning.

Also, thanks for the list of places I need to beware of. That's very useful.

> You appear to have missed a few places that have to be adapted to the
> new codes:
> 
> callframe.tcl (callFrameMovesAfter):
> 
>      Operations that return FAIL have to be called out here.  Ignore
>      the 'default' case - the mention of 'directSet' and friends there
>      is erroneous. (Moreover, I've found a bug there, I think. Will
>      fix,)

FWIW, if an operation produces a FAIL then surely that's something that 
can be known from the type and not from being explicitly identified? 
That's assuming that the rule is as simple as that, of course. ;-)

> constfold.tcl {constfold):
> 
>      We know that all the direct variable access operations cannot
>      be subject to constant folding. That's the default behaviour,
>      but there's a noisy warning message that will be silenced by
>      enumerating the instructions here.

I found and fixed that one as it happens.

> deadcode.tcl (unkillable):
> 
>      Operations with side effects such as modifying nonlocal variables
>      cannot be removed even if their results are unused, and have to be
>      enumerated here.
> 
> upvar.tcl (upvarProcEffect):
> 
>      Operations that access nonlocal variables have to be
>      enumerated here. They have to set the 'readsGlobal' and/or
>      'writesGlobal' attributes of the result as appropriate;
>      the result is no longer 'pure', and if the operation makes the
>      proc have side effects, the result is no longer 'killable'.
> 
> I'll go in and fix these - they're all easy, except for the bug I
> spotted in callframe.tcl - but I'd have appreciated a note to the
> mailing list about what's going on!

The real problem is that these are things which I didn't know about the 
existence of. That's why I have a default case in the big switch in 
codegen/compile.tcl that throws a noisy error; anything unexpected will 
definitely force me to take action. :-)

> Moving forward with array accesses -
> 
> What you're doing is kind of a half-fix. I don't think we're painting
> ourselves into a corner with it, exactly, quite, but we do need to
> think about it moving forward, given the number of things we've
> already implemented with the replacement of arrays with dicts.

I know it is a hacky half-fix. We need the operations on my side anyway 
(because there's quite a bit of code that accesses some arrays by global 
name) but there's going to need to be a lot more analysis. (If nothing 
else, they could modify any current callframe if they don't include a 
namespace separator in the name, and I don't know how to express the 
determination of that.)

> What concerns me here is the other end of code generation, where
> someone says 'gets stdin line($lineno)'. That's not going to get fixed
> with a simple rule that says 'direct accesses are actual arrays,
> other accesses are dicts.' The good news is that I already have the
> logic at least to identify that the 'gets' will be accessing an
> unknown var in the callframe. (I have more information than that,
> that I'm not using, because of course the variable name is the result
> of 'strcat' that could be deconstructed to determine that it *is* an
> array and the array name is constant. But that's a bridge too far
> initially.)

Yes. The local array logic is currently wrong as it doesn't produce 
variables of the correct form. It'll need a whole tranche of new opcodes 
to fix. Thinking about it, what I'd actually like is an ARRAY type that 
is a reference type which is not a subtype of STRING (but does admit 
intermixing with FAIL and NEXIST) and which I can convert the array 
operations to use; that'd be practical to put in and remove from a 
Tcl_Var in a callframe efficiently, yet wouldn't require the callframe 
to be present.

Implementation-wise, it'd be a pointer to a TclVarHashTable.

In fact, the more I think about this idea (which I thought of during 
breakfast this morning ;-)) the more I like it. Provided we assume we 
are free of traces (as we do in many other places) it'll work pretty great.

> I'm wondering if The Right Thing is to say that arrays may be demoted
> to dicts if (a) they are callframe-local, (b) there's nothing about,
> anywhere in the proc, that causes unknown callframe vars (or
> specifically those named vars) to be modified. We'd have translate.tcl
> generate the array operations always initially, and then a compiler
> pass (to be designed) could do the rewrite to use dict operations when
> it's known to be safe to do so.

Yes, but using an ARRAY type as outlined above is perhaps a bit better 
as it is more easily mixed with callframes and the cost is pretty 
similar. I suspect (but don't know) that it'll be easier to analyse too.

> The whole stack of additional analysis makes the project kind of
> nasty.  We've put this off too long and piled too much other stuff on
> top of the existing logic. But that's the way of "do the simplest
> thing that could possibly work."

That's why I think the hack with using dicts for arrays can't stand, at 
least not if we're targeting a runtime of any 8.* Tcl. (9.0 could just 
declare that local arrays are dictionaries and then the problems would, 
well, not go away but be different…)

Donal.

Re: [tclquadcode] 'expand' branch merged

[Donal K. F. <don...@ma...>]

On 11/11/2017 22:54, Kevin Kenny wrote:
> I've done a 'merge --integrate' of the 'expand' branch into trunk.
> Quadcode now understands most uses of {*} and can even do a competent
> job of optimization with some of them.
> 
> I might not be doing all that much more quadcode development for a few
> days. In the meantime, let's try to triage what should come next. Are
> we trying next to go to full integration with Tcl's arrays? (I haven't
> thought much about that, but that's the way Donal seems to be
> heading.)

Actually, I've been working on padding out support for as much of Tcl's 
bytecode as possible, going broad rather than deep. :-) I've got a list 
of operations that remain to be done, and I'm aiming to get at least 
something sensible done for the majority of user code, even if the 
semantics are not entirely correct. I've got five bytecode ops left to 
do and they all touch callframes; e.g., evalStk and exprStk can pretty 
much do anything (especially for non-constant inputs) so they're going 
to remain expensive, but at least we can have a chance to generate 
correct code for them now that we have callframes. The support for [dict 
with] is rather more awkward. :-(

I don't think we should plan to do NRE or TclOO (a total of another 10 
bytecode ops) in the near future. They both require major surgery to 
things both for analysis and for code generation, and will take quite a 
while to implement.

In any case, the support for expansion is important and I'm intending to 
think more about local arrays (the hack I did simply won't work with 
[upvar] about, especially with uncompiled code) so there's that. It'll 
probably result in more quadcode operations (or allowing changing the 
possible number of arguments to them; I don't know which is better) but 
may work. I'll need to have a deeper look. One of the problems is that 
we've got typically two input operations (i.e., by LVT index and by 
name) and yet three basic implementation strategies: direct, via LVT and 
Tcl's irritatingly expensive boxed types, and via full name lookup 
(which is pretty much just using Tcl's standard C API).

Also, apologies for not saying much about this recently; I've mostly 
been working in my mornings, at times when anyone sensible in the US 
would be asleep, and I've kept getting interrupted before I wrote things 
up. It happens. :-)

Donal.

[tclquadcode] 'expand' branch merged

[Kevin K. <kev...@gm...>]

I've done a 'merge --integrate' of the 'expand' branch into trunk.
Quadcode now understands most uses of {*} and can even do a competent
job of optimization with some of them.

I might not be doing all that much more quadcode development for a few
days. In the meantime, let's try to triage what should come next. Are
we trying next to go to full integration with Tcl's arrays? (I haven't
thought much about that, but that's the way Donal seems to be
heading.)

Kevin

Re: [tclquadcode] New stuff for array accesses

[Kevin K. <kev...@gm...>]

On Sat, Nov 11, 2017 at 11:52 AM, Kevin Kenny <kev...@gm...> wrote:
> I see you're running ahead of me again (which is fine, I'm slow).
>
> You added a bunch of new quadcode instructions for implementing the
> ARRAY_STK bytecodes:

I managed to get hacks into all the methods that I mentioned in
my earlier message so that things at least appear to work for the
simple cases.

It would somewhat my life if we could change the specification
of the direct* operations so that directArrayGet, directArrayExists,
directGet, directExists are callframe accessors and the other direct*
operations are callframe mutators. At the earliest points that
I'm processing these operations, I can't be sure whether they've
been emitted because someone did 'set ::x $y' or something more
like 'set $x $y' - and for the latter case, I'll definitely need the callframe!
Those are both compiled to INST_STORE_STK4, so there's
no way to tell them apart until I know what the thing on the
stack represents.

I can try to take this one on as well, but I'd like to get back on
'invokeExpanded' first and get it to where I'm comfortable
merging. (All of your test cases pass, but I've got a couple of
more evil ones in my sandbox.)

[tclquadcode] New stuff for array accesses

[Kevin K. <kev...@gm...>]

Donal,

I see you're running ahead of me again (which is fine, I'm slow).

You added a bunch of new quadcode instructions for implementing the
ARRAY_STK bytecodes:

    directArrayAppend
    directArrayExists
    directArrayGet
    directArrayLappend
    directArrayLappendList
    directArraySet
    directArrayUnset

(I think that's the correct set; am I right?)

You added the correct result type to types.tcl - that's good.

You appear to have missed a few places that have to be adapted to the
new codes:

callframe.tcl (callFrameMovesAfter):

    Operations that return FAIL have to be called out here.  Ignore
    the 'default' case - the mention of 'directSet' and friends there
    is erroneous. (Moreover, I've found a bug there, I think. Will
    fix,)

constfold.tcl {constfold):

    We know that all the direct variable access operations cannot
    be subject to constant folding. That's the default behaviour,
    but there's a noisy warning message that will be silenced by
    enumerating the instructions here.

deadcode.tcl (unkillable):

    Operations with side effects such as modifying nonlocal variables
    cannot be removed even if their results are unused, and have to be
    enumerated here.

upvar.tcl (upvarProcEffect):

    Operations that access nonlocal variables have to be
    enumerated here. They have to set the 'readsGlobal' and/or
    'writesGlobal' attributes of the result as appropriate;
    the result is no longer 'pure', and if the operation makes the
    proc have side effects, the result is no longer 'killable'.

I'll go in and fix these - they're all easy, except for the bug I
spotted in callframe.tcl - but I'd have appreciated a note to the
mailing list about what's going on!


Moving forward with array accesses -

What you're doing is kind of a half-fix. I don't think we're painting
ourselves into a corner with it, exactly, quite, but we do need to
think about it moving forward, given the number of things we've
already implemented with the replacement of arrays with dicts.

What concerns me here is the other end of code generation, where
someone says 'gets stdin line($lineno)'. That's not going to get fixed
with a simple rule that says 'direct accesses are actual arrays,
other accesses are dicts.' The good news is that I already have the
logic at least to identify that the 'gets' will be accessing an
unknown var in the callframe. (I have more information than that,
that I'm not using, because of course the variable name is the result
of 'strcat' that could be deconstructed to determine that it *is* an
array and the array name is constant. But that's a bridge too far
initially.)

I'm wondering if The Right Thing is to say that arrays may be demoted
to dicts if (a) they are callframe-local, (b) there's nothing about,
anywhere in the proc, that causes unknown callframe vars (or
specifically those named vars) to be modified. We'd have translate.tcl
generate the array operations always initially, and then a compiler
pass (to be designed) could do the rewrite to use dict operations when
it's known to be safe to do so.

The safety analysis winds up spilling over into interprocedural
analysis as well. I've not put a lot of thought into it yet, but in
the common case where procedure A calls procedure B, passing the name
of a local variable that is an array, we want to be able to generated
a version of B that will accept "array as dict" if possible, and have
A call that. This would be done with the usual chaotic iteration in
the specializer: initially, every proc would accept any of its args
with 'array as dict' unless there's a local reason not to do so, and
then procs would retract their 'array as dict' contract if they are
detected to call anything that precludes it. Retraction of that
contract implies that their callers will have to be checked and issue
their own retractions if necessary. This is all good - it's how type
analysis works, how we track that calling a procedure might affect
nonlocal variables or scribble in the callframe, and so on.

The whole stack of additional analysis makes the project kind of
nasty.  We've put this off too long and piled too much other stuff on
top of the existing logic. But that's the way of "do the simplest
thing that could possibly work."

Re: [tclquadcode] Details, details, details.

[Kevin K. <kev...@gm...>]

I've now got the code in place to translate 'invokeExpanded' into
'invoke' when the target is a compiled command. This runs the
'expandtest' cases, except tor test4/test5, which would depend
on a correct implementation of loop exception ranges and
[return -code continue]. I'm happy to leave those out for now.
It doesn't appear to have an adverse impact on any other
test cases.

Converting to [invoke] means that doing [invokeExpanded]
on a compiled procedure will get the callframe effect
right, without any extra work from me.

I'm still not ready to merge because:

(1) I'm not converting [invokeExpanded] of Core commands
    to [invoke].  I could do so, but that will depend on having
    a table of the arity of the Core commands. Ones with
    fixed arity can be converted to [invoke] easily. There is
    also a largish set of commands with a single optional
    argument, which also should be handled. Truly variadic
    Core commands will still have to go through
    'invokeExpanded'.

(2) I'm not yet even trying to compute the callframe effects
    of [invokeExpanded], so we'll likely get aliased variables
    wrong in the cases that I can't optimize.

Lower priority stuff would include optimizing away
expansion for known singletons and known empty
objects.

I'd rather merge sooner than later, but the lack of
analysis for core commands with {*} makes me quite
uneasy.

Re: [tclquadcode] throwing an error

[Donal K. F. <don...@ma...>]

On 06/11/2017 04:06, Kevin Kenny wrote:
> My question: what to do about a 'wrong # args' error? In the runtime
> logic to repack the args, I can detect that it's gone wrong by
> assembling them into a list, doing listLength, and then finding
> out that I don't have enough args to fill the mandatory parameters,
> or that I have too many args for a proc that doesn't take 'args'.
> What's the path of least resistance to getting the error thrown?

Well, if we know the key leading words (usually the command name, and 
the subcommand/method name for ensembles and object calls) and the 
descriptive part then we can generate the rest easily enough.

Or we can just punt on handling this case cleverly, use a full dispatch, 
and let Tcl and the generic thunk pick up the pieces (which should now 
Just Work™). A sort of “I've not detected that I can safely optimise 
this, so I won't” attitude. It's the simplest thing that could possibly 
work, and yes, it reduces the optimisations possible, but we can't help 
that. (I do hope that it won't come up often so the cost of the approach 
won't be borne too much.)

All procedures that we compile will get a STRING[]->FAIL STRING version 
asked for as that's what plugs into the back of the thunks and I can't 
prove that there will never be calls to a procedure from scripts.

> I'm guessing that I *don't*  want to try to construct a value of type
> CALLFRAME FAIL STRING- and in fact, I'm not by any means certain that
> particular unholy combination will survive going through a phi.

I can send it through a phi easily enough, as it corresponds to 
something like:

   struct {
       CallFrame*   callframe;
       struct {
           int      returncode;
           Tcl_Obj* value;
       };
   };

I have to have a real thing for CALLFRAME elsewhere; though the current 
code is messy with regard to it being a compilation-run global vs a 
genuine value, it is in principle separated out. (Inlining will be fine 
as long as I can pick up the requisite construction metadata from the 
right place.)

> I'm guessing that I want actually to recognize that five-instruction
> codeburst so that I can get the address of the catch block from the
> jumpMaybe and jump there, having done an 'initException'. But I've not
> done that before, and there seems to be some deep wizardry there.

If you want a 'throwWrongNumArgs' instruction, just ask. The main things 
I'd want (if it is talking about generating it for a command we know 
*that doesn't have subcommands*) is the user's name of the command I'm 
generating for (e.g., was there a “::” at the beginning?) and what 
procedure it is for (preferably by FQN) so I can look up the metadata 
for it and generate the argument descriptor.

> I suppose that I could just retain the 'invokeExpanded' as well as the
> 'invoke', and let the thunk report the problem, but that seems
> awkward. In any case, "I know statically that i have an error here"
> seems to be something that I've hit multiple times without a good
> approach.

That'd be my first approach. It's what I do with the semi-optimised 
bytecode in 8.6 too; there are places where we know we're going to be in 
an error case, but where generating the right error message during 
compilation is annoying. I'm pretty content to just leave things alone.

Of course, the other thing is that with a call that is definitely going 
to generate a wrong-num-args message, we don't actually need a CALLFRAME 
since those messages never actually touch the frame...

Donal.

[tclquadcode] throwing an error

[Kevin K. <kev...@gm...>]

Donal,

I'm working on the code that repacks arg lists to make
'invokeExpanded' into a plain 'invoke' when possible.

Assume that I have all the mechanism of [info args], [info default]
and suchlike at my disposal.  I'm working in the context of a
codeburst that I know will look like:

    invokeExpanded callframe1 callframe0 proc arg arg arg....
    retrieveResult temp1 callframe1
    extractCallFrame callframe2 callframe1
    jumpMaybe catchblock temp1
    jump (next instruction)

    extractMaybe temp2, temp1
    (execution continues...)

I'm going to be doing major surgery in front of the invoke, to repack
the args. (Essentially, it's 'copy nonexpanded ones until the first
expansion, then listConcat/listAppend the rest together, and
walk down the resulting list with listIndex to fill out the rest of the
args.')

My question: what to do about a 'wrong # args' error? In the runtime
logic to repack the args, I can detect that it's gone wrong by
assembling them into a list, doing listLength, and then finding
out that I don't have enough args to fill the mandatory parameters,
or that I have too many args for a proc that doesn't take 'args'.
What's the path of least resistance to getting the error thrown?

I'm guessing that I *don't* want to try to construct a value of type
CALLFRAME FAIL STRING- and in fact, I'm not by any means certain that
particular unholy combination will survive going through a phi.  I'm
guessing that I want actually to recognize that five-instruction
codeburst so that I can get the address of the catch block from the
jumpMaybe and jump there, having done an 'initException'. But I've not
done that before, and there seems to be some deep wizardry there.

I suppose that I could just retain the 'invokeExpanded' as well as the
'invoke', and let the thunk report the problem, but that seems
awkward. In any case, "I know statically that i have an error here"
seems to be something that I've hit multiple times without a good
approach.

Any pointers?

Thanks,

Kevin

Re: [tclquadcode] Details, details, details.

[Donal K. F. <don...@ma...>]

On 04/11/2017 23:11, Kevin Kenny wrote:
> That came down into a strange change that seemed to be piggybacked
> onto the commit that added 'verifyList'. Coming along for the ride was
> a change to the 'return' instruction in tcl.list.length and
> tcl.dict.size, and that caused the return value not to match the
> declared return type of those two procedures.

I don't know what I was thinking there. The only reason I'd not noticed 
in recent testing was that I'd been garbage collecting those functions 
before running the verifier... :-}

Anyway, expanding invokes now work provided they are dispatched to 
something that doesn't have a compiled 'args'; there's something bad in 
there, but that's not the expansion mechanism per se. (I've checked with 
an [lreplace] where the compiler could not know what going on enough to 
get magical on me.) The code to do the expansion is a bit nasty as it 
currently uses a double-poke inside the boxed representation, but it 
works (and gets optimised fairly well, thankfully).

I've got a little more cleanup to do (mostly commenting code) but the 
big thing blocking the merge is that we're not doing args handling. As 
far as I can tell, the problem there is that we've got to deal with the 
fact that the argument list at the level of doing a direct call is not 
the same as the argument list at the Tcl level. I've known there was a 
problem in the thunk generation for some time (which I will fix; it 
should be much simpler than expansion), but you'll have to also figure 
out what you want to do about it. One of the easier ways would be for 
you to inject the list creation and then use the current invoke 
mechanism when you know we're going to a known procedure. (It's 
massively more difficult for me to handle at that point; I get 
mismatched type signatures and everything goes boom.)

Args is always a LIST, so there's that. ;-)

> OK, so I committed that, and then tried to compile the
> 'expandtest::test5' procedure again - and found that the 'expandDrop'
> instructions were in 'unreachable' code. The code is 'unreachable'
> because 'invoke' doesn't yet handle return codes other than OK and
> ERROR.

I thought I'd fixed that. :-( Looking at the code, it was done for stuff 
coming through the Tcl-based invoke path, but not for function invokes 
that can only "fail". (If I remember right, the problem is that I can't 
— or at least couldn't once — tuple a return code with nothing at all; 
it trips a sanity check rule I've got elsewhere.)

Donal.

[tclquadcode] Details, details, details.

[Kevin K. <kev...@gm...>]

I tried to make a little more progress on {*} today and got sidetracked.

It began when I wanted to see whether 'expandDrop' was actually being
translated adequately. My existing cases didn't actually generate
'expandDrop', so I added expandtest::test5 to make sure we had one.

In the course of trying to debug that, I wound up doing a major
refactoring of 'bytecode-to-quads' and the associated procedures, to
get them out of the global namespace and into ::quadcode::transformer.
That, in turn, took some debugging of its own.

Once I had that working on trunk, I merged it into the 'expand'
branch, and tried to run the tester, without the 'expandtest'
procedures, and it promptly crashed!

That came down into a strange change that seemed to be piggybacked
onto the commit that added 'verifyList'. Coming along for the ride was
a change to the 'return' instruction in tcl.list.length and
tcl.dict.size, and that caused the return value not to match the
declared return type of those two procedures.

OK, so I committed that, and then tried to compile the
'expandtest::test5' procedure again - and found that the 'expandDrop'
instructions were in 'unreachable' code. The code is 'unreachable'
because 'invoke' doesn't yet handle return codes other than OK and
ERROR.

So that bit is going to go untested for a while. I think we're now
smart enough about exception ranges and about the results of
procedures that it wouldn't be impossible to add the stuff - now that
the bytecode compiler adjusts stack depth properly! Essentially, we
would need to track

   (1) whether a compiled procedure can return an unusual result
(BREAK, CONTINUE, RETURN, or a funny exception number).

   (2) whether an 'invoke' has a loop exception range inside the
innermost catch.

If we're invoking a procedure that could return BREAK or CONTINUE,
we'd need to emit checks for those to go to the appropriate branch in
the loop exception range. If it could return RETURN or an odd code, we
would need to emit code to handle that.

I think it's all doable, but it's obviously considerably less urgent -
[return -code] is generally pretty obscure.

I'll open a ticket for loop exceptions coming out of INVOKE, but right
now I'm being called away.