tcl-core

[TCLCORE] Question on coroutines

[miguel s. <mig...@gm...>]

I am working on fixing some of the missing things in coros.

For instance, as of today, HEAD can tailcall out of a coro.

The missing thing is to nre-enable coros: if you call
   coroutine B $body {*}$arglist
from within a coroutine A, it would be good ig coroutine A would remove 
itself from the callstacks before running coroutine B.

I now have a good idea as to how to do this ... sorry it took so long. 
It should be in HEAD in the next few days.

The same tech makes possible a new command [yieldTo]: essentially 
[tailcall] but yielding instead of returning - that is, saving the state 
of the current coro, so that it can be reawakened later on. I am 
planning to put this into ::tcl::unsupported.

I think that this command could find very interesting applications. The 
big Q is: shall I try to TIP it as an amendment to TIP #328? I DO know 
it is rather late ...

Cheers
Miguel

Re: [TCLCORE] Question on coroutines

[Donal K. F. <don...@ma...>]

miguel sofer wrote:
> The same tech makes possible a new command [yieldTo]: essentially 
> [tailcall] but yielding instead of returning - that is, saving the state 
> of the current coro, so that it can be reawakened later on. I am 
> planning to put this into ::tcl::unsupported.

That's the classic definition of yield in a coroutine. What we have/had 
before was really a "generator".

> I think that this command could find very interesting applications. The 
> big Q is: shall I try to TIP it as an amendment to TIP #328? I DO know 
> it is rather late ...

For unsupported? No need. For supported? It would be good to see what 
you propose in that case. Off-the-wall suggestion: maybe a "-target" 
option to normal [yield] would be better...?

Donal.

Re: [TCLCORE] Question on coroutines

[Kevin K. <kev...@gm...>]

> miguel sofer wrote:
>> The same tech makes possible a new command [yieldTo]: essentially 
>> [tailcall] but yielding instead of returning - that is, saving the state 
>> of the current coro, so that it can be reawakened later on. I am 
>> planning to put this into ::tcl::unsupported.

Donal K. Fellows wrote:
> For unsupported? No need. For supported? It would be good to see what 
> you propose in that case. Off-the-wall suggestion: maybe a "-target" 
> option to normal [yield] would be better...?

Not a formal TIP, but I was talking with Miguel about this, and I
*think* I understand what he's proposing. The idea is that [yieldto]
will accept a complete Tcl command as its arguments. This command is
not evaluated in the coroutine; rather, it is evaluated in the
coroutine's *caller*. The command will usually be an invocation of
another coroutine, but the last is not a requirement.

As an example:

proc pro {file} {
     yield
     while {1} {
         set result {[gets $file $line]}
	yieldto consumer [list $result $line]
	if {$result < 0} { return }
     }
}
proc con {} {
     while {1} {
	lassign [yieldto producer] result line
         if {$result < 0} {
	    return
         }
	puts $line
     }

}
set file [open "myfile.txt" r]
coroutine producer pro $file
coroutine consumer con
producer;   # invoke one last time to shut it down.
close $file

Here we have created two coroutines, [producer] and [consumer].
[producer] initially yieds back to the main routine, while [consumer]
starts running immediately.

The first action that [consumer] takes is [yieldto producer].  The
[consumer] coroutine's state is frozen, and control returns to the
main coroutine, which immediately invokes [producer].  The [producer]
coroutine reads the first line of the file, and calls [yieldto]
giving it [producer {$result $line}].

The two ping-pong back and forth, each specifying the next action,
until finally [producer] sends over the -1 result at end of file.
At that point [consumer] returns to the main routine, and one
last call to [producer] causes it to hit its [return]; now everything
is shut down cleanly.

Closing the file is all that remains.

I don't claim fully to understand the implications of permitting
an arbitrary command, rather than just a coroutine, as an
argument to [yieldto]. But it's fairly obvious from my discussion
with Miguel that having [yieldto] accept an arbitrary command is
by far the easiest way to implement this beast. Since using an
arbitrary command is more general, why not?

This might be a nice structure for implementing things like
compilers (the lexer yields to the parser; parser yields to
lexer and semantic engine; semantic engine yields to parser),
or other things that follow the "pipes and filters" pattern.
I think that for ordinary "green threads" I'd avoid yieldto in
favour of a scheduler in the main routine - the way coronet
does it.

This idea looks sound as far as it goes; I'm musing on how many
more iterations it'll take us to get to first-class continuations,
which is where this road seems to be heading.
--
73 de ke9tv/2, Kevin

Re: [TCLCORE] Question on coroutines

[Larry M. <lm...@bi...>]

I'm coming into this late so maybe I don't get it.  But aren't coroutines
basically busted from the start?

Long ago I did a coroutine system and it was really pleasant, it was the
first place I wrote swtch() (the context switch code that every real
hacker has to write), but it was all a waste of time.

Why?  Because the threads blocked on IO.  Is that true with the tcl
coroutines?  If it is, aren't they kind of a joke compared to threads
that handle IO?
-- 
---
Larry McVoy                lm at bitmover.com           http://www.bitkeeper.com

Re: [TCLCORE] Question on coroutines

[Kevin K. <kev...@gm...>]

Larry McVoy wrote:
> I'm coming into this late so maybe I don't get it.  But aren't coroutines
> basically busted from the start?
> 
> Long ago I did a coroutine system and it was really pleasant, it was the
> first place I wrote swtch() (the context switch code that every real
> hacker has to write), but it was all a waste of time.
> 
> Why?  Because the threads blocked on IO.  Is that true with the tcl
> coroutines?  If it is, aren't they kind of a joke compared to threads
> that handle IO?

Tcl's coroutines are essentially co-operative ("green") threads.
If you look at http://wiki.tcl.tk/22231, you'll see that a simple
(tens of lines of code per function) layer can be put on top of
coroutines to make *them* appear like threads that are waiting
on blocking I/O, when what they are actually doing is setting
fileevents and yielding to the event loop. Wub is built that way,
and once it was over its 'infant mortality' phase of performance
bugs, it seems to behave quite nicely without spinning a vast
tangle of native threads.

What green threads *won't* do is let you have CPU-burning loops
that don't yield.  On the other hand, green threads also are a good bit
more predictable than native ones about atomicity of actions - since
they never are preempted, a lot of the need for mutex (not all, but
a lot) goes away.

Also, there are other uses (e.g., "pipes and filters" programming,
discrete event simulation, decomposition of state machines)
where coroutines are useful but don't actually represent a threading
abstraction.

And, if you want kernel-level threads, you know where to find them.
The Thread extension runs, as far as I know, on every major platform.
(Windows, Mac, Linux, and anything with pthreads.) There's nothing
that says we can't have both green threads *and* native threads.

-- 
73 de ke9tv/2, Kevin

Re: [TCLCORE] Question on coroutines

[Neil M. <ne...@Cs...>]

On 7 Dec 2009, at 02:47, Kevin Kenny <kev...@gm...> wrote:

>> miguel sofer wrote:
>>> The same tech makes possible a new command [yieldTo]: essentially
>>> [tailcall] but yielding instead of returning - that is, saving the  
>>> state
>>> of the current coro, so that it can be reawakened later on. I am
>>> planning to put this into ::tcl::unsupported.

I'm not sure I understand how this differs from a normal yield. A  
tailcall reuses the current stack frame. A yield saves the current  
stack. How can you do both?


>
> Donal K. Fellows wrote:
>> For unsupported? No need. For supported? It would be good to see what
>> you propose in that case. Off-the-wall suggestion: maybe a "-target"
>> option to normal [yield] would be better...?
>
> Not a formal TIP, but I was talking with Miguel about this, and I
> *think* I understand what he's proposing. The idea is that [yieldto]
> will accept a complete Tcl command as its arguments. This command is
> not evaluated in the coroutine; rather, it is evaluated in the
> coroutine's *caller*. The command will usually be an invocation of
> another coroutine, but the last is not a requirement.

Is this essentially the same as the following code:

while 1 {
     set cmd [coro]
     eval $cmd
}


?


>
> As an example:
>
> proc pro {file} {
>     yield
>     while {1} {
>         set result {[gets $file $line]}
>    yieldto consumer [list $result $line]
>    if {$result < 0} { return }
>     }
> }
> proc con {} {
>     while {1} {
>    lassign [yieldto producer] result line
>         if {$result < 0} {
>        return
>         }
>    puts $line
>     }
>
> }
> set file [open "myfile.txt" r]
> coroutine producer pro $file
> coroutine consumer con
> producer;   # invoke one last time to shut it down.
> close $file
>
> Here we have created two coroutines, [producer] and [consumer].
> [producer] initially yieds back to the main routine, while [consumer]
> starts running immediately.
>
> The first action that [consumer] takes is [yieldto producer].  The
> [consumer] coroutine's state is frozen, and control returns to the
> main coroutine, which immediately invokes [producer].  The [producer]
> coroutine reads the first line of the file, and calls [yieldto]
> giving it [producer {$result $line}].
>
> The two ping-pong back and forth, each specifying the next action,
> until finally [producer] sends over the -1 result at end of file.
> At that point [consumer] returns to the main routine, and one
> last call to [producer] causes it to hit its [return]; now everything
> is shut down cleanly.
>
> Closing the file is all that remains.

But this is just the same as what the existing mechanism provides,  
modulo syntax.


>
> I don't claim fully to understand the implications of permitting
> an arbitrary command, rather than just a coroutine, as an
> argument to [yieldto]. But it's fairly obvious from my discussion
> with Miguel that having [yieldto] accept an arbitrary command is
> by far the easiest way to implement this beast. Since using an
> arbitrary command is more general, why not?
>
> This might be a nice structure for implementing things like
> compilers (the lexer yields to the parser; parser yields to
> lexer and semantic engine; semantic engine yields to parser),
> or other things that follow the "pipes and filters" pattern.
> I think that for ordinary "green threads" I'd avoid yieldto in
> favour of a scheduler in the main routine - the way coronet
> does it.

Right. And the pipes and filters approach works fine now, so what am I  
missing?

> This idea looks sound as far as it goes; I'm musing on how many
> more iterations it'll take us to get to first-class continuations,
> which is where this road seems to be heading.

If you could arrange for a coroutine to resume from the same yield  
multiple times, then I believe we have the full power of continuations.

Neil

Re: [TCLCORE] Question on coroutines

[Donal K. F. <don...@ma...>]

Kevin Kenny wrote:
> Not a formal TIP, but I was talking with Miguel about this, and I
> *think* I understand what he's proposing. The idea is that [yieldto]
> will accept a complete Tcl command as its arguments. This command is
> not evaluated in the coroutine; rather, it is evaluated in the
> coroutine's *caller*. The command will usually be an invocation of
> another coroutine, but the last is not a requirement.

I was interpreting it as the name of another coroutine (in the yielded 
state) to which the other argument would be passed as result. At that 
point, it becomes trivial to do instead as a -target argument to normal 
[yield].

> I don't claim fully to understand the implications of permitting
> an arbitrary command, rather than just a coroutine, as an
> argument to [yieldto]. But it's fairly obvious from my discussion
> with Miguel that having [yieldto] accept an arbitrary command is
> by far the easiest way to implement this beast. Since using an
> arbitrary command is more general, why not?

How do you yield to a non-coroutine anyway? (And can I yield to the 
"main coroutine"?)

Donal.

Re: [TCLCORE] Question on coroutines

[miguel s. <mig...@gm...>]

Donal K. Fellows wrote:
> Kevin Kenny wrote:
>> Not a formal TIP, but I was talking with Miguel about this, and I
>> *think* I understand what he's proposing. The idea is that [yieldto]
>> will accept a complete Tcl command as its arguments. This command is
>> not evaluated in the coroutine; rather, it is evaluated in the
>> coroutine's *caller*. The command will usually be an invocation of
>> another coroutine, but the last is not a requirement.
> 
> I was interpreting it as the name of another coroutine (in the yielded 
> state) to which the other argument would be passed as result. At that 
> point, it becomes trivial to do instead as a -target argument to normal 
> [yield].
> 
>> I don't claim fully to understand the implications of permitting
>> an arbitrary command, rather than just a coroutine, as an
>> argument to [yieldto]. But it's fairly obvious from my discussion
>> with Miguel that having [yieldto] accept an arbitrary command is
>> by far the easiest way to implement this beast. Since using an
>> arbitrary command is more general, why not?
> 
> How do you yield to a non-coroutine anyway? (And can I yield to the 
> "main coroutine"?)

OK, it was not clear at all :(

The idea is that
     yieldTo foo bar sum
will:
   1. suspend the currently running coro
   2. cause the coro's caller to invoke [foo bar sum] in the place of 
the current coro, and take it's return value as the coro's return value

IOW: it works EXACTLY like [tailcall] in the sense that the currently 
running thing is removed from the call stack and replaced by a new 
command. The difference is that where [tailcall] terminates the 
currently running proc, [yieldTo] suspends the currently running coro.

You can yieldTo any command, it can be a suspended coro but doesn't have 
to be

Re: [TCLCORE] Question on coroutines

[Joe E. <jen...@fl...>]

miguel sofer wrote:

> The idea is that
>      yieldTo foo bar sum
> will:
>    1. suspend the currently running coro
>    2. cause the coro's caller to invoke [foo bar sum] in the place of
> the current coro, and take it's return value as the coro's return value
>
> IOW: it works EXACTLY like [tailcall] in the sense that the currently
> running thing is removed from the call stack and replaced by a new
> command. The difference is that where [tailcall] terminates the
> currently running proc, [yieldTo] suspends the currently running coro.

OK, so if you have:

    proc A {...} {
	coroutine coro1 B ...
    }

then in proc B:

    yieldto C args...

is the same as:

    yield [C args...]

except that:

    (a) [uplevel 1] in the body of proc C will refer to proc A,
        not proc B; and:
    (b) C may call coro1.

Do I have that right?


--Joe English

  jen...@fl...

Re: [TCLCORE] Question on coroutines

[miguel s. <mig...@gm...>]

Joe English wrote:
> miguel sofer wrote:
> 
>> The idea is that
>>      yieldTo foo bar sum
>> will:
>>    1. suspend the currently running coro
>>    2. cause the coro's caller to invoke [foo bar sum] in the place of
>> the current coro, and take it's return value as the coro's return value
>>
>> IOW: it works EXACTLY like [tailcall] in the sense that the currently
>> running thing is removed from the call stack and replaced by a new
>> command. The difference is that where [tailcall] terminates the
>> currently running proc, [yieldTo] suspends the currently running coro.
> 
> OK, so if you have:
> 
>     proc A {...} {
> 	coroutine coro1 B ...
>     }
> 
> then in proc B:
> 
>     yieldto C args...
> 
> is the same as:
> 
>     yield [C args...]
> 
> except that:
> 
>     (a) [uplevel 1] in the body of proc C will refer to proc A,
>         not proc B; and:
>     (b) C may call coro1.
> 
> Do I have that right?

Yes. There are of course additional internal details in terms of stack 
consumption and suchlike, but irrelevant to the semantics.

Re: [TCLCORE] Question on coroutines

[miguel s. <mig...@gm...>]

Neil Madden wrote:
> On 7 Dec 2009, at 02:47, Kevin Kenny <kev...@gm...> wrote:
> 
>>> miguel sofer wrote:
>>>> The same tech makes possible a new command [yieldTo]: essentially
>>>> [tailcall] but yielding instead of returning - that is, saving the 
>>>> state
>>>> of the current coro, so that it can be reawakened later on. I am
>>>> planning to put this into ::tcl::unsupported.
> 
> I'm not sure I understand how this differs from a normal yield. A 
> tailcall reuses the current stack frame. A yield saves the current 
> stack. How can you do both?

A tailcall tricks the caller by swapping the current command with a new 
one; so does yieldTo.

The difference is that tailcall zaps the environment of the currently 
running proc, whereas yieldTo suspends and saves the state of the 
currently running coro.

<snip>
> But this is just the same as what the existing mechanism provides, 
> modulo syntax.

You can do the same thing now if you use the event loop (or something 
specifically coded for that purpose) as manager: you can indeed suspend 
and specify who to call next. The diff is that this does not require a 
manager.

For background, see Lua's paper
http://www.inf.puc-rio.br/~roberto/docs/MCC15-04.pdf
In their terms: we have now "full asymetric coroutines", [yieldTo] gives 
us also "full symmetric coroutines". At essentially zero cost.

<cite>
A well-known classification of coroutines concerns the control-transfer 
operations that are provided and distinguishes the concepts of symmetric 
and asymmetric coroutines. Symmetric coroutine facilities provide a 
single control-transfer operation that allows coroutines to explicitly 
pass control between themselves. Asymmetric coroutine mechanisms (more 
commonly denoted as semi-symmetric or semi coroutines [Dahl et al. 
1972]) provide two control-transfer operations: one for invoking a 
coroutine and one for suspending it, the latter returning control to the 
coroutine invoker. While symmetric coroutines operate at the same 
hierarchical level, an asymmetric coroutine can be regarded as 
subordinate to its caller, the relationship between them being somewhat 
similar to that between a called and a calling routine.
</cite>

<cite>
The basic characteristic of symmetric coroutine facilities is the 
provision of a single control-transfer operation that allows coroutines 
to pass control explicitly among themselves. Marlin [1980] and Pauli and 
Soffa [1980] argued that symmetric and asymmetric coroutines have no 
equivalent power and that general coroutine facilities should provide 
both constructs. However, it is easy to demonstrate that we can provide 
any of these mechanisms using the other; therefore, no expressive power 
is lost if only asymmetric coroutines are provided in a language.
     The implementation of symmetric coroutines on top of asymmetric 
facilities is straightforward. Symmetrical transfers of control between 
asymmetric coroutines can be easily simulated with pairs of yield–resume 
operations and an auxiliary dispatching loop that acts as an 
intermediary in the switch of control between the two coroutines. When a 
coroutine wishes to transfer control, it yields to the dispatching loop, 
which in turn resumes the coroutine that must be reactivated.
</cite>

> 
>>
>> I don't claim fully to understand the implications of permitting
>> an arbitrary command, rather than just a coroutine, as an
>> argument to [yieldto]. But it's fairly obvious from my discussion
>> with Miguel that having [yieldto] accept an arbitrary command is
>> by far the easiest way to implement this beast. Since using an
>> arbitrary command is more general, why not?
>>
>> This might be a nice structure for implementing things like
>> compilers (the lexer yields to the parser; parser yields to
>> lexer and semantic engine; semantic engine yields to parser),
>> or other things that follow the "pipes and filters" pattern.
>> I think that for ordinary "green threads" I'd avoid yieldto in
>> favour of a scheduler in the main routine - the way coronet
>> does it.
> 
> Right. And the pipes and filters approach works fine now, so what am I 
> missing?
> 
>> This idea looks sound as far as it goes; I'm musing on how many
>> more iterations it'll take us to get to first-class continuations,
>> which is where this road seems to be heading.
> 
> If you could arrange for a coroutine to resume from the same yield 
> multiple times, then I believe we have the full power of continuations.
> 
> Neil

Re: [TCLCORE] Question on coroutines

[Kevin K. <kev...@gm...>]

miguel sofer wrote: [quoting
http://www.inf.puc-rio.br/~roberto/docs/MCC15-04.pdf]:

> <cite>
> The basic characteristic of symmetric coroutine facilities is the 
> provision of a single control-transfer operation that allows coroutines 
> to pass control explicitly among themselves. Marlin [1980] and Pauli and 
> Soffa [1980] argued that symmetric and asymmetric coroutines have no 
> equivalent power and that general coroutine facilities should provide 
> both constructs. However, it is easy to demonstrate that we can provide 
> any of these mechanisms using the other; therefore, no expressive power 
> is lost if only asymmetric coroutines are provided in a language.
>     The implementation of symmetric coroutines on top of asymmetric 
> facilities is straightforward. Symmetrical transfers of control between 
> asymmetric coroutines can be easily simulated with pairs of yield–resume 
> operations and an auxiliary dispatching loop that acts as an 
> intermediary in the switch of control between the two coroutines. When a 
> coroutine wishes to transfer control, it yields to the dispatching loop, 
> which in turn resumes the coroutine that must be reactivated.
> </cite>

At first, I found Miguel's terse explanation to be understandable
only at a "my brain hurts!" level.  Once I understood it, it looks
to be a cheap and elegant way to implement full-symmetric coroutines
without a dispatcher in the main routine.

The question that remains is whether this functionality is what we
want. As the Lua paper mentions, symmetric and asymmetric coroutines
are equivalent. (I believe that I read the Pauli and Soffa paper at
the time, and wondered why they asserted that they were not.) Since
[yieldto] is obviously a difficult concept to explain - a couple of
very smart people here were puzzled at first - the fact that comparable
functionality is easily done with existing facilities would seem to
argue against including it.

That said, the trick of having [yieldto] work by substituting the
call to the coroutine with another call is clever. And the fact that
the other call might be any other command, not just another coroutine,
might prove to be useful. (I can't imagine *how*, just at the moment,
but that's most likely a limitation of my imagination.) So I'm certainly
fine with experimenting with this in 'unsupported' and letting the
imaginative people see what they can make of it.

-- 
73 de ke9tv/2, Kevin

Re: [TCLCORE] Question on coroutines

[Donal K. F. <don...@ma...>]

miguel sofer wrote:
> OK, it was not clear at all :(

That's no problem. We have this discussion so things can be made clear. :-)

> The idea is that
>     yieldTo foo bar sum
> will:
>   1. suspend the currently running coro
>   2. cause the coro's caller to invoke [foo bar sum] in the place of the 
> current coro, and take it's return value as the coro's return value
> 
> IOW: it works EXACTLY like [tailcall] in the sense that the currently 
> running thing is removed from the call stack and replaced by a new 
> command. The difference is that where [tailcall] terminates the 
> currently running proc, [yieldTo] suspends the currently running coro.
> 
> You can yieldTo any command, it can be a suspended coro but doesn't have 
> to be

So, the following should be equivalent, for any [foo]?

   yieldTo foo bar
   yield [foo bar]

Except that [foo] might see a different stack context? What will be the 
cost of making a new short-lived context in the non-coro case? What will 
[info coroutine] return inside the body of [foo] in the non-coro case?

Donal.

Re: [TCLCORE] Question on coroutines

[Kevin K. <kev...@gm...>]

Donal K. Fellows wrote:
> So, the following should be equivalent, for any [foo]?
> 
>   yieldTo foo bar
>   yield [foo bar]
> 
> Except that [foo] might see a different stack context? What will be the 
> cost of making a new short-lived context in the non-coro case? What will 
> [info coroutine] return inside the body of [foo] in the non-coro case?

As Miguel explained it to me, [yieldto] executes the yielded-to command
in place of the call to the current coroutine and in the caller's scope.
(Except that the command is resolved in the current scope before
yielding?)

So I'd expect that [info coroutine], if [foo] is not a coroutine, will
be whatever coroutine called the one that yielded.

Oh, by the way, answering your earlier question:

yieldTo the main coroutine will *not* be possible,
because you can't yield to (or otherwise call) a coroutine that is
on the stack, and the main coroutine is always there.

-- 
73 de ke9tv/2, Kevin

Re: [TCLCORE] Question on coroutines

[miguel s. <mig...@gm...>]

Donal K. Fellows wrote:
> miguel sofer wrote:
>> OK, it was not clear at all :(
> 
> That's no problem. We have this discussion so things can be made clear. :-)
> 
>> The idea is that
>>     yieldTo foo bar sum
>> will:
>>   1. suspend the currently running coro
>>   2. cause the coro's caller to invoke [foo bar sum] in the place of 
>> the current coro, and take it's return value as the coro's return value
>>
>> IOW: it works EXACTLY like [tailcall] in the sense that the currently 
>> running thing is removed from the call stack and replaced by a new 
>> command. The difference is that where [tailcall] terminates the 
>> currently running proc, [yieldTo] suspends the currently running coro.
>>
>> You can yieldTo any command, it can be a suspended coro but doesn't 
>> have to be
> 
> So, the following should be equivalent, for any [foo]?
> 
>   yieldTo foo bar
>   yield [foo bar]
> Except that [foo] might see a different stack context? What will be the 
> cost of making a new short-lived context in the non-coro case? What will 
> [info coroutine] return inside the body of [foo] in the non-coro case?

In terms of stack: as long as I don't find the way to nre-enable coros 
(still fighting), each coro invocation creates a new TEBC instance.

So (assuming foo is a suspended coroutine), what happens with the new 
option is that the current coro (letś call it cur) is gone from the C 
stack while foo runs.

While foo is running (assuming it is a coro) the (simplified) C-stack 
using your [yield] alternative looks like
    (cur's caller)
    NRRunCallbacks
    cur's TEBC
    NRRunCallbacks
    foo's TEBC

With yieldTo it would be
    (cur's caller)
    NRRunCallbacks
    foo's TEBC

In terms of Tcl context, assuming now that foo is a proc not a coro: 
[uplevel 1] and [upvar 1] from within foo's body see
   - foo's caller context within cur using the [yield] alternative
   - cur's CALLER context using [yieldTo]

There is no short-lived context being created with yieldTo, it uses 
foo's caller directly: cur's context is taken out of the C stack and 
into the freezer.

Maybe the best is for me to provide the code: I'll commit it in a matter 
of hours (within unsupported) and things should be clearer then.

Re: [TCLCORE] Question on coroutines

[miguel s. <mig...@gm...>]

miguel sofer wrote:
> Maybe the best is for me to provide the code: I'll commit it in a matter 
> of hours (within unsupported) and things should be clearer then.

Done: head has it. Patch at sf, #2910056

Re: [TCLCORE] Question on coroutines

[Neil M. <ne...@Cs...>]

On 7 Dec 2009, at 14:09, Donal K. Fellows wrote:
> [...]
>> The idea is that
>>    yieldTo foo bar sum
>> will:
>>  1. suspend the currently running coro
>>  2. cause the coro's caller to invoke [foo bar sum] in the place of  
>> the
>> current coro, and take it's return value as the coro's return value
>>
>> IOW: it works EXACTLY like [tailcall] in the sense that the currently
>> running thing is removed from the call stack and replaced by a new
>> command. The difference is that where [tailcall] terminates the
>> currently running proc, [yieldTo] suspends the currently running  
>> coro.
>>
>> You can yieldTo any command, it can be a suspended coro but doesn't  
>> have
>> to be
>
> So, the following should be equivalent, for any [foo]?
>
>   yieldTo foo bar
>   yield [foo bar]

Or perhaps "yield [list foo bar]" and have the caller do "eval  
[$coro]"? Isn't that exactly what yieldTo does, but hiding the [eval],  
or am I still missing something?

-- Neil

Re: [TCLCORE] Question on coroutines

[Kevin K. <kev...@gm...>]

Neil Madden wrote:
> Or perhaps "yield [list foo bar]" and have the caller do "eval [$coro]"? 
> Isn't that exactly what yieldTo does, but hiding the [eval], or am I 
> still missing something?

You're not missing much. It's a minor simplification; if the coro were
always invoked with [eval [$coro]], then [yield] would behave the same
as [yieldto] without it.  In that case, though, if the coro is actually
ready to return a value, it would have to execute something like
[yield [list return -level 0 $value]].  And the tangle gets more
complicated, because the dispatcher isn't a simple [eval [$coro]];
it has to continue evaluating a whole chain of control transfers
(and still be able to escape with [return -level 0] or whatever at
the end).

Equivalent power, different notation.

-- 
73 de ke9tv/2, Kevin

Re: [TCLCORE] Question on coroutines

[Neil M. <Nei...@no...>]

On 7 Dec 2009, at 14:44, Kevin Kenny wrote:
> Neil Madden wrote:
>> Or perhaps "yield [list foo bar]" and have the caller do "eval [$coro]"? Isn't that exactly what yieldTo does, but hiding the [eval], or am I still missing something?
> 
> You're not missing much. It's a minor simplification; if the coro were
> always invoked with [eval [$coro]], then [yield] would behave the same
> as [yieldto] without it.  In that case, though, if the coro is actually
> ready to return a value, it would have to execute something like
> [yield [list return -level 0 $value]].  And the tangle gets more
> complicated, because the dispatcher isn't a simple [eval [$coro]];
> it has to continue evaluating a whole chain of control transfers
> (and still be able to escape with [return -level 0] or whatever at
> the end).
> 
> Equivalent power, different notation.

OK, I sort-of see now. Wrapping all the coroutine yields in an [eval] is easily accomplished:

 proc coro {name args} {
    coroutine $name.coro {*}$args
    interp alias {} $name {} dispatch $name.coro
 }
 proc dispatch {coro arg} { eval [$coro $arg] }

To yield a value, you can just do (as you say):

 proc value val { list return -level 0 $val }
 yield [value $foo]

I'm not sure I see in what case the dispatcher needs to do more than a simple [eval], though. In what situation do you get a chain of control transfers with yieldTo? (My understanding is that this behaves like [tailcall], so there should be no chains).

Not that I am antagonistic to the proposal, I'm just trying to determine if it is a lack of expressive power in the original coroutine specification, or a syntactic nicety.

NeilThis message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

Re: [TCLCORE] Question on coroutines

[Donald G P. <dg...@ni...>]

Neil Madden wrote:
> I'm not sure I see in what case the dispatcher needs to do more than a  > simple [eval], though. In what situation do you get a chain of 
control > transfers with yieldTo? (My understanding is that this behaves 
like [tailcall],
> so there should be no chains).

Others can correct if I'm mistaken, but there's a subtle difference 
between [tailcall] and having the caller [eval] a script that is
returned, beyond the convenience of syntax.  [tailcall] resolves
the command name in the context of the [tailcall] command, while
an [eval] in the caller would resolve in the caller context.  Many
times this does not matter, but sometimes it does.

I was assuming from the prior messages that [yieldTo] would follow
the same pattern as [tailcall].

-- 
| Don Porter          Mathematical and Computational Sciences Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

Re: [TCLCORE] Question on coroutines

[miguel s. <mig...@gm...>]

Donald G Porter wrote:
> Neil Madden wrote:
>> I'm not sure I see in what case the dispatcher needs to do more than 
>> a  > simple [eval], though. In what situation do you get a chain of 
> control > transfers with yieldTo? (My understanding is that this behaves 
> like [tailcall],
>> so there should be no chains).
> 
> Others can correct if I'm mistaken, but there's a subtle difference 
> between [tailcall] and having the caller [eval] a script that is
> returned, beyond the convenience of syntax.  [tailcall] resolves
> the command name in the context of the [tailcall] command, while
> an [eval] in the caller would resolve in the caller context.  Many
> times this does not matter, but sometimes it does.
> 
> I was assuming from the prior messages that [yieldTo] would follow
> the same pattern as [tailcall].

Your assumption is correct. Under the cover this really does use the 
tailcall machinery - the NRTailcallEval callback.

But most of that can also be done with a bit of [namespace which] or 
[namespace eval] and stuff.

The real diff is that you do not need a scheduler - be it the event loop 
or a specially code one.