Menu
▾
▴
Re: [TCLCORE] Coroutines in HEAD
|
From: Lars H. <Lar...@re...> - 2008-08-27 14:01:05
|
miguel sofer skrev:
> Lars Hellström wrote:
>> To continue where I left off...
>>
>>
>> C API
>>
>> (Not my strong side, but one that has to be addressed.) Since the idea
>> is mostly to give new meaning to things that can already happen, there
>> isn't /that/ much that has to be added on the C side, but one major
>> thing is the prototype for the function that is called when a command
>> is resumed; let's call this a resumeProc:
>>
>> typedef int Tcl_ObjResumeProc(
>> ClientData clientData,
>> Tcl_Interp *interp,
>> int objc,
>> Tcl_Obj *const objv[],
>> int resumeIdx,
>> Tcl_Obj *const resumeData[] );
>>
>> The idea here is that the resumeData should be the objv from applying
>> Tcl_ListObjGetElements to the result caught with the TCL_SUSPEND, and
>> resumeIdx is the index into this array of the element which is
>> relevant for this command; the resumeProc will eventually call the
>> resumeProc of the command it received TCL_SUSPEND from with the same
>> resumeData but resumeIdx-1.
>>
>> For practical convenience, it is probably also useful to have a
>> library function for calling the resumeProc of a particular command,
>> so Tcl_EvalObjEx (or whichever is the core one these days) would get
>> the cousin
>>
>> int Tcl_ResumeObjEx(interp, objPtr, flags, resumeIdx, resumeData)
>>
>>
>> The tricky part is of course that all these resumeProcs must be
>> provided when the command is created. For a long time I thought that
>> this meant nothing of this could be implemented before Tcl 9 anyway
>> (hence there would be no rush to generate a string representation :-]
>> of the suspend/resume idea), but now I'm not so sure -- it's perfectly
>> reasonable to have e.g. Tcl_CreateObjCommand create commands without a
>> resumeProc (resumeProc==NULL), and instead introduce a new function
>>
>> Tcl_Command Tcl_CreateSuspendableCommand(
>> interp, cmdName, proc, clientData, deleteProc, resumeProc
>> )
>>
>> for creating commands that do have it. Calling Tcl_ResumeObjEx for a
>> command that doesn't have a resumeProc won't work, but we can simply
>> report that as an error, so it's no big deal (at the C level).
>>
>> However, rather than reporting such errors at the resuming stage, it
>> would be better to detect them at the suspend stage. In principle,
>> that could be done by having Tcl_Eval* verify that any command
>> returning a TCL_SUSPEND also has a resumeProc, or else convert that
>> TCL_SUSPEND into an appropriate error. As far as I can tell, that
>> would be the only ***potential incompatibility*** of this scheme
>> against current Tcl.
>
> OK, this means that you can suspend a *command*, but not suspend in a
> (say) proc that is called from your command. Right?
Wrong -- each level of Tcl_Eval* around the [suspend] contributes
another element to the continuation (kept in interp's objResultPtr,
which is a list).
When a command's /resumeProc/ determines that "my internal state is
that I had just called X", then it should complete its resumption by
calling
Tcl_ResumeObjEx(interp, objPtr, flags, resumeIdx-1, resumeData)
to have also that command resume itself (which will then
Tcl_ResumeObjEx whatever it had called, recursively up to the
[suspend]). If the command X is a procedure, then (AFAICT) the /proc/
of it is TclObjInterpProc, so is suppose the /resumeProc/ would be some
TclObjInterpResumeProc.
> And if I
> misunderstood and can you can do suspend in nested calls ... where does
> evaluation continue after the suspension, how does your system know if
> it is the current command, its caller, the caller's caller, etc?
In
proc nest1 {} {
A
suspend
B
}
proc nest2 {} {nest1}
proc nest3 {cont} {
C
resume $cont
D
}
catch {nest2} res opt here
E
nest3 $res
the single letter commands are evaluated in the order:
A, E, C, B, D.
A and B both agree (when consulting [info level] or [info frame])
they're being called from [nest1], which in turn is called from
[nest2]. They don't agree about [nest3], which is seen on the call
stack by B but not by A.
> And
> what is the return value (ie, the interp's objResultPtr) seen by whoever
> gets to continue execution after a command it called was suspended?
That would be the value specified in the [resume]. In the above case
(no value specified) it defaults to the empty string (as with
[return]). This value is placed there by the resumeProc of the
[suspend] command, which finds it in resumeData[0].
>> COMPARISON OF [coroutine]/[yield] AND [suspend]/[resume]
>>
[snip]
>> *Storage.* Since [suspend] starts putting data in the interpreter
>> result, everything has to be put under a Tcl_Obj wrapper. This is
>> potentially a slow operation (but probably not so slow that it becomes
>> a major issue when suspending the handler for an event). By contrast,
>> [coroutine]/[yield] hides everything under the opaque handle of a
>> command.
>
> Also full of conceptual and implementation problems. Do you plan to
> serialize the complete state of (say) the :: namespace at the time of
> suspension - what the commands are (builtin, procs, which extension
> provides them?), variables, traces? How about the state of the interp
> (command traces come to mind). How about children namespaces?
No, one shouldn't serialize ::, or any other namespace for that matter;
command redefinitions should take effect. Consider
proc A {} {
B
C
B
}
If C redefines B, then the second B in A does something different than
the first did. It shouldn't make a difference whether C redefines B by
an explicit [proc], by [yield]ing for a while and letting some other
part of the program do the redefinition, or by [suspend]ing itself
while some other part of the program redefines B.
However, it is similarly true that redefinitions of A while inside C
should not change the effective body of that A being evaluated. From
this follows that the /resumeData/ for a [proc] will probably need to
contain the definition of that proc which was in force when it was
called (which, BTW, need not be the same as was in force at the time of
[suspend]), perhaps in the form of a lambda.
More generally, it is a problem of this scheme that a command being
resumed need not be the same as the command that was suspended; the
original command might have been renamed, or completely deleted, and
something quite different might have been put in its place. Still, I
think the sensible thing to do is to call the /resumeProc/ of the
command of the right name which exists at the time of resumption, and
then have this /resumeProc/ throw an error if the /resumeData/ entry
does not make sense for it. This would allow replacing a proc by
another proc (same /resumeProc/, and all other details of current
definition are ignored), and also resuming a command which by some
higher level item (higher /resumeIdx/ value) was first destroyed at
suspend-time and then recreated at resume-time.
This is thus a "sharp tool, you may cut yourself" situation, but the
worst that can happen is that you get an error.
> To be
> portable only to other threads/interps in the same process might make
> this simpler and smaller by serializing pointers (yuck), but to make it
> completely portable ... ouch.
It requires consistent methods of encoding /resumeData/ elements, yes.
For a proc, there's quite a lot of state that would need to be
preserved -- body, local variables and their values (should be possible
to roll up as a dict), current position in body (perhaps character
offset is sufficient for this?), [upvar]ing of variables, [trace]s on
variables, and probably plenty of other obscure details I'm unfamiliar
with -- but I see no reason why it shouldn't be doable. (Whether it is
ultimately *worth* doing, especially now that NRE has been done, is
another matter. See below.)
When it comes to extension-defined commands, I actually think things
would often be fairly straightforward, with rather few instances where
"serializing pointers" should seem like the natural way forward. Take
the TclX [scanfile] command, for an example. Besides argument parsing,
it does most of its work in the ScanFile function. That contains
exactly two Tcl_EvalObj calls, so there are only two places in which
this command might need to be resumed; all other state is in the local
variables. Most of these are simple integers or strings:
Tcl_DString lineBuf, uniLineBuf;
int result, matchedAtLeastOne;
scanData_t data;
int matchStat;
(although several are bundled into a scanData_t struct:
typedef struct {
int storedLine;
scanContext_t *contextPtr;
Tcl_Channel channel;
char *line;
Tcl_UniChar *uniLine;
int uniLineLen;
off_t offset;
long bytesRead;
long lineNum;
matchDef_t *matchPtr;
} scanData_t;
), and a few (/contextPtr/ and /channel/) can be looked up from the
arguments [scanfile] was called with. The only thing which is not
immediately serializable is the /matchPtr/, which points to the current
item in a linked list. This can however be encoded as the index into
this list, so [scanfile] could easily record its internal state in a
dict, and thereby avoid introducing a custom Tcl_ObjType.
It's a bit of work, but not incomparable to the work of rewriting
[scanfile] to take advantage of NRE.
> OTOH, you could restrict suspendability to purely functional commands,
That's not something I'm interested in. Is it even feasible to try to
make the distinction? (I.e., determine whether we are to error out
because of being non-functional?)
[snip]
> As stated above, I do not know that I agree with "obviously possible" if
> restricted by "with reasonable expense".
Well... Back in 2005 when I started thinking about [suspend]/[resume],
I think the general opinion (at least in the discussion which prompted
the whole thing) was that something like NRE could not be done with
reasonable expense either. ;-)
>> *Emulation.* [coroutine] and [yield] can be emulated using
>> [suspend]/[resume], as follows (rough implementation):
[snip]
>> I don't see how the converse would be possible (but since NRE is more
>> than just [coroutine], that needn't be a problem).
>
> Something somewhat similar to suspend/resume can be done using
> coroutines and the event loop, I think. Again, restricting all
> considerations to a single interp.
Yes, Neil has demonstrated (http://wiki.tcl.tk/21532) how to basically
do the motivating [suspend vwait] using coroutines, so the elementary
functionality is covered by both; I stand partly corrected on that
point. Intervention, duplication, and continuation modification are
however features of [suspend]/[resume] that remain unattained by NRE.
[snip]
>> miguel sofer skrev:
>>> The thing I implemented is 100% compatible, extensions that do not
>>> adapt keep on working but may block suspending. There is an api for
>>> extenders to adapt.
>>
>> No big difference there, I'd say.
>
> It depends on how you are planning to implement things?
Realistically: I don't expect to actually implement these things -- I
know I wouldn't be able to produce all the LOC it would take.
What I primarily expect to come out of this is instead a sharpening of
NRE: if the [suspend]/[resume] primitives can be shown to do X, or
handle situation Y in a nice way, then there is an incentive for NRE to
manage that too (even if in a completely different way). A little
competition can spur significant improvements, even if that competition
is only against a cardboard cut-out.
That [suspend] and [resume] as sketched here should make it into the
core is not so likely, but they remain as a possibility, should it in
the future be felt that the "no recursion" condition to be [yield]able
is too onerous or incomprehensible.
> I am not yet
> sure if your design allows "nested suspending", in the sense that "a
> calls b which suspends both b and a, exec continues with a's caller". If
> it does, a command that calls an evaluation and receives a TCL_SUSPEND
> has to be prepared to deal with it properly.
Yes. I believe this was stated already in my original posting.
> So you will need a strategy
> to block that result to ever reach it if it is not suspendable. How?
Not block, but throw an error, since it would be an error to try to
suspend something which is not resumable (just like NRE has it be an
error to [yield] where there has been recursion -- there could be a
difference as to where this error is first thrown, though: not in the
[suspend], but when TCL_SUSPEND hits the unresumable command). I wrote
about having Tcl_Eval* check that commands returning TCL_SUSPEND also
have a /resumeProc/, did I not?
Lars Hellström
|
