From: Bruno H. <br...@cl...> - 2017-12-31 21:10:09
|
Hi Jörg, > What I propose for LOOP in clisp is to apply the exact same scoping rules that govern > WITH (possibly joined by AND) to named variables in FOR clauses. > ... > They should have distinguished for-as-equals-then and for-as-equals. > ... > Regarding a future Lisp, my vote would be We really should distinguish what can be done * in a future Lisp versus * in an ANSI CL compliant implementation. The right place to discuss the future Lisp are these Cliki pages: http://cliki.net/Proposed%20ANSI%20Changes What we do in clisp, should IMO be consistent with ANSI CL. > As the discussion around issue 222 shows, the committee did not reach a consensus, to > the texts were not changed. The ancient MIT LOOP implements this behaviour. You make it sound like the committee's decision of LOOP-INITFORM-ENVIRONMENT:PARTIAL-INTERLEAVING-VAGUE was a bad or incomplete one and should therefore disregarded. However, two people have stated the rationale for it: JonL: "I claim this ambiguity is purposeful -- the "interleaving" of binding and initialization clauses, which in some implementations has performance consequences" Kent Pitman: "... the part of LOOP that should _not_ be exposed, which is what is the right and most efficient expansion for any given implementation in order to benchmark well. The more you try to pin down that latter part, the more you just require Common Lisp to be slow. Some parts are meant to be kept abstract." In fact, for implementations with a complex compiler (with tons of optimizations) the differences won't matter. It's for implementations with a simple compiler, like clisp, that this argument has the biggest effect. Like what Sam observed: that adding some temporary variable has the effect of producing larger bytecode. > clisp's current loop contains complicated logic in an attempt to satisfy requirements > on lexical environments, global left to right evaluation, and optimization ideas Excellent remark. To fix this, we should try to find ways to execute the forms in given scopes where ANSI CL mandates it, so that the execution order becomes independent from scoping considerations. > Proposal: FOR-AS-EQUALS-THEN-HAS-INIT-FORM > Consider for-as-equals-then as > FOR # = init-form THEN form > i.e. apply aforementioned handling about variable scoping and init forms. > > Benefits: > - Named variables are immediately bound, hence values are readily available in every > subsequent clause as well as INITIALLY clauses. > - left-to-right evaluation across all init-forms is globally preserved. > - Easy to grasp and to explain. > - Implements ANSI issue 222 (which didn't reach consensus). > http://clhs.lisp.se/Issues/iss222_w.htm > > However, this proposal appears incompatible with CLHS 6.1.1.4: "form1 and form2 in > for-as-equals-then form include the lexical environment of all loop variables." So this proposal is not usable for clisp. > Proposal: FOR-AS-EQUALS-FIRST-THEN-LATER > Consider for-as-equals-then as > FOR # = form1 THEN form2 > > Because per CLHS 6.1.1.4 (quoted above), first-iteration need be evaluated in an > environment including all loop variables, the implementor of clisp's current loop > IMHO wrongly concluded that all following FOR clauses (here including those joined by > AND) must be evaluated there as well, to respect left to right evaluation order > requirements, causing known bugs and scoping rules very hard to explain or even > understand, as scoping now depends on former clauses. > > Instead, this proposal follow CLHS by the letter to "initialize the variables [...] by > setting it to the result of evaluating form1 on the first iteration, then ..." > -- Initialize, not bind! I.e. bind to NIL, later initialize to form1. > > That definition can work, considering that LOOP users need be aware of code motion > anyway -- Think how (LOOP INITIALLY (princ 2) WITH a = (princ 1) RETURN a) prints 12 > (can anybody reveal an implementation that produces another output?). This result follows from 6.1.1.6 (WITH produces bindings, whereas INITIALLY code goes into the loop prologue) with 6.1.1.4. > In effect, it will be very similar to for-as-in-list, for-as-across or for-as-hash: > - The named variables need default bindings when declared; > - It is only upon entering iteration that actual values will be supplied. > > (loop for a = (princ 1) then b for b below (princ 2) do (princ (list a b))) > would print 21 by that proposal. With this sample, you look at order of execution, not scope. Order of execution is specified in 6.1.1.6. While clisp surely has bugs in this area, in this case: the form '(princ 1)' is an initialization, whereas the form '(princ 2)' is not an initialization. => (princ 1) must be executed first. So this proposal is not usable for clisp either. > Proposal: FOR-AS-EQUALS-THEN-EARLY > In for-as-equals-then, initialize the variables prior to any INITIALLY. > (yet still in full lexical scope of all loop variables in the loop prologue). > > Benefits: > - Named variables are ready for use within INITIALLY. > - which distinguishes it from for-as-equals without THEN, so the user has the choice. Do you have a test case for this? I believe clisp implements this in some case. > Proposal: FOR-AS-EQUALS-THEN-LATE > In for-as-equals-then, initialize the variables just upon entering the loop > body. INITIALLY clauses will only see default values. > > Benefits: > - Very close to for-as-equal without then, where I don't plan at all to duplicate the > step form in the loop prologue and the loop body. > > Disadvantages: > - Precisely that it's late, so INITIALLY only ever sees default values. I think this proposal will become obsolete when we decouple scope from execution order. > Proposal: FOR-AS-EQUALS-THEN-VERY-LATE > In for-as-equals-then, initialize the variables within the loop body, after > evaluating the termination tests of preceding FOR-AS clauses. > > clisp did/does this. > (loop for vars on vars for x = (error) then 2 return vars) would not error out. sbcl still signals an error on this sample, regardless what we do in clisp. This sample now signals a warning before the error: WARNING: Reference to VARS is implementation-dependent, per ANSI CL 6.1.1.4. This is the best we can do: tell the programmer to change their code. So this proposal is pointless for clisp. > Proposal: FOR-AS-INITIALLY-INTERLEAVE-LET > The LET forms resulting from FOR-AS initforms and WITH clauses could alternate > easily. That would make it trivial to globally respect left to right evaluation across > FOR-AS, WITH and even INITIALLY clauses. > > This is not strictly conforming to CLtL2 or CLHS. They require INITIALLY to lie in the > loop prologue, which is inside the TAGBODY, hence after all LET forms, instead of amid > LET forms. > ... > Disadvantage > - Not the letter of CLtL2 or CLHS. So this proposal is not usable for clisp. > Proposal: FOR-AS-INITIALLY-GROUP-PROLOGUE > Group all INITIALLY clauses in the prologue, following all variable initializations. > > Benefits: > - Values of driver variables readily accessible in INITIALLY > - Easy to explain (after grasping code movement) > > Disadvantage: > - Code movement, hence no global left to right evaluation order between INITIALLY and FOR-AS/WITH initforms > Do you interpret ANSI-CL to expect a global order of evaluation across INITIALLY and FOR/AS-WITH clauses? Can you provide a sample, and review it in the light of 6.1.1.6? > Proposal: FOR-AS-INITIALLY-INTERLEAVE-PROLOGUE > Use dummy LET bindings, assign variables later in the prologue, which enables an > interleaving with INITIALLY clauses, hence benefitting left to right evaluation. > > This is somehow what clisp's LOOP currently does. > > Benefits: > - Global left to right evaluation order between INITIALLY and FOR-AS/WITH initforms > > Disadvantages: > - LOOP for VARS on VARS won't work (without further tricks, e.g. see FLET below). > - INITIALLY evaluates in an environment including all variables, yet values are not > yet available, depending on clause order, so it's somehow unreliable. Can you give samples? > Proposal: FOR-AS-EQUALS-THEN-INTERLEAVE > Interleave the initialization of for-as-equals-then variables with INITIALLY forms. > > Benefits: > - left to right evaluation is obeyed. > > Disadvantages: > - It may appear inconsistent to obey left to right with for-as-equals-then only, > when all other for-as forms init-forms are evaluated and bindings were established > long before any INITIALLY. Can you give samples? > Proposal: FOR-AS-EQUALS-THEN-FLET-FOR-SCOPE > Maintain the complexity in clisp's current loop, and fix some scoping bugs by adding > even more complexity. FLET can be used to evaluate init-forms in an outer scope not > comprising all loop variables. > > (LOOP FOR a = (princ 1) FOR vars ON (princ vars) RETURN (list a b)) > would expand to something containing > (let ((A nil)) > (flet ((#:on-init () (princ VARS))) > (let ((VARS nil)) > (tagbody ... (setq A (princ 1)) (setq VARS (#:on-init)) ...)))) > > Benefits: > - At least the scoping rules become explainable. > - Visible order of evaluation is obeyed. > > Disadvantage: > - Poor performance, likely needs closure. This is a nice idea, but needs compiler work first. > Proposal: FOR-AS-PROLOGUE-FIX > clisp's current loop contains complicated logic in an attempt to satisfy requirements > on lexical environments, global left to right evaluation, and optimization ideas > (binding with actual values is faster than producing dummy bindings and initializing > later). It uses LET bindings first, then degrades to (P)SETQ inside TAGBODY. > Fix that logic not to degrade in unexpected (and unwarranted) cases. > > Benefits: > - Visible order of evaluation is globally obeyed. > > Disadvantages: > - (loop FOR x = # THEN # FOR vars ON vars ...) becomes unable to satisfy user expectations, e.g. bug #375; > (loop FOR vars ON vars FOR x = # THEN # ...) would work as expected(!). User expectations were flawed in this case. The warning will help. > - initforms sometimes evaluated in an environment including all variables, sometimes > not. How to document and explain that to users? It does not need to be explained. In those cases where it matters, the users will see the warning. In summary, although I haven't said "yes, great, please implement it now" to any particular proposal, your analysis has brought light on what we need to do next, to get rid of the maze of constraints. Thanks!! Bruno |