clisp-devel — closed list for developers and testers

Re: release plans and schedule

[Bruno H. <br...@cl...>]

Hi Sam and Jörg,

> >   - fix remaining high-priority issues: bug 718.

Done.

> 1. regexp module #691
> 2. Hopefully loop.
> My TODO contains only LOOP.
> I think 719 is also high-priority.

OK, that's clear. And many more commits have gone in since beginning of
December 2017. All this is too fresh to go into the release that everyone
is waiting for.

I'm going to put the following into the release branch:
all commits <= 16183
16185..16209
16211..16212
16214..16215
16219..16222
16224..16226
16229
16231
16234
16236
16238..16240
16242..16243
16245..16252
16254
16261
16264..16267
and the remaining asm fixes.

> I think 719 is also high-priority.
> It is unconscionable that CLISP compiles its own libunistring and
> libregexp on linux.

This can wait for 2.51.

> At that point I will either say "LOOP is fixed" or "please create the
> branch and roll back 52d209b7d29d".

I don't like to roll back on the branch (because that makes "hg annotate"
results harder to understand). Instead, I'm picking a branch point and
cherry-picking from the main branch.

Bruno

Re: multiple unsequenced modifications

[Bruno H. <br...@cl...>]

Sam asked on 2017-12-07:
> >> modules/rawsock/rawsock.c:893:48: warning: multiple unsequenced
> >>   modifications to 'STACK' [-Wunsequenced]
> >> It is worth bothering with?
> >
> > Definitely!
> 
> How is this different from -Wsequence-point ?

Look up the manuals: GCC [1] has -Wsequence-point, whereas clang [2]
has -Wsequence-point and -Wunsequenced, and they are synonymous [3].

Bruno

[1] https://gcc.gnu.org/onlinedocs/gcc-7.2.0/gcc/Warning-Options.html
[2] https://clang.llvm.org/docs/DiagnosticsReference.html
[3] https://clang.llvm.org/docs/DiagnosticsReference.html#wsequence-point

Re: About driver and named variables in LOOP, esp. for-as-arithmetic

[Bruno H. <br...@cl...>]

Hi Jörg,

> Here's some vocabulary I made up:
> 
> Driver Variable(s)
> The main variable(s) that permit iteration, e.g. the array index FOR # ACROSS.
> One may or not consider the vector that is constant through the iteration as a
> driver variable too.
> 
> Derived Variables
> Those are derived from driver variables, e.g. all variables in a true destructuring pattern.
> A "true" destructuring pattern is a list, i.e. either nil or a cons, not another symbol.
> 
> Named variables
> Those explicitly named by the user of LOOP.
> Note that this also includes collecting INTO # etc. but that's irrelevant for now.
> 
> Clearly in FOR x IN ..., x is nevertheless a derived variable of some internal
> driver variable necessary to hold the list whose CAR is set to x during iteration.
> 
> FOR x ON ... is peculiar, because users expect x to *be* the driver variable. So we
> shall meet that expectation as a special rule.
> 
> Regarding for-as-arithmetic, it can be debated whether we should make driver and named variable collapse.

I don't think we have much of a choice. The wording in CLHS 6.1.2.1.1 matters.

It says: "The variable var is bound to the value of form1 in the first iteration
  and is stepped[1] by the value of form3 in each succeeding iteration, or by 1
  if form3 is not provided."
So "stepping" means to modify the value of var.

Case UPTO:
  (loop for i from 0.5 upto 10 finally (return i))
The text is clear: "the loop terminates when the variable var passes the value
of form2". So, in your terms, the driver variable is the named variable.
The expected result is 10.5. clisp and sbcl get this right.

Case BELOW:
  (loop for i from 0.5 below 10 finally (return i))
The text is clear: "These keywords stop iteration just before the value of the
variable var reaches the value supplied by form2". So, either the driver variable
is different from the named variable, or the addition during each stepping is
performed twice.
The expected result is 9.5. clisp and sbcl both get this wrong.

Case TO:
  (loop for i from 0.5 to 10 finally (return i))
The wording is not clear, but it is probably meant to execute like UPTO.
clisp and sbcl return the expected value 10.5.

Bruno

Re: About how to initialize in LOOP

[Bruno H. <br...@cl...>]

Hi Jörg,

> What I propose for LOOP in clisp is to apply the exact same scoping rules that govern
> WITH (possibly joined by AND) to named variables in FOR clauses.
> ...
> They should have distinguished for-as-equals-then and for-as-equals.
> ...
> Regarding a future Lisp, my vote would be

We really should distinguish what can be done
  * in a future Lisp
versus
  * in an ANSI CL compliant implementation.

The right place to discuss the future Lisp are these Cliki pages:
  http://cliki.net/Proposed%20ANSI%20Changes

What we do in clisp, should IMO be consistent with ANSI CL.

> As the discussion around issue 222 shows, the committee did not reach a consensus, to
> the texts were not changed. The ancient MIT LOOP implements this behaviour.

You make it sound like the committee's decision of
LOOP-INITFORM-ENVIRONMENT:PARTIAL-INTERLEAVING-VAGUE was a bad or incomplete one
and should therefore disregarded.

However, two people have stated the rationale for it:
JonL:
  "I claim this ambiguity is purposeful -- the "interleaving" of binding
   and initialization clauses, which in some implementations has 
   performance consequences"
Kent Pitman:
  "... the part of LOOP that should _not_ be exposed, which is what
   is the right and most efficient expansion for any given implementation in
   order to benchmark well.  The more you try to pin down that latter part,
   the more you just require Common Lisp to be slow.  Some parts are
   meant to be kept abstract."

In fact, for implementations with a complex compiler (with tons of optimizations)
the differences won't matter. It's for implementations with a simple compiler,
like clisp, that this argument has the biggest effect. Like what Sam observed:
that adding some temporary variable has the effect of producing larger bytecode.

> clisp's current loop contains complicated logic in an attempt to satisfy requirements
> on lexical environments, global left to right evaluation, and optimization ideas

Excellent remark. To fix this, we should try to find ways to execute the forms
in given scopes where ANSI CL mandates it, so that the execution order becomes
independent from scoping considerations.

> Proposal: FOR-AS-EQUALS-THEN-HAS-INIT-FORM
> Consider for-as-equals-then as
> FOR # = init-form THEN form
> i.e. apply aforementioned handling about variable scoping and init forms.
> 
> Benefits:
>  - Named variables are immediately bound, hence values are readily available in every
>     subsequent clause as well as INITIALLY clauses.
>  - left-to-right evaluation across all init-forms is globally preserved.
>  - Easy to grasp and to explain.
>  - Implements ANSI issue 222 (which didn't reach consensus).
>    http://clhs.lisp.se/Issues/iss222_w.htm
> 
> However, this proposal appears incompatible with CLHS 6.1.1.4: "form1 and form2 in
> for-as-equals-then form include the lexical environment of all loop variables."

So this proposal is not usable for clisp.

> Proposal: FOR-AS-EQUALS-FIRST-THEN-LATER
> Consider for-as-equals-then as
> FOR # = form1 THEN form2
> 
> Because per CLHS 6.1.1.4 (quoted above), first-iteration need be evaluated in an
> environment including all loop variables, the implementor of clisp's current loop
> IMHO wrongly concluded that all following FOR clauses (here including those joined by
> AND) must be evaluated there as well, to respect left to right evaluation order
> requirements, causing known bugs and scoping rules very hard to explain or even
> understand, as scoping now depends on former clauses.
> 
> Instead, this proposal follow CLHS by the letter to "initialize the variables [...]  by
> setting it to the result of evaluating form1 on the first iteration, then ..."
> -- Initialize, not bind! I.e. bind to NIL, later initialize to form1.
> 
> That definition can work, considering that LOOP users need be aware of code motion
> anyway -- Think how (LOOP INITIALLY (princ 2) WITH a = (princ 1) RETURN a) prints 12
> (can anybody reveal an implementation that produces another output?).

This result follows from 6.1.1.6 (WITH produces bindings, whereas INITIALLY code
goes into the loop prologue) with 6.1.1.4.

> In effect, it will be very similar to for-as-in-list, for-as-across or for-as-hash:
>  - The named variables need default bindings when declared;
>  - It is only upon entering iteration that actual values will be supplied.
> 
> (loop for a = (princ 1) then b for b below (princ 2) do (princ (list a b)))
> would print 21 by that proposal.

With this sample, you look at order of execution, not scope. Order of execution
is specified in 6.1.1.6. While clisp surely has bugs in this area, in this case:
the form '(princ 1)' is an initialization, whereas the form '(princ 2)' is not
an initialization. => (princ 1) must be executed first.

So this proposal is not usable for clisp either.

> Proposal: FOR-AS-EQUALS-THEN-EARLY
> In for-as-equals-then, initialize the variables prior to any INITIALLY.
> (yet still in full lexical scope of all loop variables in the loop prologue).
> 
> Benefits:
> - Named variables are ready for use within INITIALLY.
> - which distinguishes it from for-as-equals without THEN, so the user has the choice.

Do you have a test case for this? I believe clisp implements this in some case.

> Proposal: FOR-AS-EQUALS-THEN-LATE
> In for-as-equals-then, initialize the variables just upon entering the loop
> body. INITIALLY clauses will only see default values.
> 
> Benefits:
> - Very close to for-as-equal without then, where I don't plan at all to duplicate the
>   step form in the loop prologue and the loop body.
> 
> Disadvantages:
> - Precisely that it's late, so INITIALLY only ever sees default values.

I think this proposal will become obsolete when we decouple scope from
execution order.

> Proposal: FOR-AS-EQUALS-THEN-VERY-LATE
> In for-as-equals-then, initialize the variables within the loop body, after
> evaluating the termination tests of preceding FOR-AS clauses.
> 
> clisp did/does this.
> (loop for vars on vars for x = (error) then 2 return vars) would not error out.

sbcl still signals an error on this sample, regardless what we do in clisp.
This sample now signals a warning before the error:
  WARNING: Reference to VARS is implementation-dependent, per ANSI CL 6.1.1.4.
This is the best we can do: tell the programmer to change their code.

So this proposal is pointless for clisp.

> Proposal: FOR-AS-INITIALLY-INTERLEAVE-LET
> The LET forms resulting from FOR-AS initforms and WITH clauses could alternate
> easily. That would make it trivial to globally respect left to right evaluation across
> FOR-AS, WITH and even INITIALLY clauses.
> 
> This is not strictly conforming to CLtL2 or CLHS. They require INITIALLY to lie in the
> loop prologue, which is inside the TAGBODY, hence after all LET forms, instead of amid
> LET forms.
> ...
> Disadvantage
>  - Not the letter of CLtL2 or CLHS.

So this proposal is not usable for clisp.

> Proposal: FOR-AS-INITIALLY-GROUP-PROLOGUE
> Group all INITIALLY clauses in the prologue, following all variable initializations.
> 
> Benefits:
> - Values of driver variables readily accessible in INITIALLY
> - Easy to explain (after grasping code movement)
> 
> Disadvantage:
> - Code movement, hence no global left to right evaluation order between INITIALLY and FOR-AS/WITH initforms
> Do you interpret ANSI-CL to expect a global order of evaluation across INITIALLY and FOR/AS-WITH clauses?

Can you provide a sample, and review it in the light of 6.1.1.6?

> Proposal: FOR-AS-INITIALLY-INTERLEAVE-PROLOGUE
> Use dummy LET bindings, assign variables later in the prologue, which enables an
> interleaving with INITIALLY clauses, hence benefitting left to right evaluation.
> 
> This is somehow what clisp's LOOP currently does.
> 
> Benefits:
> - Global left to right evaluation order between INITIALLY and FOR-AS/WITH initforms
> 
> Disadvantages:
> - LOOP for VARS on VARS won't work (without further tricks, e.g. see FLET below).
> - INITIALLY evaluates in an environment including all variables, yet values are not
>   yet available, depending on clause order, so it's somehow unreliable.

Can you give samples?

> Proposal: FOR-AS-EQUALS-THEN-INTERLEAVE
> Interleave the initialization of for-as-equals-then variables with INITIALLY forms.
> 
> Benefits:
> - left to right evaluation is obeyed.
> 
> Disadvantages:
> - It may appear inconsistent to obey left to right with for-as-equals-then only,
>   when all other for-as forms init-forms are evaluated and bindings were established
>   long before any INITIALLY.

Can you give samples?

> Proposal: FOR-AS-EQUALS-THEN-FLET-FOR-SCOPE
> Maintain the complexity in clisp's current loop, and fix some scoping bugs by adding
> even more complexity. FLET can be used to evaluate init-forms in an outer scope not
> comprising all loop variables.
> 
> (LOOP FOR a = (princ 1) FOR vars ON (princ vars) RETURN (list a b))
> would expand to something containing
> (let ((A nil))
>   (flet ((#:on-init () (princ VARS)))
>     (let ((VARS nil))
>       (tagbody ... (setq A (princ 1)) (setq VARS (#:on-init)) ...))))
> 
> Benefits:
>  - At least the scoping rules become explainable.
>  - Visible order of evaluation is obeyed.
> 
> Disadvantage:
>  - Poor performance, likely needs closure.

This is a nice idea, but needs compiler work first.

> Proposal: FOR-AS-PROLOGUE-FIX
> clisp's current loop contains complicated logic in an attempt to satisfy requirements
> on lexical environments, global left to right evaluation, and optimization ideas
> (binding with actual values is faster than producing dummy bindings and initializing
> later).  It uses LET bindings first, then degrades to (P)SETQ inside TAGBODY.
> Fix that logic not to degrade in unexpected (and unwarranted) cases.
> 
> Benefits:
>  - Visible order of evaluation is globally obeyed.
> 
> Disadvantages:
>  - (loop FOR x = # THEN # FOR vars ON vars ...) becomes unable to satisfy user expectations, e.g. bug #375;
>    (loop FOR vars ON vars FOR x = # THEN # ...) would work as expected(!).

User expectations were flawed in this case. The warning will help.

>  - initforms sometimes evaluated in an environment including all variables, sometimes
>    not. How to document and explain that to users?

It does not need to be explained. In those cases where it matters, the
users will see the warning.

In summary, although I haven't said "yes, great, please implement it now" to any
particular proposal, your analysis has brought light on what we need to do next,
to get rid of the maze of constraints. Thanks!!

Bruno

Re: lazy-let & loop

[Bruno H. <br...@cl...>]

Sam wrote:
> >> The problem with loop is that under certain circumstances we separate
> >> initializing and binding of a variable.
> >
> > This is not a problem by itself. It is explicitly allowed by CLHS
> > 6.1.1.4 "Implementations can interleave the setting of initial values
> > with the bindings."
> 
> If I understand you correctly, this merely implies that the naive user
> expectations are not guaranteed. Sure. Why not give them what they want
> if we can though?

The ANSI CL standard is explicitly vague in this area, so that implementations
can optimize loops well.

Look how GCC handles such a situation: Recently you encountered code which
  - was not portable across C implementations,
  - did things that the original author _thought_ would be OK but it was in
    fact risky code.
GCC showed a warning "multiple unsequenced modifications". Then we modified
the code to avoid the area of pitfalls.

The same approach is good here as well. I've added a warning
  Reference to <symbol> is implementation-dependent, per ANSI CL 6.1.1.4.
This has two benefits:
  * Users have an opportunity to change their code in such a way that it is
    guaranteed to be ANSI CL compliant (and therefore portable to ANSI CL
    implementation).
  * We don't need to modify clisp to improve such uses of LOOP, when it
    would be detrimental to the performance of other uses of LOOP (that don't
    generate a warning).

Bruno

New Spanish PO file for 'clisp' (version 2.49.60)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'clisp' has been submitted
by the Spanish team of translators.  The file is available at:

    http://translationproject.org/latest/clisp/es.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/clisp/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/clisp.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

Re: QuickLisp fails

[Sam S. <sd...@gn...>]

> * Blake McBride <oy...@zp...zr> [2017-12-25 11:52:00 -0600]:
>
> Are you telling me that quicklisp hasn't worked since 2008?

I am telling you that since 2008 quicklisp compilation produced
a deprecation warning.

This is what is meant by:

"The old names are still available, but deprecated."

(see the news item quoted below).


> On Mon, Dec 25, 2017 at 11:44 AM, Sam Steingold <sd...@gn...> wrote:
>
>> > * Blake McBride <oy...@zp...zr> [2017-12-25 09:04:39 -0600]:
>> >
>> >> It would be good if someone could file a pull request for updating
>> >> quicklisp to use DELETE-DIRECTORY instead of DELETE-DIR.
>> >
>> > I think the code should test and work in either case since some people
>> > may be using the old, released CLISP, and other would be using the
>> > current, HG, version.
>>
>> From clisp/src/NEWS:
>>
>> --8<---------------cut here---------------start------------->8---
>> 2.44 (2008-02-02)
>> =================
>> ...
>> * Rename EXT:DELETE-DIR, EXT:MAKE-DIR, and EXT:RENAME-DIR to
>>   EXT:DELETE-DIRECTORY, EXT:MAKE-DIRECTORY, and EXT:RENAME-DIRECTORY,
>>   respectively, for consistency with EXT:PROBE-DIRECTORY,
>>   EXT:DEFAULT-DIRECTORY and CL:PATHNAME-DIRECTORY.
>>   The old names are still available, but deprecated.
>> --8<---------------cut here---------------end--------------->8---
>>
>> Do you think there are pre-2.44 CLISP installation around?
>>
>> Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net
http://memri.org http://jij.org http://camera.org https://jihadwatch.org
Yellow wine is called "white" because it is made out of green grapes.

Re: QuickLisp fails

[Blake M. <bl...@mc...>]

Are you telling me that quicklisp hasn't worked since 2008?

On Mon, Dec 25, 2017 at 11:44 AM, Sam Steingold <sd...@gn...> wrote:

> > * Blake McBride <oy...@zp...zr> [2017-12-25 09:04:39 -0600]:
> >
> >> It would be good if someone could file a pull request for updating
> >> quicklisp to use DELETE-DIRECTORY instead of DELETE-DIR.
> >
> > I think the code should test and work in either case since some people
> > may be using the old, released CLISP, and other would be using the
> > current, HG, version.
>
> From clisp/src/NEWS:
>
> --8<---------------cut here---------------start------------->8---
> 2.44 (2008-02-02)
> =================
> ...
> * Rename EXT:DELETE-DIR, EXT:MAKE-DIR, and EXT:RENAME-DIR to
>   EXT:DELETE-DIRECTORY, EXT:MAKE-DIRECTORY, and EXT:RENAME-DIRECTORY,
>   respectively, for consistency with EXT:PROBE-DIRECTORY,
>   EXT:DEFAULT-DIRECTORY and CL:PATHNAME-DIRECTORY.
>   The old names are still available, but deprecated.
> --8<---------------cut here---------------end--------------->8---
>
> Do you think there are pre-2.44 CLISP installation around?
>
> Thanks.
>
> --
> Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
> http://steingoldpsychology.com http://www.childpsy.net https://ffii.org
> http://think-israel.org http://camera.org http://islamexposedonline.com
> Who is General Failure and why is he reading my hard disk?
>

Re: QuickLisp fails

[Sam S. <sd...@gn...>]

> * Blake McBride <oy...@zp...zr> [2017-12-25 09:04:39 -0600]:
>
>> It would be good if someone could file a pull request for updating
>> quicklisp to use DELETE-DIRECTORY instead of DELETE-DIR.
>
> I think the code should test and work in either case since some people
> may be using the old, released CLISP, and other would be using the
> current, HG, version.

>From clisp/src/NEWS:

--8<---------------cut here---------------start------------->8---
2.44 (2008-02-02)
=================
...
* Rename EXT:DELETE-DIR, EXT:MAKE-DIR, and EXT:RENAME-DIR to
  EXT:DELETE-DIRECTORY, EXT:MAKE-DIRECTORY, and EXT:RENAME-DIRECTORY,
  respectively, for consistency with EXT:PROBE-DIRECTORY,
  EXT:DEFAULT-DIRECTORY and CL:PATHNAME-DIRECTORY.
  The old names are still available, but deprecated.
--8<---------------cut here---------------end--------------->8---

Do you think there are pre-2.44 CLISP installation around?

Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net https://ffii.org
http://think-israel.org http://camera.org http://islamexposedonline.com
Who is General Failure and why is he reading my hard disk?

Re: QuickLisp fails

[Blake M. <bl...@mc...>]

On Mon, Dec 25, 2017 at 7:05 AM, Bruno Haible <br...@cl...> wrote:

> Blake McBride wrote:
> > I noticed that QuickLisp fails on the current CLISP but succeeds in a
> > variety of other Lisp's I tried.  What I get is
> > as follows:
> > ** - Continuable Error
> > DEFPACKAGE QL-CLISP: There is no symbol EXT::DELETE-DIR .
>
> Indeed, https://github.com/quicklisp/quicklisp-client/blob/master/
> quicklisp/impl.lisp
> and https://github.com/quicklisp/quicklisp-client/blob/master/
> quicklisp/impl-util.lisp
> reference the EXT:DELETE-DIR function, which we have now removed.
>
> Jean Louis wrote:
> > Did you try the latest development version?
> >
> > I did and now it is working. Make the latest one.
>
> Which latest development version of quicklisp do you mean?
>
> It would be good if someone could file a pull request for updating
> quicklisp to use DELETE-DIRECTORY instead of DELETE-DIR.
>
> Bruno
>

I think the code should test and work in either case since some people may
be using the old, released CLISP, and other would be using the current, HG,
version.

Thanks.

Blake

Re: QuickLisp fails

[Jean L. <bu...@gn...>]

On Mon, Dec 25, 2017 at 02:05:35PM +0100, Bruno Haible wrote:
> Blake McBride wrote:
> > I noticed that QuickLisp fails on the current CLISP but succeeds in a
> > variety of other Lisp's I tried.  What I get is
> > as follows:
> > ** - Continuable Error
> > DEFPACKAGE QL-CLISP: There is no symbol EXT::DELETE-DIR .
> 
> Indeed, https://github.com/quicklisp/quicklisp-client/blob/master/quicklisp/impl.lisp
> and https://github.com/quicklisp/quicklisp-client/blob/master/quicklisp/impl-util.lisp
> reference the EXT:DELETE-DIR function, which we have now removed.
> 
> Jean Louis wrote:
> > Did you try the latest development version?
> > 
> > I did and now it is working. Make the latest one.
> 
> Which latest development version of quicklisp do you mean?
> 
> It would be good if someone could file a pull request for updating
> quicklisp to use DELETE-DIRECTORY instead of DELETE-DIR.

I was using CLISP development version. And I also
remember changing that delete-directory in the
source to get it working. Copy to Zach to update
it. 

-- 
Jean Louis

Re: QuickLisp fails

[Bruno H. <br...@cl...>]

Blake McBride wrote:
> I noticed that QuickLisp fails on the current CLISP but succeeds in a
> variety of other Lisp's I tried.  What I get is
> as follows:
> ** - Continuable Error
> DEFPACKAGE QL-CLISP: There is no symbol EXT::DELETE-DIR .

Indeed, https://github.com/quicklisp/quicklisp-client/blob/master/quicklisp/impl.lisp
and https://github.com/quicklisp/quicklisp-client/blob/master/quicklisp/impl-util.lisp
reference the EXT:DELETE-DIR function, which we have now removed.

Jean Louis wrote:
> Did you try the latest development version?
> 
> I did and now it is working. Make the latest one.

Which latest development version of quicklisp do you mean?

It would be good if someone could file a pull request for updating
quicklisp to use DELETE-DIRECTORY instead of DELETE-DIR.

Bruno

more on MT

[<don...@is...>]

I'm trying to expand my use of MT, inc. with-timeout, and so
want to understand better where interrupts can occur.
I'm imagining that the following idiom should work:
(catch 'timeout
  (with-timeout (10 (throw 'timeout)) (do-stuff)))
There should be no problem if do-stuff reads data, or writes
data that does not survive the call to do-stuff.
So I'm mainly worrying about writing data that will survive the
possibly interrupted call.  

One kind of data I want to write is the result of whatever was
found in the 10 seconds allowed, e.g.,
(let (results)
  (catch 'timeout
    (with-timeout (10 (throw 'timeout)) (do-stuff)))
where do-stuff includes things like
 (push newresult results)
I'm expecting the (setf results) par of this will either be done 
completely or not done at all no matter where an interrupt
occurs.  (I'm actually viewing it as an atomic write into memory.)
Am I ok so far?

I'd like the same to be true of any setf of a single place,
such as writing into a hash table.  I'm hoping that this
is similarly atomic in the sense that the table remains in a 
valid state after an interrupt and throw, either the state it was
in before the (setf gethash) or the state that would be expected
after the (setf gethash).
Is that true in the current implementation?
If not, does it solve the problem to put the setf inside
with-deferred-interrupts ?
(Is there some way I should have known the answer from the doc?)
Note that I'm already protecting that hash table from being written
by more than one thread at a time.

What does preemption mean in this doc?
MT:WITH-DEFERRED-INTERRUPTS &BODY body)
    Defer thread interrupts (but not thread preemption)

Does that mean that the OS can stop running this thread at a point 
where the table is not in one of those two valid states, and that 
either the thread might never be resumed (and the table might end
up in an undesirable state) or later when the thread is resumed it
might be interrupted in an undesirable state?

Impnotes contains a very useful warning pointing out that interrupts
can occur in cleanup forms.
I suggest that be changed -- almost all cleanup forms you really 
want to complete.
This is especially true for functions like with-open-file that I imagine
use unwind-protect internally and the user has no way to add protection
to the cleanup forms (other than protecting the entire with-open-file).
Is it already the case that such internal cleanup forms are protected 
from interrupts?

Is it even possible to protect (only) the cleanup forms with
deferred-interrupts?
 (unwind-protect (foo) (with-deferred-interrupts (bar)))
could still interrupt just before (with-deferred-interrupts (bar)),
right?  Or is that not one of the places GC could occur?

I'm guessing that even without worrying about cleanup forms,
something like
 (with-open-file (f filename :direction :output)(print x f))
could be interrupted during the print, and if aborted, could end up 
with pretty much any part of x in the file.  So generally these 
forms will require some sort of unwind protect that tests whether 
the file is expected to be in a satisfactory state and tries to 
recover if not.

It would be nice to provide a variant of with-open-file for the 
common case that checks whether the stuff inside with-open-file
was completed, and if not deletes the file (if it was created)
or resets the file length (if it was appended) or restores the
previous version (if a new version was created).  

The section on GC suggests that a loop that never conses or makes
system calls might be uninterruptible.  Is this true?  I hope not.
Would it make sense to make any backward jump or function call
interruptible?

BTW, it would be really nice to use threads to do GC while other 
threads continue to run.  What's the prospect of that?

Sorry this message is so long and wide ranging.  
I'll be happy to get answers or discussion on small subsets of
the questions here.

Re: QuickLisp fails

[Jean L. <bu...@gn...>]

On Sun, Dec 24, 2017 at 10:15:44PM -0600, Blake McBride wrote:
> Greetings,
> 
> I noticed that QuickLisp fails on the current CLISP but succeeds in a
> variety of other Lisp's I tried.  What I get is
> as follows:

Did you try the latest development version?

I did and now it is working. Make the latest one.

-- 
Jean Louis

QuickLisp fails

[Blake M. <bl...@mc...>]

Greetings,

I noticed that QuickLisp fails on the current CLISP but succeeds in a
variety of other Lisp's I tried.  What I get is as follows:

$ clisp
  i i i i i i i       ooooo    o        ooooooo   ooooo   ooooo
  I I I I I I I      8     8   8           8     8     o  8    8
  I  \ `+' /  I      8         8           8     8        8    8
   \  `-+-'  /       8         8           8      ooooo   8oooo
    `-__|__-'        8         8           8           8  8
        |            8     o   8           8     o     8  8
  ------+------       ooooo    8oooooo  ooo8ooo   ooooo   8

Welcome to GNU CLISP 2.49.60+ (2017-06-25) <http://clisp.org/>

Copyright (c) Bruno Haible, Michael Stoll 1992, 1993
Copyright (c) Bruno Haible, Marcus Daniels 1994-1997
Copyright (c) Bruno Haible, Pierpaolo Bernardi, Sam Steingold 1998
Copyright (c) Bruno Haible, Sam Steingold 1999-2000
Copyright (c) Sam Steingold, Bruno Haible 2001-2010

Type :h and hit Enter for context help.

[1]> (load "quicklisp")
;; Loading file /home/blake/Backup/QuickLisp/quicklisp.lisp ...

  ==== quicklisp quickstart 2015-01-28 loaded ====

    To continue with installation, evaluate: (quicklisp-quickstart:install)

    For installation options, evaluate: (quicklisp-quickstart:help)

;; Loaded file /home/blake/Backup/QuickLisp/quicklisp.lisp
#P"/home/blake/Backup/QuickLisp/quicklisp.lisp"
[2]> (quicklisp-quickstart:install)
; Fetching #<URL "http://beta.quicklisp.org/client/quicklisp.sexp">
; 0.82KB
==================================================
838 bytes in 0.02 seconds (35.80KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/client/2017-03-06/quicklisp.tar
">
; 250.00KB
==================================================
256,000 bytes in 0.22 seconds (1139.87KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/client/2015-09-24/setup.lisp">
; 4.94KB
==================================================
5,054 bytes in 0.00 seconds (2042.01KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/asdf/2.26/asdf.lisp">
; 194.07KB
==================================================
198,729 bytes in 0.18 seconds (1083.28KB/sec)
** - Continuable Error
DEFPACKAGE QL-CLISP: There is no symbol EXT::DELETE-DIR .
If you continue (by typing 'continue'): This symbol will be created.
The following restarts are also available:
RETRY          :R1      Retry compiling #<CL-SOURCE-FILE "quicklisp"
"impl">.
ACCEPT         :R2      Continue, treating compiling #<CL-SOURCE-FILE
"quicklisp" "impl"> as having been successful.
SKIP           :R3      skip (LET # #)
RETRY          :R4      retry (LET # #)
STOP           :R5      stop loading file /home/blake/quicklisp/setup.lisp
ABORT          :R6      Abort main loop
Break 1 QL-IMPL[3]>

Re: 1 extra bytecode instruction per variable per loop

[Bruno H. <br...@cl...>]

Hi Sam,

> --8<---------------cut here---------------start------------->8---
> Disassembly of function :LAMBDA
> (CONST 0) = 100
> (CONST 1) = B
> (CONST 2) = Y
> 0 required arguments
> 0 optional arguments
> No rest parameter
> No keyword parameters
> reads special variables: Y B
> 27 byte-code instructions:
> 0     (CONST&PUSH 0)                      ; 100
> 1     (PUSH-NIL 4)
> 3     (GETVALUE 1)                        ; B
> 5     (STORE 3)
> 6     (GETVALUE 2)                        ; Y
> 8     (STORE 2)
> 9     (JMP L26)
> 11    L11
> --8<---------------cut here---------------end--------------->8---
> 
> becomes
> 
> --8<---------------cut here---------------start------------->8---
> Disassembly of function :LAMBDA
> (CONST 0) = B
> (CONST 1) = Y
> (CONST 2) = 100
> 0 required arguments
> 0 optional arguments
> No rest parameter
> No keyword parameters
> reads special variables: Y B
> 29 byte-code instructions:
> 0     (GETVALUE&PUSH 0)                   ; B
> 2     (GETVALUE&PUSH 1)                   ; Y
> 4     (CONST&PUSH 2)                      ; 100
> 5     (PUSH-NIL 4)
> 7     (LOAD 6)
> 8     (STORE 3)
> 9     (LOAD 5)
> 10    (STORE 2)
> 11    (JMP L28)
> 13    L13
> --8<---------------cut here---------------end--------------->8---
> 
> The cost is probably trivial, but I wonder if there is some simple
> compilation optimization missing here...

The missing compiler optimization is a full data flow analysis.

clisp's compiler supports data flow analysis for simple cases (e.g.
a variable that is only bound to an initform and never assigned).
More complex cases that would require multiple passes are not
implemented.

The LOOP macro (and other macros) have been written in a way to
generate good bytecode.

Bruno

Re: WG: Recovering the once true meaning of type NIL in LOOP

[Bruno H. <br...@cl...>]

Don Cohen wrote:
> I notice the MIT loop macro (which is what I normally use) gives
> 
> (macroexpand '(loop for (a b c) of-type (integer) in x do nil))
> 
> (LET ((A 0) (B NIL) (C NIL) (#:G10173004 X))
>  (DECLARE (TYPE LIST #:G10173004) (TYPE INTEGER A))
>  ...
> 
> I guess you'd view it as non-compliant

I didn't say that this was non-compliant. If you write code
where you bind variables B and C and declare that these variables
have no valid values, an implementation is free to
  - ignore the "obviously" wrong declaration - this is what MIT LOOP does,
  - or emit a warning,
  - or signal an error,
  - or launch a rocket to the moon.

What I'm saying is that not all implementations are required to do the
same thing.

> So I'll now return to my tradition of avoiding loop issues by 
> sticking with MIT loop.

Yuck. Everyone has a different way of dealing with the oddities of CL LOOP.
Mine is to stick to the "simple" cases...

Bruno

Re: lazy-let & loop

[Bruno H. <br...@cl...>]

Hi Sam,

> > Your proposal introduces a new approach to scoping: Have the scope
> > depend on whether the access to a variable is a read or a write
> > access.
> 
> Not really - it merely allows control over binding activation.
> ...
> This activates the binding and assigns the inner binding to 1+outer.

This is nearly as bad. If we have a scoping problem in LOOP, the "solution"
to rebind the variable with initially the same value looks really like a
makeshift, and the "at the first SETQ" rule will not do what you expected
when the code flow is not linear (which can happen, given that the LOOP
macro is reshuffling user-provided forms).

The way to proceed, IMO, is to pursue Jörg's "About how to initialize in LOOP"
thought.

> > Instead, maybe we should provide explicit control of the environment in
> > which a form will be evaluated/compiled. For example:
> >
> >   (let ((a 1))
> >     (environment-let env-with-a
> >       (let ((b 2))
> >         (in-enviroment env-with-a (list a b)))))
> 
> (declare (name-environment env-with-a))
> 
> and
> 
> (declare (use-environment env-with-a))
> 
> looks better to me.

But code walkers usually ignore declarations, which they are allowed to do,
per CLHS 3.3.1.

> >> The problem with loop is that under certain circumstances we separate
> >> initializing and binding of a variable.
> >
> > This is not a problem by itself. It is explicitly allowed by CLHS
> > 6.1.1.4 "Implementations can interleave the setting of initial values
> > with the bindings."
> 
> If I understand you correctly, this merely implies that the naive user
> expectations are not guaranteed. Sure. Why not give them what they want
> if we can though?
> ...
> I am trying to implement "naive user expectations" (as long as they are
> conforming, of course).

If it is easy to do without introducing user-visible bugs or de-optimizations,
I am all for implementing "naive user expectations". (Recall that the
rationale for LOOP-INITFORM-ENVIRONMENT:PARTIAL-INTERLEAVING-VAGUE was
to allow implementations to optimize, according to Kent Pitman.)

In this case, however, Jörg's summary indicates that there are some
constraints, and there is no clear winner among the proposals he has
investigated so far.

Bruno

Re: WG: Recovering the once true meaning of type NIL in LOOP

[<don...@is...>]

I notice the MIT loop macro (which is what I normally use) gives

(macroexpand '(loop for (a b c) of-type (integer) in x do nil))

(LET ((A 0) (B NIL) (C NIL) (#:G10173004 X))
 (DECLARE (TYPE LIST #:G10173004) (TYPE INTEGER A))
 ...

I guess you'd view it as non-compliant, though I thought one of the
goals of the spec was to codify existing practice, which suggests
that in this case it's the spec that's at fault.

I'm also surprised not to see anything about the interpretation of 
nil as a type in the spec (not that I've searched it very hard), but
I'm pretty sure this has been discussed before, along with, for that 
matter, the degree to which MIT and other implementations comply.

So I'll now return to my tradition of avoiding loop issues by 
sticking with MIT loop.

Re: lazy-let & loop

[Sam S. <sd...@gn...>]

Hi Bruno,

> * Bruno Haible <oe...@py...t> [2017-12-23 22:31:05 +0100]:
>
> Your proposal introduces a new approach to scoping: Have the scope
> depend on whether the access to a variable is a read or a write
> access.

Not really - it merely allows control over binding activation.

> 1) The purpose of the scopes in the language definition is that by looking
>    a the (fully macroexpanded) source code of a function, the user can
>    tell to which binding a certain variable reference refers.
>    If the user furthermore has to _scan_ the source code for assignments,
>    this lookup is significantly more complicated.

CL is already complicated enough.
This is a small extra cherry on the side. ;-)
Jörg already implemented LAZY-LET in pure CL.
What I am suggesting is a mere optimization.

> 2) What about conditional accesses?
>      (lazy-let (a)
>        (if (> (random) 0.5)
>          (setq a 10))
>        (lambda (a) (print a)))

What you probably meant was lambda without arguments (otherwise the
question is trivial), and then we get random results:

(let ((a 1))
  (values
   (funcall (lazy-let1 (a)
              (when (zerop (random 2))
                (setq a 10))
              (lambda () (print a))))
   a))
==> 1 or 10 randomly as the 1st value; 1 always as the second value.


> 3) There are forms which combine a read access and a write access, like
>    (INCF b).

This activates the binding and assigns the inner binding to 1+outer.

> Instead, maybe we should provide explicit control of the environment in
> which a form will be evaluated/compiled. For example:
>
>   (let ((a 1))
>     (environment-let env-with-a
>       (let ((b 2))
>         (in-enviroment env-with-a (list a b)))))

(declare (name-environment env-with-a))

and

(declare (use-environment env-with-a))

looks better to me.

> Such a clisp specific set of special-operators ENVIRONMENT-LET and
> IN-ENVIRONMENT could be emulated for other implementations through a
> code walker.

Since lazy-let can be implemented via a simple macro, code walkers will
be happy.

> Still, I don't know whether such a thing is really needed for the LOOP
> macro, where we have a finite set of initforms and bindings.

So (loop for var in var do ...) will expand to something like

(macrolet ...
  (declare (name-environment outer))
  (let ((var nil))
    (tagbody ...
       (setq var (locally (declare (use-environment outer)) var))
       ...)))

this is easy, but I am afraid more complicated things will be harder.

(loop for x in x for y in x for z = y then x ...)

>> The problem with loop is that under certain circumstances we separate
>> initializing and binding of a variable.
>
> This is not a problem by itself. It is explicitly allowed by CLHS
> 6.1.1.4 "Implementations can interleave the setting of initial values
> with the bindings."

If I understand you correctly, this merely implies that the naive user
expectations are not guaranteed. Sure. Why not give them what they want
if we can though?

> What I see as a problem is that users are not warned when they write
> unportable code. That is, we should have a warning (in the spirit of
> "gcc -fsanitize=undefined" for C) whenever the interpretation of an
> initform actually depends on the environment.

You are looking at the problem from the portability/compliance POV.
I am trying to implement "naive user expectations" (as long as they are
conforming, of course).

Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://camera.org
http://jij.org http://islamexposedonline.com http://think-israel.org
How do you prove a cotheorem? Using rollaries.

Re: WG: Recovering the once true meaning of type NIL in LOOP

[Bruno H. <br...@cl...>]

Hi Don,

>  > Here you have assumed that the type specifier has a list structure
>  > with some kind of "optional" semantics. However, 6.1.1.7 is clear about:
> 
> Not sure what you mean by optional semantics - if there is nothing
> matching some variable then I don't want it assigned a type.

Yes, this is what I mean. This is not written in CLHS 6.1.1.7.

What is written is:
  (A) A type specifier for a destructuring pattern is a tree of type specifiers
  (B) same shape as the tree of variable names, with the following exceptions:
  (C) When aligning the trees, an atom in the tree of type specifiers that
      matches a cons in the variable tree declares the same type for each
      variable in the subtree rooted at the cons.
  (D) A cons in the tree of type specifiers that matches an atom in the tree of
      variable names is a compound type specifier.

By (A), we have to view the variables and type specifier as trees:
  (i . (j . (k . nil)))  and  (fixnum . nil)

By (B), fixnum matches i
and     nil    matches (j . (k . nil))

By (C), nil    matches j
and     nil    matches (k . nil)

By (C) again,  nil matches k.

> You could argue that (j) matches nil, but in that case I'd expect
> that the semantics of a single symbol matching a list should 
> specifically exclude nil as the symbol.

The rules don't talk about "a list". They talk about trees, atoms, and conses.

Bruno

Re: WG: Recovering the once true meaning of type NIL in LOOP

[<don...@is...>]

 > > but in (LOOP for (i j k) of-type (fixnum) ...)
 > > I would expect the atom fixnum to match only i,
 > 
 > Yes, agree.
 > 
 > > and I would NOT
 > > expect of-type (fixnum) to mean the same as of-type (fixnum nil nil)
 > > so I would expect j and k to be of unspecified type.
 > 
 > Here you have assumed that the type specifier has a list structure
 > with some kind of "optional" semantics. However, 6.1.1.7 is clear about:

Not sure what you mean by optional semantics - if there is nothing
matching some variable then I don't want it assigned a type.

 >   tree n. 1. a binary recursive data structure made up of conses and atoms
 > So the type specifier (FIXNUM) has to be read/interpreted as (FIXNUM . NIL).

Yes, but that's supposed to match a tree of variables, so 
(fixnum . nil) matches (i . nil) with i matching fixnum.
But I don't think that (i j) matches (fixnum) with j matching nil.
You could argue that (j) matches nil, but in that case I'd expect
that the semantics of a single symbol matching a list should 
specifically exclude nil as the symbol.
Something like this:
(defun match(x y)
 (cond ((and x (symbolp x) y (symbolp y))(list x y))
       ((and (consp x) y (symbolp y)) (list x y))
       ((and (consp y) y (symbolp x)) (list x y))
       ((and (consp x)(consp y))
        (cons (match (car x)(car y))(match (cdr x)(cdr y))))))
MATCH
[2]> (match '(i j k) 'integer)
((I J K) INTEGER)
[3]> (match '(i j k) '(integer))
((I INTEGER))

Re: WG: Recovering the once true meaning of type NIL in LOOP

[Bruno H. <br...@cl...>]

Hi Don,

> Bruno Haible writes:
>  > > Consider (macroexpand-1'
>  > > (LOOP for (i j k) of-type (fixnum) across #())
>  > 
>  > Who says that this should expand to something reasonable?
>  > 
>  > Citing 6.1.1.7 "Destructuring"
> 
>  >   "When aligning the trees, an atom in the tree of type specifiers
>  >    that matches a cons in the variable tree declares the same type
>  >    for each variable in the subtree rooted at the cons."
>  > 
>  > So this form declares I as being of type FIXNUM and J and K as
>  > being of type NIL.
> 
> My interpretation of
>  an atom in the type specifier matches a cons in the variable tree
> would be fixnum matching (i j k) in
>  (LOOP for (i j k) of-type fixnum ...)
> so in that case i,j,k would all be declared fixnum,

Yes.

> but in (LOOP for (i j k) of-type (fixnum) ...)
> I would expect the atom fixnum to match only i,

Yes, agree.

> and I would NOT
> expect of-type (fixnum) to mean the same as of-type (fixnum nil nil)
> so I would expect j and k to be of unspecified type.

Here you have assumed that the type specifier has a list structure
with some kind of "optional" semantics. However, 6.1.1.7 is clear about:

  A type specifier for a destructuring pattern is a tree of type specifiers ...

with

  tree n. 1. a binary recursive data structure made up of conses and atoms

So the type specifier (FIXNUM) has to be read/interpreted as (FIXNUM . NIL).

Bruno

Re: lazy-let & loop

[Bruno H. <br...@cl...>]

Jörg wrote:
> I've not yet attacked the problem of iteration in LOOP. I'm afraid there will be no way to
> make everybody happen w.r.t. the value of variables in FINALLY clauses.

There's already an issue write-up at http://cliki.net/Issue%20LOOP-FINALLY-VARIABLES .

Feel free to add write-ups for the other issues in LOOP !

Bruno

Re: lazy-let & loop

[Bruno H. <br...@cl...>]

Hi Sam,

> I wonder if something like "lazy-binding" could help: bind the variable
> but not initialize its value. until it is explicitly set, accessing it
> reaches up in the environment. This would require something like
> `lazy-let` with the following test case:
> 
> --8<---------------cut here---------------start------------->8---
> (let ((a 1))
>   (lazy-let (a b)
>     (print a) ; prints 1
>     (print b) ; error - unbound variable
>     (setq a 10) ; inner binding of a is activated
>     (print a) ; prints 10
>     (setq b 22)) ; binding of b is activated
>   (print b) ; error - unbound variable
>   (print a)) ; prints 1
> --8<---------------cut here---------------end--------------->8---
> 
> while it is easy to avoid `lazy-let` above by putting `let` after the
> second `print`, it is not so easy with `loop` because we need the
> binding to exist around the main `tagbody` but to do initialization in
> the preamble inside `tagbody`.

Your proposal introduces a new approach to scoping: Have the scope depend
on whether the access to a variable is a read or a write access.

While this may provide a small advantage in implementing LOOP, I believe
that it is a really bad idea in general.

1) The purpose of the scopes in the language definition is that by looking
   a the (fully macroexpanded) source code of a function, the user can
   tell to which binding a certain variable reference refers.
   If the user furthermore has to _scan_ the source code for assignments,
   this lookup is significantly more complicated.

2) What about conditional accesses?
     (lazy-let (a)
       (if (> (random) 0.5)
         (setq a 10))
       (lambda (a) (print a)))

3) There are forms which combine a read access and a write access, like
   (INCF b).

Instead, maybe we should provide explicit control of the environment in
which a form will be evaluated/compiled. For example:

  (let ((a 1))
    (environment-let env-with-a
      (let ((b 2))
        (in-enviroment env-with-a (list a b)))))

will evaluate (list a b) in an environment that contains the binding of a
but not the binding of b. This would be equivalent to

  (let ((a 1))
    (let ((#:g1 (lambda () (list a b))))
      (let ((b 2))
        (funcall #:g1))))

Such a clisp specific set of special-operators ENVIRONMENT-LET and
IN-ENVIRONMENT could be emulated for other implementations through a
code walker.

Still, I don't know whether such a thing is really needed for the LOOP
macro, where we have a finite set of initforms and bindings.

> The problem with loop is that under certain circumstances we separate
> initializing and binding of a variable.

This is not a problem by itself. It is explicitly allowed by CLHS 6.1.1.4
"Implementations can interleave the setting of initial values with the bindings."

What I see as a problem is that users are not warned when they write
unportable code. That is, we should have a warning (in the spirit of
"gcc -fsanitize=undefined" for C) whenever the interpretation of an initform
actually depends on the environment.

Bruno