clisp-devel — closed list for developers and testers

when do we remove deprecated features?

[Sam S. <sd...@gn...>]

Hi,

The following is our `sys::*deprecated-functions-alist*`:
--8<---------------cut here---------------start------------->8---
((WILDCARD:wildcard-matcher "Use ~S instead." FNMATCH-MATCHER)
 (WILDCARD:match "Use ~S instead." FNMATCH)
 (SOCKET-SERVICE-PORT "Use ~S instead." SERVICE)
 (RENAME-DIR "Use ~S instead." RENAME-DIRECTORY)
 (MAKE-DIR "Use ~S instead." MAKE-DIRECTORY)
 (DELETE-DIR "Use ~S instead." DELETE-DIRECTORY)
 (FFI:FOREIGN-ADDRESS-NULL "The FFI now returns C NULL pointers as Lisp NIL. Use the function ~S instead." NULL)
 (DEFINE-SETF-METHOD "Use ~S instead." DEFINE-SETF-EXPANDER)
 (GET-SETF-METHOD-MULTIPLE-VALUE "Use ~S instead." GET-SETF-EXPANSION)
 (SPECIAL-FORM-P "Use ~S instead." SPECIAL-OPERATOR-P)
 (TYPE-EXPAND-1 "Use ~S instead." TYPE-EXPAND)
 (GENTEMP "This function creates symbols that cannot be garbage-collected. Use ~S instead." GENSYM)
 (SET "This function name is anachronistic. Use ~S ~S instead." SETF SYMBOL-VALUE))
--8<---------------cut here---------------end--------------->8---

1. We should keep the CLtL2 and ANSI CL deprecated symbols.

2. We can drop our extensions:

type-expand-1:        deprecated since 2002-01-08
foreign-address-null: deprecated since 2003-08-05
rename-dir, make-dir, delete-dir:      2007-12-31
socket-service-port:  deprecated since 2005-11-09

3. We should keep wildcard (deprecated since 2011-05-17, after the last release)

WDYT?

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net https://ffii.org
https://jihadwatch.org http://honestreporting.com http://www.memritv.org
Our business is run on trust.  We trust you will pay in advance.

FFI: optional argument in foreign calls

[Sam S. <sd...@gn...>]

Hi,

How do we handle access to variadic foreign (C) functions?

E.g., in PARI/GP:

--8<---------------cut here---------------start------------->8---
factor(x,{lim}): factorization of x. lim is optional and can be set
whenever x is of (possibly recursive) rational type. If lim is set
return partial factorization, using primes < lim.
--8<---------------cut here---------------end--------------->8---

Is there a better way than

--8<---------------cut here---------------start------------->8---
(pari-call-out factor1 "factor" (x))
(pari-call-out factor2 "factor" (x (lim long)))
(defun factor (x &optional (lim nil limp))
  (if limp
      (factor2 x lim)
      (factor1 x)))
--8<---------------cut here---------------end--------------->8---

(I can probably wrap this into the pari-call-out macro, but the point is
-- do we really need *two* def-call-out forms?)

Thanks!

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://iris.org.il
http://jij.org http://mideasttruth.com https://ffii.org http://think-israel.org
"A pint of sweat will save a gallon of blood."          --George S. Patton

A Case for O(locale_encoding)

[<Joe...@t-...>]

Hi,

I wanted to query the list about how to deal with the encodings that libraries uses.

For instance, gnulib/regex uses the following code to store an error message
into a user-supplied buffer:

regerror (int errcode, ..., char *_Restrict_ errbuf, size_t errbuf_size) {
  ... msg = gettext (__re_error_msgid + __re_error_msgid_idx[errcode]);
}
But how was that text encoded into the library? What will gettext select?

Wouldn't clisp need to provide and export a new object named O(locale_encoding)?

Currently, clisp initializes O(misc_encoding) and all the others according to locale
information *only* if no command line overrides that setting,
witness encoding.c around line 2650:
  begin_system_call();
  locale_encoding = locale_charset(); /* depends on environment variables */
  end_system_call();
  pushSTACK(encoding_from_name(locale_encoding,"locale"));
[...]
  O(misc_encoding) =
    (argv_encoding_misc == NULL ? (object)STACK_0
     : encoding_from_name(argv_encoding_misc,"*MISC-ENCODING*"));
Hence, if you use -E, you cannot easily find out what that original locale is that
the libraries use.

It looks like O(locale_encoding) is the one that would best match the above
function regerror(), not the existing O(misc_encoding).

I've not given thought to what can happen when a program changes LANG or LC_xyz after
the start. Depending on the library's code, the initial binding may remain in effect
(assuming they never rebind). Hence there's no provision for changing O(locale_encoding)
(except it needs to be overwritten when starting with an image).

Any comments?
Is moving away from misc_encoding too much?

Would it be wiser to allow clisp's early command line parsing code to set a different
locale, i.e. not(!) obey $LANG&LC_*, before any foreign modules initialize? Then
O(misc_encoding) already fits the bill, except from a missing call to setlocale().

	Jörg

DEF-CALL-OUT now accepts the :GUARD argument

[<Joe...@t-...>]

Hi,

>FFI:def-call-out
>(:guard "defined(__USE_GNU)")

<Bikeshedding on>

IMHO we should have used e.g. c-guard for this and kept :guard reserved for a concept IIRC
sketched years ago: a kind dependable type, useful for variable sized arrays -- something
more on the Lisp level.
Here :guard clearly introduces C level preprocessor #ifdef, so why not use :c-* in the name,
like e.g. c-array-ptr etc.?

Regards,
	Jörg

Re: makemake.in configuration (os--hw) -- what default settings?

[Bruno H. <br...@cl...>]

Jörg wrote:
> OTOH, the default "portability" (which includes -DWIDE internally) costs 33% performance
> against the best (so far) optimized result (as seen by make benchmark), on my 32bit
> machine.

The performance is not essential for the --enable-portability setting. The
bit-rot rate is what matters for this setting. Half a year ago, when I
introduced --enable-portability, it was the *only* working configuration on
about 1/3 of the platforms. Optimizations can have bugs or can make assumptions
that no longer hold 5 years later. This is the issue that --enable-portability
solves.

> >The first thing to check is whether the address range in processes on your Mac is similar
> >enough as the one on Mac OS X/x86_64 with 32-bit ABI, that I have access to. See the
> >unix/PLATFORMS file, section "Address space layout".
> AFAICT, it looked identical

Good, then you can proceed to the next step I detailed.

> Are you aware that adding or not -O is useless, because somewhere down the line,
> -O2 and lots of -W get added anyway, when compiling clisp?

Indeed. Adding -O2 when -O was specified in CFLAGS may be a violation of the
GNU standards, but it is a low-priority one, IMO.

Bruno

Re: makemake.in configuration (os--hw) -- what default settings?

[<Joe...@t-...>]

Hi,

Bruno wrote:

>The result should be better in the sense: "Better a clisp that works and is slow,
>than a clisp that is fully optimized and crashes during the test suite or even
>during the build."
Probably that's a better setting for distributors, who cannot know with what settings
users may reconfigure or recompile their kernels (only few actually do that).

OTOH, the default "portability" (which includes -DWIDE internally) costs 33% performance
against the best (so far) optimized result (as seen by make benchmark), on my 32bit
machine. But it's a bit early for a summary.
Certainly there are means to provide less severe, yet broadly working settings.
Let's find out.

>The first thing to check is whether the address range in processes on your Mac is similar
>enough as the one on Mac OS X/x86_64 with 32-bit ABI, that I have access to. See the
>unix/PLATFORMS file, section "Address space layout".
AFAICT, it looked identical (even though seeing *ADDRESS_RANGE 0x00000000 initially made
me feel  like something not working). Solely the comment need by changed to mention
i686 (no i386 because Apple started their Intel line with Core Solo/Duo machines).

>Then find out which parameters you need in the environment variable MULTIBUILD_32_OPTIONS
>so that $ make -k -f Makefile.devel build-porting32-gcc-portability succeeds.


BTW, in one of your patches, or more generally, in makemake
+build-porting32-...:
+	[...]
+	CFLAGS="-O $(MULTIBUILD_CFLAGS)" \

Are you aware that adding or not -O is useless, because somewhere down the line,
-O2 and lots of -W get added anyway, when compiling clisp?
(Loop at the cbc*.log for the exact gcc command line).

Regards,
	Jörg

Re: makemake.in configuration (os--hw) -- what default settings?

[Bruno H. <br...@cl...>]

Sam wrote:
> Of course, I am not opposing the new approach, but rather explaining the
> rationale for the old one.

The old approach was based on two assumptions:
  * When a user encounters a crash, they will report it.
  * Bruno is always available to investigate a crash.

Both assumptions were wrong.

> With the current approach it is quite possible that no one will ever
> bother to check whether the optimizations actually work on a specific
> platform

Yes, it is our task now to check whether the optimizations work. And the
Makefile.devel multibuild-* targets make this quite comfortable (assuming
you have the patience for waiting for ca. 20 builds to complete).

Bruno

Re: makemake.in configuration (os--hw) -- what default settings?

[Sam S. <sd...@gn...>]

Hi Bruno,

> * Bruno Haible <oe...@py...t> [2017-11-24 20:28:03 +0100]:
>
> Now, by default, all possible NO_* optimizations blockers are enabled, i.e.
> the optimizations are disabled. And we turn them on when we have verified
> they cause no harm. The result should be better in the sense: "Better a
> clisp that works and is slow, than a clisp that is fully optimized and
> crashes during the test suite or even during the build."

This is refreshingly user-centric ;-)

The previous approach was "better" in the sense that the users were
forced to report the failures and thus we got feedback faster.
With the current approach it is quite possible that no one will ever
bother to check whether the optimizations actually work on a specific
platform, so they will remain disabled for no good reason.

Of course, I am not opposing the new approach, but rather explaining the
rationale for the old one.

Incidentally, this reminds me of an old discussion of whether it is
better to be aggressive with free() (risking a segfault -- easier
debugging but less immediate usability) or not (risking a memory leak --
harder to debug but probably more useful to the users).

>> - The most performant settings that happen work on my machine, after
>> running make benchmark in each of the build-gcc-spvw* directories?
>
> Ignore "make benchmark" for the moment. Assume that each optimization
> actually improves speed.

The big question is "by how much".
Clearly, this depends on one's usage pattern, and "make benchmark" is
hardly useful here.

Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://camera.org
http://islamexposedonline.com http://www.dhimmitude.org http://mideasttruth.com
There is an exception to every rule, including this one.

Re: makemake.in configuration (os--hw) -- what default settings?

[Bruno H. <br...@cl...>]

Hi Jörg and all,

> I'm a bit puzzled by what still needs to be done about makemake.in.
> Despite the numerous recent patches and successful compilation attempts
> on lots of systems, that file still contains lots of empty/default
> sections, e.g.
> 
>       # Mac OS X/x86_64 with 32-bit ABI
>       darwin*--i386) # TODO

Glad that you ask. The approach to portability now has changed. Before,
we enabled all optimizations by default, and reacted when people reported
a crash. And we did nothing if we could not reproduce the problem.

Now, by default, all possible NO_* optimizations blockers are enabled, i.e.
the optimizations are disabled. And we turn them on when we have verified
they cause no harm. The result should be better in the sense: "Better a
clisp that works and is slow, than a clisp that is fully optimized and
crashes during the test suite or even during the build."

The stubs here are records of how far we have come, for each platform.

I am currently working on the Linux/*, GNU/kFreeBSD, and Hurd platforms,
therefore work on these would duplicate work of mine. (I am currently
hanging on issues with Linux/sparc64.) But for other operating systems,
you can determine the progress.

The overall process of a port to a new platform is described in
unix/PLATFORMS.

> For instance, I have an 11 year old MacBook Pro. It's i386, 32bit only,
> and limited by apple to Snow Leopard OSX 10.6.x. No x86_64, despite the
> above comment.

I see [1][2]. The first thing to check is whether the address range in
processes on your Mac is similar enough as the one on Mac OS X/x86_64 with
32-bit ABI, that I have access to. See the unix/PLATFORMS file, section
"Address space layout". If it's similar enough, adjust the section for
defined(UNIX_MACOSX) && defined(I80386) (around line 2400 of lispbibl.d),
otherwise create a new one.

Then find out which parameters you need in the environment variable
MULTIBUILD_32_OPTIONS so that
  $ make -k -f Makefile.devel build-porting32-gcc-portability
succeeds.

Then try
  $ make -k -f Makefile.devel multibuild-macosx-i386
and analyze the build failures, so see which are failures that are expected
and unavoidable vs. failures that you can fix (through code changes or
by reducing the gcc optimizations).

Once you have determined this set, remove most optimization blocker flags
in makemake.in and redo
  $ make -k -f Makefile.devel multibuild-macosx-i386
and verify that all builds that succeeded earlier still succeed.

> What should go into the above section?
> - The most performant settings that happen work on my machine, after running
>   make benchmark in each of the build-gcc-spvw* directories?

Ignore "make benchmark" for the moment. Assume that each optimization
actually improves speed.

> - The most conservative settings, because my machine is just one among many darwin*--i386?
>   For instance, it currently contains NO_ADDRESS_SPACE_ASSUMPTIONS

NO_ADDRESS_SPACE_ASSUMPTIONS is certainly an optimization blocker
that you can remove.

> Furthermore, despite dispatching upon the OS and architecture, this section
> typically contains compiler-specific settings, e.g. -DNO_ARI_ASM. Is that
> really the correct place to put these?

NO_ARI_ASM is not compiler-specific, but NO_ASM is. Whether it's the correct
place, can be debated. Patches will be considered.

> That machine reports:
> 
> $ gcc -v
> Using built-in specs.
> Target: i686-apple-darwin10
> Configured with: /var/tmp/gcc/gcc-5666.3~6/src/configure --disable-checking --enable-werror --prefix=/usr --mandir=/usr/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin10 --with-gxx-include-dir=/usr/include/c++/4.2.1 --host=i686-apple-darwin10 --target=i686-apple-darwin10
> Thread model: posix
> gcc version 4.2.1 (Apple Inc. build 5666) (dot 3)
> 
> $ clang -v
> Apple clang version 1.7 (tags/Apple/clang-77) (based on LLVM 2.9svn)
> Target: i386-apple-darwin10
> Thread model: posix

I would ignore clang on this platform. Too old. The current version is 4.0 or so.

Bruno

[1] https://support.apple.com/en-us/HT201948
[2] https://apple.stackexchange.com/questions/12666/how-to-check-whether-my-intel-based-mac-is-32-bit-or-64-bit

Re: parallel stepping in loop, but what about termination tests?

[<Joe...@t-...>]

Pascal,

Thank you for pointing at a missed error in CLISP and another case which will help me
make up my mind about the intermingling of stepping and termination tests in LOOP.

>>(loop for x across "abc" and for z on (list 1) do (princ x) finally (return (cons x z)))
>The first form seems to be a syntax error; other implementations signal it:
>Z is an unknown keyword in FOR or AS clause in LOOP. Current LOOP context:
>  FOR X ACROSS "abc" AND FOR Z ON (LIST 1).
Oops. I'll have to check whether CLtL2 allowed my bogus syntax.
 
>One case where it makes a difference is:
>$ clall -r  '(loop for x across #((1 2 3) (4 5 6) (7 8 9)) and z := nil :then x finally (return (list :x x :z z)))'
>CLISP                           --> (:X (7 8 9) :Z (7 8 9))
>[All other OSS implementations] --> (:X (7 8 9) :Z (4 5 6))

Ugh, room for improvement. I don't like the fact that the final x and z do not stem from
The same iteration, even though ANSI-CL LOOP probably can't forbid CLISP's result.

	Jörg

Re: parallel stepping in loop, but what about termination tests?

[Pascal B. <pj...@in...>]

> On 24 Nov 2017, at 16:09, <Joe...@t-...> <Joe...@t-...> wrote:
> 
> Hi,
> 
> The following two loops produce identical results and side-effects. Surprise?
> 
> (loop for x across "abc" and for z on (list 1) do (princ x) finally (return (cons x z)))
> (loop for x across "abc"     for z on (list 1) do (princ x) finally (return (cons x z)))
> a
> -> (#\b)
> 
> One *might* have expected "AND", implying parallel stepping, to assign either all values,
> or nothing when iteration terminates. However, in the above case, x clearly has the value
> #\b from a second round, but there's only one iteration -- one would think.

The first form seems to be a syntax error; other implementations signal it:

$ clall -r '(loop for x across "abc" and for z on (list 1) do (princ x) finally (return (cons x z)))' '(loop for x across "abc"     for z on (list 1) do (princ x) finally (return (cons x z)))'

Armed Bear Common Lisp         Z is an unknown keyword in FOR or AS clause in LOOP. Current LOOP context: FOR X ACROSS "abc" AND FOR Z ON (LIST 1).
Armed Bear Common Lisp         --> (#\b)
Clozure Common Lisp            Z is an unknown keyword in FOR or AS clause in LOOP. Current LOOP context: FOR X ACROSS "abc" AND FOR Z ON (LIST 1).
Clozure Common Lisp            --> (#\b)
CLISP                          --> (#\b)
CLISP                          --> (#\b)
ECL                            Z is an unknown keyword in FOR or AS clause in LOOP. Current LOOP context: FOR X ACROSS "abc" AND FOR Z ON (LIST 1).
ECL                            --> (#\b)
SBCL                           Z is an unknown keyword in FOR or AS clause in LOOP. current LOOP context: FOR X ACROSS "abc" AND FOR Z ON (LIST 1).
SBCL                           --> (#\b)


$ clall  '(loop for x across "abc" and z on (list 1) do (princ x) finally (return (cons x z)))' '(loop for x across "abc"  as z on (list 1) do (princ x) finally (return (cons x z)))'



Armed Bear Common Lisp:a
--> (#\b)


Armed Bear Common Lisp:a
--> (#\b)


Clozure Common Lisp:a
--> (#\b)


Clozure Common Lisp:a
--> (#\b)


CLISP:a
--> (#\b)


CLISP:a
--> (#\b)


ECL:a
--> (#\b)


ECL:a
--> (#\b)


SBCL:a
--> (#\b)


SBCL:a
--> (#\b)


Here are the 4 occurrences of the keyword AND in the syntax of LOOP:

with-clause::= with var1 [type-spec] [= form1] {and var2 [type-spec] [= form2]}* 
conditional::= {if | when | unless} form selectable-clause {and selectable-clause}*  
               [else selectable-clause {and selectable-clause}*]  
               [end] 
for-as-clause::= {for | as} for-as-subclause {and for-as-subclause}* 


The semantics of AND vs. FOR|AS are specified in 6.1.2.1:

If multiple iteration clauses are used to control iteration, variable initialization and stepping[1] occur sequentially <http://www.lispworks.com/documentation/HyperSpec/Body/26_glo_s.htm#sequentially> by default. The and construct can be used to connect two or more iteration clauses when sequential <http://www.lispworks.com/documentation/HyperSpec/Body/26_glo_s.htm#sequential> binding <http://www.lispworks.com/documentation/HyperSpec/Body/26_glo_b.htm#binding> and stepping[1] are not necessary. The iteration behavior of clauses joined by and is analogous to the behavior of the macro do <http://www.lispworks.com/documentation/HyperSpec/Body/m_do_do.htm#do>with respect to do* <http://www.lispworks.com/documentation/HyperSpec/Body/m_do_do.htm#doST>.


This means that if you don’t have references to the previous iteration variable in the increment of the current clause, AND will make no different with FOR or AS.  The results above are therefore correct.

One case where it makes a difference is:

$ clall -r  '(loop for x across #((1 2 3) (4 5 6) (7 8 9)) and z := nil :then x finally (return (list :x x :z z)))'

Armed Bear Common Lisp         --> (:X (7 8 9) :Z (4 5 6))
Clozure Common Lisp            --> (:X (7 8 9) :Z (4 5 6))
CLISP                          --> (:X (7 8 9) :Z (7 8 9))
ECL                            --> (:X (7 8 9) :Z (4 5 6))
SBCL                           --> (:X (7 8 9) :Z (4 5 6))

$ clall -r  '(loop for x across #((1 2 3) (4 5 6) (7 8 9)) for z := nil :then x finally (return (list :x x :z z)))'

Armed Bear Common Lisp         --> (:X (7 8 9) :Z (7 8 9))
Clozure Common Lisp            --> (:X (7 8 9) :Z (7 8 9))
CLISP                          --> (:X (7 8 9) :Z (7 8 9))
ECL                            --> (:X (7 8 9) :Z (7 8 9))
SBCL                           --> (:X (7 8 9) :Z (7 8 9))


> This contributes to the topic of non-obvious values of variables within FINALLY clauses.
> 
> So the question IMHO is: when, if ever, shall termination tests be combined, such that
> all tests (from FOR clauses) are performed prior to assigning new values?
> 
> Testing varying inputs with clisp's LOOP macro, I've seen tests sometimes combined,
> others not. The picture is not yet clear to me.
> 
> Combined test & stepping might look like this:
> (when (OR (<= (length vector) index) (atom z)) (LOOP-FINISH))
> (PSETQ x (AREF vector index) z (cdr z))
> 
> ... (INCF index)
> 
> [From an implementation POV, combined tests might require more internal variables
> (cf. e.g. internals of FOR HASH), thus be slower.]
> 
> Regards,
> 	Jörg




-- 
__Pascal J. Bourguignon__

makemake.in configuration (os--hw) -- what default settings?

[<Joe...@t-...>]

Hi,

I'm a bit puzzled by what still needs to be done about makemake.in. Despite the numerous recent patches and successful compilation attempts on lots of systems, that file still contains lots of empty/default sections, e.g.

      # Mac OS X/x86_64 with 32-bit ABI
      darwin*--i386) # TODO

For instance, I have an 11 year old MacBook Pro. It's i386, 32bit only, and limited by apple to Snow Leopard OSX 10.6.x. No x86_64, despite the above comment.

What should go into the above section?
- The most performant settings that happen work on my machine, after running
  make benchmark in each of the build-gcc-spvw* directories?
- The most conservative settings, because my machine is just one among many darwin*--i386?
  For instance, it currently contains NO_ADDRESS_SPACE_ASSUMPTIONS
- ??

Furthermore, despite dispatching upon the OS and architecture, this section typically contains compiler-specific settings, e.g. -DNO_ARI_ASM. Is that really the correct place to put these?
For instance, I looked at NO_SP_ASM, and despite being defined, it seems to be ignored when IIRC GNU is defined, e.g. when compiling with gcc.

That machine reports:

$ gcc -v
Using built-in specs.
Target: i686-apple-darwin10
Configured with: /var/tmp/gcc/gcc-5666.3~6/src/configure --disable-checking --enable-werror --prefix=/usr --mandir=/usr/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin10 --with-gxx-include-dir=/usr/include/c++/4.2.1 --host=i686-apple-darwin10 --target=i686-apple-darwin10
Thread model: posix
gcc version 4.2.1 (Apple Inc. build 5666) (dot 3)

$ clang -v
Apple clang version 1.7 (tags/Apple/clang-77) (based on LLVM 2.9svn)
Target: i386-apple-darwin10
Thread model: posix

Regards,
	Jörg Höhle

parallel stepping in loop, but what about termination tests?

[<Joe...@t-...>]

Hi,

The following two loops produce identical results and side-effects. Surprise?

(loop for x across "abc" and for z on (list 1) do (princ x) finally (return (cons x z)))
(loop for x across "abc"     for z on (list 1) do (princ x) finally (return (cons x z)))
a
-> (#\b)

One *might* have expected "AND", implying parallel stepping, to assign either all values,
or nothing when iteration terminates. However, in the above case, x clearly has the value
#\b from a second round, but there's only one iteration -- one would think.

This contributes to the topic of non-obvious values of variables within FINALLY clauses.

So the question IMHO is: when, if ever, shall termination tests be combined, such that
all tests (from FOR clauses) are performed prior to assigning new values?

Testing varying inputs with clisp's LOOP macro, I've seen tests sometimes combined,
others not. The picture is not yet clear to me.

Combined test & stepping might look like this:
(when (OR (<= (length vector) index) (atom z)) (LOOP-FINISH))
(PSETQ x (AREF vector index) z (cdr z))

... (INCF index)

[From an implementation POV, combined tests might require more internal variables
(cf. e.g. internals of FOR HASH), thus be slower.]

Regards,
	Jörg

Re: [clisp:bugs] #375 loop (...) for vars on vars misbehaving

[<Joe...@t-...>]

Sam,

My 2005 query "loop variable value in finally forms?" to comp.lang.lisp produced nothing, alas:
https://groups.google.com/forum/#!msg/comp.lang.lisp/VC4fWvqt3y8/yvLNZF7cIQwJ;context-place=topic/comp.lang.lisp/Z7VtKCZ3xkQ

Generally, I feel Barry Margolin is right in the response to your query:
https://groups.google.com/forum/#!topic/comp.lang.lisp/U5LqiM5nKq8

In my example as well, the value within finally depends on whether the user-visible variable was assigned, then tested, or whether some internal variable was assigned, then tested, inside each iteration, such that the assignment of the user-visible variable is skipped when the test causes the iteration to terminate.


One could make the case that values are not available within the epilogue(!), as follows:
6.1.2.2 http://clhs.lisp.se/Body/06_abb.htm
"When a loop form is executed, the local variables are bound and are initialized to some value.
These local variables exist until loop iteration terminates, at which point they cease to exist."

6.1.1.4 http://clhs.lisp.se/Body/06_aad.htm
"Loop epilogue
    The loop epilogue contains forms that are executed after iteration terminates, such as finally clauses [...]"

Unambiguously, the local variables do not exist anymore within the epilogue!
Of course, that renders many examples of COLLECT INTO useless.

One might argue what "local variables" are, however
6.1.2.2
"The with construct initializes variables that are local to a loop."
Clearly, WITH variables are unavailable within FINALLY. Ouch. Fun.


About clisp, we shall ask ourselves whether we want clisp's loop to assign the user-visible variables prior to the test, and document when that's applicable:
+ FOR x ON ; x a symbol, not nil, no destructuring
+ FOR-AS arithmetic (as I argued elsewhere)

- FOR ... IN list ; might retain the last iteration's value, if any
- FOR ... ACROSS, likewise
- FOR ... HASH, likewise
- FOR s PACKAGE ; -- assign s to NIL at end, saving the internal "SYMBOL-" variable?

o FOR-AS = THEN ; -- "does not provide any termination test."


That's not all there's to document. The next is whether to document whether termination tests are combined or not, esp. in FOR AND FOR. That also influences the values available in FINALLY.
See another mail.

Regards,
	Jörg

Re: 1 extra bytecode instruction per variable per loop

[<Joe...@t-...>]

Hi,

Sam Steingold wrote:
>The fix for https://sourceforge.net/p/clisp/bugs/375/
>(https://sourceforge.net/p/clisp/clisp/ci/52d209b7d29ddf1125d2d89f4185c20c2f09da12/)
>introduces an extra binding per init expression.
>This results in an extra bytecode instruction in the preamble, e.g., for
>(loop repeat 100 for a on b for x in y collect (list a x)):

Next to a cost in bytecode, there's also a memory footprint. Given
that clisp's compiler doesn't perform liveness analysis,
  (LET ((s b)) (LOOP/PROG/TAGBODY #))
means that the binding for s will remain on the stack (where the value
of s is held) for the whole duration of the loop. CLISP doesn't
notice that s is only used once, in the loop initialization code and
never afterwards.

This reminds me of the discussion of GC leaks in functional programming.

Hence it would be best if
  (loop repeat 100 for a on b for x in y collect (list a x))
could expand to
  (let* ((a b) (list-x y)(x #)) (prog ...))
without much intermediates.

Regards,
	Jörg

1 extra bytecode instruction per variable per loop

[Sam S. <sd...@gn...>]

Hi Bruno,

The fix for https://sourceforge.net/p/clisp/bugs/375/
(https://sourceforge.net/p/clisp/clisp/ci/52d209b7d29ddf1125d2d89f4185c20c2f09da12/)
introduces an extra binding per init expression.
This results in an extra bytecode instruction in the preamble, e.g.,
for (loop repeat 100 for a on b for x in y collect (list a x)):

--8<---------------cut here---------------start------------->8---
Disassembly of function :LAMBDA
(CONST 0) = 100
(CONST 1) = B
(CONST 2) = Y
0 required arguments
0 optional arguments
No rest parameter
No keyword parameters
reads special variables: Y B
27 byte-code instructions:
0     (CONST&PUSH 0)                      ; 100
1     (PUSH-NIL 4)
3     (GETVALUE 1)                        ; B
5     (STORE 3)
6     (GETVALUE 2)                        ; Y
8     (STORE 2)
9     (JMP L26)
11    L11
--8<---------------cut here---------------end--------------->8---

becomes

--8<---------------cut here---------------start------------->8---
Disassembly of function :LAMBDA
(CONST 0) = B
(CONST 1) = Y
(CONST 2) = 100
0 required arguments
0 optional arguments
No rest parameter
No keyword parameters
reads special variables: Y B
29 byte-code instructions:
0     (GETVALUE&PUSH 0)                   ; B
2     (GETVALUE&PUSH 1)                   ; Y
4     (CONST&PUSH 2)                      ; 100
5     (PUSH-NIL 4)
7     (LOAD 6)
8     (STORE 3)
9     (LOAD 5)
10    (STORE 2)
11    (JMP L28)
13    L13
--8<---------------cut here---------------end--------------->8---

The cost is probably trivial, but I wonder if there is some simple
compilation optimization missing here...

Thanks

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://www.memritv.org
http://iris.org.il http://americancensorship.org https://jihadwatch.org
"Sustainability" without "population growth control" is hypocrisy.

Re: NetBSD/sparc64 port

[<Joe...@t-...>]

Hi,

>It means that if you have a SPARC64 hardware (64-bit, i.e. UltraSPARC or newer, i.e.
>for a Sun4: Sun-4u or newer) and you can install NetBSD on it, you can insert
>the appropriate figures here.
We only have some Sun sparcstation 10 and one Ultra 1 here.
Never tried a foreign OS (they are not mine).

Regards,
	Jörg

Re: NetBSD/sparc64 port

[Bruno H. <br...@cl...>]

Hi Jörg,

> +          #if defined(UNIX_NETBSD) && defined(SPARC64) /* NetBSD/sparc64 */
> +            #define TYPECODES_WITH_MALLOC_WORKS ?
> +          #endif
> 
> What's this '?'

It means that if you have a SPARC64 hardware (64-bit, i.e. UltraSPARC or newer,
i.e. for a Sun4: Sun-4u or newer) and you can install NetBSD on it, you can
insert the appropriate figures here.

NetBSD/sparc64 under QEMU is not stable enough, and I understand that it will
not change any time soon. [1]

Bruno

[1] http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=52657

Re: [clisp:clisp] 20 new commits to clisp

[<Joe...@t-...>]

Hi,

In commit: Find good parameters for using TYPECODES model, for each platform.

+          #if defined(UNIX_NETBSD) && defined(SPARC64) /* NetBSD/sparc64 */
+            #define TYPECODES_WITH_TRIVIALMAP_WORKS ?
+          #endif

+          #if defined(UNIX_NETBSD) && defined(SPARC64) /* NetBSD/sparc64 */
+            #define TYPECODES_WITH_MALLOC_WORKS ?
+          #endif

What's this '?'

Likewise in: Record whether GENERIC64C_HEAPCODES works.

+    #if defined(UNIX_NETBSD) && defined(SPARC64) /* NetBSD/sparc64 */
+      #define GENERIC64C_HEAPCODES_WORKS ?
+    #endif

	Jörg

no more need for --enable-portability

[Bruno H. <br...@cl...>]

Hi all,

When building clisp, you don't need to pass the option --enable-portability
any more.

I introduced this option before the 2.49.50 prerelease, in order to avoid risky
options. The notion which options are risky is now defined in makemake.in and
(to a small extent) lispbibl.d, and the risky options are no longer enabled by
default.

The only use of --enable-portability is now if you are on a platform where
some optimization options have been found to be not risky and you want to
avoid them _nevertheless_.

Bruno

Re: nightly build error

[Bruno H. <br...@cl...>]

Hi Don,

> after:
>  pulling from http://hg.code.sf.net/p/clisp/clisp
>  searching for changes
>  adding changesets
>  adding manifests
>  adding file changes
>  added 4 changesets with 9 changes to 4 files
> 
> make ends like this:
>  make[1]: Leaving directory '/home/don/hg/clisp/build-dir/po'
>  rm -rf data
>  mkdir data
>  cd data && ln -s ../../utils/unicode/UnicodeDataFull.txt .
>  cd data && ln -s ../../doc/Symbol-Table.text .
>  gcc -m64 -g -O2 -no-integrated-cpp -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral -Wno-shift-negative-value -fwrapv -fno-strict-aliasing -ggdb -O0 -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE -DDYNAMIC_FFI -DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o i18n.o foreign.o unixaux.o built.o modules.o libgnu.a  -lreadline -lncurses -ldl /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a  -lsigsegv  -o lisp.run
>  ./lisp.run -marc > marc.out
>  ./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m 2MW -lp ../src/ -x '(and (load "../src/init.lisp") (sys::%saveinitmem) (ext::exit)) (ext::exit t)'
>  STACK size: 262078 [0x300001ffe00 0x30000000010]
> ...
>  make: *** [interpreted.mem] Segmentation fault (core dumped)

Thanks for the report! It was a major blunder of mine. Fix pushed.

And thanks for the rapid feedback. Part of the tests that I've been running
since yesterday have been pointless, due to this bug; therefore I am very
grateful that your rapid report saved me a number of other useless tests.

Bruno

nightly build error

[<don...@is...>]

after:
 pulling from http://hg.code.sf.net/p/clisp/clisp
 searching for changes
 adding changesets
 adding manifests
 adding file changes
 added 4 changesets with 9 changes to 4 files

make ends like this:
 make[1]: Leaving directory '/home/don/hg/clisp/build-dir/po'
 rm -rf data
 mkdir data
 cd data && ln -s ../../utils/unicode/UnicodeDataFull.txt .
 cd data && ln -s ../../doc/Symbol-Table.text .
 gcc -m64 -g -O2 -no-integrated-cpp -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral -Wno-shift-negative-value -fwrapv -fno-strict-aliasing -ggdb -O0 -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE -DDYNAMIC_FFI -DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o i18n.o foreign.o unixaux.o built.o modules.o libgnu.a  -lreadline -lncurses -ldl /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a  -lsigsegv  -o lisp.run
 ./lisp.run -marc > marc.out
 ./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m 2MW -lp ../src/ -x '(and (load "../src/init.lisp") (sys::%saveinitmem) (ext::exit)) (ext::exit t)'
 STACK size: 262078 [0x300001ffe00 0x30000000010]
 READ-FROM-STRING

Re: cannot enable executable stack as shared object requires

[Bruno H. <br...@cl...>]

Sam Steingold wrote on 2016-11-16:
> Any chance you know what this means?
> http://stackoverflow.com/q/40646703/850781

Took me a lot of time to understand it, but now it's fixed (in libffcall-2.0).

Thanks Sam for the note; I've answered it on stackoverflow.

Bruno

Re: workarounds around GCC miscompilation

[Bruno H. <br...@cl...>]

Hi Jörg,

> Just names. Perhaps "HACK:" as a prefix in the one-liner log message would be
> enough to raise attention.

Whereas I thought about mentioning these things in the advice for porters file,
namely unix/PLATFORMS...

> Likewise, I felt uncomfortable about this log message:
>  < Make intparam.c compile with g++ >= 6.
> -> Make intparam.c compile with g++ >= 6. The "auto" specifier changed with C++11.
> Intparam and g++ 6 are symptoms -- or noise. "auto" is the root cause.

Yes, "auto" is the root cause. But root cause analysis and other technical
explanations belong in comments. A log message should indicate what has changed
(from the point of view of the user). In this case: intparam.c did not compile
and now it compiles again.

> in C++14 because of the redundancy: either you write e.g. int i =..., or you
> let the compiler derive the type, and write auto i = ...;

Thanks for explaining. They have reused a keyword to mean a completely different
thing. As usual in C++... Whereas in C# and Java it's called 'var'.

Bruno

Re: workarounds around GCC miscompilation

[Sam S. <sd...@gn...>]

Hi Jörg,

> *  <Wbret-Plevy.Ubruyr@g-flfgrzf.pbz> [2017-10-25 14:22:17 +0000]:
>
> Likewise, I felt uncomfortable about this log message:
>  < Make intparam.c compile with g++ >= 6.
> -> Make intparam.c compile with g++ >= 6. The "auto" specifier changed with C++11.
> Intparam and g++ 6 are symptoms -- or noise. "auto" is the root cause.

Yes, I requested on multiple occasions that Bruno includes the full ChangeLog
entry into the Mercurial commit message, like they do in, e.g., Emacs.

Thanks for your support.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://www.memritv.org
http://no2bds.org http://islamexposedonline.com http://thereligionofpeace.com
Lisp is a way of life.  C is a way of death.