clisp-devel — closed list for developers and testers

Re: Problem building on LinuxMint 19

[Blake M. <bl...@mc...>]

That did it.  Thanks!

On Sun, Nov 25, 2018 at 4:49 AM Bruno Haible <br...@cl...> wrote:

> Blake McBride wrote:
> > $ sh -vx configure --with-threads=POSIX_THREADS with-gcc
> > ...
> > ;; Loading file ../src/defmacro.lisp ...
> > *** - handle_fault error2 ! address = 0x8 not in
> > [0x1000000c0000,0x1000000c0000) !
> > SIGSEGV cannot be cured. Fault address = 0x8.
>
> Yes, we know that --with-threads is currently not stable.
> './configure --help' marks it as "highly experimental".
>
> Bruno
>
>

Re: Problem building on LinuxMint 19

[Bruno H. <br...@cl...>]

Blake McBride wrote:
> $ sh -vx configure --with-threads=POSIX_THREADS with-gcc
> ...
> ;; Loading file ../src/defmacro.lisp ...
> *** - handle_fault error2 ! address = 0x8 not in
> [0x1000000c0000,0x1000000c0000) !
> SIGSEGV cannot be cured. Fault address = 0x8.

Yes, we know that --with-threads is currently not stable.
'./configure --help' marks it as "highly experimental".

Bruno

Problem building on LinuxMint 19

[Blake M. <bl...@mc...>]

Greetings,

I am trying to build the latest GIT clone of CLISP from
https://gitlab.com/gnu-clisp/clisp.git on a 64 bit LinuxMint 19 box.  What
I am getting on a clean checkout is:

$ sh -vx configure --with-threads=POSIX_THREADS with-gcc
[...]
Configure findings:
  FFI:        ${cl_cv_have_ffcall} (user requested: ${do_ffi})
  readline:   ${ac_cv_have_readline} (user requested: ${ac_cv_use_readline})
  libsigsegv: ${gl_cv_lib_sigsegv}
EOF
+ cat
Configure findings:
  FFI:        yes (user requested: default)
  readline:   yes (user requested: default)
  libsigsegv: yes

# The user can set MAKE. In particular, on IRIX, VPATH builds require GNU
make.
if test -z "${MAKE}"; then
  MAKE=make
fi
+ test -z
+ MAKE=make

# are any FFI modules requested?
ffi_modules=''
+ ffi_modules=
for module in ${all_modules}; do
  mdir=${ABS_SRCDIR}/modules/${module}
  test -d ${mdir} || fail "$0: ${mdir} is not an existing directory"
  if test -r ${mdir}/configure.in; then
    grep 'CL_MODULE_COMMON_CHECKS.*ffi' ${mdir}/configure.in > /dev/null
2>&1 && \
      ffi_modules=${ffi_modules}" ${module}"
  elif grep '.*\.c.*\.fas.*:\|.*\.fas.*\.c.*:' ${mdir}/Makefile > /dev/null
2>&1 ; then
    ffi_modules=${ffi_modules}" ${module}"
  elif grep ':use.*"FFI"' ${mdir}/*.lisp > /dev/null 2>&1; then
    ffi_modules=${ffi_modules}" ${module}"
  fi
done
if test -n "${ffi_modules}"; then
  test ${do_ffi} = no && \
    fail "$0: --without-ffcall is incompatible with requested
module(s):${ffi_modules}"
  test "${cl_cv_have_ffcall}" = yes || \
    fail "$0: modules${ffi_modules} require FFI"
fi
+ test -n
if [ ${do_ffi} != no -a "${cl_cv_have_ffcall}" = yes ]; then
  makemake_args="--with-dynamic-ffi ${makemake_args}"
fi
+ [ default != no -a yes = yes ]
+ makemake_args=--with-dynamic-ffi  --with-threads=POSIX_THREADS

if [ "${gl_cv_lib_sigsegv}" != "yes" ]; then
  if [ "${ignore_absence_of_libsigsegv}" = "yes" ]; then
    echo "As you requested, we will proceed without libsigsegv..."
  else
    if [ "$ac_cv_build" = "$ac_cv_host" ]; then host_arg="";
    else host_arg=" --host=$ac_cv_host --build=$ac_cv_build";
    fi
    cat <<EOF 1>&2
$0: libsigsegv was not detected, thus some features, such as
  generational garbage collection and
  stack overflow detection in interpreted Lisp code
cannot be provided.
Please install libsigsegv like this:
EOF
    if [ "${CC+set}" = "set" ]; then
      echo "  CC='$CC'; export CC" 1>&2
    fi
    libsigsegv_targz=`echo "$libsigsegv_url" | sed -e 's|^.*/||'`
    libsigsegv_dirname=`echo "$libsigsegv_targz" | sed -e 's|\.tar\.gz$||'`
    cat <<EOF 1>&2
  mkdir prerequisites; cd prerequisites; prefix=\`pwd\`/${ac_cv_host}
  wget ${libsigsegv_url}
  tar xfz ${libsigsegv_targz}
  cd ${libsigsegv_dirname}
  ./configure${host_arg} --prefix=\${prefix} && ${MAKE} && ${MAKE} check &&
${MAKE} install
  cd ../..
  rm -f ${DIRNAME}/config.cache
  ./configure --with-libsigsegv-prefix=\${prefix} $*
If you insist on building without libsigsegv, please pass
  --ignore-absence-of-libsigsegv
to this script:
  ./configure --ignore-absence-of-libsigsegv $*
If you have installed libsigsegv, but clisp does not detect it,
you might have installed it incorrectly, see section 2 in in unix/INSTALL.
EOF
    exit 1;
  fi
fi
+ [ yes != yes ]

# CLISP needs a lot of stack space for bootstrapping,
# and insufficient stack space manifests itself via arbitrary GC errors.
# It was believed that 8192 is enough until power5 came along:
# https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=166347
# But this affects only the powerpc CPU, because it has lots of reserved
space
# in its stack frame layout (see section 2.2.2 of
# "Power Architecture 64-Bit ELF V2 ABI Specification"
# https://members.openpowerfoundation.org/document/dl/576 ).
case ${ac_cv_host} in
  *-*-rs6000 | *-*-rs6000-* | *-*-powerpc* | *-*-ppc*)
    STACK_LIMIT=16384 ;;
  *)
    STACK_LIMIT=8192 ;;
esac
+ STACK_LIMIT=8192
stacksizelimit=`ulimit -s 2>/dev/null || :` # cygwin /bin/sh ulimit is
broken
+ ulimit -s
+ stacksizelimit=8192
# need 3 separate test calls because of "integer expression expected" errors
# when $stacksizelimit is "" or "unlimited" (no short-circuiting!)
set +e;
+ set +e
test -z "$stacksizelimit" || { test "$stacksizelimit" != unlimited && test
"$stacksizelimit" -lt ${STACK_LIMIT}; }
+ test -z 8192
+ test 8192 != unlimited
+ test 8192 -lt 8192
STACK_TOO_SMALL=$?      # 0=true => need to reset; 1=false => big enough
+ STACK_TOO_SMALL=1
set -e
+ set -e

cd "$ABS_DIRNAME"
+ cd /home/blake/Backups/clisp.git/with-gcc
echo "./makemake $makemake_args > Makefile"
+ echo ./makemake --with-dynamic-ffi  --with-threads=POSIX_THREADS >
Makefile
./makemake --with-dynamic-ffi  --with-threads=POSIX_THREADS > Makefile
./makemake $makemake_args > Makefile
+ ./makemake --with-dynamic-ffi --with-threads=POSIX_THREADS
${MAKE} config.lisp
+ make config.lisp
cp -p ../src/cfgunix.lisp config.lisp
chmod +w config.lisp
echo '(setq *clhs-root-default* "
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/")' >>
config.lisp

if test -z "$do_cbc"; then
  echo
  echo "To continue building CLISP, the following commands are recommended"
  echo "  (cf. unix/INSTALL step 4 ff):"
  if test "$DIRNAME" != "."; then
    echo "    cd $DIRNAME"
  fi
  echo "    ${EDITOR:-vi} config.lisp"
  if [ "${STACK_TOO_SMALL}" = 0 ]; then
    cat <<EOF
# The default stack size on your platform is insufficient
# and must be increased to at least ${STACK_LIMIT}.  You must do either
# 'ulimit -s ${STACK_LIMIT}' (for Bourne shell derivatives, e.g., bash and
zsh)
# or 'limit stacksize ${STACK_LIMIT}' (for C shell derivarives, e.g., tcsh)
EOF
  fi
  cat <<EOF
    ${MAKE}
# CLISP self-test:
    ${MAKE} check
# Test non-portable features and modules
    ${MAKE} extracheck mod-check
EOF
else
  if [ -n "${edit_config}" ]; then
    ${MAKE} config.lisp
    ${EDITOR:-vi} config.lisp
  fi
  if [ "${STACK_TOO_SMALL}" = 0 ]; then
    set +e; ulimit -s ${STACK_LIMIT} 2>/dev/null; set -e;
  fi
  run 2 ${MAKE}
  run 3 ${MAKE} check
  if test "$do_cbc" = 2; then
    run 4 ${MAKE} extracheck
    run 5 ${MAKE} base-mod-check
    # Disabled because the rawsock tests hang on Solaris.
    # run 6 ${MAKE} full-mod-check
  fi
  # report results:
  for log in 1 2 3 4 5 # 6
  do
    stepresult=`sed -n -e '$p' cbcstep${log}.log`
    echo "Step ${log} ${stepresult}"
  done
  # Fail if one of the commands "make" or "make check" failed.
  # "make extracheck" and "make mod-check" are allowed to fail.
  sed -n -e '$p' cbcstep2.log | grep ' SUCCEEDED$' >/dev/null || exit 1
  sed -n -e '$p' cbcstep3.log | grep ' SUCCEEDED$' >/dev/null || exit 1
  if test "$do_cbc" != 2; then
    cat <<EOF
# Now you can test non-portable features and modules:
    ${MAKE} extracheck mod-check
EOF
  fi
fi
+ test -z
+ echo

+ echo To continue building CLISP, the following commands are recommended
To continue building CLISP, the following commands are recommended
+ echo   (cf. unix/INSTALL step 4 ff):
  (cf. unix/INSTALL step 4 ff):
+ test with-gcc != .
+ echo     cd with-gcc
    cd with-gcc
+ echo     vi config.lisp
    vi config.lisp
+ [ 1 = 0 ]
+ cat
    make
# CLISP self-test:
    make check
# Test non-portable features and modules
    make extracheck mod-check
$ cd with-gcc
$ make
[...]
mkdir data
cd data && ln -s ../../utils/unicode/UnicodeDataFull.txt .
cd data && ln -s ../../doc/Symbol-Table.text .
gcc -g -O2 -no-integrated-cpp -W -Wswitch -Wcomment -Wpointer-arith
-Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare
-Wno-format-nonliteral -Wno-shift-negative-value -O -fwrapv -pthread
-fno-strict-aliasing -DNO_ASM -DENABLE_UNICODE -DMULTITHREAD
-DPOSIX_THREADS -DDYNAMIC_FFI -DDYNAMIC_MODULES  -Wl,--export-dynamic
spvw.o spvwtabf.o spvwtabs.o spvwtabo.o eval.o control.o encoding.o
pathname.o stream.o socket.o io.o funarg.o array.o hashtabl.o list.o
package.o record.o weak.o sequence.o charstrg.o debug.o error.o misc.o
time.o predtype.o symbol.o lisparit.o i18n.o foreign.o unixaux.o zthread.o
built.o modules.o libgnu.a  -lreadline -ltermcap -ldl -lffcall  -lsigsegv
 -o lisp.run
./lisp.run -marc > marc.out
./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m 2MW
-lp ../src/ -x '(and (load "../src/init.lisp") (sys::%saveinitmem)
(ext::exit)) (ext::exit t)'
  i i i i i i i       ooooo    o        ooooooo   ooooo   ooooo
  I I I I I I I      8     8   8           8     8     o  8    8
  I  \ `+' /  I      8         8           8     8        8    8
   \  `-+-'  /       8         8           8      ooooo   8oooo
    `-__|__-'        8         8           8           8  8
        |            8     o   8           8     o     8  8
  ------+------       ooooo    8oooooo  ooo8ooo   ooooo   8

Welcome to GNU CLISP 2.49.93+ (2018-02-18) <http://clisp.org/>

Copyright (c) Bruno Haible, Michael Stoll 1992-1993
Copyright (c) Bruno Haible, Marcus Daniels 1994-1997
Copyright (c) Bruno Haible, Pierpaolo Bernardi, Sam Steingold 1998
Copyright (c) Bruno Haible, Sam Steingold 1999-2000
Copyright (c) Sam Steingold, Bruno Haible 2001-2018

Type :h and hit Enter for context help.

;; Loading file ../src/defseq.lisp ...
;; Loaded file ../src/defseq.lisp
;; Loading file ../src/backquote.lisp ...
;; Loaded file ../src/backquote.lisp
;; Loading file ../src/defmacro.lisp ...
*** - handle_fault error2 ! address = 0x8 not in
[0x1000000c0000,0x1000000c0000) !
SIGSEGV cannot be cured. Fault address = 0x8.
GC count: 2
Space collected by GC: 324488
Run time: 0 83247
Real time: 0 834816
GC time: 0 2327
Permanently allocated: 192224 bytes.
Currently in use: 724080 bytes.
Free space: 0 bytes.
Makefile:1472: recipe for target 'interpreted.mem' failed
make: *** [interpreted.mem] Segmentation fault (core dumped)
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31583
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31583
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


Thanks!

Blake McBride

Re: with-open-file in clisp vs sbcl

[Sam S. <sd...@gn...>]

On Sun, Nov 11, 2018 at 6:50 PM Bruno Haible <br...@cl...> wrote:
> Sam asked:
> > I discovered
> > that `with-open-file' is implemented differently in CLISP and SBCL:
> >
> > (LET ((A (OPEN "f")))
> >   (DECLARE (SYSTEM::READ-ONLY A))
> >   (UNWIND-PROTECT (MULTIPLE-VALUE-PROG1 (PROGN (PRINT 1 A)) (WHEN A (CLOSE A)))
> >     (WHEN A (CLOSE A :ABORT T))))
> >
> > vs
> >
> > (LET ((A (OPEN "f")) (#:G368 T))
> >   (UNWIND-PROTECT (MULTIPLE-VALUE-PROG1 (PROGN (PRINT 1 A)) (SETQ #:G368 NIL))
> >     (WHEN A (CLOSE A :ABORT #:G368))))
> >
> > The second version produces a marginally smaller bytecode (probably
> > unimportant) and seems slightly more aesthetic (calls CLOSE only once).
> >
> > Why did you choose the first version?
>
> I did so for reliability. When the disk gets full, this produces an error
> condition. Most often, the error condition will occur during the body that
> produces output. But recall that a file-stream is most often buffered, and
> that means that some output is still waiting in the buffer until the final
> (CLOSE A) call; if the disk full error only occurs in this last file system
> access,
>   * the clisp variant will execute the protection clause (CLOSE A :ABORT T)
>     and thus do as specified in ANSI CL [1]: "the file system is left, so far
>     as possible, as if the file had never been opened."
>   * the sbcl variant will not execute (CLOSE A :ABORT T) and thus leave a
>     file on the (full) disk that is truncated.
>
> WITH-OPEN-FILE was designed to avoid leaving truncated files on disk, and
> an implementation should better achieve this goal in 100%, not 99%, of the
> cases.
> [1] http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/mac_with-open-file.html

Ah, now I remember asking this question 10 or 20 years ago ;-)
However, looking at stream.d:low_close_handle, it does not look like
your impeccable theoretical justification has a practical
underpinning.
IOW, if ENOSP is raised in (CLOSE A), the subsequent (CLOSE A :ABORT
T) does not seem to be removing the file.
(note that, in fact, removing the file is not necessarily TRT if the
file existed before OPEN or if FINISH-OUTPUT has been called).
If your point it that the "recovery" should be done at the OS level,
then it does not look like any existing OS distinguishes between ABORT
and non-ABORT close(), so the issue is actually moot.

Another weirdness is that `with-open-file` is defined separately
instead of via `with-open-stream`...

-- 
Sam Steingold <http://sds.podval.org> <http://www.childpsy.net>
<http://steingoldpsychology.com>

Re: with-open-file in clisp vs sbcl

[Bruno H. <br...@cl...>]

Sam asked:

> I discovered
> that `with-open-file' is implemented differently in CLISP and SBCL:
> 
> (LET ((A (OPEN "f")))
>   (DECLARE (SYSTEM::READ-ONLY A))
>   (UNWIND-PROTECT (MULTIPLE-VALUE-PROG1 (PROGN (PRINT 1 A)) (WHEN A (CLOSE A)))
>     (WHEN A (CLOSE A :ABORT T))))
> 
> vs
> 
> (LET ((A (OPEN "f")) (#:G368 T))
>   (UNWIND-PROTECT (MULTIPLE-VALUE-PROG1 (PROGN (PRINT 1 A)) (SETQ #:G368 NIL))
>     (WHEN A (CLOSE A :ABORT #:G368))))
> 
> The second version produces a marginally smaller bytecode (probably
> unimportant) and seems slightly more aesthetic (calls CLOSE only once).
> 
> Why did you choose the first version?

I did so for reliability. When the disk gets full, this produces an error
condition. Most often, the error condition will occur during the body that
produces output. But recall that a file-stream is most often buffered, and
that means that some output is still waiting in the buffer until the final
(CLOSE A) call; if the disk full error only occurs in this last file system
access,
  * the clisp variant will execute the protection clause (CLOSE A :ABORT T)
    and thus do as specified in ANSI CL [1]: "the file system is left, so far
    as possible, as if the file had never been opened."
  * the sbcl variant will not execute (CLOSE A :ABORT T) and thus leave a
    file on the (full) disk that is truncated.

WITH-OPEN-FILE was designed to avoid leaving truncated files on disk, and
an implementation should better achieve this goal in 100%, not 99%, of the
cases.

Bruno

[1] http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/mac_with-open-file.html

Re: git update schedule

[<don...@is...>]

I was looking in the old hg subdirectory.
It turns out that parts of my nightly build script were
also doing that.  Now corrected.

My previously reported MT crash recipe still crashes 
using current sources.  

Re: git update schedule

[Bruno H. <br...@cl...>]

Hi Don,

> and the output ended with this:
> 
>  Updating 90b3631..228b90a
>  Fast-forward
>   src/ChangeLog         | 11 +++++++++++
>   src/NEWS              |  1 +
>   src/aclocal.m4        | 12 ++++++++----
>   src/configure         | 10 +++++++---
>   src/m4/filecharset.m4 | 12 ++++++++----
>   src/pathname.d        | 33 ++++++++++++++++++++-------------
>   6 files changed, 55 insertions(+), 24 deletions(-)

These are changes that I pushed two days ago.

> However, Changelog starts out like this:
>  2018-04-08  Bruno Haible  <br...@cl...>  
> 
> Or am I using the wrong repository? (how can I tell?)

Maybe you're on the wrong branch? Try 'git status'
and 'git branch'. Also, review your .git/config file.

Or maybe your text editor did not realize that the src/ChangeLog
file has changed?

Bruno

Re: git update schedule

[Karsten P. <Kar...@gm...>]

On 24.09.18 15:06, Don Cohen wrote:

> 
> Are changes from 6 months ago just now being distributed?
> Or am I using the wrong repository? (how can I tell?)
The following seems correct


git remote -v
origin	https://gitlab.com/gnu-clisp/clisp.git (fetch)
origin	https://gitlab.com/gnu-clisp/clisp.git (push)
git log
commit 228b90a52bc8d23328e289ff4d5742a90a7a750d (HEAD -> master, 
origin/master, origin/HEAD)
Author: Bruno Haible <br...@cl...>
Date:   Sat Sep 22 18:54:39 2018 +0200

     Fix problem with non-ASCII characters in file names on Windows, part 2.

commit 35fa6b73567a1eead27c528d2af3a95a763321c1
Author: Bruno Haible <br...@cl...>
Date:   Sat Sep 22 17:02:41 2018 +0200

     Fix problem with non-ASCII characters in file names on Windows.
.....
git status
On branch master
Your branch is up to date with 'origin/master'.

git update schedule

[<don...@is...>]

My nightly build of clisp did this:
 git pull
at the following time:
 Sun Sep 23 03:17:02 PDT 2018

and the output ended with this:

 Updating 90b3631..228b90a
 Fast-forward
  src/ChangeLog         | 11 +++++++++++
  src/NEWS              |  1 +
  src/aclocal.m4        | 12 ++++++++----
  src/configure         | 10 +++++++---
  src/m4/filecharset.m4 | 12 ++++++++----
  src/pathname.d        | 33 ++++++++++++++++++++-------------
  6 files changed, 55 insertions(+), 24 deletions(-)

However, Changelog starts out like this:
 2018-04-08  Bruno Haible  <br...@cl...>  

Are changes from 6 months ago just now being distributed?
Or am I using the wrong repository? (how can I tell?)

reproducible MT segfault

[<don...@is...>]

I've lucked into an easily reproducible example!

This is in MT clisp newly built from source in fedora 24.
More details are available if they'll help.
I look forward to hearing about the diagnosis and cure 
from the MT experts.

in shell:
  ./lisp.run -M ./lispinit.mem
  (load ".../plain-dbg-server.lisp")

(now it's waiting at command line with prompt [2]>)

Here's the source for plain-dbg-server:
====
#-(and clisp MT)(error "this file needs MT") 
 
(setf *debug-server-port* 8225) 
 
(push :usemp *features*) 
 
(defun serve-one-debugger(socket) 
  (let ((tlist (loop for x in (mt:list-threads) with i = 0 
                 when (mt:thread-active-p x) collect (cons (incf i) x))) 
        ans) 
    (print tlist socket) 
    (format socket 
            "~&enter the number of a thread to interrupt/debug: ") 
    (setf ans (or (cdr (assoc (read socket) tlist)) 
                  ;; try to put in package ap5? 
                  (mt:current-thread))) 
    (mt:thread-interrupt 
     ans 
     :function 
     (lambda nil 
       (let ((*standard-input* socket) 
             (*standard-output* socket) 
             (*debug-io* socket) 
             (*error-output* socket) 
             (*trace-output* socket) 
             (*query-io* socket)) 
         (unwind-protect 
             (break "debug") ;; close socket when finished with break 
           (close socket))))))) 
(defun show-ut (&optional (ut (get-universal-time)))  
  (multiple-value-bind  
   (s m h d mo y) (decode-universal-time ut)  
   (format nil "~d-~d-~d ~2,'0d:~2,'0d:~2,'0d" y mo d h m s))) 
(defun new-debugger-name() 
  (format nil "debugger-~a" (show-ut))) 
(defun debug-server() 
  (let ((server (socket:socket-server *debug-server-port* 
                                      :interface "localhost"))) 
       (unwind-protect 
           (loop 
            (let ((socket (socket:socket-accept server :buffered nil))) 
              (mt:make-thread #'(lambda ()(serve-one-debugger socket)) 
                              :name (new-debugger-name)))) 
         (socket:socket-server-close server)))) 
 
(mt:make-thread #'debug-server :name "debug-server") 
====

Now in another shell:

 telnet localhost 8225 
 Trying ::1... 
 telnet: connect to address ::1: Connection refused 
 Trying 127.0.0.1... 
 Connected to localhost. 
 Escape character is '^]'. 
  
 ((1 . #<THREAD "debugger-2018-9-20 14:34:32">) (2 . #<THREAD "debug-server">) 
  (3 . #<THREAD "main thread">))  
 enter the number of a thread to interrupt/debug: 1 
 1 


[the date naming the first thread will be different, but type 1]

  
 ** - Continuable Error 
 debug 
 If you continue (by typing 'continue'): Return from BREAK loop 
 The following restarts are also available: 
 ABORT          :R1      ABORT 
 Break 1 [1]>  

Now from the break I do this defun and it segfaults.
I know this code is gibberish.  I was trying to remove junk that
didn't affect whether it crashes, and might have gone further on
that path, but was eager to send this when I discovered that all 
I needed to cause the crash was the debug server.

(defun andgenerator (vars wff gen) 
  (and 
    gen 
    `( 
      (initialstate 
 ,(unless *in-relationsize* 
  `(let (,.(setq statevars 
        (loop for g in gens unless (assoc 'test (genprops g)) 
       collect (gensym "ST"))) 
       (start-flg t) ,@(when saved '(saved-tables))) ,@(when saved '(saved-tabl\
es)) 
   #'(lambda nil 
       (prog , (setq bind (list '|Exhausted |)) 
  (cond (start-flg (setq start-flg nil) (go start)) 
        (t (go cont))) 
  GEN0CONT 
  (return t)  ;exhausted 
  start 
  ,@(let (ans check gen1 wff1) 
      (loop for g on gens do 
        (cond ((not (assoc 'test (genprops (car g)))) 
    (setq gen1 (sgen (car g)) wff1 (wff (car g))) 
    (setq check nil) 
    (push 
      `(multiple-value-setq 
         (|Exhausted | 
          ,@(cond 
       ((compoundwffp wff1) 
        (loop for var in (cadr (assoc 'output gen1)) collect nil)) 
       (t 
        (loop for var in (cdr wff1) as pos from 0 
       as temp in (cadr (assoc 'template gen1)) 
       when (eq temp 'output) collect 
       (cond 
      ((variable-p var)
       (push (list (make-evalvar 
              :evalvarname (gensym "CHK") 
              :evalvarcompare 
              (varcompare var)) 
            (name-of-var var)) 
             check) 
       (nconc bind 
       (list (evalvarname (caar check)))) 
       (evalvarname (caar check)))))))) 
         (funcall ,(pop statevars))) 
      ans) 
    (setq remainingvars (fldifference remainingvars 
          (soutput (car g))))) 
   (t 
    (push `(cond ((?? ,.(vars&rels-to-names-wff (wff (car g)))) nil) 
          (t (go ,(pack* 'gen label 'cont)))) 
          ans)))) 
      (reverse ans)) 
                (return (values nil ,@(vars-to-names 
     (soutput gen)))))))))))) 

Re: explanations around cleavir based compiler

[Charles Z. <ka...@be...>]

Hello Bruno,

I've updated the README to point to the reader where to find more
about Cleavir. The buildable document should bring you up to speed
about what HIR is and most of everything else about Cleavir.

I don't think edge split graph is what is meant by the thesis in that
link. I mean edge split graph to come from the process of edge
splitting following the axiomatic definition I gave, which is
described in compiler texts like Appel.

I wrote a comment describing unphi. It's an approach I made up to
convert SSA to a stack machine.

Also, the whole CST-to-AST and Generate-AST issue is a little
complicated and a long story. Robert Strandh has deprecated
Generate-AST which also has some closure conversion bugs. I discovered
the bug is also present in CST-to-AST but just brushed under the rug.
I went ahead and fixed it but it required a bunch of changes to CLISP
to support the proper fix to the bug, allowing me to reduce the number
of closures allocated. To make things more complicated, that pull
request hasn't been merged yet and still needs to be tested by Clasp.

In summary, things should be working and only using CST-to-AST once
the pull request is merged, which should be soon.

The graph-form tool is fairly simple. You invoke it on a quoted lambda
form and it should draw the optimized HIR graph in GraphViz for you.
It is a great way of getting familiar with HIR, the flow graph IR
which gets translated to CLISP. The optimization function is specified
by the keyword optimize-fun.

You can learn about catch/unwind and its weirdness from the Cleavir
doc (if its been updated, it was recently redesigned in June). It's
the HIR way of representing non-local exits.

Value numbering was a prototype value numbering pass I wrote to test
removing redundant computations like in (+ (* x y) (* x y)). I
abandoned it to work on more pressing issues I found with correctness
and closures.

Charles


On Sun, Aug 12, 2018 at 10:23 PM, Bruno Haible <br...@cl...> wrote:
> Hi Charles,
>
> Thank you for having added file-level comments to the files.
> These comments are very useful, because I wouldn't have guessed
> these things.
>
> The comment I loved most is how to make the compiler self-hosting
> (in README) - never guessed it would be so easy.
>
> But there are more things that comments could explain:
>
> - Where is readable documentation about the SICL terms? For example,
>   while I know that IR means "internal representation", 'HIR' is
>   unknown to me.
>
> - What is unphi for? What does the unphi-instruction stand for?
>
> - Is the "edge-split graph" the same thing as defined in
>   Definition 2.11 of
>   https://dspace.library.uu.nl/bitstream/handle/1874/336410/thesis.pdf ?
>   You seem to use an axiomatic definition of "edge-split graph" instead?
>
> - What are the differences between CST and AST? Why does using CST help
>   reducing the number of cells of closure vectors?
>
> - What do you mean with "cells are stuck in the strange
>   structure of catch-unwind branches"? Can you add an example
>   (as source code or IR or byte code, I don't know)?
>
> - When and how do you use graph-form in tools.lisp? What insights does
>   it bring?
>
> - What is value-numbering.lisp about?
>
> Bruno
>

explanations around cleavir based compiler

[Bruno H. <br...@cl...>]

Hi Charles,

Thank you for having added file-level comments to the files.
These comments are very useful, because I wouldn't have guessed
these things.

The comment I loved most is how to make the compiler self-hosting
(in README) - never guessed it would be so easy.

But there are more things that comments could explain:

- Where is readable documentation about the SICL terms? For example,
  while I know that IR means "internal representation", 'HIR' is
  unknown to me.

- What is unphi for? What does the unphi-instruction stand for?

- Is the "edge-split graph" the same thing as defined in
  Definition 2.11 of
  https://dspace.library.uu.nl/bitstream/handle/1874/336410/thesis.pdf ?
  You seem to use an axiomatic definition of "edge-split graph" instead?

- What are the differences between CST and AST? Why does using CST help
  reducing the number of cells of closure vectors?

- What do you mean with "cells are stuck in the strange
  structure of catch-unwind branches"? Can you add an example
  (as source code or IR or byte code, I don't know)?

- When and how do you use graph-form in tools.lisp? What insights does
  it bring?

- What is value-numbering.lisp about?

Bruno

Re: gsoc: New compiler (almost) ANSI conforming, progress report.

[Charles Z. <ka...@be...>]

Hello Bruno,

> Oh, I would have thought that dead code elimination, copy propagation,
> and constant propagation would be built-in in cleavir? Can some of
> this code be propagated back into SICL?

SICL is very new, so it is lacking in many optimizations. The existing
code in SICL is enough to get a correct compiler working. There is
also a local inlining pass. So, I've needed to write almost all the
optimizations passes myself. I've spent much of this period sending
pull requests upstream to make local inlining faster and contributing
some components I've written such as:

- Basic blocks
- Flow graph loop code
- Much faster local inlining and compilation times with CSTs.

I am working on upstreaming more components so that there is less
maintenance work on CLISP for things that are independent of the
Cleavir/CLISP interface.

Charles

On Wed, Jul 18, 2018 at 8:46 PM, Bruno Haible <br...@cl...> wrote:
> Hi Charles,
>
> Replying to your mail from June 3:
>> See some issues I opened on GitLab. ...
>> So far, I have handled macroexpansion by just calling macroexpanders
>> with the null macroexpansion environment #(NIL NIL). Since I am trying
>> to leverage CLISP's macro definitions as much as possible, I'd like to
>> know for what type of macros this will break down for CLISP. I'm not
>> quite sure which information I really need to make all macros work.
>
> If a macro expander ignores its macroexpansion environment (like you
> discovered is the case for MULTIPLE-VALUE-SETQ), it is a bug.
>
>> About the compiler, the code produced currently looks like it is
>> promising in some ways, and bad in others. For example, since I am
>> translating an SSA flow graph to byte code directly, the code looks
>> like it is trying to use the stack as a register machine, and
>> therefore at times LOADs and and PUSHs a lot. On the other hand, flow
>> sensitive information is readily apparent and it should be easy to
>> write previously impossible optimizations. To demonstrate ...
>
> Very nice!
>
>> the Cleavir code does not mention the constant 'e at all, because it's
>> obvious from the SSA flow graph that 'e is never used
>
> Marvellous! This is really what the old clisp compiler is not good at.
>
>> The passes that I've already written myself
>> are the SSA conversion, dead code elimination, and copy propogation
>> passes.
>
> Oh, I would have thought that dead code elimination, copy propagation,
> and constant propagation would be built-in in cleavir? Can some of
> this code be propagated back into SICL?
>
>> For now, I am still focusing on correctness, before jumping to more
>> sophisticated optimizations like conditional constant propogation or
>> fixing some things like special casing for the various branch
>> instructions. I wonder how I should integrate this new compiler to
>> toplevel functions like COMPILE-FILE/EVAL. Would it make sense to have
>> a dynamic variable like *use-cleavir* that switches on the new
>> compiler, or a keyword parameter to COMPILE/COMPILE-FILE? When I tried
>> using a dynamic variable, I ran into the issue where CLOS methods
>> would try and randomly recompile using Cleavir when I had only meant
>> to compile one form with Cleavir, since it's heavily CLOS based.
>
> When you start using a dynamic variable like *use-cleavir*, immediately
> the question comes up how to limit it to a particular scope. Then one
> starts to write
>   (defun foo ...
>     (compiler-let ((*use-cleavir* t))
>       ...))
> and it gets ugly. Source code should not contain operational instructions
> for the compiler, only declarations.
>
> At some point we could have a logic that evaluates the OPTIMIZE declarations.
> But it is also good to have a completely unambiguous way to say
> "this code is to be compiled by cleavir".
>
> clisp already has a non-standardized declaration (COMPILE)
> https://clisp.sourceforge.io/impnotes/declarations.html#compile-decl
> It says "this code is to be compiled".
>
> Maybe we could have a declaration (COMPILER [basic|cleavir]) that specifies
> which compiler to use, when compilation is requested. Like this:
>
> Or maybe add an argument to the COMPILE declaration:
>   (locally (declare (compile) (compiler basic)) ...)   ; uses the original compiler
>   (locally (declare (compile) (compiler cleavir)) ...) ; uses the cleavir compiler
>
> For COMPILE-FILE, one needs a global setting. This global setting
> would be available through DECLAIM:
>   (declaim (compiler cleavir))
>
> With this proposal,
>   - The compiler can be chosen locally or globally.
>   - The mechanism integrates with the Common Lisp declaration system.
>   - Interpreted code can continue to be interpreted.
>   - The logic that evaluates the OPTIMIZE declarations can be decided upon
>     independently.
>
> Bruno
>

New Swedish PO file for 'clisp' (version 2.49.60)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'clisp' has been submitted
by the Swedish team of translators.  The file is available at:

    http://translationproject.org/latest/clisp/sv.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/clisp/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/clisp.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

New Swedish PO file for 'clisp' (version 2.49.60)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'clisp' has been submitted
by the Swedish team of translators.  The file is available at:

    http://translationproject.org/latest/clisp/sv.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/clisp/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/clisp.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

Re: correct sicl-git.version?

[Charles Z. <ka...@be...>]

Hello Bruno,

I've updated the file; it should be at 2b2a73ccc5b29fc8f76f402d2c1b5962b3aac461.

Unfortunately, SICL is still moving ahead very quickly, with frequent
breaking changes. It doesn't help that I am also contributing some
code/bugfixes upstream for problems I run into. I will try to keep
that file in sync.

Also, that new git version might try to pull in some new dependencies
like Eclector, etc.

And if you want to try self compiling, you will need to add #-cleavir
above this form:
https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/check-special-form-syntax.lisp#L66
(to handle multi-arg FUNCTION in clisp).
then
(push :cleavir *features*)
(setf *use-cleavir* t)
(time (let ((sys::*load-compiling* t)) (load ...)))
(time (let ((sys::*load-compiling* t)) (load ...)))

I am currently porting the compiler to use the new CSTs
https://github.com/robert-strandh/Concrete-Syntax-Tree for generating
ASTs instead of Generate-AST since that framework is now obsolete.
This will remove the need for that hack as well.

Again, bleeding edge stuff. SICL is not quite stable yet. I will try
to add more documentation about self-compiling once the port settles
down and of course general documentation in the meantime.

Charles

On Thu, Jul 19, 2018 at 6:47 PM, Bruno Haible <br...@cl...> wrote:
> Hi Charles,
>
> I'm trying to load SICL/cleavir into clisp, and encountering a problem:
> Your sicl-git.version file contains the commit id
>   bf5e4e2ec273b1e8e6d14de3a0f606846c7cad24
> which is from 2018-06-10. So I chose this SICL version. Now, when I load
> your work, trace-lap-blocks.lisp attempts to access the exported symbol
> cleavir-basic-blocks:basic-block. But this symbol does not exist,
> it was only added on 2018-06-13, see
>   https://github.com/robert-strandh/SICL/commits/master/Code/Cleavir/Basic-blocks/packages.lisp
>
> What's the new SICL git version that you can recommend?
>
> Bruno
>

correct sicl-git.version?

[Bruno H. <br...@cl...>]

Hi Charles,

I'm trying to load SICL/cleavir into clisp, and encountering a problem:
Your sicl-git.version file contains the commit id
  bf5e4e2ec273b1e8e6d14de3a0f606846c7cad24
which is from 2018-06-10. So I chose this SICL version. Now, when I load
your work, trace-lap-blocks.lisp attempts to access the exported symbol
cleavir-basic-blocks:basic-block. But this symbol does not exist,
it was only added on 2018-06-13, see
  https://github.com/robert-strandh/SICL/commits/master/Code/Cleavir/Basic-blocks/packages.lisp

What's the new SICL git version that you can recommend?

Bruno

Re: gsoc: New compiler (almost) ANSI conforming, progress report.

[Bruno H. <br...@cl...>]

Charles Zhang wrote on June 5:
> I noticed that there is no way to make #<SPECIAL REFERENCE> objects,
> as that's only marked in C

There is exactly one such object, and you can access it through
sys::specdecl (defined in compiler.lisp through some dirty hack).
Code that accesses it and stores it in local variables needs to be
compiled, not interpreted (because the interpreter would interpret
this value #<SPECIAL REFERENCE> as an indication to look on the
symbol-value of the variable...).

Bruno

Re: gsoc: New compiler (almost) ANSI conforming, progress report.

[Bruno H. <br...@cl...>]

Hi Charles,

Replying to your mail from June 3:
> See some issues I opened on GitLab. ...
> So far, I have handled macroexpansion by just calling macroexpanders
> with the null macroexpansion environment #(NIL NIL). Since I am trying
> to leverage CLISP's macro definitions as much as possible, I'd like to
> know for what type of macros this will break down for CLISP. I'm not
> quite sure which information I really need to make all macros work.

If a macro expander ignores its macroexpansion environment (like you
discovered is the case for MULTIPLE-VALUE-SETQ), it is a bug.

> About the compiler, the code produced currently looks like it is
> promising in some ways, and bad in others. For example, since I am
> translating an SSA flow graph to byte code directly, the code looks
> like it is trying to use the stack as a register machine, and
> therefore at times LOADs and and PUSHs a lot. On the other hand, flow
> sensitive information is readily apparent and it should be easy to
> write previously impossible optimizations. To demonstrate ...

Very nice!

> the Cleavir code does not mention the constant 'e at all, because it's
> obvious from the SSA flow graph that 'e is never used

Marvellous! This is really what the old clisp compiler is not good at.

> The passes that I've already written myself
> are the SSA conversion, dead code elimination, and copy propogation
> passes.

Oh, I would have thought that dead code elimination, copy propagation,
and constant propagation would be built-in in cleavir? Can some of
this code be propagated back into SICL?

> For now, I am still focusing on correctness, before jumping to more
> sophisticated optimizations like conditional constant propogation or
> fixing some things like special casing for the various branch
> instructions. I wonder how I should integrate this new compiler to
> toplevel functions like COMPILE-FILE/EVAL. Would it make sense to have
> a dynamic variable like *use-cleavir* that switches on the new
> compiler, or a keyword parameter to COMPILE/COMPILE-FILE? When I tried
> using a dynamic variable, I ran into the issue where CLOS methods
> would try and randomly recompile using Cleavir when I had only meant
> to compile one form with Cleavir, since it's heavily CLOS based.

When you start using a dynamic variable like *use-cleavir*, immediately
the question comes up how to limit it to a particular scope. Then one
starts to write
  (defun foo ...
    (compiler-let ((*use-cleavir* t))
      ...))
and it gets ugly. Source code should not contain operational instructions
for the compiler, only declarations.

At some point we could have a logic that evaluates the OPTIMIZE declarations.
But it is also good to have a completely unambiguous way to say
"this code is to be compiled by cleavir".

clisp already has a non-standardized declaration (COMPILE)
https://clisp.sourceforge.io/impnotes/declarations.html#compile-decl
It says "this code is to be compiled".

Maybe we could have a declaration (COMPILER [basic|cleavir]) that specifies
which compiler to use, when compilation is requested. Like this:

Or maybe add an argument to the COMPILE declaration:
  (locally (declare (compile) (compiler basic)) ...)   ; uses the original compiler
  (locally (declare (compile) (compiler cleavir)) ...) ; uses the cleavir compiler

For COMPILE-FILE, one needs a global setting. This global setting
would be available through DECLAIM:
  (declaim (compiler cleavir))

With this proposal,
  - The compiler can be chosen locally or globally.
  - The mechanism integrates with the Common Lisp declaration system.
  - Interpreted code can continue to be interpreted.
  - The logic that evaluates the OPTIMIZE declarations can be decided upon
    independently.

Bruno

notifications on gitlab

[Bruno H. <br...@cl...>]

Hi all,

I don't see an immediate way to configure gitlab in such a way that commits
and changes to the issue tracker would yield notifications to this
mailing list.

Therefore, if you are interested in receiving emails from the issue tracker,
do this:
  - Go to https://gitlab.com/profile . Log in.
  - On the left-hand side, pick "Notifications".
  - For group 'clisp', pick the level "Watch" instead of "Global".

Bruno

Local function inlining works.

[Charles Z. <ka...@be...>]

Hello clisp, Bruno, Sam,

It is straightforward to do local inlining with the new compiler.
(There isn't policy to control this yet; its all or nothing).

A fancy example:
(cleavir-compile '(lambda ()
                   (let ((x 0))
                     (block a
                       (flet ((%f (y)
                                (flet ((%g (z)
                                         (return-from a (+ x y z))))
                                  (%g 10))
                                'bad))
                         (%f 14)
                         'badder)
                       'super-bad))))

CLEAVIR-CLISP> (disassemble *)
Disassembly of function NIL
(CONST 0) = 24
0 required arguments
0 optional arguments
No rest parameter
No keyword parameters
2 byte-code instructions:
0     (CONST 0)                           ; 24
1     (SKIP&RET 1)

And Sam's example from a long time ago:
(cleavir-compile '(lambda ()
                                  (labels ((bar () t)) (bar))
                                  (labels ((bar () nil)) (bar))))

Disassembly of function NIL
(CONST 0) = NIL
0 required arguments
0 optional arguments
No rest parameter
No keyword parameters
2 byte-code instructions:
0     (CONST 0)                           ; NIL
1     (SKIP&RET 1)

Local inlining triggers more optimizations and eliminates consing from
closure creation. So it is well worth it. This can also allow making
function-macro-let just an alias to labels. Local inlining can handle
inlining call-next-method et al.

Charles

Re: gsoc: 2/3 progress, summary

[Jean L. <bu...@gn...>]

Where do I get that development version?

Jean

On Mon, Jul 09, 2018 at 03:24:40PM -0400, Charles Zhang wrote:
> Hello clisp,
> 
> For this period many correctness bugs in the new compiler were fixed,
> with multiple self compiles possible without any detectable oddities.
> 
> A few new dataflow passes and better code generation has been
> implemented. Some highlights:
> 
> - SSA conditional constant propagation, which can fold code like
> (lambda () (let ((i 0) a b c d)
>                           (values
>                            (and (setq a (setq i (1+ i)))
>                                 (setq b (setq i (1+ i)))
>                                 (setq c (setq i (1+ i)))
>                                 (setq d (setq i (1+ i))))
>                            i a b c d)))
> into (values 4 4 1 2 3 4). Very useful for macro heavy code.
> - Loop invariant hoisting. Useful for numeric code with arrays/matrices.
> - Prototype value numbering with Kildall. Good for some forms of
> redundant expression elimination.
> - Better dead code elimination detection with respect to multiple values.
> Code generation improvements:
> - Better instruction selection for branches
> - Inlined multiple-values instructions. No more consing and improved
> self compile time from 55 seconds to 42 seconds.
> - Inlined funcall
> - Much better instruction selection for various CALL instructions
> - Some form of basic block scheduling, to remove JMPs from LAP.
> - Extra LOAD/PUSH/STOREs eliminated. The code looks more like stack
> machine code.
> 
> Some optimizations were made to make the compiler faster as well, by
> paying more attention to the speed of convergence of some dataflow
> analysis algorithms.
> 
> Some goals for next period:
> - Using escape analysis to see when closures can be eliminated; e.g.
> by lambda lifting closed over values in non escaping nested functions.
> - Integrating value numbering analysis
> - Loop induction variable analysis
> - Making the compiler faster by ordering flow nodes properly in
> dataflow analysis.
> - Try to experiment with untyped instructions and type inference.
> - Local function inlining with Robert Strandh's partial inlining
> technique, which was recently presented at ELS.
> - Inlining in general.
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> clisp-devel mailing list
> cli...@li...
> https://lists.sourceforge.net/lists/listinfo/clisp-devel

-- 
Jean Louis

gsoc: 2/3 progress, summary

[Charles Z. <ka...@be...>]

Hello clisp,

For this period many correctness bugs in the new compiler were fixed,
with multiple self compiles possible without any detectable oddities.

A few new dataflow passes and better code generation has been
implemented. Some highlights:

- SSA conditional constant propagation, which can fold code like
(lambda () (let ((i 0) a b c d)
                          (values
                           (and (setq a (setq i (1+ i)))
                                (setq b (setq i (1+ i)))
                                (setq c (setq i (1+ i)))
                                (setq d (setq i (1+ i))))
                           i a b c d)))
into (values 4 4 1 2 3 4). Very useful for macro heavy code.
- Loop invariant hoisting. Useful for numeric code with arrays/matrices.
- Prototype value numbering with Kildall. Good for some forms of
redundant expression elimination.
- Better dead code elimination detection with respect to multiple values.
Code generation improvements:
- Better instruction selection for branches
- Inlined multiple-values instructions. No more consing and improved
self compile time from 55 seconds to 42 seconds.
- Inlined funcall
- Much better instruction selection for various CALL instructions
- Some form of basic block scheduling, to remove JMPs from LAP.
- Extra LOAD/PUSH/STOREs eliminated. The code looks more like stack
machine code.

Some optimizations were made to make the compiler faster as well, by
paying more attention to the speed of convergence of some dataflow
analysis algorithms.

Some goals for next period:
- Using escape analysis to see when closures can be eliminated; e.g.
by lambda lifting closed over values in non escaping nested functions.
- Integrating value numbering analysis
- Loop induction variable analysis
- Making the compiler faster by ordering flow nodes properly in
dataflow analysis.
- Try to experiment with untyped instructions and type inference.
- Local function inlining with Robert Strandh's partial inlining
technique, which was recently presented at ELS.
- Inlining in general.

gsoc: type inference, again

[Charles Z. <ka...@be...>]

Hello Bruno and Sam,

Asking again because I don't think I got a reply from a few weeks ago.

I'd like to start leveraging the type inferencing in SICL's compiler,
but CLISP's bytecodes abstract away the  type checking into C. It
would be nice to eliminate provably unnecessary type checks with the
inferred information. So, would it make sense to add raw instructions
to do things like CAR/CDR without type checking? These can be inserted
only when the compiler can prove safety; otherwise, it can fallback to
the old inlie CAR/CDR instructions to improve space, for example. The
same holds for other system functions implemented in C.

I know CLISP just currently ignores type declarations, but it might
also be useful to add type annotations for the standard library
functions at least.

Charles

gsoc: Improved code generation

[Charles Z. <ka...@be...>]

Hello clisp, Bruno, Sam,

I have spent some time optimizing bytecode generation, including:
- using all the various specialized CALL instructions more effectively
- inlining VALUES, MULTIPLE-VALUE-SETQ/BIND
- a very bad basic block scheduler
- reduce LOAD/STOREs (without peephole optimizing)

Together with data flow optimizations, the new compiler is usually
able to generate better code than the existing compiler.

For example, (from cl-ansi-test-suite)
> (cleavir-compile '(lambda ()
                   (let* ((x (list 'a 'b 'c 'd 'e))
                          (y (list 'f 'g 'h 'i 'j))
                          (p1 1) (p2 2) (len 3)
                          (z '(10 11 12)))
                     (rotatef (subseq x p1 (+ p1 len))
                              (subseq y p2 (+ p2 len))
                              z)
                     (values x y z))))

Disassembly of function NIL
(CONST 0) = A
(CONST 1) = B
(CONST 2) = C
(CONST 3) = D
(CONST 4) = E
(CONST 5) = F
(CONST 6) = G
(CONST 7) = H
(CONST 8) = I
(CONST 9) = J
(CONST 10) = 1
(CONST 11) = 4
(CONST 12) = 2
(CONST 13) = 5
(CONST 14) = (10 11 12)
(CONST 15) = :START1
(CONST 16) = :END1
(CONST 17) = :START1
(CONST 18) = :END1
0 required arguments
0 optional arguments
No rest parameter
No keyword parameters
42 byte-code instructions:
0     (CONST&PUSH 0)                      ; A
1     (CONST&PUSH 1)                      ; B
2     (CONST&PUSH 2)                      ; C
3     (CONST&PUSH 3)                      ; D
4     (CONST&PUSH 4)                      ; E
5     (LIST&PUSH 5)
7     (CONST&PUSH 5)                      ; F
8     (CONST&PUSH 6)                      ; G
9     (CONST&PUSH 7)                      ; H
10    (CONST&PUSH 8)                      ; I
11    (CONST&PUSH 9)                      ; J
12    (LIST&PUSH 5)
14    (LOAD&PUSH 1)
15    (CONST&PUSH 10)                     ; 1
16    (CONST&PUSH 11)                     ; 4
17    (CALLS2&PUSH 96)                    ; SUBSEQ
19    (LOAD&PUSH 1)
20    (CONST&PUSH 12)                     ; 2
21    (CONST&PUSH 13)                     ; 5
22    (CALLS2&PUSH 96)                    ; SUBSEQ
24    (CONST&PUSH 14)                     ; (10 11 12)
25    (LOAD&PUSH 4)
26    (LOAD&PUSH 2)
27    (PUSH-UNBOUND 4)
29    (CONST 10)                          ; 1
30    (STORE 3)
31    (CONST 11)                          ; 4
32    (STORE 2)
33    (CALLS2 104)                        ; REPLACE
35    (LOAD&PUSH 3)
36    (LOAD&PUSH 1)
37    (PUSH-UNBOUND 4)
39    (CONST 12)                          ; 2
40    (STORE 3)
41    (CONST 13)                          ; 5
42    (STORE 2)
43    (CALLS2 104)                        ; REPLACE
45    (LOAD&PUSH 4)
46    (LOAD&PUSH 4)
47    (LOAD&PUSH 4)
48    (STACK-TO-MV 3)
50    (SKIP&RET 6)

vs CLISP's
Disassembly of function NIL
(CONST 0) = A
(CONST 1) = B
(CONST 2) = C
(CONST 3) = D
(CONST 4) = E
(CONST 5) = F
(CONST 6) = G
(CONST 7) = H
(CONST 8) = I
(CONST 9) = J
(CONST 10) = (10 11 12)
(CONST 11) = 1
(CONST 12) = 3
(CONST 13) = 2
0 required arguments
0 optional arguments
No rest parameter
No keyword parameters
48 byte-code instructions:
0     (CONST&PUSH 0)                      ; A
1     (CONST&PUSH 1)                      ; B
2     (CONST&PUSH 2)                      ; C
3     (CONST&PUSH 3)                      ; D
4     (CONST&PUSH 4)                      ; E
5     (LIST&PUSH 5)
7     (CONST&PUSH 5)                      ; F
8     (CONST&PUSH 6)                      ; G
9     (CONST&PUSH 7)                      ; H
10    (CONST&PUSH 8)                      ; I
11    (CONST&PUSH 9)                      ; J
12    (LIST&PUSH 5)
14    (CONST&PUSH 10)                     ; (10 11 12)
15    (CONST&PUSH 11)                     ; 1
16    (CONST&PUSH 12)                     ; 3
17    (CALLSR&PUSH 2 55)                  ; +
20    (CONST&PUSH 13)                     ; 2
21    (CONST&PUSH 12)                     ; 3
22    (CALLSR&PUSH 2 55)                  ; +
25    (LOAD&PUSH 4)
26    (CONST&PUSH 11)                     ; 1
27    (LOAD&PUSH 3)
28    (CALLS2&PUSH 96)                    ; SUBSEQ
30    (LOAD&PUSH 4)
31    (CONST&PUSH 13)                     ; 2
32    (LOAD&PUSH 3)
33    (CALLS2&PUSH 96)                    ; SUBSEQ
35    (LOAD&PUSH 4)
36    (LOAD&PUSH 7)
37    (LOAD&PUSH 2)
38    (CONST&PUSH 11)                     ; 1
39    (LOAD&PUSH 7)
40    (PUSH-UNBOUND 2)
42    (CALLS2 104)                        ; REPLACE
44    (LOAD&PUSH 6)
45    (LOAD&PUSH 1)
46    (CONST&PUSH 13)                     ; 2
47    (LOAD&PUSH 6)
48    (PUSH-UNBOUND 2)
50    (CALLS2 104)                        ; REPLACE
52    (LOAD 2)
53    (STORE 5)
54    (SKIP 5)
56    (LOAD&PUSH 2)
57    (LOAD&PUSH 2)
58    (LOAD&PUSH 2)
59    (STACK-TO-MV 3)
61    (SKIP&RET 4)

An example of the suboptimal basic block scheduler:
CLEAVIR-CLISP> (cleavir-compile '(lambda (n) (dotimes (i n) (print i))))
#<COMPILED-FUNCTION NIL>
CLEAVIR-CLISP> (disassemble *)

Disassembly of function NIL
(CONST 0) = 0
(CONST 1) = NIL
1 required argument
0 optional arguments
No rest parameter
No keyword parameters
15 byte-code instructions:
0     (CONST&PUSH 0)                      ; 0
1     L1
1     (LOAD&PUSH 0)
2     (LOAD&PUSH 3)
3     (CALLSR&PUSH 1 52)                  ; >=
6     (LOAD&JMPIF 0 L20)
9     (LOAD&PUSH 1)
10    (PUSH-UNBOUND 1)
12    (CALLS1 142)                        ; PRINT
14    (LOAD&INC&STORE 1)
16    (SKIP 1)
18    (JMP L1)
20    L20
20    (CONST 1)                           ; NIL
21    (SKIP&RET 4)

Recursive functions don't use JSR in the new compiler, but rather
close over themselves.
CLEAVIR-CLISP> (cleavir-compile '(lambda ()
                   (labels ((fact (n)
                              (if (zerop n)
                                  1
                                  (* n (fact (1- n))))))
                     #'fact)))
#<COMPILED-FUNCTION NIL>
CLEAVIR-CLISP> (funcall * )
#<COMPILED-FUNCTION NIL>
CLEAVIR-CLISP> (disassemble *)

Disassembly of function NIL
(CONST 0) = #(NIL #<COMPILED-FUNCTION NIL>)
(CONST 1) = 1
1 required argument
0 optional arguments
No rest parameter
No keyword parameters
13 byte-code instructions:
0     (LOAD&PUSH 1)
1     (CALLS2&PUSH 172)                   ; ZEROP
3     (LOAD&JMPIF 0 L20)
6     (LOADV&PUSH 0 1)
9     (LOAD&DEC&PUSH 3)
11    (FUNCALL&PUSH 1)
13    (LOAD&PUSH 3)
14    (LOAD&PUSH 1)
15    (CALLSR 2 57)                       ; *
18    (SKIP&RET 4)
20    L20
20    (CONST 1)                           ; 1
21    (SKIP&RET 3)

Charles