Steel Bank Common Lisp

On 18 August 2010 14:49, James Fleming <lisp@...> wrote:
> The very short version of the question: what changed between 1.0.39 and
> 1.0.40, regarding the expected locations of source files, and when/how
> they're expected to be copied there during asdf::perform operations?

ASDF version supplied with SBCL changing is the probable culprit for
at least some of this -- adding asdf-devel to CC.

(It seems the NEWS entry for ASDF2 was forgotten, oops...)

However, this bit right here

> ; compiling file
> "/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-grovel/def-to-lisp.lisp" (written 15

indicates that something else is wrong too: loading SB-GROVEL should
not cause it to be compiled. If your ASDF-LOAD uses :FORCE, that's a
the obvious reason, but if not I suspect something funny is going on
with timestamps on those files since ASDF feels the need to recompile
them.

Cheers,

 -- Nikodemus

> The context: I'm writing a wiki (yes, yet another one), and its back-end
> consists of plain-text files while I get to grips with writing a parser.
>
> The text-file wrangling calls on cl-fad, which uses sb-posix, which in
> turn makes use of sb-grovel.
> Somewhere between 1.0.39 and 1.0.40, a change was made that broke
> sb-posix, at least on the two machines I use - both run 64-bit Linux, one
> on AMD and one on Intel.
>
> The symptom: sb-grovel's def-to-lisp fails to compile at line 244, where
> it tries to evaluate (compile-file tmp-constants :output-file output-file)
> after defining 'tmp-constants in this manner:
> (let ((tmp-constants (merge-pathnames #p"constants.lisp-temp"
>                                         real-output-file)))
>
> It seems to come down to where source-files are expected to be, i.e. the
> ~/.cache tree, and how they're copied there. The problem can be reproduced
> on cue by invoking (asdf:oos 'asdf:load-op :sb-posix). This appears as
> (asdf-load :sb-posix) below because I'm a lazy git and defined that
> abbreviation in my .sbclrc.
>
> I've been trying to follow along in the source-code, and the nearest
> relevant changes appear to be in contrib/asdf/asdf.lisp, but my thinking
> has been addled by a nasty head-cold. Could somebody please give me a
> pointer to what I need to look at next? What have I missed? Help!
>
>
> Thanks in advance for any useful tips,
> James
>
>
> P.S. For those who'd like to see exactly what happens, a bleeding great
> copy-and-paste follows...
>
> * (asdf-load :sb-posix)
>
> ; compiling file
> "/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-grovel/def-to-lisp.lisp" (written 15
> AUG 2010 07:51:57 PM):
> ; compiling (IN-PACKAGE #:SB-GROVEL)
> ; compiling (DEFVAR *DEFAULT-C-STREAM* ...)
> ; compiling (DEFUN ESCAPE-FOR-STRING ...)
> ; compiling (DEFUN SPLIT-CFLAGS ...)
> ; compiling (DEFUN C-ESCAPE ...)
> ; compiling (DEFUN AS-C ...)
> ; compiling (DEFUN PRINTF ...)
> ; compiling (DEFUN C-FOR-ENUM ...)
> ; compiling (DEFUN C-FOR-STRUCTURE ...)
> ; compiling (DEFUN PRINT-C-SOURCE ...)
> ; compiling (DEFUN C-CONSTANTS-EXTRACT ...)
> ; compiling (DEFCLASS GROVEL-CONSTANTS-FILE ...)
> ; compiling (DEFINE-CONDITION C-COMPILE-FAILED ...)
> ; compiling (DEFINE-CONDITION A-DOT-OUT-FAILED ...)
> ; compiling (DEFMETHOD PERFORM ...)
>
> ;
> /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-grovel/ASDF-TMP-def-to-lisp.fasl
> written
> ; compilation finished in 0:00:00.558
> (/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp
>  /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.fasl
>  /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.fasl
>  /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/foo.c
>  /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/a.out
>  /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp)
>
> debugger invoked on a SB-INT:SIMPLE-FILE-ERROR in thread #<THREAD
>                                                           "initial
> thread" RUNNING
>                                                           {1002AC20A1}>:
>  failed to find the TRUENAME of
> /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp:
>    No such file or directory
>
> Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL.
>
> restarts (invokable by number or by possibly-abbreviated name):
>  0: [TRY-RECOMPILING] Try recompiling constants
>  1: [RETRY          ] Retry performing #<ASDF:COMPILE-OP NIL
> {1003B23C31}> on
>                       #<SB-GROVEL:GROVEL-CONSTANTS-FILE
>                         "sb-posix" "constants">.
>  2: [ACCEPT         ] Continue, treating #<ASDF:COMPILE-OP NIL {1003B23C31}>
>                       on
>                       #<SB-GROVEL:GROVEL-CONSTANTS-FILE
>                         "sb-posix" "constants">
>                       as having been successful.
>  3: [ABORT          ] Exit debugger, returning to top level.
>
> (SB-IMPL::SIMPLE-FILE-PERROR
>  "failed to find the TRUENAME of ~A"
>  #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp"
>  2)
> 0] backtrace
>
> 0: (SB-IMPL::SIMPLE-FILE-PERROR
>    "failed to find the TRUENAME of ~A"
>    #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp"
>    2)
> 1: ((FLET SB-IMPL::FAIL)
>    "failed to find the TRUENAME of ~A"
>    #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp"
>    2)
> 2: (SB-IMPL::QUERY-FILE-SYSTEM
>    #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp"
>    :TRUENAME
>    T)
> 3: (TRUENAME
>    #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp")
> 4: ((FLET SB-C::TRY-WITH-TYPE)
>    #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp"
>    "lisp"
>    T)
> 5: (SB-C::VERIFY-SOURCE-FILE
>    #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp")
> 6: (SB-C::VERIFY-SOURCE-FILE
>    #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp")[:EXTERNAL]
> 7: (COMPILE-FILE
>    #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp")[:EXTERNAL]
> 8: ((SB-PCL::FAST-METHOD ASDF:PERFORM
>     (ASDF:COMPILE-OP SB-GROVEL:GROVEL-CONSTANTS-FILE))
>    #<unavailable argument>
>    #<unavailable argument>
>    #<ASDF:COMPILE-OP NIL {1003B23C31}>
>    #<SB-GROVEL:GROVEL-CONSTANTS-FILE "sb-posix" "constants">)
> 9: ((LAMBDA
>        (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0.
>         SB-PCL::.ARG1.))
>    #<unavailable argument>
>    #<unavailable argument>
>    #<ASDF:COMPILE-OP NIL {1003B23C31}>
>    #<SB-GROVEL:GROVEL-CONSTANTS-FILE "sb-posix" "constants">)
> 10: ((SB-PCL::FAST-METHOD ASDF::PERFORM-WITH-RESTARTS
>      (ASDF:COMPILE-OP ASDF:CL-SOURCE-FILE))
>     #<unavailable argument>
>     #S(SB-PCL::FAST-METHOD-CALL
>        :FUNCTION #<FUNCTION #>
>        :PV NIL
>        :NEXT-METHOD-CALL NIL
>        :ARG-INFO (2))
>     #<ASDF:COMPILE-OP NIL {1003B23C31}>
>     #<SB-GROVEL:GROVEL-CONSTANTS-FILE "sb-posix" "constants">)
> 11: ((LAMBDA ()))
> 12: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK))
> 13: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-RECURSIVE-LOCK]324))
> 14: (SB-THREAD::CALL-WITH-RECURSIVE-LOCK
>     #<CLOSURE (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK) {7FFFF79AB6C9}>
>     #S(SB-THREAD:MUTEX
>        :NAME "World Lock"
>        :%OWNER #<SB-THREAD:THREAD "initial thread" RUNNING {1002AC20A1}>
>        :STATE 1))
> 15: ((FLET SB-C::WITH-IT))
> 16: ((SB-PCL::FAST-METHOD ASDF:OPERATE (T T))
>     #<unavailable argument>
>     #<unavailable argument>
>     ASDF:LOAD-OP
>     :SB-POSIX)[:EXTERNAL]
> 17: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ASDF-LOAD :SB-POSIX) #<NULL-LEXENV>)
> 18: (INTERACTIVE-EVAL (ASDF-LOAD :SB-POSIX))[:EXTERNAL]
> 19: (SB-IMPL::REPL-FUN NIL)
> 20: ((LAMBDA ()))
> 21: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #<CLOSURE (LAMBDA #) {1002EBD309}>)
> 22: (SB-IMPL::TOPLEVEL-REPL NIL)
> 23: (SB-IMPL::TOPLEVEL-INIT)
> 24: ((LABELS SB-IMPL::RESTART-LISP))
>
> 0]
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by
>
> Make an app they can't live without
> Enter the BlackBerry Developer Challenge
> http://p.sf.net/sfu/RIM-dev2dev
> _______________________________________________
> Sbcl-help mailing list
> Sbcl-help@...
> https://lists.sourceforge.net/lists/listinfo/sbcl-help
>

I can't reproduce your issue.

sbcl --eval '(require :asdf)' --eval '(trace compile-file load
asdf::perform sb-ext:run-program)' --eval '(require :sb-posix)'

sbcl --eval '(require :asdf)' --eval '(trace compile-file load
asdf::perform sb-ext:run-program)' --eval '(asdf:load-system
:sb-posix)'

Both work fine for me.

What ASDF configuration do you use?

Could it be that you're running from a dumped image and that you're
not exporting SBCL_HOME to a proper value? ASDF does something magic
with SBCL_HOME for those output files.

But yeah, I shall submit a patch to SB-GROVEL so that things should
work even in screwy cases like that.

Note that you don't need to define a ASDF-LOAD function now that ASDF
has ASDF:LOAD-SYSTEM.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Please leave the State in the toilets where you found it.

(Adding sbcl-devel to a thread started on sbcl-help.)

> Could it be that you're running from a dumped image and that you're
> not exporting SBCL_HOME to a proper value? ASDF does something magic
> with SBCL_HOME for those output files.
>
> But yeah, I shall submit a patch to SB-GROVEL so that things should
> work even in screwy cases like that.
>
Easier said than done.
the perform method in sb-grovel/def-to-lisp.lisp has a short-circuit
do-not-grovel, set in sb-posix.asd and sb-bsd-sockets.asd. If I
replace its value from
                 :do-not-grovel #.(progn #-sb-building-contrib t)
to
                 :do-not-grovel #.(and #-sb-building-contrib t
(sb-int:sbcl-homedir-pathname))
loading the stuff loses with:
The CC environment variable has not been set in SB-GROVEL. Since this
variable should always be set during the SBCL build process, this
might indicate an SBCL with a broken contrib installation.

Note: I could reproduce the bug this way:
rlwrap xcvb repl
(progn (dolist (x (list :sb-bsd-sockets-system :sb-posix-system
:sb-posix :sb-grovel)) (delete-package x)) (dolist (x (quote
(:sb-posix :sb-grovel))) (asdf::clear-system
x))(asdf:initialize-source-registry
"/home/fare/src/sbcl//:")(asdf:initialize-output-translations)(asdf::load-system
:sb-grovel)(trace sb-grovel::c-constants-extract) (asdf:load-system
:sb-posix))

A more complete solution would involve
(1) having a warning instead of an error there
(2) remembering the build-time value of CC and CFLAGS, etc., and
hoping that they are still valid at runtime, if not overridden by the
user (?)

OR, it may involve
(3) having a way to save a default sb-homedir-pathname in the dumped image, etc.

In any case, it's no trivial endeavor, it requires munging inside the
SBCL build process.

WORKAROUND: be sure to include sb-posix and sb-bsd-sockets in your dumped image.

--#f
Multiple instances of a same hacker with different context in his mental
cache count as multiple hackers wrt documentation and testing needs.

> I can't reproduce your issue.

I figured there was a good chance that it was something quirky in my
environment; if I'm the first to report it, then it's probably not that
widespread an issue.


> What ASDF configuration do you use?
---
My .sbclrc includes these lines:
(require 'asdf)

(setf asdf:*central-registry* (list "/opt/pkgs/lisp-asdf/systems/"))

(defun asdf-load (pkg)
  (asdf:oos 'asdf:load-op pkg))
(export 'asdf-load)
---


> Could it be that you're running from a dumped image and that you're
> not exporting SBCL_HOME to a proper value? ASDF does something magic
> with SBCL_HOME for those output files.

It's entirely possible.
I'm a big fan of graft for installations, at least on my own machines. I
compile and install to /opt/pkgs/<application>-<version>/ and then graft
it to /usr/local/, and set SBCL_HOME to /usr/local/lib/sbcl.

I'll try compiling and installing directly to /usr/local/, and see if that
fixes the problem.


> But yeah, I shall submit a patch to SB-GROVEL so that things should
> work even in screwy cases like that.

Marvellous. I was hoping to provide this myself, but haven't yet figured
out what it is I need to rearrange.


> Note that you don't need to define a ASDF-LOAD function now that ASDF
> has ASDF:LOAD-SYSTEM.

Sweet! Thanks for the tip. That's one nonstandard thing I can ditch.


> [ François-René ÐVB Rideau | Reflection&Cybernethics |
> http://fare.tunes.org ]
> Please leave the State in the toilets where you found it.

>> What ASDF configuration do you use?
> (setf asdf:*central-registry* (list "/opt/pkgs/lisp-asdf/systems/"))
>
With ASDF2, you can instead
  mkdir -p /etc/common-lisp/source-registry.conf.d/
echo > /etc/common-lisp/source-registry.conf.d/opt-pkgs.conf \
  '(:directory "/opt/pkgs/lisp-asdf/systems/")'
OR you may skip the link farm altogether and directly do something like
echo > /etc/common-lisp/source-registry.conf.d/opt-pkgs.conf \
  '(:tree "/opt/pkgs/lisp-asdf/sources/")'

>> Could it be that you're running from a dumped image and that you're
>> not exporting SBCL_HOME to a proper value? ASDF does something magic
>> with SBCL_HOME for those output files.
>
> It's entirely possible.
> I'm a big fan of graft for installations, at least on my own machines. I
> compile and install to /opt/pkgs/<application>-<version>/ and then graft
> it to /usr/local/, and set SBCL_HOME to /usr/local/lib/sbcl.
>
Weird. How are you starting sbcl, and what does
  (sb-ext:posix-getenv "SBCL_HOME")
and/or
  (sb-int:sbcl-homedir-pathname)
return?
If they return some path, is there something at said path?

> I'll try compiling and installing directly to /usr/local/, and see if that
> fixes the problem.
>
For the record, I compile and install in /usr/local/stow/sbcl-1.0.41
then use stow to create appropriate symlinks.

>> But yeah, I shall submit a patch to SB-GROVEL so that things should
>> work even in screwy cases like that.
>
> Marvellous. I was hoping to provide this myself, but haven't yet figured
> out what it is I need to rearrange.
>
Unhappily (see previous email), there is no easy solution to that problem.
Maybe a future version of ASDF could selectively disable output-translations
for some systems?

--#f
Peace is an extension of war by political means. Plenty of elbowroom
is pleasanter — and much safer.
        — Robert Heinlein, "Time Enough For Love"

> With ASDF2, you can instead
<snip>
Thanks again; I'll try those.


> Weird. How are you starting sbcl

sbcl --dynamic-space-size 256 --core /usr/local/lib/sbcl/sbcl.core


> what does
>   (sb-ext:posix-getenv "SBCL_HOME")

"/usr/local/lib/sbcl"

>   (sb-int:sbcl-homedir-pathname)

#P"/usr/local/lib/sbcl/"


> If they return some path, is there something at said path?

Indeed there is, and it's exactly what's expected; they're the correct paths.


>> I'll try compiling and installing directly to /usr/local/, and see if
>> that fixes the problem.

...and it did. sb-posix and cl-fad now load cleanly. So _graft_ broke it.


> For the record, I compile and install in /usr/local/stow/sbcl-1.0.41
> then use stow to create appropriate symlinks.

Weirder and weirder. I wonder what the practical difference between stow
and graft is?


> Unhappily (see previous email), there is no easy solution to that problem.

I always was a source of trouble.


> Maybe a future version of ASDF could selectively disable
> output-translations for some systems?

A simpler workaround for the problem?


Well, now we know how to cause and fix the problem. I'm happy to play
guinea-pig for any patches you come up with; I found the problem, so I may
as well help with its solution :)


Thanks kindly,
James

From: "James Fleming" <lisp@...>
Subject: [Sbcl-help] cl-fad, sb-grovel and failure to compile constants.lisp-temp
Date: Wed, 18 Aug 2010 21:49:32 +1000 (EST)

> The symptom: sb-grovel's def-to-lisp fails to compile at line 244, where
> it tries to evaluate (compile-file tmp-constants :output-file output-file)
> after defining 'tmp-constants in this manner:
> (let ((tmp-constants (merge-pathnames #p"constants.lisp-temp"
>                                          real-output-file)))

I've run into this problem as well:

darcs get http://common-lisp.net/project/clbuild/clbuild
cd clbuild
chmod a+x clbuild
./clbuild update sbcl
./clbuild compile-implementation sbcl
./clbuild lisp --eval "(progn (print (lisp-implementation-version)) (asdf:oos 'asdf:load-op :sb-posix))"

"1.0.42.52" 

debugger invoked on a SB-INT:SIMPLE-FILE-ERROR in thread #<THREAD
                                                           "initial thread" RUNNING
                                                           {1002AAFEA1}>:
  failed to find the TRUENAME of /home/petter/.fasls/sbcl-1.0.42.52-linux-x86-64/tmp/clbuild-test/clbuild/target/lib/sbcl/sb-posix/constants.lisp-temp:
    No such file or directory


Was this issue resolved somehow?

//Petter

> I've run into this problem as well:
...
> Was this issue resolved somehow?

It wasn't resolved, so much as I found a workaround, which was to install
it directly under /usr/local.
I haven't the knowledge of the internals, nor the time (now less than
ever) to wrap my head around the root cause or resolution, unfortunately.


Cheers,
James

On 23 September 2010 11:27, Petter Gustad <sbcl@...> wrote:

>  failed to find the TRUENAME of /home/petter/.fasls/sbcl-1.0.42.52-linux-x86-64/tmp/clbuild-test/clbuild/target/lib/sbcl/sb-posix/constants.lisp-temp:
>    No such file or directory

> Was this issue resolved somehow?

Not that I am aware -- not that I was actively aware of the entire issue.

It seems to be a bad interaction between the way clbuild
builds/installs SBCL, and the way ASDF2 does things.

Cheers,

 -- Nikodemus

On 23 September 2010 15:49, Petter Gustad <clbuild@...> wrote:

> Yes, but I'm little uncertain where it should be fixed. I'll check the
> clbuild mailing list and see if there are any new updates available to
> resolve this issue.

Well, if you can reduce it to a semi-standard SBCL installation, along
the lines of

 sh make.sh
 INSTALL_ROOT=/place sh install.sh

 SBCL_HOME=/place/lib/sbcl/ /place/bin/sbcl --no-userinit --no-sysinit
 (require :sb-posix) ; this in the repl

then SBCL or ASDF is almost certainly the right place to fix it.

Cheers,

 -- Nikodemus