Kai-Florian Richter writes:
> Intriguided by the promise of significant increases in speed in
> dealing with files and sequences I updated to SBCL 1.0.13 today (this
> is x86 Linux).
> However, it turned out that none of the loaders I have for my various
> projects worked anymore. The problem is related to using logical
There are a lot of things going on here. Some of what you're seeing are
consequences of changes I made that I consider to be improvements and/or
corrections (that is, if the exact code you show below used to work, it
was because of bugs in older SBCL versions, though bugs of which I was
unaware when I rewrote LOAD); some of the problems you're seeing are
consequences of you doing questionable things with logical pathnames;
and some of them have to do with flaws in the pathnames system itself.
> More precisely, apparently the newest version of SBCL expects to have
> explicit information about file-type and file-version, i.e.
> pathname-type and pathname-version must be defined. It seems there are
> no (sensible) default values for them (anymore?). At least not when
> translating logical pathnames.
This is an incorrect interpretation of what's happening. I explain below.
> As an example, consider the following:
> In the SBCL init file, logical pathname translations are defined.
> (setf (logical-pathname-translations "obscure-project")
> '((";**;*" "/home/user/projects/obscure-project/**/")
> ("**;*" "/home/user/projects/obscure-project/**/")))
> Up to version 1.0.12 of SBCL, the following worked just fine:
> (load #p"obscure-project:;bar;foo")
> This would load /home/user/projects/obscure-project/bar/foo.fasl if a
> fasl exists, otherwise foo.lisp from the same directory gets loaded.
This was incorrect behavior on SBCL's part. Please keep reading.
> Now, in 1.0.13, it fails
> The pathname gets correctly translated:
> (translate-logical-pathname #p"obscure-project:;bar;foo") =>
This is a correct translation, but it's not the name of the file that we
should be trying to open when you supply #P"obscure-project:;bar;foo" to
> probe-file, however, returns NIL.
This is correct behavior: there is no such file.
> Accordingly, the load command fails with a "file not found"
> error. There is no default rule for setting file-type, it seems.
This is an incorrect assessment of what's happening.
> Adding a file type does not help, however.
> (load #p"obscure-project:;bar;foo.lisp")
> error opening #P"OBSCURE-PROJECT:HOME;USER;BAR;FOO.LISP.NEWEST":
> No such file or directory
> [Condition of type SB-INT:SIMPLE-FILE-ERROR]
Look at the pathname in the error message: it's a logical pathname with
host "OBSCURE-PROJECT" and directory "HOME;USER;BAR;". Examine the
translation of this pathname:
Notice that that file does not exist.
What's going on here is that whenever we perform an operation that
accepts a pathname and operates on the file system (i.e., LOAD,
COMPILE-FILE, OPEN, PROBE-FILE, TRUENAME, FILE-AUTHOR, FILE-WRITE-DATE,
DELETE-FILE, RENAME-FILE, DIRECTORY, ENSURE-DIRECTORIES-EXIST), we are
supposed to call MERGE-PATHNAMES on the argument before accessing the
file system (see CLHS 19.2.3). That is,
(setf *default-pathname-defaults* (user-homedir-pathname))
(with-open-file (f "foo" :if-does-not-exist :create)
We do this whether the pathname is a logical pathname or a physical
pathname; in particular, we call MERGE-PATHNAMES before
(setf *default-pathname-defaults* (merge-pathnames
(make-pathname :type "LISP"
=> #<LOGICAL-PATHNAME (with no namestring)
:HOST #<SB-KERNEL:LOGICAL-HOST "SYS">
:DIRECTORY (:ABSOLUTE "SRC" "CODE")
(with-open-file (f (make-pathname :host "SYS" :name "TARGET-LOAD"))
(values (pathname f)
What's happening in your case is that you're using a logical pathname
whose directory is (:RELATIVE "BAR"), but your
*DEFAULT-PATHNAME-DEFAULTS* has a directory of (:ABSOLUTE "home" "user")
give or take lettercase, and so the MERGE-PATHNAMES implicit in OPEN
produces the logical pathname in the error message, and that logical
pathname's translation does not name any file.
As to why this "worked" in previous versions: the old code for LOAD did
probing and defaulting and merging before calling OPEN, and so LOAD and
OPEN were not consistent:
(setf *default-pathname-defaults* (user-homedir-pathname))
(with-open-file (f (ensure-directories-exist "/tmp/dir/foo.lisp")
:direction :output :if-exists :supersede)
(print '(print *load-pathname*) f)
(print '(print *load-truename*) f)
(setf (logical-pathname-translations "FOO")
=> ((";**;*" "/tmp/**/") ("**;*" "/tmp/**/"))
!! debugger invoked on a SB-INT:SIMPLE-FILE-ERROR:
no translation for #P"FOO:HOME;KREUTER;DIR;FOO.LISP.NEWEST"
The new version of LOAD is relies on OPEN to do all merging and
translating, and so it should turn out that except for the type
defaulting that LOAD does when the argument pathname's type is NIL and
no such file exists, LOAD and OPEN should agree now.
(Notice also that the old load bound *LOAD-PATHNAME* incorrectly too; it
should not have been translated.)
> Apparently, SBCL assumes as default that there is a versioning
> file-system and each file has a version number, including "newest",
> rather then having "nil" (no versioning) as default assumption.
SBCL makes no such assumption per se. However, we parse Unix and
Windows namestrings not ending in a directory delimiter into pathnames
whose version is :NEWEST (I shall say more about :NEWEST below). What
is your *DEFAULT-PATHNAME-DEFAULTS*? I suspect the :NEWEST is being
merged in from that.
> Is this a bug or intended behavior? I think it's nice to have an
> automatic selection of ".fasl if exist, else .lisp". And having "newest"
> as default value for pathname-version doesn't seem to make much sense
It is an open empirical question whether it's possible to have a
pathnames implementation that is simultaneously sensible and conforming,
or whether the pathnames specification in the standard is inherently
nonsensical. (Additionally, there are some details where the pathnames
specification is inconsistent with itself, and so it becomes hard to
judge which details it's okay to get wrong.)
In the present case, there is no specification of what the version
component should be for a pathname corresponding to a file system with
no versions, but there are requirements on the value of the version
component in various places: PARSE-NAMESTRING is required to return a
pathname whose version is NIL when the namestring is "", and
USER-HOMEDIR-PATHNAME is required to return a pathname whose version is
NIL. But an implementation is required to accept :NEWEST as a version
component to MAKE-PATHNAME, though MAKE-PATHNAME is permitted to
canonicalize pathname components at construction-time. However, :NEWEST
and NIL are distinct for MERGE-PATHNAMES, for OPEN, for
similarity-as-a-constant and so for printing pathnames readably,
possibly for EQUAL, possibly for namestringification, probably for the
special case of ENOUGH-NAMESTRING, and maybe more. So I think it might
turn out that even if the file system doesn't support versions, the
implementation has to support a distinct :NEWEST value for versions,
since programs might depend on the "algebra" of pathnames treating
:NEWEST differently from NIL: that is, it might turn out that a
pathnames system implementation is obliged to distinguish pathname
components that have no meaning for the host file system, which is why I
suggest that a conforming pathnames system implementation might be
required to be nonsensical.
> On the good side, finally my loaders should be in a state that is OS-
> and lisp-distribution independent :)
Well, I think the required-to-be-portable uses of logical pathnames and
logical pathname namestrings are extremely limited. I've listed some
problems of which I am aware below; don't bother reading it unless
you've a strong stomach.
== Portability problems having to do with logical pathnames ==
First, note that logical pathnames with a relative directory are quite
likely not to work reliably across implementations unless you ensure
that *DEFAULT-PATHNAME-DEFAULTS* is a suitable logical pathname. In
general, I'd advise you to avoid relative logical pathnames entirely.
Second, the translation of a logical pathname with NIL for any component
is not required to denote a filename, because a physical pathname with
NIL for any component need not be usable as a filename: there are hints
of this, for example, in the statements about NIL in the specifications
for DELETE-FILE and RENAME-FILE. Here is a run-down of the cases:
* NIL for the device or directory, or a directory that's a list whose
first element is :RELATIVE: while a Lisp running Windows or Unix with
access only to the local host's file system(s) will have one or more
"current working directories" with respect to which the operating
system will resolve non-absolute filenames during system calls, a Lisp
that accesses a file system via some remote file access protocol
cannot be assumed to have a current working directory, and so a Lisp
pathname whose device or directory is NIL, or whose directory is a
list starting with :RELATIVE is not required to be usable as a
* NIL for the name: I think it's relatively uncontentious that a
physical pathname with NIL for the name and non-NIL type or version is
unlikely to denote a filename (it doesn't on SBCL, for instance).
Further nothing in Common Lisp requires an implementation to treat
directories as files, and so a pathname with NIL for all of the name,
type, and version is not required to be useable as a file (OPEN'ed,
RENAME-FILE'd, PROBE-FILE'd, etc.). As I understand things,
directories really weren't files under some historical file systems.
* NIL for the type or version: a physical pathname with a NIL type or
version need not denote a filename on a file system that requires
filenames to have a type or a version. Even under Unix and Windows
systems, whose filenames have no real versions and are not required to
have extensions, a conforming Lisp implementation might represent
filenames with no extension using :UNSPECIFIC for the type, and say
that a pathname having NIL for the type does not denote a filename.
So even if a logical pathname with NIL components has a translation to a
physical pathname, I think it turns out that the physical pathname is
not required to be usable as an object that denotes a filename. (Notice
that it's not possible to write a set of logical pathname translations
that treats NIL specially, because logical pathname translations are
selected as if by PATHNAME-MATCH-P, and PATHNAME-MATCH-P fills in NIL
components in the wildcard pathname with :WILD.)
Third, I don't see anything in the standard that requires an
implementation to support wild physical pathnames, so I don't think that
a wild logical pathname is required to have a translation.
Consequently, I think that DIRECTORY on a wild logical pathname has no
Fourth, note that logical pathname translation is defined in terms of
PATHNAME-MATCH-P, and PATHNAME-MATCH-P is defined to fill in NIL
components, but implementations differ about whether the absence of a
directory parses to NIL or (:ABSOLUTE), so logical pathname namestrings
with no directory might parse to pathnames that translate quite
differently under different implementations (I think both
interpretations might be conforming and that the standard is
Fifth, a logical pathname namestring that has a semicolon followed by no
directories might parse to a pathname with directory (:RELATIVE), or it
might parse to a pathname with directory NIL; these have identical
properties for MERGE-PATHNAMES, but not for PATHNAME-MATCH-P.
Consequently, logical pathname namestrings with a semicolon and no
following directories don't necessarily work portably. (Or it might be
that parsing any namestring to a pathname with (:RELATIVE) for the
directory is non-conforming: 22.214.171.124.3 says that (:RELATIVE) "is not
used", but doesn't say what "is not used" means: signal an error?
canonicalize to NIL?)
Sixth, while logical pathname namestring parsing is supposed to
canonicalize alphabetic characters to upper case, the syntax of logical
pathname namestrings specifies that alphabetic characters must be
uppercase, and so an implementation is not required to recognize a
string containing any lowercase characters as a logical pathname
Seventh, it's not clear what PARSE-NAMESTRING is supposed to do when
JUNK-ALLOWED is true and a string is being parsed as a logical pathname
namestring but contains some bad characters. Does the parse return NIL?
Does it return whatever could have been parsed up to the bad character,
or up to the last delimiter before the last bad character?
And so on. So I think the upshot of this is that the only
required-to-be-portable uses of logical pathnames are really quite
limited, and that it's best to think of the logical pathname "file
system" as one that implicitly requires all pathnames to have non-NIL
components for all fields and a directory starting with :ABSOLUTE, and
whose namestring syntax requires all-uppercase alphabetics, and
explicit, preferably absolute directories.