Steel Bank Common Lisp

On 30 September 2010 10:37, Didier Verna <didier@...> wrote:

> it seems that when you use "sbcl --script" (or at least create an SBCL
> shell script with it), the file
> ~/.config/common-lisp/source-registry.conf is read by the script.
>
> In fact, I happen to like this behavior but I was surprised by it
> because it appears in contradiction with the semantics of --script,
> which is to *not* read any user init file.

ASDF2 does the reading, so --script has no effect.

The most SBCL can do is to provide a way to detect if it has been
invoked with --script.

Cheers,

 -- Nikodemus

On 9/30/10 Sep 30 -3:59 AM, Nikodemus Siivola wrote:
> On 30 September 2010 10:37, Didier Verna <didier@...> wrote:
> 
>> it seems that when you use "sbcl --script" (or at least create an SBCL
>> shell script with it), the file
>> ~/.config/common-lisp/source-registry.conf is read by the script.
>>
>> In fact, I happen to like this behavior but I was surprised by it
>> because it appears in contradiction with the semantics of --script,
>> which is to *not* read any user init file.
> 
> ASDF2 does the reading, so --script has no effect.
> 
> The most SBCL can do is to provide a way to detect if it has been
> invoked with --script.

Dumb question:  "can do"?  Does that mean "SBCL could be modified to
provide a way to detect if it has been invoked with --script" or "SBCL
now provides a way to detect if it has been invoked with --script"?

thanks,
r

Nikodemus Siivola wrote:

> ASDF2 does the reading, so --script has no effect.

  Yeah, I figured that much. But my concern was more along the lines of
whether the current behavior is going to stay, or ...

> The most SBCL can do is to provide a way to detect if it has been
> invoked with --script.

  ... whether something like this will be used in future SBCL to prevent
ASDF from reading the config file, which again would seem more coherent
to me.

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

On 30 September 2010 12:51, Didier Verna <didier@...> wrote:
> Nikodemus Siivola wrote:
>
>> ASDF2 does the reading, so --script has no effect.
>
>  Yeah, I figured that much. But my concern was more along the lines of
> whether the current behavior is going to stay, or ...
>
>> The most SBCL can do is to provide a way to detect if it has been
>> invoked with --script.
>
>  ... whether something like this will be used in future SBCL to prevent
> ASDF from reading the config file, which again would seem more coherent
> to me.
>
You could open a bug for ASDF to change its behavior, but what exactly
do you expect ASDF to do? Skip the user-configured systems but honor
the system-configured systems? Honor or not honor the
CL_SOURCE_REGISTRY environment variable? Start with builtin defaults
only? Start with defaults somehow configured at the time that SBCL is
compiled? Whichever way you put it, I don't see how you can justify
any deviation from the normal behaviour based on the sole --script
argument. How do you distinguish between scripts written by the user
and scripts that come with the system?

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
"If men are good, you don't need government; if men are evil or ambivalent,
you don't dare have one." — Robert LeFevre

On 9/30/10 Sep 30 -12:01 PM, Faré wrote:
> On 30 September 2010 12:51, Didier Verna <didier@...> wrote:
>> Nikodemus Siivola wrote:
>>
>>> ASDF2 does the reading, so --script has no effect.
>>
>>  Yeah, I figured that much. But my concern was more along the lines of
>> whether the current behavior is going to stay, or ...
>>
>>> The most SBCL can do is to provide a way to detect if it has been
>>> invoked with --script.
>>
>>  ... whether something like this will be used in future SBCL to prevent
>> ASDF from reading the config file, which again would seem more coherent
>> to me.
>>
> You could open a bug for ASDF to change its behavior, but what exactly
> do you expect ASDF to do? Skip the user-configured systems but honor
> the system-configured systems? Honor or not honor the
> CL_SOURCE_REGISTRY environment variable? Start with builtin defaults
> only? Start with defaults somehow configured at the time that SBCL is
> compiled? Whichever way you put it, I don't see how you can justify
> any deviation from the normal behaviour based on the sole --script
> argument. How do you distinguish between scripts written by the user
> and scripts that come with the system?

This is a good point, but the --script argument is supposed to make
behavior independent of user-specific configuration.

It occurs to me that /all/ lisp implementations I know of are going to
have some flavor of this problem (e.g., allegro has arguments for
suppressing user- and site-specific configurations).

This is probably a place where it would be useful to open a discussion
with the implementers.  I figure the best thing for ASDF to do is to
expose an API that allows the implementations to disable user- and site-
specific configuration at will.

Probably we should move this to ASDF launchpad ticket, when we figure
out what is The Right Thing.

note that this is a complication that is unique to ASDF2.  In ASDF1, all
configuration is done in the user's normal lisp initialization, so
user-configuration suppression will honor the implementations'
configuration control API.

Best,
r

On 30 September 2010 13:09, Robert Goldman <rpgoldman@...> wrote:
> note that this is a complication that is unique to ASDF2.  In ASDF1, all
> configuration is done in the user's normal lisp initialization, so
> user-configuration suppression will honor the implementations'
> configuration control API.
>
Whatever the script is doing that was ensuring the systems will be
found despite lack of configuration in ASDF1 will keep making things
work in ASDF2; the user configuration file may be read, but whatever
makes ASDF1 work will make those settings irrelevant.

But yes, if there were a protocol to specify that we should skip user
and/or system configuration, that would be nice. Problem being: if
ASDF is to remain an optional component, then we'll need some
non-portable thing that fetches this information differently from each
and every of the supported implementations. Sigh.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
The reason truth is stranger than fiction is that fiction has to make sense.

From: Robert Goldman <rpgoldman@...>
Subject: Re: [Sbcl-devel] --script and ASDF 2
Date: Thu, 30 Sep 2010 12:09:18 -0500

> This is probably a place where it would be useful to open a discussion
> with the implementers.  I figure the best thing for ASDF to do is to
> expose an API that allows the implementations to disable user- and site-
> specific configuration at will.

You can do this to some extent by modifying
asdf:*system-definition-search-functions* and calling
asdf:initialize-source-registry

I recently had some problems with ASDF loading old system wide
libraries on my machine despite that I was running a local SBCL in my
home directory with both the--no-userinit and --no-sysinit given on
the command line.

The solution was to call initialize-source-registry (thanks to Zach
for pointing it out to me):

(asdf:initialize-source-registry 
   '(:source-registry :ignore-inherited-configuration))


Petter

On 9/30/10 Sep 30 -12:56 PM, Faré wrote:
> On 30 September 2010 13:09, Robert Goldman <rpgoldman@...> wrote:
>> note that this is a complication that is unique to ASDF2.  In ASDF1, all
>> configuration is done in the user's normal lisp initialization, so
>> user-configuration suppression will honor the implementations'
>> configuration control API.
>>
> Whatever the script is doing that was ensuring the systems will be
> found despite lack of configuration in ASDF1 will keep making things
> work in ASDF2; the user configuration file may be read, but whatever
> makes ASDF1 work will make those settings irrelevant.
> 
> But yes, if there were a protocol to specify that we should skip user
> and/or system configuration, that would be nice. Problem being: if
> ASDF is to remain an optional component, then we'll need some
> non-portable thing that fetches this information differently from each
> and every of the supported implementations. Sigh.

Question:  can't we simply support an api that toggles the two types of
configuration on and off, and then leave it to the implementations to
call this API when processing command-line arguments?

I'm not sure about the internals of this configuration stuff, so I might
be wrong.

r

On 30 September 2010 14:01, Robert Goldman <rpgoldman@...> wrote:
> On 9/30/10 Sep 30 -12:56 PM, Faré wrote:
>> But yes, if there were a protocol to specify that we should skip user
>> and/or system configuration, that would be nice. Problem being: if
>> ASDF is to remain an optional component, then we'll need some
>> non-portable thing that fetches this information differently from each
>> and every of the supported implementations. Sigh.
>
> Question:  can't we simply support an api that toggles the two types of
> configuration on and off, and then leave it to the implementations to
> call this API when processing command-line arguments?
>
That only works if the implementation does the loading of ASDF. Should
--script always require ASDF? Or something like --script --asdf or
--asdf-script?
Meh.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
It’s amazing that the amount of news that happens in the world everyday
always just exactly fits the newspaper.

Faré <fahree@...> wrote:

> On 30 September 2010 12:51, Didier Verna <didier@...> wrote:
>> Nikodemus Siivola wrote:
>>
>>> ASDF2 does the reading, so --script has no effect.
>>
>>  Yeah, I figured that much. But my concern was more along the lines of
>> whether the current behavior is going to stay, or ...
>>
>>> The most SBCL can do is to provide a way to detect if it has been
>>> invoked with --script.
>>
>>  ... whether something like this will be used in future SBCL to prevent
>> ASDF from reading the config file, which again would seem more coherent
>> to me.
>>
> You could open a bug for ASDF to change its behavior, but what exactly
> do you expect ASDF to do?

  I don't know, really :-) Like I said, I happen to like the current
behavior, because it makes my life simpler, but there is something not
totally coherent wrt the way SBCL used to behave with ASDF 1. Let me
explain:

- with SBCL / ASDF 1, the only way a user could configure the source
registry was to set *central-registry* from a lisp file, typically
.sbclrc. So when using scbl --script, that file is skipped, and so is
the ASDF configuration.

- now, the user would set the source registry in the config file, but
then sbcl --script will still read it (because in fact, it is ASDF that
reads it).

So the consequence is that from one version of SBCL to another, there
will be a change in behavior. I just thought the issue was important
enough to be raised, eventhough I'm not totally clear on which behavior
makes more sense.

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

Robert Goldman <rpgoldman@...> wrote:

> This is probably a place where it would be useful to open a discussion
> with the implementers. I figure the best thing for ASDF to do is to
> expose an API that allows the implementations to disable user- and
> site- specific configuration at will.

  Yup.

> note that this is a complication that is unique to ASDF2. In ASDF1,
> all configuration is done in the user's normal lisp initialization, so
> user-configuration suppression will honor the implementations'
> configuration control API.

  Exactly.

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

On 9/30/10 Sep 30 -1:06 PM, Faré wrote:
> On 30 September 2010 14:01, Robert Goldman <rpgoldman@...> wrote:
>> On 9/30/10 Sep 30 -12:56 PM, Faré wrote:
>>> But yes, if there were a protocol to specify that we should skip user
>>> and/or system configuration, that would be nice. Problem being: if
>>> ASDF is to remain an optional component, then we'll need some
>>> non-portable thing that fetches this information differently from each
>>> and every of the supported implementations. Sigh.
>>
>> Question:  can't we simply support an api that toggles the two types of
>> configuration on and off, and then leave it to the implementations to
>> call this API when processing command-line arguments?
>>
> That only works if the implementation does the loading of ASDF. Should
> --script always require ASDF? Or something like --script --asdf or
> --asdf-script?
> Meh.

Ugh.  I don't think --asdf-script is going to be appropriate.

Sigh.  The only thing I can think of is to have some variable(s), in
some known and pre-existing package, control this.

This pre-configuration issue is one where I have always bonked my head
on CL's package system.  How do you permit the user to configure a
system (in a package) that's not yet loaded?  Emacs lisp doesn't have
this issue, because it doesn't have packages, so simply using SETQ and
DEFVAR correctly does The Right Thing.

But I've never figured out anything better than either

(1) crapping up CL-USER with variables for configuration or
(2) crapping up the *FEATURES*.

In this case, I think *FEATURES* is pretty inappropriate, so maybe we
need to have

CL-USER::*ASDF-LOAD-USER-CONFIG-P* and CL-USER::*ASDF-LOAD-SYSTEM-CONFIG-P*

ick.

r

On 30 September 2010 21:01, Robert Goldman <rpgoldman@...> wrote:

> Question:  can't we simply support an api that toggles the two types
> of configuration on and off, and then leave it to the implementations
> to call this API when processing command-line arguments?

I don't know how: unless the implementation has ASDF already loaded,
it isn't going to process any ASDF configs at startup.

It seems to me that the issue here is really the configuration ASDF
does when it is loaded -- by (REQUIRE :ASDF) or whatever.

In SBCL, for example, --script implies both --no-userinit and
--no-sysinit (among other things). The intention is to make things
less dependent on user specific configurations.

However, being able to find installed libraries is probably not such a
configuration issue.

Prior to ASDF2 for --script to have access to ASDF systems they pretty
much had to be symlinked to ~/.sbcl/systems on
/usr/local/lib/sbcl/systems.

It is not obvious to me that seeing the source-registry is a downside.

However, that said, I wonder how to make sure that ASDF configuration
in general is consistent: consider for example a user who normally
disables output-translations in initfiles. Since --script disables
those initfiles, any ASDF systems used by them will use .cache --
presumably even if there were perfectly good fasl-files in the source
directory already.

Cheers,

 -- Nikodemus

On 9/30/10 Sep 30 -4:09 PM, Nikodemus Siivola wrote:
> On 30 September 2010 21:01, Robert Goldman <rpgoldman@...> wrote:
> 
>> Question:  can't we simply support an api that toggles the two types
>> of configuration on and off, and then leave it to the implementations
>> to call this API when processing command-line arguments?
> 
> I don't know how: unless the implementation has ASDF already loaded,
> it isn't going to process any ASDF configs at startup.

I think that's why we have to use (yuck) a global variable in the
CL-USER package as the "api."  See earlier email.
> 
> It seems to me that the issue here is really the configuration ASDF
> does when it is loaded -- by (REQUIRE :ASDF) or whatever.
> 
> In SBCL, for example, --script implies both --no-userinit and
> --no-sysinit (among other things). The intention is to make things
> less dependent on user specific configurations.
> 
> However, being able to find installed libraries is probably not such a
> configuration issue.

I don't necessarily agree.  It's easy for me to imagine a script that
wants absolute control of what systems are going to be loaded.  E.g., a
script that's distributed with a fixed set of libraries.

> 
> Prior to ASDF2 for --script to have access to ASDF systems they pretty
> much had to be symlinked to ~/.sbcl/systems on
> /usr/local/lib/sbcl/systems.
> 
> It is not obvious to me that seeing the source-registry is a downside.
> 
> However, that said, I wonder how to make sure that ASDF configuration
> in general is consistent: consider for example a user who normally
> disables output-translations in initfiles. Since --script disables
> those initfiles, any ASDF systems used by them will use .cache --
> presumably even if there were perfectly good fasl-files in the source
> directory already.

Ouch.  That suggests that what seemed like a sensible default case for
interactive programmers might be really bad for use with --script.

Unnecessary recompilation seems the least of our worries.  Another
possibility would be inappropriately crushing the user's existing fasls
with fasls compiled under a different, --script-imposed configuration.

best,
r

Robert Goldman <rpgoldman@...> wrote:

> This pre-configuration issue is one where I have always bonked my head
> on CL's package system. How do you permit the user to configure a
> system (in a package) that's not yet loaded? Emacs lisp doesn't have
> this issue, because it doesn't have packages, so simply using SETQ and
> DEFVAR correctly does The Right Thing.

  You delay the configuration until the package is loaded. I even have a
very small library for doing that (cl-rcfiles).

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

> This pre-configuration issue is one where I have always bonked my head
> on CL's package system.  How do you permit the user to configure a
> system (in a package) that's not yet loaded?  Emacs lisp doesn't have
> this issue, because it doesn't have packages, so simply using SETQ and
> DEFVAR correctly does The Right Thing.

(eval-always
  (unless (find-package :foo)
    (make-package :foo)))

(defparameter foo::*var* 42)

(asdf:load-system :foo)

and the :foo lib should use defvar's.

the only issue can be that it may emit warnings. at least SBCL only
warns if a defpackage form contains less stuff than the currently
existing package.

-- 
 attila

On 10/1/10 Oct 1 -3:57 AM, Attila Lendvai wrote:
>> This pre-configuration issue is one where I have always bonked my head
>> on CL's package system.  How do you permit the user to configure a
>> system (in a package) that's not yet loaded?  Emacs lisp doesn't have
>> this issue, because it doesn't have packages, so simply using SETQ and
>> DEFVAR correctly does The Right Thing.
> 
> (eval-always
>   (unless (find-package :foo)
>     (make-package :foo)))
> 
> (defparameter foo::*var* 42)
> 
> (asdf:load-system :foo)
> 
> and the :foo lib should use defvar's.
> 
> the only issue can be that it may emit warnings. at least SBCL only
> warns if a defpackage form contains less stuff than the currently
> existing package.
> 

Is this really safe?

>From the CLHS definition of DEFPACKAGE:

If defined-package-name already refers to an existing package, the
name-to-package mapping for that name is not changed. If the new
definition is at variance with the current state of that package, the
consequences are undefined; an implementation might choose to modify the
existing package to reflect the new definition.

So it looks like your snippet relies on undefined consequences.

Best,
r

Sorry to join the party late.


On Thu, 30 Sep 2010, Robert Goldman wrote:
> On 9/30/10 Sep 30 -4:09 PM, Nikodemus Siivola wrote:
>> In SBCL, for example, --script implies both --no-userinit and
>> --no-sysinit (among other things). The intention is to make things
>> less dependent on user specific configurations.
>>
>> However, being able to find installed libraries is probably not such a
>> configuration issue.
>
> I don't necessarily agree.  It's easy for me to imagine a script that
> wants absolute control of what systems are going to be loaded.  E.g., a
> script that's distributed with a fixed set of libraries.

And such a script should not rely on sbcl being invoked with --script in 
the first place.  It should use a trick similar to Zach's as listed 
earlier.

(asdf:initialize-source-registry
    '(:source-registry :ignore-inherited-configuration))


>> However, that said, I wonder how to make sure that ASDF configuration
>> in general is consistent: consider for example a user who normally
>> disables output-translations in initfiles. Since --script disables
>> those initfiles, any ASDF systems used by them will use .cache --
>> presumably even if there were perfectly good fasl-files in the source
>> directory already.
>
> Ouch.  That suggests that what seemed like a sensible default case for
> interactive programmers might be really bad for use with --script.
>
> Unnecessary recompilation seems the least of our worries.  Another
> possibility would be inappropriately crushing the user's existing fasls
> with fasls compiled under a different, --script-imposed configuration.

Yet another reason a build system should facilitate fasl-only (no sources 
present) loading.  No chance of inappropriate recompilation.

Though presumably a script using the above trick could also set all the 
other asdf parameters as appropriate.

- Daniel

I take your points, Daniel, but mere technical ability to perform some task doesn't seem to me to entail a useful ability to do so. The expedients you propose seem to me likely to push script authoring into the realm of practical infeasibility.

I wish I had a better answer. For reasons related to these I have,  so far, decided (somewhat reluctantly) to stick to *central-registry*.
R

"Daniel Herring" <dherring@...> wrote:

>Sorry to join the party late.
>
>
>On Thu, 30 Sep 2010, Robert Goldman wrote:
>> On 9/30/10 Sep 30 -4:09 PM, Nikodemus Siivola wrote:
>>> In SBCL, for example, --script implies both --no-userinit and
>>> --no-sysinit (among other things). The intention is to make things
>>> less dependent on user specific configurations.
>>>
>>> However, being able to find installed libraries is probably not such a
>>> configuration issue.
>>
>> I don't necessarily agree.  It's easy for me to imagine a script that
>> wants absolute control of what systems are going to be loaded.  E.g., a
>> script that's distributed with a fixed set of libraries.
>
>And such a script should not rely on sbcl being invoked with --script in 
>the first place.  It should use a trick similar to Zach's as listed 
>earlier.
>
>(asdf:initialize-source-registry
>    '(:source-registry :ignore-inherited-configuration))
>
>
>>> However, that said, I wonder how to make sure that ASDF configuration
>>> in general is consistent: consider for example a user who normally
>>> disables output-translations in initfiles. Since --script disables
>>> those initfiles, any ASDF systems used by them will use .cache --
>>> presumably even if there were perfectly good fasl-files in the source
>>> directory already.
>>
>> Ouch.  That suggests that what seemed like a sensible default case for
>> interactive programmers might be really bad for use with --script.
>>
>> Unnecessary recompilation seems the least of our worries.  Another
>> possibility would be inappropriately crushing the user's existing fasls
>> with fasls compiled under a different, --script-imposed configuration.
>
>Yet another reason a build system should facilitate fasl-only (no sources 
>present) loading.  No chance of inappropriate recompilation.
>
>Though presumably a script using the above trick could also set all the 
>other asdf parameters as appropriate.
>
>- Daniel

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

I think my stance on the environmental guarantees provided by --script is that:

 1. SBCL initfiles are not processed.

 2. SBCL contribs should be available.

 3. If a script uses non-SBCL provided ASDF systems,
    then the onus to make them visible without initfiles
    is split between the user of the script and the writer.

The only sensible alternative to this that I can see is to split SBCL
initfiles into "interactive" and "non-interactive" initfiles, and make
--script choose use the latter.

Cheers,

 -- Nikodemus

On 10/10/10 Oct 10 -9:20 AM, Nikodemus Siivola wrote:
> I think my stance on the environmental guarantees provided by --script is that:
> 
>  1. SBCL initfiles are not processed.
> 
>  2. SBCL contribs should be available.
> 
>  3. If a script uses non-SBCL provided ASDF systems,
>     then the onus to make them visible without initfiles
>     is split between the user of the script and the writer.
> 
> The only sensible alternative to this that I can see is to split SBCL
> initfiles into "interactive" and "non-interactive" initfiles, and make
> --script choose use the latter.
> 
> Cheers,
> 
>  -- Nikodemus

What about my proposal that the implementation be able to set two
variables that would indicate to ASDF whether or not to read user and
site initialization when output translations are initialized?

The SBCL code that processes the --script arguments (or other
command-line arguments) could handle these accordingly.

best,
r

On 10/10/10 Oct 10 -12:13 PM, Robert Goldman wrote:
> On 10/10/10 Oct 10 -9:20 AM, Nikodemus Siivola wrote:
>> I think my stance on the environmental guarantees provided by --script is that:
>>
>>  1. SBCL initfiles are not processed.
>>
>>  2. SBCL contribs should be available.
>>
>>  3. If a script uses non-SBCL provided ASDF systems,
>>     then the onus to make them visible without initfiles
>>     is split between the user of the script and the writer.
>>
>> The only sensible alternative to this that I can see is to split SBCL
>> initfiles into "interactive" and "non-interactive" initfiles, and make
>> --script choose use the latter.
>>
>> Cheers,
>>
>>  -- Nikodemus
> 
> What about my proposal that the implementation be able to set two
> variables that would indicate to ASDF whether or not to read user and
> site initialization when output translations are initialized?
> 
> The SBCL code that processes the --script arguments (or other
> command-line arguments) could handle these accordingly.

The above really should have said INPUT LOCATIONS, not output...

This does raise an issue, though --- even the venerable old
asdf-binary-locations potentially could interact poorly with processing
like --script, scattering binaries where they were not expected in
normal processing....

cheers,
r

Maybe it's time to standardize a common-lisp-configuration package
that could eventually be provided by all implementations, with a few
utility functions and variables, allowing to communicate with the
environment.

Candidate variables:
*use-user-configuration*
*use-system-configuration*

Candidate functions:
command-line-arguments ()
argv0 ()
getenv (name) ==> get the environment variable, or NIL
setenv (name value) ==> set the environment variable, or unset (if
value is NIL).
callsym (package symbol &rest arguments) ==> call that function, or error
getsym (package symbol &optional default) ==> value T NIL, or default NIL reason

Maybe also some macros to enable or disable certain semantically
significant "optimizations", like proper tail-calls or slot
type-checking.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Demand Truly Free Health Care: where doctors need neither pay, food, sleep
nor training, and hospitals grow on trees!

On 10/10/10 22:37, Faré wrote:
> Maybe it's time to standardize a common-lisp-configuration package
> that could eventually be provided by all implementations, with a few
> utility functions and variables, allowing to communicate with the
> environment.
>
> Candidate variables:
> *use-user-configuration*
> *use-system-configuration*
>
> Candidate functions:
> command-line-arguments ()
> argv0 ()
> getenv (name) ==>  get the environment variable, or NIL
> setenv (name value) ==>  set the environment variable, or unset (if
> value is NIL).
> callsym (package symbol&rest arguments) ==>  call that function, or error
> getsym (package symbol&optional default) ==>  value T NIL, or default NIL reason
>
> Maybe also some macros to enable or disable certain semantically
> significant "optimizations", like proper tail-calls or slot
> type-checking.

As a user, i see value in this. Could this solve the 
undefined-package-configuration problem too ? There must be something to 
generalize in a pragmatic way about configurations and their loading in 
various contexts.

I'm not sure where to discuss this, maybe this is a broader subject than 
just sbcl-devel.

-- 
Thomas de Grivel
http://b.lowh.net/billitch

On 10/10/10 Oct 10 -3:37 PM, Faré wrote:
> Maybe it's time to standardize a common-lisp-configuration package
> that could eventually be provided by all implementations, with a few
> utility functions and variables, allowing to communicate with the
> environment.
> 
> Candidate variables:
> *use-user-configuration*
> *use-system-configuration*
> 
> Candidate functions:
> command-line-arguments ()
> argv0 ()
> getenv (name) ==> get the environment variable, or NIL
> setenv (name value) ==> set the environment variable, or unset (if
> value is NIL).
> callsym (package symbol &rest arguments) ==> call that function, or error
> getsym (package symbol &optional default) ==> value T NIL, or default NIL reason
> 
> Maybe also some macros to enable or disable certain semantically
> significant "optimizations", like proper tail-calls or slot
> type-checking.
> 
> [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
> Demand Truly Free Health Care: where doctors need neither pay, food, sleep
> nor training, and hospitals grow on trees!

I like this.  I would only propose that it be narrowed in scope.  The
functions CALLSYM and GETSYM are both possible already with portable CL,
so I'd propose they be dropped, to make this minimal.

I also confess that I have misgivings about the macros for enabling and
disabling optimizations --- I get why one would want this, but (1) I
think they may be provided in different enough ways that this would
raise the bar for implementers and (2) they seem semantically distinct
from the others, which all seem like load-time and execute-time (as
opposed to compile-time) items.  So I would be inclined to put them in
some other library.

Maybe someone should set up a BoF session at ILC for things like this?
Maybe a portability session?

best,
r

Nikodemus Siivola wrote:

> In SBCL, for example, --script implies both --no-userinit and
> --no-sysinit (among other things). The intention is to make things
> less dependent on user specific configurations.

  In fact, I think this is the problem. Because --script implies
--no-userinit and --no-sysinit, there is no simple way to distribute
SBCL shell scripts that would purposedly depend on the target system or user
configuration. If you want to do that, you're back the old way, which is
to use #!/path/to/sbcl without the --script option, and require the now
obsolete shebang machinery in the target user's .sbclrc.

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

On 12 October 2010 18:20, Didier Verna <didier@...> wrote:
> Nikodemus Siivola wrote:
>
>> In SBCL, for example, --script implies both --no-userinit and
>> --no-sysinit (among other things). The intention is to make things
>> less dependent on user specific configurations.
>
>  In fact, I think this is the problem. Because --script implies
> --no-userinit and --no-sysinit, there is no simple way to distribute
> SBCL shell scripts that would purposedly depend on the target system or user
> configuration. If you want to do that, you're back the old way, which is
> to use #!/path/to/sbcl without the --script option, and require the now
> obsolete shebang machinery in the target user's .sbclrc.

Hm. I have to say I never considered that someone would want to
distribute a --script non-trivial enough to demand non-contrib
libraries.

You can, of course do

 (load (merge-pathnames ".sbclrc" (user-homedir-pathname)))

in a script, but that isn't the pretties thing ever, no.

I'm starting to think we really want to split initialization files
into "interactive" and "script" versions.

Cheers,

 -- Nikodemus

Nikodemus Siivola wrote:

> Hm. I have to say I never considered that someone would want to
> distribute a --script non-trivial enough to demand non-contrib
> libraries.

  Consider a script that needs to load (ASDF sense) the package in which
it is distributed. I do this in Declt and Clon for instance, in order to
be able to generate their respective reference manuals.

In order to make the user's life easy, you need to let him untar the
package wherever he wants (including in a non-standard directory that
ASDF sees because of his personal configuration) and have the script
find the package independently of that location.

Now that we have ASDF 2, this is going to work for users hacking their
.config/common-lisp/source-registry because even with --script, this
file is read in. However, for ASDF 1 OR ASDF 2 with a call to the API in
.sbclrc, this won't.


> You can, of course do
>
>  (load (merge-pathnames ".sbclrc" (user-homedir-pathname)))
>
> in a script, but that isn't the pretties thing ever, no.

  Exactly.


> I'm starting to think we really want to split initialization files
> into "interactive" and "script" versions.

  Or perhaps we need different script options, like --standalone,
--system-script and --user-script ?

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

On 12 October 2010 12:39, Didier Verna <didier@...> wrote:
>
>  Consider a script that needs to load (ASDF sense) the package in which
> it is distributed. I do this in Declt and Clon for instance, in order to
> be able to generate their respective reference manuals.
>
Or you could be using cl-launch. It can handle ASDF configuration for
you, and works not just with SBCL, but also CCL, CLISP, ECL, etc.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
The last ten percent of any reform is the most difficult to
achieve. Moreover, it is often harmful.  — John McCarthy

This is all starting to sound really complicated, and that seems unnecessary.

Let's take Python as an example of what makes sense to me: upon startup, it sets its sys.path (where to load packages from) to a standard set of paths: first, the contents of the PYTHONPATH environment variable. Then, the implementation-internal paths, systemwide paths, and user-specific paths. Those are not configurable with a config file, they're set as part of the installation.

So, normally, users will either install packages in the site-wide location, or the user-specific location. And things will just work, without any configuration.

If users install things into another private non-default location, they can set PYTHONPATH to point to it, or else have their program simply modify sys.path as the first thing it does, prior to loading packages.

That seems like a sane model, that ASDF and SBCL should follow.

James

Abstract:  SBCL does not need to change --script for this issue.  The main 
problem is learning how to use ASDF 2, with some difficulty reliably 
bootstrapping it.


On Tue, 12 Oct 2010, James Y Knight wrote:

> This is all starting to sound really complicated, and that seems unnecessary.

Agreed on that point.  The proposed features were not sounding clear or 
orthogonal.


> Let's take Python ... a standard set of paths: first, the contents of 
> the PYTHONPATH environment variable. Then, the implementation-internal 
> paths, systemwide paths, and user-specific paths. Those are not 
> configurable with a config file, they're set as part of the 
> installation.
>
> So, normally, users will either install packages in the site-wide 
> location, or the user-specific location. And things will just work, 
> without any configuration.

This is heavily inspired by the C loader system, a proven technology.  I 
suggest that everyone read `man ld.so` to see what customization points 
it provides.

http://www.kernel.org/doc/man-pages/online/pages/man8/ld.so.8.html


Then there's the super-simple approach:  what is so evil with embedding 
paths into the script?  Prior art: shebang lines, RPATH, crontabs 
and classic CGI scripts, ...


Concluding points:

- There should be a mechanism for specifying library paths, independent of 
the general "sbclrc" support.  ASDF 2 provides this.

- There should be ways to specify more specific paths for a particular 
user or script.  ASDF 2 and in-script paths provide most/all of this[1].

- Sometimes a pig really is just a pig.  There is no magic solution to 
library paths, but common patterns keep appearing becuase they can handle 
most cases nicely.  Many of the exceptional cases really are asking too 
much (such as over-the-shoulder, intuition-guided inconsistency).

- We do need a guaranteed way of having a "library loader" properly 
configured when a "script"/"standalone app" starts.  Since ASDF itself is 
just a library, there is a potential bootstrapping issue here.  That is 
eased by getting some version of ASDF 2 baked in to all the impls[2].

- Its about time to start a "how to configure ASDF" thread on asdf-devel, 
with guidelines on what goes where, when to use the different hooks, etc. 
I'm not quite confident, but getting close.

Later,
Daniel

[1] Big thanks to Fare for implementing a better config mechanism for 
ASDF, even though we're still not 100% sure how to use it.  It seems that 
SAVE-LISP-AND-DIE and fasl concatenation are the "static linking" options 
for SBCL.  There are a few remaining gaps.
- LD_PRELOAD and RTLD_NEXT
   PRELOAD forces an exact library to be used.  Think of NEXT as :around
   for libraries.  Great power tools.
- LD_TRACE_LOADED_OBJECTS, LD_DEBUG, and LD_VERBOSE
   Its nice to have the loader say what its loading, from where, etc.
   Maybe ASDF has stuff for this? Even when default streams are redirected?
- RPATH info embedded in libs
   helps find dependencies in unusual places (esp. customized forms of
   common libs)

[2] Is upgrading SBCL's ASDF contrib as simple as overwriting 
lib/sbcl/asdf/asdf.fasl with a newer version?  I was surprised that ASDF 
did not add a safeguard to prevent an accidental downgrade of the library. 
This complicates efforts of tools like quicklisp and libcl to "always 
provide a current copy".

P.S.  Bonus OT:  The core feature should be a "library loader" with 
semantics similar to ld.so or the much maligned, underspecified and 
underimplemented REQUIRE.  ASDF has too many features (pseudo-random 
recompilation wrt discrete apps) and requires complete control of the 
world (no easy way for "some loader file" to satisfy an ASDF dependency) 
to be the universal foundation for CL loading.

On Wed, Oct 13, 2010 at 12:34:36AM -0400, Daniel Herring wrote:
> [1] Big thanks to Fare for implementing a better config mechanism for 
> ASDF, even though we're still not 100% sure how to use it.  It seems that 
> SAVE-LISP-AND-DIE and fasl concatenation are the "static linking" options 
> for SBCL.  There are a few remaining gaps.
> - LD_PRELOAD and RTLD_NEXT
>    PRELOAD forces an exact library to be used.  Think of NEXT as :around
>    for libraries.  Great power tools.
> - LD_TRACE_LOADED_OBJECTS, LD_DEBUG, and LD_VERBOSE
>    Its nice to have the loader say what its loading, from where, etc.
>    Maybe ASDF has stuff for this? Even when default streams are redirected?
> - RPATH info embedded in libs
>    helps find dependencies in unusual places (esp. customized forms of
>    common libs)

I respond to this portion of the email only because I've directly run
into it.  

First, some background: I've written a CL master/slave library that
developers can use to build for fine to medium task grained distributed
applications.  Once a developer finishes an application built against
it, they can produce a lisp binary of it for execution by other batch
processing systems like Condor, PBS, LSF, etc, etc, etc. The application
will likely be shipped to thousands of different execute machines all
outside of the developer's administrative control.  These machines will
very likely not have SBCL even installed on them let alone other lisp
environments or libraries.

It is ok if the produced lisp binary from save-lisp-and-die needs some
shared libraries since those can be shipped along with the lisp image
by the batch system, but the search path locations of them have to be
changable at execution time and should default to the current working
directory. For my purposes, I've written an SBCL specific piece of code
which produces this binary and dumps the associated shared libraries
(found by doing a similar search algorithm to dlopen) as well as the lisp
image executable.  This code is ugly, and likely fragile, but it is easy
for the application developer to get right and works in the cases I need.

>From a user's point of view. I don't care what the internals look like
for a solution to the above general problem or how the shard libraries
came to be in the lisp image (CFFI, manuall called dlopen through CFFI,
etc). Just make it really really really easy to use and robust and follow
the law of least surprises!

In the case of a lisp script, I'd be happy to simply package the needed
libraries into a sub directory along with the executable and

#! /usr/bin/sbcl --script --prepend-libraries-path ./stuff --fasl-dir ./fasls
(format t "HI~%")

so I can run my script on machines that have SBCL, but might not have
my needed libraries, so I include them and a place to find/build them.

In the usual case, these contexts apply:

a) User running script on self built/known SBCL and known libraries.
b) User running script on someone else's SBCL with unknown library
   availability.
c) User building executable to run on their machine with known good SBCL and 
   libraries.
d) User building executable to run on other machines with unknown SBCL 
   install and unknown/none libraries.

Make it easy to do the above with the least knobs possible and you'll
have a winner. :)

Thank you.

-pete

James Y Knight <foom@...> wrote:

> So, normally, users will either install packages in the site-wide
> location, or the user-specific location. And things will just work,
> without any configuration.

  This would of course be ideal, but would require that *all*
implementation agree with each other (Python is one language, CL is
many). Problem is that there is no such thing as a default site-wide or
even user-specific location for CL code.

Based on the machines I use, it seems that:

/usr/share/common-lisp/source
/usr/local/share/common-lisp/source
~/share/common-lisp/source

would be a good start.


> If users install things into another private non-default location,
> they can set PYTHONPATH to point to it, or else have their program
> simply modify sys.path as the first thing it does, prior to loading
> packages.

  Except for the bootstrapping issue with ASDF which is itself a
library ;-)

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

Daniel Herring wrote:

> Abstract: SBCL does not need to change --script for this issue. The
> main problem is learning how to use ASDF 2, with some difficulty
> reliably bootstrapping it.

  I still think it's wrong for --script to imply no user/sys
initialization[1], but I agree that the question is more how to properly
use ASDF 2, or maybe *documenting* the (intended) proper way to use it.

Given how it currently interacts with SBCL --script, it seems that the
distinction between non-interactive / interactive behavior is already
here: either you configure the source registry in the conf file, or you
do so with the API in .sbclrc.




Footnotes: 
[1]  because executing a zsh script doesn't prevent zsh from reading
.zshenv, and here I rejoin Nikodemus on the interactive/non-interactive
distinction.

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

On 13 October 2010 11:22, Didier Verna <didier@...> wrote:

> [1]  because executing a zsh script doesn't prevent zsh from reading
> .zshenv, and here I rejoin Nikodemus on the interactive/non-interactive
> distinction.

Yep. I have to kinds of things in my .sbclrc: tools for interactive
use, and basic environment configuration. (Setting up ASDF and
Quicklisp, pretty much.)

The latter could very well live in hypothetical .sbclenv.

Cheers,

 -- Nikodemus

> - Its about time to start a "how to configure ASDF" thread on asdf-devel,
> with guidelines on what goes where, when to use the different hooks, etc.
> I'm not quite confident, but getting close.


http://common-lisp.net/project/asdf/asdf.html#Configuring-ASDF

the discussion about the topic of ASDF configuration can go to the
list, but please open a wiki page somewhere for the documentation body
itself (or just edit the doc in the asdf repo, but that has a slower
turnaround and needs ASDF maintainers to update the doc on the project
page).

btw, i've found the above ASDF documentation too verbose compared to
what it's about. or at least too much in the form of prose... but i'll
write a short feedback in a mail to asdf-devel.

-- 
 attila

On Oct 13, 2010, at 3:42 AM, Peter Keller wrote:

> #! /usr/bin/sbcl --script --prepend-libraries-path ./stuff --fasl-dir ./fasls
> (format t "HI~%")
> 
> so I can run my script on machines that have SBCL, but might not have
> my needed libraries, so I include them and a place to find/build them.

It's not necessary for all that to be a command-line option: you should be able to set variables/call functions to effect that at the top of your script code. They just need to be well-documented.

On Oct 13, 2010, at 4:22 AM, Didier Verna wrote:
> [1]  because executing a zsh script doesn't prevent zsh from reading
> .zshenv, and here I rejoin Nikodemus on the interactive/non-interactive
> distinction.

Yeah, but that's a strange and unique zsh behavior. And it's definitely caused strange problems for me in the past (my fault, yes, but still). Neither traditional sh nor bash source *any* config files at all when running scripts. And I don't see why SBCL and ASDF really need to either, so long as it starts out life looking for libraries in $LISPPATH, /usr/somelocation, and ~/.somelocation.

James

James Y Knight <foom@...> wrote:

> Yeah, but that's a strange and unique zsh behavior.

  Strange, I don't know. Useful I think so :-)


> And I don't see why SBCL and ASDF really need to either, so long as it
> starts out life looking for libraries in $LISPPATH, /usr/somelocation,
> and ~/.somelocation.

  In an ideal world, that solution (which I would also like to see
happening) would not only concern SBCL but all other lisp
implementations.

I'm not sure how likely it is that all lisp implementations agree on a
standard set of default paths. But for the time being, you can provide
your own in the rc files (I currently have a .clrc with global code that
all implementation load through their own rc file).

Now, this is even better when you can split this from the rest of your
initialization code that's mostly aimed at interacting in the repl.

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

On 13 October 2010 16:44, Didier Verna <didier@...> wrote:

> I'm not sure how likely it is that all lisp implementations agree on a
> standard set of default paths. But for the time being, you can provide
> your own in the rc files (I currently have a .clrc with global code that
> all implementation load through their own rc file).

They don't have to. Only ASDF needs to have it's story straight, I think.

Cheers,

 -- Nikodemus

Nikodemus Siivola wrote:

> On 13 October 2010 16:44, Didier Verna <didier@...> wrote:
>
>> I'm not sure how likely it is that all lisp implementations agree on a
>> standard set of default paths. But for the time being, you can provide
>> your own in the rc files (I currently have a .clrc with global code that
>> all implementation load through their own rc file).
>
> They don't have to. Only ASDF needs to have it's story straight, I think.

  Except maybe if you want the ability to handle non-ASDF packages, or
even implementation-specific additions ?

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

On 13 October 2010 11:07, Didier Verna <didier@...> wrote:
> Nikodemus Siivola wrote:
>
>> On 13 October 2010 16:44, Didier Verna <didier@...> wrote:
>>
>>> I'm not sure how likely it is that all lisp implementations agree on a
>>> standard set of default paths. But for the time being, you can provide
>>> your own in the rc files (I currently have a .clrc with global code that
>>> all implementation load through their own rc file).
>>
>> They don't have to. Only ASDF needs to have it's story straight, I think.
>
>  Except maybe if you want the ability to handle non-ASDF packages, or
> even implementation-specific additions ?
>
These don't require implementations to agree with each other.

So yes, only ASDF has to have proper defaults.
I agree ASDF configuration is more complex than needs be if we can
have consensus on where things should go. If you think ASDF defaults
should be changed, please discuss on asdf-devel.

As for making a Wiki page on ASDF configuration as Attila suggested,
if you start one on Cliki, I'll link to it from the ASDF  page.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
La vie n'est pas un restaurant mais un buffet. Levez-vous pour vous servir.
        Life is not a restaurant but a buffet. Stand up to serve yourself.
                — Dominique Glocheux

> As for making a Wiki page on ASDF configuration as Attila suggested,
> if you start one on Cliki, I'll link to it from the ASDF  page.


well, what i wanted to propose was to set up a free-to-edit
semi-public document focusing on the configuration part of ASDF.

when things settled down, it could be merged back into the ASDF documentation.

-- 
 attila