[Sbcl-devel] Re: 0.6.5 on linux, and multi-proc

[William H. N. <wil...@ai...>]

(The original message was on the sbcl-help list, but I think
it's more appropriate for the sbcl-devel list, so I've put this reply
both on sbcl-help and on sbcl-devel, and I suggest that future
replies be on sbcl-devel.)

On Wed, Jun 07, 2000 at 11:12:00AM -0700, Craig Brozefsky wrote:
> First, I wanna let you know that I built 0.6.5 on linux using
> sbcl-0.6.4.  I haven't tried building with CMUCL on that platform
> yet.  I'm trying to get multi-proc suport back in, and am making some
> progress.  I have a few questions tho.
 
> Would there be any issues with having the stem loader just compile
> files that have changed source since the last time they were compiled.
> Have that turned on an off by a switch, so that we can still get clean
> builds.  Maybe it could be more sophisticated and compile every file
> after the first changed file it finds in the stem list.  I am still
> trying to work out the whole build process, so I may be suggesting
> something impossible.

I agree that this is painful. I'm not sure what the best solution is,
though. Incremental recompilation is hard to do cleanly, because there
are so many dependencies on macros, package structure, specialness of
variables, and so forth.

On other projects I have tried the approach of compiling every file
after the first changed file, and it's easy to implement and works OK.
If you're making a lot of target-system-only changes it might be worth
implementing something like that for SBCL.

When I want to (relatively) quickly test small target-system-only
changes, I usually try one of two ways. Neither is 100% reliable,
but they can be worth it to save an hour or more of rebuild time.
  1. If your SBCL gets through boot, and you just want to modify
     some high level thing, then run SBCL in the root of your SBCL
     source tree and do (LOAD "src/cold/chill.lisp"). Now you can
     COMPILE-FILE and LOAD your modified source files.
  2. If something is messed up in the SBCL boot sequence, so that 
     you can't do the first approach, you can still use 
     output/after-xc.core (if you built with :SB-SHOW enabled
     so that you have output/after-xc.core). Run SBCL with this
     core file
	$ sbcl --core output/after-xc.core
     then in this SBCL, call TARGET-COMPILE-STEM to rebuild the modified
     source file(s), e.g.
	(target-compile-stem "src/code/multi-proc")
     them re-run GENESIS.

> In general, I'm looking for ways to break the build process down a
> little bit so that my work in fixing MP is a little quicker.  Right
> now it takes a good chunk of time to compile and load everything up
> till multi-proc.lisp only to find another problem.  Then I gotta do it
> all over against when I fix it.  Any suggestions would be welcome.

Yes. This has been a pain for me too. The suggestions above are
the best I have, but if someone else has better suggestions,
I'd welcome them too.

I suspect that this is how CMU CL was pushed into its weird
bootstrapping process in the first place, since even with Y2K hardware
it's painful to bootstrap from scratch. I can only imagine how awful
this must've been on 1987 hardware..
 
> When are def!struct and other def!<blah> constructs needed, and when
> are they not.  Do I need to convert the defstructs in multi-proc.lisp
> to them?

Ah.

Currently things with "bangs" (exclamation marks, '!') in their names
are have separate in-the-cross-compiler and in-the-target-system
versions. Thus
   * The #!+ readmacro corresponds to the #+ readmacro.
   * The SB!C package corresponds to the SB-C package.
   * The DEF!MACRO macro corresponds to the DEF!MACRO macro.
Someday I tentatively plan to add another class of uses of exclamation
marks, for names of things which are only needed when the system
is being built, and can be uninterned and otherwise undefined (e.g.
removed from the globaldb.lisp tables) once the system has been
built, e.g. the *REVERSED-TYPE-INFO-INIT-FORMS* variable and 
the VOP macro, but I haven't done anything like that yet. (When/if I
do, I expect they might look like *!REVERSED-TYPE-INFO-INIT-FORMS!*
and !VOP!, and there will be code at the end of boot to automatically
delete anything whose name has one of these magical forms.)

DEF!FOO things in general are used for target system things which 
the cross-compiler needs to know about and manipulate somehow.
They may also take care of saving up definitions through part of
initialization until the system is ready to deal with them. Thus
   * DEF!TYPE defines a target system type which the cross-compiler 
     knows about.
   * DEF!MACRO defines a macro which exists both in the cross-compilation
     host and in the target system.
   * DEF!STRUCT defines a structure which exists both in the 
     cross-compilation host and in the target system (with the 
     appropriate magic to dump an instance which exists in the
     cross-compilation host into a target system FASL file).
   * DEF!METHOD defines a method which exists both in the cross-compilation
     host and in the target system.
 
DEF!FOO macros in target system code are supposed to act basically
like the corresponding DEFFOO macro, modulo any magic needed to save
up things like method definitions until the system is mature enough to
understand them.

If the multi-proc.lisp code only needs to be cross-compiled (i.e. can
remain marked :NOT-HOST in stems-and-flags.lisp-expr, i.e. does not
need to also be compiled as code to run in the cross-compilation host)
then I don't expect that it should need DEF!FOO things. The DEF!FOO
things are intended for stuff like DEF!MACRO DO, DEF!STRUCT PACKAGE,
DEF!TYPE SC-OFFSET, DEF!METHOD MAKE-LOAD-FORM, and other fundamental
things which need to execute at build-the-cross-compiler time as well
as at build-the-target-system time. (And just now, looking for all the
uses of DEF!TYPE, it looks as though those might actually not be very
fundamental -- perhaps DEF!TYPE could go away with a little work.)

> Is there any problem with breaking multi-proc.lisp into multiple
> files?  One for processes, another for stacks groups.

Nothing that I know of. But I have hardly looked at the CMUCL/SBCL
multiprocessing stuff at all, and I know hardly anything about
multiprocessing in SBCL (or in any other Lisp).

> Thanx for SBCL!

You're welcome. Thanks for helping!

-- 
William Harold Newman <wil...@ai...>
software consultant
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C  B9 25 FB EE E0 C3 E5 7C