I've just reached the point in my experimental thread support that I
can run a repl in a thread, start new threads from it, and watch them
run. So, I thought this would be a good time to put a few words down
on phosphor describing what it currently looks like: people will
probably have different opinions regarding to what extent (if any) an
MP implementation could/would/should affect users who only want/need a
The basic outline is at http://ww.telent.net/sbcl-internals/Threading
The code changes to achieve this are as follows
1) Each "process context" is represented by a stack_group structure.
This is defined as a primitive-object to make it simple to keep
accessors consistent between the Lisp and C world, but it lives in C
space. It includes pointers to control, number, binding stacks, a
buffer to save register contents in, an interrupt_contexts array, and
so on; on thread switch we reload the registers appropriately to
point to new stacks
(I'm doing this initially on Alpha, which is of course non-x86. Toy
computer users should probably read "special variable" where I say
(Yeah yeah. "Toy". My P3 laptop outperforms my Alpha, but the Alpha
is still more fun)
This is a change from CMUCL, which uses a single memory area for each
stack for the active thread and copies its contents to and from
'save' areas on thread switch
Runtime changes were needed to GC and purify (to make sure we attack
all stack_groups and not just the current one) and call_into_lisp,
call_into_c and fake_foreign_function_call to sopy a few bits around.
I also chopped the startup around a bit so that the runtime called
allocate_stack_group to get the control and binding stacks mmaped
This should have practically no maintainability or performance
implications for a single-threaded lisp, so my inclination would be to
make these changes unconditional.
2) Thread switch is done using a couple of VOPs to save/restore
registers, and some Lisp functions based on the CMUCL MP system to
deal with the binding stack and switch all the other context around.
I've put these into an SB!MP package, for the time being, but ...
3) The initial thread is created by the runtime at process startup.
If you want, you can run a thread-enabled SBCL without starting any
more threads. This is not done just to appease people ;-) - it also makes
it much easier to work with when all of cold init can happen
When you do go multithreaded, you start a new thread to run a repl in,
and the initial thread becomes the idle thread - basically, just sits
in a loop calling the next ready process or poll(2) if none are, to
wait until one is
Of course, if you expected to run multi-threaded all of the time, you
could dump an image with initial function such as
(create-process "listener" (lambda () (sb-impl::toplevel-repl)))
4) File IO was one of the least pleasant bits of the CMUCL
implementation (the other was stack copying), and I'm dealing with
that really quite differently. Now when a thread (other than the
initial/idle thread, which is special-cased) attempts to do IO to a
file descriptor which is not ready, it gets added to a queue, and the
idle thread is responsible for calling poll(2) (the new-world
equivalment of select(2)) on _all_ waited-for file descriptors and
restarting whichever thread should be OK to continue.
This is distinct from the CMUCL way of doing things, in which each
waiting thread would do its own select() call - giving n kernel calls
where only one is needed.
This makes the fd-stream code a lot clearer (IMO) as all the
OUTPUT-LATER stuff can be replaced with (Lispish pseudocode)
(loop until buffer empty
(sb!mp:wait-for-descriptor-io fd '(:output)))
and similarly for input
I killed serve-event in the process, but the serve-event interface
should be easily and cleanly reimplemented in terms of threads; I'll
get to that eventually.
Now, this could be merged as unconditional changes, or
conditiionalized on #+sb-mp or something. My preference (again) is to
change it unconditionally, as it does make fd-stream.lisp rather
shorter and easier to follow. As wait-for-descriptor-io special-cases
the idle thread, these changes continue to work even in an SBCL with
only one thread running, so this needn't actually break anything.
5) I have done nothing yet to tackle the issue of thread safety in
SBCL functions generally. Yes, I know that needs doing too.
Tim Moore is also looking at that for CMUCL, so with any luck we'll
be able to steal each others code.
6) This is a milestone but is not by any means news of impending
completion: there's still a whole lot of stuff to do before I'd
consider it ready to be let out. If people are interested in seeing
it, I'll stick up a diff (against something in the region of 0.7.1.45,
as it happens); if people are interested in collaborating (in
particular, volunteers to write and test register saving vops on
sparc, and do general porting stuff for x86) then I'm going to have to
figure out something with CVS. Bill, how would you feel about
allowing creation of a thread branch in the sourceforge cvs, if this
http://ww.telent.net/cliki/ - Link farm for free CL-on-Unix resources