[Sbcl-devel] Threads and Lutexes on Mac OS X for Intel Status Update

[Cyrus H. <ch...@bo...>]

Nathan Froyd and I have been working on getting threads up and  
running on Mac OS X for Intel. Back in November or so, Nathan  
originally proposed the concept of a lutex, or a lisp mutex lock,  
that would be API compatible with the current linux futex-based  
locking code.

Once we had SBCL up and running on darwin/x86, I decided to see if I  
could take the lutex code and make it work on Darwin/x86. In addition  
to the GENCGC and the locking primitives, the other (another) key  
piece is OS and architecture support for thread-local storage (TLS).  
Turns out Apple doesn't really go out of their way to make life  
difficult for us here, it merely seemed so at first.

The i386_set_ldt function can be used to set up the descriptor tables  
that describe how the segment registers should map to memory  
addresses, and the %fs register is available for our use here. All  
seemed good, until we realized that %fs was getting clobbered in  
signal handlers. Well it turns out that Apple was doing this on  
purpose but that we can get the %fs that was loaded when the signal  
was raised by using the %fs slot of the machine context. So, now  
we're back in business.

After some intense #lisp-based hacking, we managed to build a quasi- 
working version #+sb-thread and #+sb-lutex. The code for this has  
been checked in on the lutex-branch branch of the main SBCL source  
repository. In order to make sure that you really have to go out of  
your way to build this, lest someone end up tangling with this beast  
by accident, you must check out the branch AND edit base-target- 
features.lisp-expr and add :sb-thread and :sb-lutex to the features  
list.

That's the good news. The bad news is that there are a number of  
loose ends:

1. gc.impure.lisp fails as it seems without-interrupts is blocking  
SIG_STOP_FOR_GC.

2. SLIME dies at the end of its initialization sequence with a  
SIGILL. This is perhaps a sign that we are executing code somewhere  
with the wrong value of %fs. (Yes, that seems to be the error that  
results in accessing bogus values off of %fs).

3. slam.sh dies with a bus error if we use a threaded version to do a  
full build and then attempt to run slam.sh on the newly built  
threaded version.

Back to good news, we can rebuild #+(and sb-thread sb-lutex) with  
itself and we pass all 14 contrib tests. So things are looking good,  
but there is still more work to do. In particular, I'm not sure we're  
doing the right things in purify.c and in the interrupt handling  
code. Patch review and more testing would be greatly appreciated. In  
the ideal case, we well be able to resolve these issues shortly and  
be in position to merge this onto the trunk shortly after this months  
code freeze and have this ready to go in time for 0.9.12.

If this is successful it also suggests that threads on PPC/darwin  
might be feasible as well. The architecture support is now the  
missing piece on PPC, given that we have GENCGC, and at least limited  
darwin threads on Intel, and enough registers to work with. It should  
be relatively straightforward to free up a GPR for TLS in the  
existing ppc/linux code. If that works, merging the OS code from  
darwin/x86/threads and the arch code from ppc/linux/threads would put  
us in good position to make threads work on darwin/ppc. Volunteers  
wanted for the ppc/linux port.

Thanks to Nathan for the lutexes and for jumping in once I got in  
over my head with the %fs stuff. And, as always, thanks to the #lisp  
for crew for telepathic debugging and moral support.

Cyrus