A development branch is up at https://github.com/pkhuong/sbcl/tree/no-world-lock-pcl. The patches take care of two issues:
1. PCL no longer *has* to call COMPILE to function to generate functions
2. Avoid grabbing the world lock in CPL-OR-NIL and in CHECK-WRAPPER-VALIDITY
It seems to work: not only does it still pass all tests on linux/x86-64 (according to Stelian), but it also avoids deadlocks when building iolib under SWANK (Stelian's tests, again).
However, 1) is extremely hacky, and 2) feels like random band aids. Test cases, comments, review and suggestions welcome (:
1) is implemented by attempting to grab the world lock, without blocking if it isn't available. If we can't acquire the world lock, PCL instead uses SB-EVAL to return interpreted functions that are opportunistically converted to native code each time they're called. That's all in src/pcl/macros.lisp and it seems Rube Goldberg-y. It also depends a bit on a patch to sb-eval to support the compilation of interpreted closures; I'm not convinced it's the best way to do it.
The net result is that, rather than deadlocking, we have slower functions. Better, in the long run, these slower functions are converted into funcallable instances that jump to compiled code (overhead: a single indirect jump). I'm also doubtful of the policy of punting to SB-EVAL whenever the world lock is held by another thread.
2) seems obviously correct, but I don't know what else could be similarly improved. What do you think of switching the world-lock to an upgradable reader/writer lock? It would let us more easily exploit our read-mostly common cases, with less hand-rolled code duplication (but we'd have to handle write upgrade failures).
Obviously, nikodemus's work to eliminate the world lock would be even better, but, in the meantime, this lets people build useful systems.
I'm not that confident about the changes, especially without tests, so I'll wait and think a couple days before committing.