[Sbcl-devel] bug 311: tokeniser not thread safe

[Gabor M. <me...@ho...>]

The attached patch binds the buffer related global vars to fresh values on=
=20
non-recursive (i.e. 'toplevel') read calls if the :sb-thread feature is=20
present. Passing t for recursive-p when it is not really recursive will rai=
se=20
and unbound-variable error.

The performance drops a bit, but it is not significant for typical source=20
files. On the other hand reading integers in a loop will be noticeably=20
slower.

Other approaches considered/tested:

=2D bind them in make-thread: ugly and silently assumes that the tokeniser=
=20
cannot call read & co in an inappropriate moment

=2D maintain a pool of read buffers with or without weak pointers: test sho=
w=20
this solution to be slower (maybe with-mutex is heavier than I thought)

My tests uncovered another thread safety problem with stringify-buffer that=
=20
maintains a cache in *string-output-streams*. It is a bit harder to trigger=
=20
but it can kill sbcl. Removing the cache and creating a new=20
string-output-stream every time it is called obviously fixes the problem.=20
Same goes for read-from-string and *read-from-string-spares*.

I don't know how to provide unit tests that don't take forever to run and=20
produce somewhat dependable results, but attached you can find the small=20
tests I used to break sbcl and compare performance.

Likely there are other thread safety problems in the reader related to case=
=20
handling and maybe *sharp-equal-alist* when calling read-delimited-list.

Cheers, G=E1bor