Steel Bank Common Lisp

Robert Marlow <bobstopper@...> writes:

> 10: (SB-IMPL::MAKE-INTEGER 0)[:EXTERNAL]

This is where the error is occurring... and I suspect I know what's
going on.

The tokeniser uses global buffers to do its tokenization (have a look
at the use of *read-buffer* in src/code/reader.lisp).  This means that
if two threads attempt to tokenise at once, something much like what
you've observed will happen, for instance when DIGIT-CHAR-P (in
MAKE-INTEGER) gets called on a non-numeric character.

I don't have a ready fix right now.  The quick fix, but with possible
performance implications, is to bind *read-buffer* to a fresh array
somewhere in src/code/reader.lisp (and *read-buffer-length*, too).  If
that turns out to be too consy, then maybe a pool of buffers needs to
be implemented.

Cheers,

Christophe
-- 
http://www-jcsu.jesus.cam.ac.uk/~csr21/       +44 1223 510 299/+44 7729 383 757
(set-pprint-dispatch 'number (lambda (s o) (declare (special b)) (format s b)))
(defvar b "~&Just another Lisp hacker~%")    (pprint #36rJesusCollegeCambridge)

Thanks for the pointer

I've set about trying to rewrite reader.lisp to be thread friendly but
my attempts seem to not be playing nicely.

I've possibly not chosen the best way to do it, but how I've written it
is rewritten *read-buffer* to be a hash table and have buffers extracted
from the hash table by using #'current-thread-id as the key. Perhaps I
could have done this more efficiently?

Anyway, that's required quite a lot of rewriting (most of it just moving
the global read-buffer stuff into functional arguments for much of the
functions in the file) but now I find that during my attempts to compile
sbcl I get buffer overflows. grow-read-buffer seems like it should work
ok, but the compile dies on the first attempt to parse a >512 character
documentation string.

I've attached a diff for what I've done.

Actually, a sort of unrelated question - what's an easy way to speed up
code testing for SBCL development? There's a lot of cross compilation
stuff in reader.lisp so it doesn't make it easy for me to just load up
the file and start messing with it interactively as lisp hackers are
wont to enjoy. Is there a trick you guys have to make SBCL source
hacking quicker than a hack->full-recompile->debug cycle?


On Mon, 2004-04-19 at 23:11, Christophe Rhodes wrote:
> Robert Marlow <bobstopper@...> writes:
> 
> > 10: (SB-IMPL::MAKE-INTEGER 0)[:EXTERNAL]
> 
> This is where the error is occurring... and I suspect I know what's
> going on.
> 
> The tokeniser uses global buffers to do its tokenization (have a look
> at the use of *read-buffer* in src/code/reader.lisp).  This means that
> if two threads attempt to tokenise at once, something much like what
> you've observed will happen, for instance when DIGIT-CHAR-P (in
> MAKE-INTEGER) gets called on a non-numeric character.
> 
> I don't have a ready fix right now.  The quick fix, but with possible
> performance implications, is to bind *read-buffer* to a fresh array
> somewhere in src/code/reader.lisp (and *read-buffer-length*, too).  If
> that turns out to be too consy, then maybe a pool of buffers needs to
> be implemented.
> 
> Cheers,
> 
> Christophe
-- 
Regards,

Robert Marlow

On Tue, 11 May 2004, Robert Marlow wrote:

> Actually, a sort of unrelated question - what's an easy way to speed up
> code testing for SBCL development? There's a lot of cross compilation
> stuff in reader.lisp so it doesn't make it easy for me to just load up
> the file and start messing with it interactively as lisp hackers are
> wont to enjoy. Is there a trick you guys have to make SBCL source
> hacking quicker than a hack->full-recompile->debug cycle?

The ways I'm aware of:

 * Just load the code into a running system. May work as long as the code
   contains no def!foo, sb!foo or #! +stuff.

 * Load src/cold/chill.lisp -- this allows loading _some_ parts
   of the SBCL sources into a running image, including sending
   stuff from Slime with C-c C-c and friends. Though it seems that
   hacking on a chilled system is a good way to get Slime confused
   eventually, at least when tinkering with PCL.

 * Modify sources, cross your fingers and run "sh slam.sh <original build
   host>". Works for only some changes, though. After you get your
   stuff working you better do a full build still, since it's possible to
   slam some stuff that will break a full build -- mostly things that
   affect the cross-compilation.

   Requires having :sb-after-xc-core in features for the original
   build. This is the reason that I usually use CMUCL as a build host,
   since then I can safely install the freshly built SBCL without
   invalidating old after-xc-cores.

 * If you  really want to live dangerously, see my slam-host hack in
   the archives (around last december, iirc): just like slam, but
   occasionally able to incorporate changes that normal slam can't.
   Can result in spectacular *CRASH* *BOOM* *BANG* sound effects,
   though...

Cheers,

 -- Nikodemus

Excellent, thanks very much Nikodemus. Those pointers should save me a
lot of time with this kind of stuff :)


On Tue, 2004-05-11 at 20:32, Nikodemus Siivola wrote:
> On Tue, 11 May 2004, Robert Marlow wrote:
> 
> > Actually, a sort of unrelated question - what's an easy way to speed up
> > code testing for SBCL development? There's a lot of cross compilation
> > stuff in reader.lisp so it doesn't make it easy for me to just load up
> > the file and start messing with it interactively as lisp hackers are
> > wont to enjoy. Is there a trick you guys have to make SBCL source
> > hacking quicker than a hack->full-recompile->debug cycle?
> 
> The ways I'm aware of:
> 
>  * Just load the code into a running system. May work as long as the code
>    contains no def!foo, sb!foo or #! +stuff.
> 
>  * Load src/cold/chill.lisp -- this allows loading _some_ parts
>    of the SBCL sources into a running image, including sending
>    stuff from Slime with C-c C-c and friends. Though it seems that
>    hacking on a chilled system is a good way to get Slime confused
>    eventually, at least when tinkering with PCL.
> 
>  * Modify sources, cross your fingers and run "sh slam.sh <original build
>    host>". Works for only some changes, though. After you get your
>    stuff working you better do a full build still, since it's possible to
>    slam some stuff that will break a full build -- mostly things that
>    affect the cross-compilation.
> 
>    Requires having :sb-after-xc-core in features for the original
>    build. This is the reason that I usually use CMUCL as a build host,
>    since then I can safely install the freshly built SBCL without
>    invalidating old after-xc-cores.
> 
>  * If you  really want to live dangerously, see my slam-host hack in
>    the archives (around last december, iirc): just like slam, but
>    occasionally able to incorporate changes that normal slam can't.
>    Can result in spectacular *CRASH* *BOOM* *BANG* sound effects,
>    though...
> 
> Cheers,
> 
>  -- Nikodemus
-- 
Regards,

Robert Marlow

I think I've fixed this bug. I took dan`b's advice and locally bound
*read-buffer*, *ouch-ptr* and *inch-ptr* during thread creation in
make-thread. I've attached the diff.

I've run my test case and got no read errors where I used to get 2-3
during the run so I think it's fixed the problem.


I have still been getting one other error though: 
mmap: Cannot allocate memory
unhandled condition (of type SB-INT:SIMPLE-CONTROL-ERROR):
  attempt to RETURN-FROM a block or GO to a tag that no longer exists

Is this just SBCL running out of memory or is it possibly to do with
thread contention for memory addresses?


On Mon, 2004-04-19 at 23:11, Christophe Rhodes wrote:
> Robert Marlow <bobstopper@...> writes:
> 
> > 10: (SB-IMPL::MAKE-INTEGER 0)[:EXTERNAL]
> 
> This is where the error is occurring... and I suspect I know what's
> going on.
> 
> The tokeniser uses global buffers to do its tokenization (have a look
> at the use of *read-buffer* in src/code/reader.lisp).  This means that
> if two threads attempt to tokenise at once, something much like what
> you've observed will happen, for instance when DIGIT-CHAR-P (in
> MAKE-INTEGER) gets called on a non-numeric character.
> 
> I don't have a ready fix right now.  The quick fix, but with possible
> performance implications, is to bind *read-buffer* to a fresh array
> somewhere in src/code/reader.lisp (and *read-buffer-length*, too).  If
> that turns out to be too consy, then maybe a pool of buffers needs to
> be implemented.
> 
> Cheers,
> 
> Christophe
-- 
Regards,

Robert Marlow