Steel Bank Common Lisp

On Fri, 10 Jun 2005, John Morrison wrote:

> Short version: I get a memory fault after loading shared object
> libraries, loading callback-sbcl.lisp, and a REALLY BIG compiled file
> of sb-alien bindings.  Any advice?  I am running 0.9.0.18 (from an
> RPM), and I have a recently-released RPM of 0.9.1-1.

While building from source might help, I doubt it. LDB most certainly
won't help, as you seem to be still in a live SBCL after the memory fault 
-- LDB is strictly for post mortem.

My first guess is that you're simply running out of static space.
As a quick workaround: don't purify, and if you need to save an image,
do it with :PURIFY NIL.

If you're purifying because you need stuff that doesn't move, then 
SB-INT:MAKE-STATIC-VECTOR may be what you need: it allocates vectors of 
immediate objects directly to static space. Do not be overly alarmed by 
the SB-INT in this case: once the dust settles I think it (or something 
very much like it) is destined to move to SB-EXT. You get to beta-test it. 
;-)

If you're feeling adventurous, then twiddling the static-space-* values in 
src/compiler/target/parms.lisp may help by giving you a larger static 
space; searching sbcl-devel archives should yield some advice for that.

Similarly, building SBCL with the first PURIFY in make-target-2.sh 
commented out should also conserve _some_ static space, and may be
sufficient for your needs, but I rather doubt it.

As a longer term solution we should probably detect the end of static 
space and signal a STORAGE-CONDITION if we run out of it.

Cheers,

  -- Nikodemus              Schemer: "Buddha is small, clean, and serious."
                   Lispnik: "Buddha is big, has hairy armpits, and laughs."

John,

IIRC you're doing FFI bindings from GCCXML output. Are you doing this  
straight to sb-alien calls or through an intermediary like UFFI? One  
problem I ran into is that without the proper declarations these  
calls were wildly inefficient and had to do a whole bunch of stuff at  
runtime. It's possible you're just getting really large compiled  
functions for each of your foreign function definitions and that  
these are, as Nikodemus suggested, causing you to run out of memory.  
This is just a wild guess and I might be way-off here, but you might  
look at trying to get the proper declarations to keep these foreign  
function definitions small and more efficient at the same time.

Cyrus


On Jun 11, 2005, at 2:25 AM, Nikodemus Siivola wrote:

> On Fri, 10 Jun 2005, John Morrison wrote:
>
>
>> Short version: I get a memory fault after loading shared object
>> libraries, loading callback-sbcl.lisp, and a REALLY BIG compiled file
>> of sb-alien bindings.  Any advice?  I am running 0.9.0.18 (from an
>> RPM), and I have a recently-released RPM of 0.9.1-1.
>>
>
> While building from source might help, I doubt it. LDB most certainly
> won't help, as you seem to be still in a live SBCL after the memory  
> fault -- LDB is strictly for post mortem.
>
> My first guess is that you're simply running out of static space.
> As a quick workaround: don't purify, and if you need to save an image,
> do it with :PURIFY NIL.
>
> If you're purifying because you need stuff that doesn't move, then  
> SB-INT:MAKE-STATIC-VECTOR may be what you need: it allocates  
> vectors of immediate objects directly to static space. Do not be  
> overly alarmed by the SB-INT in this case: once the dust settles I  
> think it (or something very much like it) is destined to move to SB- 
> EXT. You get to beta-test it. ;-)
>
> If you're feeling adventurous, then twiddling the static-space-*  
> values in src/compiler/target/parms.lisp may help by giving you a  
> larger static space; searching sbcl-devel archives should yield  
> some advice for that.
>
> Similarly, building SBCL with the first PURIFY in make-target-2.sh  
> commented out should also conserve _some_ static space, and may be
> sufficient for your needs, but I rather doubt it.
>
> As a longer term solution we should probably detect the end of  
> static space and signal a STORAGE-CONDITION if we run out of it.
>
> Cheers,
>
>  -- Nikodemus              Schemer: "Buddha is small, clean, and  
> serious."
>                   Lispnik: "Buddha is big, has hairy armpits, and  
> laughs."
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: NEC IT Guy Games.  How far can  
> you shotput
> a projector? How fast can you ride your desk chair down the office  
> luge track?
> If you want to score the big prize, get to know the little guy.   
> Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
> _______________________________________________
> Sbcl-help mailing list
> Sbcl-help@...
> https://lists.sourceforge.net/lists/listinfo/sbcl-help
>

Hi;

On Saturday 11 June 2005 05:25 am, Nikodemus Siivola wrote:
> My first guess is that you're simply running out of static space.
> As a quick workaround: don't purify, and if you need to save an image,
> do it with :PURIFY NIL.

OK, I can do that ... thanks!  Two questions immediately spring to
mind: (1) Is there any way I can verify that (that it's a simple space
issue and not some Deep Dark Coding Malevolence)?  (2) Will/might
there be fewer such problems with a 64-bit version of SBCL?

> If you're purifying because you need stuff that doesn't move, then

The "callback" trampoline stuff has an advisory about that, but I
cannot recall whether either the infrastructure or all my callbacks
require the purify to keep them from moving...

> SB-INT:MAKE-STATIC-VECTOR may be what you need: it allocates vectors of
> immediate objects directly to static space. Do not be overly alarmed by
> the SB-INT in this case: once the dust settles I think it (or something
> very much like it) is destined to move to SB-EXT. You get to beta-test it.
> ;-)

OK.  Thanks again.  I will check out the callback/trampoline code.

> If you're feeling adventurous, then twiddling the static-space-* values in
> src/compiler/target/parms.lisp may help by giving you a larger static
> space; searching sbcl-devel archives should yield some advice for that.

OK.  I will do that if I get No Joy with the first two suggestions you
have been kind enough to make.

> Similarly, building SBCL with the first PURIFY in make-target-2.sh
> commented out should also conserve _some_ static space, and may be
> sufficient for your needs, but I rather doubt it.
>
> As a longer term solution we should probably detect the end of static
> space and signal a STORAGE-CONDITION if we run out of it.

That would be cool!

Thanks again!

-jm

>
> Cheers,
>
>   -- Nikodemus              Schemer: "Buddha is small, clean, and serious."
>                    Lispnik: "Buddha is big, has hairy armpits, and laughs."

-- 
==== John Morrison
==== MAK Technologies Inc.
==== 10 Fawcett Street, Cambridge, MA 02138
==== http://www.mak.com/
==== vox:617-876-8085 x115
==== fax:617-876-9208
==== jm@...

On Saturday 11 June 2005 11:14 am, Cyrus Harmon wrote:
> IIRC you're doing FFI bindings from GCCXML output. Are you doing this
> straight to sb-alien calls or through an intermediary like UFFI? One

I am using sb-alien rather than UFFI, but currently (and I know this
is Poor Form) I wrap each define-alien-routine with some extra
eval-when stuff. 

> problem I ran into is that without the proper declarations these
> calls were wildly inefficient and had to do a whole bunch of stuff at
> runtime. It's possible you're just getting really large compiled
> functions for each of your foreign function definitions and that
> these are, as Nikodemus suggested, causing you to run out of memory.

It sounds like this is largely self-inflicted, so I shall attempt to
make this code unnecessary (I did it because I wanted to know about
redefining symbols multiple times, as one might get when one binds
that 8 or do different C++ atan routines (overloading) -- the right
thing to do would be to detect this at binding generation time).

> This is just a wild guess and I might be way-off here, but you might
> look at trying to get the proper declarations to keep these foreign
> function definitions small and more efficient at the same time.

Sounds sensible -- Thanks, and I shall check it out!

-jm


>
> Cyrus
>
> On Jun 11, 2005, at 2:25 AM, Nikodemus Siivola wrote:
> > On Fri, 10 Jun 2005, John Morrison wrote:
> >> Short version: I get a memory fault after loading shared object
> >> libraries, loading callback-sbcl.lisp, and a REALLY BIG compiled file
> >> of sb-alien bindings.  Any advice?  I am running 0.9.0.18 (from an
> >> RPM), and I have a recently-released RPM of 0.9.1-1.
> >
> > While building from source might help, I doubt it. LDB most certainly
> > won't help, as you seem to be still in a live SBCL after the memory
> > fault -- LDB is strictly for post mortem.
> >
> > My first guess is that you're simply running out of static space.
> > As a quick workaround: don't purify, and if you need to save an image,
> > do it with :PURIFY NIL.
> >
> > If you're purifying because you need stuff that doesn't move, then
> > SB-INT:MAKE-STATIC-VECTOR may be what you need: it allocates
> > vectors of immediate objects directly to static space. Do not be
> > overly alarmed by the SB-INT in this case: once the dust settles I
> > think it (or something very much like it) is destined to move to SB-
> > EXT. You get to beta-test it. ;-)
> >
> > If you're feeling adventurous, then twiddling the static-space-*
> > values in src/compiler/target/parms.lisp may help by giving you a
> > larger static space; searching sbcl-devel archives should yield
> > some advice for that.
> >
> > Similarly, building SBCL with the first PURIFY in make-target-2.sh
> > commented out should also conserve _some_ static space, and may be
> > sufficient for your needs, but I rather doubt it.
> >
> > As a longer term solution we should probably detect the end of
> > static space and signal a STORAGE-CONDITION if we run out of it.
> >
> > Cheers,
> >
> >  -- Nikodemus              Schemer: "Buddha is small, clean, and
> > serious."
> >                   Lispnik: "Buddha is big, has hairy armpits, and
> > laughs."
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: NEC IT Guy Games.  How far can
> > you shotput
> > a projector? How fast can you ride your desk chair down the office
> > luge track?
> > If you want to score the big prize, get to know the little guy.
> > Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
> > _______________________________________________
> > Sbcl-help mailing list
> > Sbcl-help@...
> > https://lists.sourceforge.net/lists/listinfo/sbcl-help
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you
> shotput a projector? How fast can you ride your desk chair down the office
> luge track? If you want to score the big prize, get to know the little guy.
> Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
> _______________________________________________
> Sbcl-help mailing list
> Sbcl-help@...
> https://lists.sourceforge.net/lists/listinfo/sbcl-help

-- 
==== John Morrison
==== MAK Technologies Inc.
==== 10 Fawcett Street, Cambridge, MA 02138
==== http://www.mak.com/
==== vox:617-876-8085 x115
==== fax:617-876-9208
==== jm@...

On Sat, 11 Jun 2005, John Morrison wrote:

>> My first guess is that you're simply running out of static space.
>> As a quick workaround: don't purify, and if you need to save an image,
>> do it with :PURIFY NIL.
>
> OK, I can do that ... thanks!  Two questions immediately spring to
> mind: (1) Is there any way I can verify that (that it's a simple space
> issue and not some Deep Dark Coding Malevolence)?

One simple but tedious way would be to add sufficient debugging output
to the signal handling in the runtime to see where the final memory fault 
comes from: it should coincide with the end of static space.

(The above could also be discerned with gdb given sufficient patience,
but I'd be more inclined to add output.)

Another option would be to mprotect the last page of static space
and detect that in the runtime, printing an appropriate flaming death 
message.

Yet another would be twiddling the spaces: if that helps, it was probably 
the cause...

> (2) Will/might
> there be fewer such problems with a 64-bit version of SBCL?

64-bitness has no direct bearing, but given the indecently large memory 
space I would suppose in effect that might be the case.

> The "callback" trampoline stuff has an advisory about that, but I
> cannot recall whether either the infrastructure or all my callbacks
> require the purify to keep them from moving...

If these are the callbacks I'm thinking about then purifying should not be 
necessary, but since I don't know which callbacks you are using...

Cheers,

  -- Nikodemus              Schemer: "Buddha is small, clean, and serious."
                   Lispnik: "Buddha is big, has hairy armpits, and laughs."