Steel Bank Common Lisp

On Wed, Sep 14, 2005 at 01:40:58AM -0400, Scott E. Fahlman wrote:
> The short version is that 64-bit SBCL on an AMD-64 machine dies if I try to
> allocate more than 8GB of dynamic space.

I'm glad to hear that allocating 8GB memory works! :-) The x86-64 port
was developed on machines with significantly less physical memory, so
the large memory space were only superficially tested. 

> It's possible that this has to do with the 8GB of physical memory on the
> machine, but I suspect that this is just a coincidence, and that we are
> either hitting a virtual-memory limit in Linux, or in the process, or in
> the SBCL allocation machinery.
>
> Can anyone tell me what's going on here, and if there's a way to enable
> SBCL to use more space?

The dynamic space limits are set at compile-time. To increase the
allocation, edit the definition of DYNAMIC-SPACE-END in
src/compiler/x86-64/parms.lisp and recompile SBCL. I'd rather not make
the space any larger than 8GB in the default distribution until some
other issues are solved, since the GENCGC page tables use too much
memory with large spaces.

-- 
Juho Snellman

On Wed, 14 Sep 2005, Juho Snellman wrote:

> The dynamic space limits are set at compile-time. To increase the
> allocation, edit the definition of DYNAMIC-SPACE-END in
> src/compiler/x86-64/parms.lisp and recompile SBCL. I'd rather not make
> the space any larger than 8GB in the default distribution until some
> other issues are solved, since the GENCGC page tables use too much
> memory with large spaces.

I was just to send a very similar question to the list. Unlike the orignal 
poster, I was not runing out of memory at "runtime", but when dumping an image
with save-lisp-and-die, I was running out of static space, since I had
a really big hashtable. I can then follow your advice and recompile sbcl
with a modified parms.lisp, but my question is:

- What is safe to modify. Obvouosly, the memory areas cannot overlap, but which
   restrictions are on these memory areas anyway.

- Why are these memory areas at fixed location? My guess would be that it has
   something with typetags to do, or that it is needed by gc-algorithm. Just
   curious.

I also use linux on AMD64.

>

-- 
Gisle Sælensminde, Phd student, Scientific programmer
Computational biology unit, University of Bergen, Norway
Email: gisle@...

On Wed, Sep 14, 2005 at 02:53:34PM +0200, Gisle S=E6lensminde wrote:
> I was just to send a very similar question to the list. Unlike the orig=
nal=20
> poster, I was not runing out of memory at "runtime", but when dumping a=
n=20
> image
> with save-lisp-and-die, I was running out of static space, since I had
> a really big hashtable. I can then follow your advice and recompile sbc=
l
> with a modified parms.lisp, but my question is:

Alternatively you can specify :PURIFY NIL to SAVE-LISP-AND-DIE to dump
everything in dynamic space (this'll become the default on GENCGC
platforms in the near future).

> - What is safe to modify. Obvouosly, the memory areas cannot overlap, b=
ut=20
> which
>   restrictions are on these memory areas anyway.

The loader shouldn't map anything into those areas before SBCL has a=20
chance to call mmap. Basically conflicting things could be:
  * The program segments
  * The brk heap=20
  * The stack
  * Shared libraries
Execute "cat /proc/pid/maps" for approximation of what's likely to be
mapped by the system and where.

Additionally on x86-64 the static (or read-only? can't remember) space
must be located in the lowest 4GB. But the <4GB space is otherwise
pretty empty, so fitting larger spaces in there should be no problem.

> - Why are these memory areas at fixed location? My guess would be that =
it has
>   something with typetags to do, or that it is needed by gc-algorithm. =
Just
>   curious.

Because the core files (which are mapped into those memory areas on
startup) contain pointers. If the memory spaces moved, we'd either
need position independent code or fix the pointers on every startup.

--=20
Juho Snellman