[Sbcl-devel] SBCL memory overflow

[Daniel L. <dla...@st...>]

I've been using SBCL for the last few days to do
bioinformatics. Essentially I'm running through a LARGE string (approx
25M chars) and generating some floating point numbers. So far I've
figured out that SBCL uses unicode and therefore have moved to using
base-string (and combining lines from the input file into chunks about
1k in size before concatenating them all together at the end so that
the allocator is happier). Now I am able to read in the string without
it crashing (and only using about 40 MB), but it still crashes at some
point before it finishes the following code.

It gets past the "made the array" but it doesn't ever finish the loop
before it runs out of memory and gives up in the GC. 

(defun score-sequence (scorealgo seq)
  "scorealgo is a function which will take a sequence and an index and
return a score for that index, seq is a genomic sequence stored as a
string"
  (declare
   (type (function (array fixnum) single-float) scorealgo))
  (let ((scores (make-array (length seq) :element-type 'single-float)))
;;    (declare (type  (simple-array single-float (*)) scores))
    (format t "Made the score array ok~%")
    (loop for i from 0 to (length seq) do
	  (setf (aref scores i)
		(the single-float (funcall scorealgo seq i))))
    scores))

I'm using the 0.9.9.0 version from debian on x86 from within SLIME. I
don't understand why there would be so much space allocated! Before
this function is entered there's about 48MB resident (including the
very long string being passed into this function as "seq"), and after
the array is created there's about 80MB, but then it climbs over about
a minute to 400 or so resident (as measured by top) and dies.

-------------- error info --------------

* Argh! gc_find_freeish_pages failed (restart_page), nbytes=89631608.
   Gen Boxed Unboxed LB   LUB  !move  Alloc  Waste   Trig    WP  GCs Mem-age
   0:     9     0     0     0     0    20440 16424  2000000    0   0  0.0000
   1:    37     1     0 52224 52260 214049144 16008 14706016    1   1  0.0594
   2:    20    10     0 21883 21900 89727008 28640  2000000    2   0  0.0000
   3:     0     0     0     0     0        0     0  2000000    0   0  0.0000
   4:     0     0     0     0     0        0     0  2000000    0   0  0.0000
   5:  1163  5794    46  5520   174 51184256 109952 53184256 1017   1  0.0000
   6:  6092     0     0     0     0 24952832     0  2000000 5809   0  0.0000
   Total bytes allocated=379933680
fatal error encountered in SBCL pid 21802(tid 3055782816):

--------------------------

In this case, the floats returned by "scorealgo" are actually only 0.0
or 1.0 (they need to be floats for later portions of the computation)
so I doubt that they're confusing the conservative collector into not
collecting something.

Here are some questions: 

Is it possible to set SBCL to use a larger address space on x86? Does
it require a recompile?

is SBCL smart enough to deal with the floating point numbers properly
given the declarations or is this boxing floats like crazy?

Would upgrading my machine to a 64 bit Sempron or Athlon solve my
problem or would it just take longer to die?

Does x86-64 use an exact generational garbage collector like the
recent PPC one?

Does anyone have suggestions for where I might look to solve my
garbage problem?

-- 
Daniel Lakeland
dla...@st...
http://www.street-artists.org/~dlakelan