Steel Bank Common Lisp

I am using sbcl for a web-server application (http://github.com/bvds/andes)
for an intelligent tutor homework system.  Each instance of a student
solving a problem has a user session associated with it.  Each session 
lasts for roughly 15 minutes (but could vary quite a bit) and uses 
roughly 1MByte of memory (on a 32 bit machine).   When the 
server is fully loaded, it can handle about 300 concurrent sessions 
without maxing out the cpu.

However, I am having lots of trouble with garbage collection, since 
the behaviour of the server badly violates the basic assumption of 
the generational garbage collector:  in my case, it is more likely that 
old data (after a session has closed) can be garbage collected 
than newer data.

For simultaneous 300 sessions, the memory usages looks like this:
Breakdown for dynamic space:
  274,297,608 bytes for 34,287,201 cons objects.
  95,976,736 bytes for 1,372,664 simple-character-string objects.
  37,470,400 bytes for 1,170,950 symbol objects.
  45,860,504 bytes for 1,064,281 other objects.
  453,605,248 bytes for 37,895,096 dynamic objects (space total.)


Any advice for what to do?  Two ideas:

1.  Modify the garbage collector so that it gcs the older data first.
(that is, do generation 0, 1, 2 in the usual way, but do generations 
3, 4, 5 backwards).

2.  Is there any way to garbage collect for a specific object
and its descendents?

Brett van de Sande

On 27 February 2010 02:49, Brett van de Sande <bvds@...> wrote:

> However, I am having lots of trouble with garbage collection, since

Can you be more specific? Are you running out of memory, are the GC
pauses too long, or what?

> the behaviour of the server badly violates the basic assumption of
> the generational garbage collector:  in my case, it is more likely that
> old data (after a session has closed) can be garbage collected
> than newer data.

I don't see how this violates the generational hypothesis, actually.
Surely "session closing" is a rare event: most of the time the older
the object is, the less likely it is to be up for collection.

Anyways.

> 1.  Modify the garbage collector so that it gcs the older data first.
> (that is, do generation 0, 1, 2 in the usual way, but do generations
> 3, 4, 5 backwards).

This is a non-starter, pretty much.

> 2.  Is there any way to garbage collect for a specific object
> and its descendents?

Nope.

What you can and want to do depends on your problem.

If this is "just" a performance issue, then a bigger nursery and
making higher GENERATION-BYTES-CONSED-BETWEEN-GCS for the top-few
generations will probably help. See section 7.13 in the fine manual
(assuming you have SBCL 1.32.33 or newer.)

If you are running out of heap space, running a full collection after
a session has closed is likely to help -- but this has obvious
performance implications.

[ Actually, one GC toggle that we don't have but could is SCHEDULE-GC
where you could specify the depth of the next GC, so that when a
session ends you could do (SCHEDULE-GC :FULL) -- so that next time GC
runs, it will be a full GC. Need to think about this. ]

Other alternatives are:

* Use stack allocation for short-lived data, so it will not cause
promotion of older objects so fast. This may require restructuring
control flow, obviously.

* Structure your large data to minimize the number of pointers: if you
can pack things into eg. (SIMPLE-ARRAY (UNSIGNED-BYTE 32) 1), it will
mean less work for the GC.

These don't radically change the equation, just ameliorate it. The
basic problem remains as you stated it: you have big chunks of old
data that become garbage rarely. The way around that is to:

* Keep long-lived data outside the heap: either in foreign memory, or
store it in a DB. Then you can release/delete it when you want.

Cheers,

 -- Nikodemus

>
>Can you be more specific? Are you running out of memory, are the GC
>pauses too long, or what?

No, I am not running out out of memory, but after about 20-30 minutes,
I get a gc pause that is several minutes long.  Also, I eventually get the error:

fatal error encountered in SBCL pid 8528(tid 46912619751744):
An mprotect call failed with ENOMEM. This probably means that the maximum amount
of separate memory mappings was exceeded. To fix the problem, either increase
the maximum with e.g. 'echo 262144 > /proc/sys/vm/max_map_count' or recompile
SBCL with a larger value for GENCGC-PAGE-BYTES in
'src/compiler/target/backend-parms.lisp'.

If I recompile with a larger value of gencgc-page-bytes, as suggested
then gc overall becomes *much* slower.

>> 1.  Modify the garbage collector so that it gcs the older data first.
>> (that is, do generation 0, 1, 2 in the usual way, but do generations
>> 3, 4, 5 backwards).
>
>This is a non-starter, pretty much.

What about doing a gc that only does generation 5?
That is, put a switch in garbage_collect so that, if desired, it only
calls  garbage_collect_generation(5,0)

>> 2.  Is there any way to garbage collect for a specific object
>> and its descendents?

OK, I understand that there is no lisp version of free().
However, would it help the garbage collection if I applied
something like the following to my session data at the close
of a session:

(defun dereference (obj)
       (cond ((consp obj)
              (dereference (car obj))
              (dereference (cdr obj))
              (setf (car obj) nil)
              (setf (cdr ojb) nil))
        ;; add similar actions for arrays, structs, etc.
))	      

Brett

>If this is "just" a performance issue, then a bigger nursery and
>making higher GENERATION-BYTES-CONSED-BETWEEN-GCS for the top-few
>generations will probably help. 

Doesn't work, see bug https://bugs.launchpad.net/sbcl/+bug/529014
Instead, I have been changing the parameters in gencgc.c itself:

josiah> diff sbcl-1.0.35/src/runtime/gencgc.c gencgc.c_original
4625c4626
<         generations[i].bytes_consed_between_gc = i<3?2*1024*1024:100*1024*1024;
---
>         generations[i].bytes_consed_between_gc = 2000000;

However, this doesn't quite work because the initial value of gc_trigger
is too small.  If I also change the initial value of gc_trigger:

josiah> diff sbcl-1.0.35/src/runtime/gencgc.c gencgc.c_original
4621c4622
<         generations[i].gc_trigger = i<3?2*1024*1024:200*1024*1024;
---
>         generations[i].gc_trigger = 2000000;
4625c4626
<         generations[i].bytes_consed_between_gc = i<3?2*1024*1024:100*1024*1024;
---
>         generations[i].bytes_consed_between_gc = 2000000;

Then sbcl crashes on startup.

>See section 7.13 in the fine manual
>(assuming you have SBCL 1.32.33 or newer.)

:-)

>If you are running out of heap space, running a full collection after
>a session has closed is likely to help -- but this has obvious
>performance implications.

I will try this, just to see what happens.

>[ Actually, one GC toggle that we don't have but could is SCHEDULE-GC
>where you could specify the depth of the next GC, so that when a
>session ends you could do (SCHEDULE-GC :FULL) -- so that next time GC
>runs, it will be a full GC. Need to think about this. ]
>
>Other alternatives are:
>
>* Use stack allocation for short-lived data, so it will not cause
>promotion of older objects so fast. This may require restructuring
>control flow, obviously.

I have been looking at this.  But it doesn't solve the big problem
of the 1MB of data that persists over the course of rougly 15 minutes.

>* Structure your large data to minimize the number of pointers: if you
>can pack things into eg. (SIMPLE-ARRAY (UNSIGNED-BYTE 32) 1), it will
>mean less work for the GC.

No dice.  Most of the memory is for lisp expressions in our AI system, 
structs, etc.  The data structure is pretty complicated.
>
>* Keep long-lived data outside the heap: either in foreign memory, or
>store it in a DB. Then you can release/delete it when you want.

See above.

BTW, if you want to see our application in action, you can try it out 
at:  http://gideon.eas.asu.edu/web-UI/login.html

Brett

On 27 February 2010 17:07, Brett van de Sande <bvds@...> wrote:
>
>>If this is "just" a performance issue, then a bigger nursery and
>>making higher GENERATION-BYTES-CONSED-BETWEEN-GCS for the top-few
>>generations will probably help.
>
> Doesn't work, see bug https://bugs.launchpad.net/sbcl/+bug/529014

Huh. I seem to have committed the wrong version. The page structure
definition on the lisp side is missing lutexes at its end, so every
generation after the first is off by one word.

Commit coming in after freeze, here's the diff in the meanwhile:

Index: src/code/gc.lisp
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/code/gc.lisp,v
retrieving revision 1.90
diff -u -r1.90 gc.lisp
--- src/code/gc.lisp    14 Feb 2010 20:17:40 -0000      1.90
+++ src/code/gc.lisp    27 Feb 2010 15:58:25 -0000
@@ -346,7 +346,9 @@
             (number-of-gcs int)
             (number-of-gcs-before-promotion int)
             (cum-sum-bytes-allocated unsigned-long)
-            (minimum-age-before-gc double)))
+            (minimum-age-before-gc double)
+            ;; The space for lutexes is reserved on non-lutex
platforms as well.
+            (lutexes (* t))))

 #!+gencgc
 (define-alien-variable generations

Cheers,

 -- Nikodemus

On 27 February 2010 16:46, Brett van de Sande <bvds@...> wrote:

> If I recompile with a larger value of gencgc-page-bytes, as suggested
> then gc overall becomes *much* slower.

What about adjusting the max_map_count?

> What about doing a gc that only does generation 5?
> That is, put a switch in garbage_collect so that, if desired, it only
> calls  garbage_collect_generation(5,0)

The way the GC works is that you cannot collect generation N+1 unless
generation N is empty -- so no. This is a fairly fundamental
assumption in the system.

> OK, I understand that there is no lisp version of free().
> However, would it help the garbage collection if I applied
> something like the following to my session data at the close
> of a session:
>
> (defun dereference (obj)
>       (cond ((consp obj)
>              (dereference (car obj))
>              (dereference (cdr obj))
>              (setf (car obj) nil)
>              (setf (cdr ojb) nil))
>        ;; add similar actions for arrays, structs, etc.
> ))

To a degree, yes.

Doing it always for everything is almost certainly a bad idea, but
doing things like CLRHASH for hash-tables or FILL for vectors can be
worthwhile.

It all depends: if cleaning out pointers means you're writing to
write-protected pages it is going to be a loss -- but if those pages
are dirty already, it is probably a win.

Cheers,

 -- Nikodemus

On 2010-02-27, at 17:04 , Nikodemus Siivola wrote:

> On 27 February 2010 16:46, Brett van de Sande <bvds@...> wrote:
>
>> If I recompile with a larger value of gencgc-page-bytes, as suggested
>> then gc overall becomes *much* slower.
>
> What about adjusting the max_map_count?
>
>> What about doing a gc that only does generation 5?
>> That is, put a switch in garbage_collect so that, if desired, it only
>> calls  garbage_collect_generation(5,0)
>
> The way the GC works is that you cannot collect generation N+1 unless
> generation N is empty -- so no. This is a fairly fundamental
> assumption in the system.
>
>> OK, I understand that there is no lisp version of free().
>> However, would it help the garbage collection if I applied
>> something like the following to my session data at the close
>> of a session:
>>
>> (defun dereference (obj)
>>       (cond ((consp obj)
>>              (dereference (car obj))
>>              (dereference (cdr obj))
>>              (setf (car obj) nil)
>>              (setf (cdr ojb) nil))
>>        ;; add similar actions for arrays, structs, etc.
>> ))

if you are in a position to do this, you could consider two options  
which are (+/-) variations of the "db" option nikodemus mentioned:
- resource (especially large) objects. that is, institute a protocol  
for them which is the equivalent of allocate&free.
- intern reappearing, but otherwise unique objects. that is, replace  
references with those to an "equal" object which will never be gc'd.

that is, even though you have already reduced the unique data per  
session to just 1MB, you may be able to reduce the data subject to gc  
a yet smaller subset of that.

>
> To a degree, yes.
>
> Doing it always for everything is almost certainly a bad idea, but
> doing things like CLRHASH for hash-tables or FILL for vectors can be
> worthwhile.
>
> It all depends: if cleaning out pointers means you're writing to
> write-protected pages it is going to be a loss -- but if those pages
> are dirty already, it is probably a win.
>
> Cheers,
>
>  -- Nikodemus
>
> ---------------------------------------------------------------------- 
> --------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Sbcl-help mailing list
> Sbcl-help@...
> https://lists.sourceforge.net/lists/listinfo/sbcl-help

On 27.02.2010, at 17:04, Nikodemus Siivola wrote:
[...]
> 
>> OK, I understand that there is no lisp version of free().
>> However, would it help the garbage collection if I applied
>> something like the following to my session data at the close
>> of a session:
>> 
>> (defun dereference (obj)
>>       (cond ((consp obj)
>>              (dereference (car obj))
>>              (dereference (cdr obj))
>>              (setf (car obj) nil)
>>              (setf (cdr ojb) nil))
>>        ;; add similar actions for arrays, structs, etc.
>> ))
> 
> To a degree, yes.
> 
> Doing it always for everything is almost certainly a bad idea, but
> doing things like CLRHASH for hash-tables or FILL for vectors can be
> worthwhile.
> 
> It all depends: if cleaning out pointers means you're writing to
> write-protected pages it is going to be a loss -- but if those pages
> are dirty already, it is probably a win.

I'm not sure it would help at all.  The way I understand the problem, we have 1MB per request x 300 requests per second = 300MB/s of memory arriving in the oldest generation.  This seems to be a pessimal memory usage pattern for our generational gc.  I can't see how manually breaking some pointers between objects in oldspace would trigger oldspace collection earlier than it does now.  (Remember, gc is a heuristic, as is process scheduling, memory allocation, disk request management and many other problems of that sort.  Applications cannot be blamed for not fulfilling the expectations of heuristic approaches.)
I dimly remember a parameter that sets the number of generations, and setting it to some small value (1? 2?) essentially reinstating non-generational behavior -- sorry I can't be more specific since I don't have the time to go spelunking in sbcl sources right now.  :(  But it might be interesting to switch off gengc, just for laughs, and see what happens.

Cheers,

Rudi

On 28 February 2010 13:14, Rudi Schlatte <rudi@...> wrote:

> I'm not sure it would help at all.  The way I understand the problem, we have 1MB per
> request x 300 requests per second = 300MB/s of memory arriving in the oldest generation.
> This seems to be a pessimal memory usage pattern for our generational gc.  I can't see how
> manually breaking some pointers between objects in oldspace would trigger oldspace
> collection earlier than it does now.  (Remember, gc is a heuristic, as is process

It won't trigger an oldspace collection earlier, no. It may, however
do several other beneficial things: allow some things to be collected
before they are promoted so far (via real pointers or accidental
conservativism); make oldspace scavenging faster (this is conditional
on so many things it's not funny -- but given the right set of
circumstances it can do so.)

> I dimly remember a parameter that sets the number of generations, and setting it to some
> small value (1? 2?) essentially reinstating non-generational behavior -- sorry I can't be
> more specific since I don't have the time to go spelunking in sbcl sources right now.  :(
> But it might be interesting to switch off gengc, just for laughs, and see what happens.

Ooh, that's right! I completely forgot that.

+highest-normal-generation+ and +pseudo-static-generation+ in
src/compiler/generic/parms.lisp

Setting the first to 0 and the second to 1 seems to at least build
just fine -- didn't test properly.

Cheers,

 -- Nikodemus

>>> OK, I understand that there is no lisp version of free().
>>> However, would it help the garbage collection if I applied
>>> something like the following to my session data at the close
>>> of a session:
>>> 
>>> (defun dereference (obj)
>>>       (cond ((consp obj)
>>>              (dereference (car obj))
>>>              (dereference (cdr obj))
>>>              (setf (car obj) nil)
>>>              (setf (cdr ojb) nil))
>>>        ;; add similar actions for arrays, structs, etc.
>>> ))
>> 
>> To a degree, yes.
>> 
>> Doing it always for everything is almost certainly a bad idea, but
>> doing things like CLRHASH for hash-tables or FILL for vectors can be
>> worthwhile.
>> 
>> It all depends: if cleaning out pointers means you're writing to
>> write-protected pages it is going to be a loss -- but if those pages
>> are dirty already, it is probably a win.
>
>I'm not sure it would help at all.  

Indeed, dereferencing objects only made the behavior of 
the garbage collector somewhat worse.  Since it does "stop-and-copy"
garbage collection steps, cleaning out pointers doesn't help.

Brett

I would like to give a status report on what I have done.
By way of refresher/update:  I am load testing a server by running
about 200 concurrent user sessions.  Each session lasts about 15
minutes and uses about 1.5 MB memory that lasts the course of an 
entire session.  Right now, I am running sbcl-1.0.36, with the 
following modifications, on 32 bit linux.

1.  Increase sb-vm:gencgc-page-bytes to 16384.  In my case, it ends up 
    that all the memory pages eventually get protected, so 
        max_map_count * gencgc-page-bytes
    must be larger than the total heap memory.

2.  Run sbcl with "--dynamic-space-size 1500"

3.  Tune garbage collector by setting:
     (def!constant +highest-normal-generation+ 2)
     (def!constant +pseudo-static-generation+ 3)
    and 
     (let ((b (* 1024 1024)))
      ;; gc rate associated with turns (about .5 sec)
      (setf (generation-bytes-consed-between-gcs 0) (* 1 b))
      ;; gc rate associated with time between overlapped turns
      (setf (generation-bytes-consed-between-gcs 1) (* 4 b))
      ;; gc rate associated with session length (15 min).
      (setf (generation-bytes-consed-between-gcs 2) (* 100 b))))


4.  If I run the server with the above changes, it appears that 
    garbage collection is working as desired:  garbage collection occurs
    every 3 minus (generation 2), doesn't use too much cpu, and seems to 
    be recovering the memory.  However, after about a half hour, I get a 
    heap exhausted error.   So, I tried to increase the heap size:
        (def!constant dynamic-space-end         #x3ffff000)
    in src/compiler/x86/parms.lisp (to meet up with bottom of shared libs)
    This gives 922742784 bytes of heap space, instead of 500K
    However the heap exhaustion error still occurs, although a bit later:
        Heap exhausted during allocation: 234094592 bytes available, 
	268435464 requested.

5.  For monitoring, I use the function:
    (defun gen-stats (&optional (str t))
           (format str "Gen consed gcs avgage~%")
           (loop for i from 0 to SB-VM:+pseudo-static-generation+ 
	       do (format str "~A ~A ~A ~A~%" i
			  (generation-bytes-allocated i)
			  (generation-number-of-gcs i)
			  (generation-average-age i))))
    Right before the heap exhaustion error, I get something like:
      Gen consed gcs avgage
      0 5878968 1 0.0d0
      1 14762664 1 2.7070593762751765d0
      2 401231264 9 2.40689281880088d0
      3 26066944 0 0.0d0
    Which indicates everything is fine.

Any help/ideas would be appreciated.  

Brett van de Sande

On Wed, Mar 10, 2010 at 9:29 PM, Brett van de Sande <bvds@...> wrote:

> 4.  If I run the server with the above changes, it appears that
>    garbage collection is working as desired:  garbage collection occurs
>    every 3 minus (generation 2), doesn't use too much cpu, and seems to
>    be recovering the memory.  However, after about a half hour, I get a
>    heap exhausted error.   So, I tried to increase the heap size:
>        (def!constant dynamic-space-end         #x3ffff000)
>    in src/compiler/x86/parms.lisp (to meet up with bottom of shared libs)
>    This gives 922742784 bytes of heap space, instead of 500K
>    However the heap exhaustion error still occurs, although a bit later:
>        Heap exhausted during allocation: 234094592 bytes available,
>        268435464 requested.
>

This doesn't sound like a crash during gc to me, but an attempt to allocate
a 256MB object (probably an array) when less heap space is available. Does
that ring any bells? If your code shouldn't be allocating single arrays that
are that large, can you get a backtrace at this point and see why one is
being allocated?

-- 
Juho Snellman

On 2010-03-10, at 21:29 , Brett van de Sande wrote:

>
> [ ... ]
>
> Any help/ideas would be appreciated.

your previous message indicated that you had tried "dereferencing"  
session-specific data.
did you try resourcing it?

On 10 March 2010 22:29, Brett van de Sande <bvds@...> wrote:

>     (let ((b (* 1024 1024)))
>      ;; gc rate associated with turns (about .5 sec)
>      (setf (generation-bytes-consed-between-gcs 0) (* 1 b))

This doesn't do what you think it does -- it does nothing. Doc says:

    Number of bytes that can be allocated to GENERATION before that
    generation is considered for garbage collection. This value is
meaningless for
    generation 0 (the nursery): see BYTES-CONSED-BETWEEN-GCS instead. Default is
    20Mb. Can be assigned to using SETF. Available on GENCGC platforms only.

(1Mb is also seems far too small by-the-seat-of-my-pants.)

> Any help/ideas would be appreciated.

You could split this into multiple instances: limit the number of
sessions per SBCL instance whatever it can comfortably handle (10? 50?
100?) Since you are keeping all the sessions in same image and all the
session data in the image, they are interfering with each other not
just via parallelism but via GC stress over time. Reduce the number of
sessions to reduce the stress: multiple instances should not consume
that much more in the way of global resources, but should provide
better control over local ones.

If you can't or won't split into instances, I still believe hacking
the data representation is the way to go. Complex data structures can
be externalized just as well as simple ones -- or at least represented
in a less pointer-intensive manner.

That said, you could try would be to rig things up for 2-space
collection, set nursery size  high (eg. 1024Mb), and manually call (GC
:FULL T) whenever a session closes. (Goal being to mostly collect only
when a whole session's dataset becomes garbage, but ASAP after that so
you don't end up with multiple garbage sessions in the tenured.)

Cheers,

 -- Nikodemus

On 10 March 2010 22:29, Brett van de Sande <bvds@...> wrote:

> 4.  If I run the server with the above changes, it appears that
>    garbage collection is working as desired:  garbage collection occurs
>    every 3 minus (generation 2), doesn't use too much cpu, and seems to
>    be recovering the memory.  However, after about a half hour, I get a
>    heap exhausted error.   So, I tried to increase the heap size:
>        (def!constant dynamic-space-end         #x3ffff000)
>    in src/compiler/x86/parms.lisp (to meet up with bottom of shared libs)
>    This gives 922742784 bytes of heap space, instead of 500K
>    However the heap exhaustion error still occurs, although a bit later:
>        Heap exhausted during allocation: 234094592 bytes available,
>        268435464 requested.

As Juho noted, this is during allocation, not GC. What's the heap
composition at this point? There should be a printout.

Cheers,

 -- Nikodemus

>
>If you can't or won't split into instances, I still believe hacking
>the data representation is the way to go. Complex data structures can
>be externalized just as well as simple ones -- or at least represented
>in a less pointer-intensive manner.

I came up with a rather interesting solution to this one (giving about
a 50% reduction in the size of my data structures).  I go through my 
data structures, which are basically large lisp expressions, and create 
a list of all distinct subexpressions.
Then I go through the data structures and remove any duplicate subexpression,
making all pointers to a given subexpression point to the same object
in memory.

Since the server never modifies these expressions, there is no
problem with several things pointing to the same subexpression.

Brett

On 13 March 2010 00:11, Brett van de Sande <bvds@...> wrote:

> I came up with a rather interesting solution to this one (giving about
> a 50% reduction in the size of my data structures).  I go through my
> data structures, which are basically large lisp expressions, and create
> a list of all distinct subexpressions.
> Then I go through the data structures and remove any duplicate subexpression,
> making all pointers to a given subexpression point to the same object
> in memory.
>
> Since the server never modifies these expressions, there is no
> problem with several things pointing to the same subexpression.

You might want to google "hash consing". (Esp. as combined with weak
hash-tables...)

Cheers

 -- Nikodemus

>On 10 March 2010 22:29, Brett van de Sande <bvds@...> wrote:
>
>> 4.  If I run the server with the above changes, it appears that
>>    garbage collection is working as desired:  garbage collection occurs
>>    every 3 minus (generation 2), doesn't use too much cpu, and seems to
>>    be recovering the memory.  However, after about a half hour, I get a
>>    heap exhausted error.   So, I tried to increase the heap size:
>>        (def!constant dynamic-space-end         #x3ffff000)
>>    in src/compiler/x86/parms.lisp (to meet up with bottom of shared libs)
>>    This gives 922742784 bytes of heap space, instead of 500K
>>    However the heap exhaustion error still occurs, although a bit later:
>>        Heap exhausted during allocation: 234094592 bytes available,
>>        268435464 requested.
>
>As Juho noted, this is during allocation, not GC. What's the heap
>composition at this point? There should be a printout.

I am still getting this problem.  What printout am I supposed to see?
How do I find the heap composition?  At this point I just get the error
message and there is no more lisp prompt:

   Heap exhausted during allocation: 91848704 bytes available, 134217736 requested.
   CORRUPTION WARNING in SBCL pid 12955(tid 2951347088):
   Signalling HEAP-EXHAUSTED in a WITHOUT-INTERRUPTS.
   The integrity of this image is possibly compromised.
   Continuing with fingers crossed.

   debugger invoked on a SB-KERNEL::HEAP-EXHAUSTED-ERROR in thread #<THREAD
                                                                  "Hunchentoot worker (client: 127.0.0.1:38527)" RUNNING
                                                                  {D5311C1}>:
     Heap exhausted (no more space for allocation).
   There are still 91848704 bytes available; the request was for 134217736 bytes.

   PROCEED WITH CAUTION.

If I look at cat /proc/12955/maps, there seems to be an allocation
that crosses dynamic-space-end at #x3ffff000:

   ...
   2a434000-2c438000 r-xp 2a434000 00:00 0 
   2c438000-35450000 rwxp 2c438000 00:00 0 
   35450000-3d454000 rwxp 35450000 00:00 0 
   3d454000-66c00000 rwxp 3d454000 00:00 0 
   ae2fc000-ae2fd000 rwxp ae2fc000 00:00 0 
   ae2fd000-ae2fe000 r-xp ae2fd000 00:00 0 
   ae2fe000-ae5fa000 rwxp ae2fe000 00:00 0 
   ...

But I have no idea how to proceed further.

Brett

>
>You might want to google "hash consing". (Esp. as combined with weak
>hash-tables...)

Thanks.  That (hash consing) is essentially what I am doing.  
The only twist is that I am saving the the lisp expressions to a 
file (along with a table of pointers) and, when I am running my 
server, I reconstruct the cons pointers as I read the expressions 
from the file.

Brett

Further update:

After running some further benchmarks, I found the best garbage
collection parameters for my application are for two generations with:

   (setf (bytes-consed-between-gcs) (* 100 1024 1024))
   (setf (generation-bytes-consed-between-gcs 1) (* 100 1024 1024))

In this case, garbage collection takes less than 10% of
the cpu time and the pauses are still short enough <1s.
(One generation is very inefficient, and three generations
 is slightly slower than two.)

Brett

>>
>> Thanks.  That (hash consing) is essentially what I am doing.
>> The only twist is that I am saving the the lisp expressions to a
>> file (along with a table of pointers) and, when I am running my
>> server, I reconstruct the cons pointers as I read the expressions
>> from the file.
>
>you are not describing everything.
>
>when is "when i am running my server"? when it starts up, or on each  
>request?
>
>in any case, your problem does not sound like one where hash consing  
>will be the complete answer.
>from what you have revealed so far, you do not want to release the  
>resources.
>
Sorry for the vagueness.  What I do is the following:  I generate
model solutions to the problems in Andes beforehand and save them
in data files.  These model solutions are represented mainly in terms
of lisp expressions.

When I am running my Andes server and a student starts solving a problem,
I read in the model solutions from the appropriate data file.
In this case, I need the model solutions to be stored efficiently in
memory, since there will be several hundred students working simultaneously on 
different problems.  Once the student is finished working on that problem,
I no longer need that model solution and the data is freed for garbage
collection.

Obviously, if several students are working on the same problem,
I could share some of the data, but I need to support the case
where all students are working on different problems.

Brett

On 2010-03-18, at 01:40 , Brett van de Sande wrote:

>>>
>>> Thanks.  That (hash consing) is essentially what I am doing.
>>> The only twist is that I am saving the the lisp expressions to a
>>> file (along with a table of pointers) and, when I am running my
>>> server, I reconstruct the cons pointers as I read the expressions
>>> from the file.
>>
>> you are not describing everything.
>>
>> when is "when i am running my server"? when it starts up, or on each
>> request?
>>
>> in any case, your problem does not sound like one where hash consing
>> will be the complete answer.
>> from what you have revealed so far, you do not want to release the
>> resources.
>>
> Sorry for the vagueness.  What I do is the following:  I generate
> model solutions to the problems in Andes beforehand and save them
> in data files.  These model solutions are represented mainly in terms
> of lisp expressions.
>
> When I am running my Andes server and a student starts solving a  
> problem,
> I read in the model solutions from the appropriate data file.
> In this case, I need the model solutions to be stored efficiently in
> memory, since there will be several hundred students working  
> simultaneously on
> different problems.  Once the student is finished working on that  
> problem,
> I no longer need that model solution and the data is freed for garbage
> collection.
>
> Obviously, if several students are working on the same problem,
> I could share some of the data, but I need to support the case
> where all students are working on different problems.

what you have so far revealed implies that

- you have persistent models which you manipulate and present
   in response to requests.
- these models combine some persistent/static data and some
   dynamic/per-session data.
- the complete set of models takes more space than the heap would allow.
- the models for any given actual simultaneous session mix do fit in  
heap.
- you load model state from external storage on demand.
- you free models when no longer used - through gc possibly combined
   with weak sets and/or releasing.

given this situation, you experience memory exhaustion despite that  
no session ever requests more than its unit resources.

? what happens if you arrange the request mix such that the models  
are never released. that is, there is always some session which uses  
each model and the session mix is limited such that no more than the  
maximum expected permitted number of models is required. does this  
eventually exhaust memory as well or does it survive?

if such a configuration survives, then one option is for you to  
arrange to never release the static model memory. instead, you  
recycle it.
when a model must be loaded, you reuse the storage from a previous,  
no longer in use model. if there is none available, then the request  
must wait. if you can keep the per-session data small enough that it  
is never promoted, then you never require more than ephemeral gc.

if the entire model is s-expressions and function objects, reuse  
would be difficult as the former are very fine-grained and the latter  
are difficult to recycle. one would need more information about the  
model. if there are higher-level composite data structures, the  
approach offers advantages.

>what you have so far revealed implies that
>
>- you have persistent models which you manipulate and present
>   in response to requests.
>- these models combine some persistent/static data and some
>   dynamic/per-session data.
>- the complete set of models takes more space than the heap would allow.
>- the models for any given actual simultaneous session mix do fit in  
>heap.
>- you load model state from external storage on demand.
>- you free models when no longer used - through gc possibly combined
>   with weak sets and/or releasing.

Yes.  I think you have described it correctly.

>given this situation, you experience memory exhaustion despite that  
>no session ever requests more than its unit resources.

If I monitor garbage collection, using (generation-bytes-allocated i), 
and memory usage, using (room), I find the dynamic space size is 
stable and that session data is being recovered by the garbage collection 
after sessions are finished.  This goes on for about 30 minutes (a session
is average 14 minutes long) with the dynamic space size staying
under 300MB.  Then I get the memory exhaustion error.
>
>? what happens if you arrange the request mix such that the models  
>are never released. that is, there is always some session which uses  
>each model and the session mix is limited such that no more than the  
>maximum expected permitted number of models is required. does this  
>eventually exhaust memory as well or does it survive?

I have thought about sharing models between sessions where there
are multiple students solving the same problem.  It is a good idea
but will require some work to implement.
>
>if the entire model is s-expressions and function objects, reuse  
>would be difficult as the former are very fine-grained and the latter  
>are difficult to recycle. one would need more information about the  
>model. if there are higher-level composite data structures, the  
>approach offers advantages.

Yes, reuse of model data structures between sessions solving differenct 
problems would be a bad idea.

Brett

On 18 March 2010 01:43, Brett van de Sande <bvds@...> wrote:

>>As Juho noted, this is during allocation, not GC. What's the heap
>>composition at this point? There should be a printout.
>
> I am still getting this problem.  What printout am I supposed to see?
> How do I find the heap composition?  At this point I just get the error
> message and there is no more lisp prompt:

Ooops, I was wrong. I thought we printed the heap composition
(print_generation_stats) in case of all heap exhaustions, but we were
only doing it only for heap exhaustion during GC.

Will fix this in CVS today.

Re. no more lisp prompt: do you get into LDB? Do you know (even
approximately) what is being allocated and why when you run out of
heap?

Using something along the lines of

(defun handle-connection (conn)
  (handler-bind ((storage-condition
                  (lambda (c)
                    (format *error-output* "~A in ~A:~%  ~A~%"
                            (type-of c)
                            conn
                            c)
                    (sb-debug:backtrace most-positive-fixnum *error-output*)
                    (return-from handle-connection nil))))
    ...))

may help -- assuming the image is not too far gone for it to run.

If it is, you can use gdb to get backtraces. See
http://www.sbcl.org/manual/Reporting-Bugs.html (section "Signal
Related Bugs") for how.

Otherwise, what happens if you just run with a bigger heap? Just
cranking up --dynamic-space-size from the command line should not
effect your performance besides giving you more space to play in: GC
frequency and effectiveness should not change.

Cheers,

 -- Nikodemus

On 18 March 2010 09:51, Nikodemus Siivola <nikodemus@...> wrote:

> Ooops, I was wrong. I thought we printed the heap composition
> (print_generation_stats) in case of all heap exhaustions, but we were
> only doing it only for heap exhaustion during GC.
>
> Will fix this in CVS today.

In 1.0.36.30.

Cheers,

 -- Nikodemus

On 2010-03-18, at 02:55 , Brett van de Sande wrote:

> [...]
> Yes, reuse of model data structures between sessions solving  
> differenct
> problems would be a bad idea.

if you could say more about the model than that it is a mix of cons,  
array, and structure objects, it might be easier to understand what  
can be shared. whether between sessions solving the same problem or  
sessions solving different ones.

>
>Re. no more lisp prompt: do you get into LDB? Do you know (even
>approximately) what is being allocated and why when you run out of
>heap?

No.  I don't even get LDB
>
>If it is, you can use gdb to get backtraces. See
>http://www.sbcl.org/manual/Reporting-Bugs.html (section "Signal
>Related Bugs") for how.

OK. I will look into this, thanks.
>
>Otherwise, what happens if you just run with a bigger heap? Just
>cranking up --dynamic-space-size from the command line should not
>effect your performance besides giving you more space to play in: GC
>frequency and effectiveness should not change.

I have already increased dynamic-space-end to #x3ffff000 
in src/compiler/x86/parms.lisp.  According to the documentation,
0x40000000 is the bottom of the shared library space in 32 bit linux.
Also, I am using --dynamic-space-size 1500.  Should I go higher?

Brett

On 18 March 2010 15:46, Brett van de Sande <bvds@...> wrote:

> I have already increased dynamic-space-end to #x3ffff000
> in src/compiler/x86/parms.lisp.  According to the documentation,
> 0x40000000 is the bottom of the shared library space in 32 bit linux.
> Also, I am using --dynamic-space-size 1500.  Should I go higher?

There is little point in adjusting dynamic-space-end if you use
--dynamic-space-size: the latter will override the first. You should
go as high as you need to, asssuming you can go as high as you need
to...

Theoretically (if no garbage is retained via conservativism) you
should in the worst case need 2*max-live-data space.

If a session should use only 1Mb memory, then ~600Mb should be enough.

However, something in the code is allocating 256Mb objects -- most
likely arrays of some sort.

If you can get 1 such live object per session, you will need a much
larger heap than 32-bit Linux can support.

I think the next step is tracking down that large allocation. Perhaps
there is an ill-adviced MAKE-HASH-TABLE :REHASH-SIZE somewhere?
(Floats larger than 2.0 there are *bad*.)

Cheers,

 -- Nikodemus

>> Will fix this in CVS today.
>
>In 1.0.36.30.
>
>Cheers,
>
> -- Nikodemus
>
OK, I re-ran my program with the version from CVS.  I also increased
the dynamic space size:   --dynamic-space-size 1800
So it ran for much longer (1 hour) before crashing.  Lisp is still
frozen after the error (no LDB, no debugger, etc.)  Here is the error
message:

Heap exhausted during allocation: 60850176 bytes available, 536870920 requested.
 Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB   LUB  !move  Alloc  Waste   Trig
   WP  GCs Mem-age
   0:  2811     0     0     0   461     4 16385 24579 41043 678133352 639384 674324808    0   1  0.0000
   1:  3380  3379     0     0 12033  4243 14937 22377 26197 866312856 11705704 295005904 10946  86  0.7072
   2:     0     0     0     0  1316   278     0     0     0 26116096     0  2000000  936   0  0.0000
   Total bytes allocated    = 1570562304
   Dynamic-space-size bytes = 1887436800
GC control variables:
   *GC-INHIBIT* = true
   *GC-PENDING* = false
   *STOP-FOR-GC-PENDING* = false
CORRUPTION WARNING in SBCL pid 26071(tid 3003935632):
Signalling HEAP-EXHAUSTED in a WITHOUT-INTERRUPTS.
The integrity of this image is possibly compromised.
Continuing with fingers crossed.

debugger invoked on a SB-KERNEL::HEAP-EXHAUSTED-ERROR in thread #<THREAD
                                                                  "Hunchentoot w           orker (client: 127.0.0.1:54874)" RUNNING
                                                                  {177F9061}>:
  Heap exhausted (no more space for allocation).
There are still 60850176 bytes available; the request was for 536870920 bytes.

PROCEED WITH CAUTION.

>> I have already increased dynamic-space-end to #x3ffff000
>> in src/compiler/x86/parms.lisp.  According to the documentation,
>> 0x40000000 is the bottom of the shared library space in 32 bit linux.
>> Also, I am using --dynamic-space-size 1500.  Should I go higher?
>
>There is little point in adjusting dynamic-space-end if you use
>--dynamic-space-size: the latter will override the first. You should
>go as high as you need to, asssuming you can go as high as you need
>to...

Thanks.  I was long confused about this point.
>
>Theoretically (if no garbage is retained via conservativism) you
>should in the worst case need 2*max-live-data space.
>
>If a session should use only 1Mb memory, then ~600Mb should be enough.
>
In my case, my dynamic memory usage stays comfortably within 500 MB
(353 MB average), max about 450 MB.

>However, something in the code is allocating 256Mb objects -- most
>likely arrays of some sort.
>
>If you can get 1 such live object per session, you will need a much
>larger heap than 32-bit Linux can support.

Actually, I don't have any large arrays or very large objects.
The total data associated with one user session is about 500kB.
>
>I think the next step is tracking down that large allocation. Perhaps
>there is an ill-adviced MAKE-HASH-TABLE :REHASH-SIZE somewhere?
>(Floats larger than 2.0 there are *bad*.)

The closest I have to that is calling (remhash session-hash *sessions*)
when a user session ends...

Brett

>>> Will fix this in CVS today.
>>
>>In 1.0.36.30.
>>
>>Cheers,
>>
>> -- Nikodemus
>>
>OK, I re-ran my program with the version from CVS.  I also increased
>the dynamic space size:   --dynamic-space-size 1800

I ran again, with --dynamic-space-size 1000 and it crashed after about 
55 minutes, with the same error.  (Plus some warning messages about 
hash tables!)  This time, I am attaching the complete
list of backtraces from gdb, as well as the error message itself.

Just to be clear, I am running the latest sbcl from cvs with the
following modifications:
  (def!constant gencgc-page-bytes (* 4 *backend-page-bytes*))
  (def!constant +highest-normal-generation+ 1)
  (def!constant +pseudo-static-generation+ 2))

Brett

****************************************************************************


298 sessions
Gen byte alloc gcs avgage
  0   58971808   1    0.0
  1  196937208  28 4.8518
  2   26116096   0    0.0
time:  2.37s gc and 52.712s total run time
Warning: no pointer at 1607c008 in hash table: this indicates non-fatal corruption caused by concurrent access to a hash-table from multiple threads. Any accesses to hash-tables shared between threads should be protected by locks.
Warning: no pointer at 222e8008 in hash table: this indicates non-fatal corruption caused by concurrent access to a hash-table from multiple threads. Any accesses to hash-tables shared between threads should be protected by locks.
Warning: no pointer at 222e8008 in hash table: this indicates non-fatal corruption caused by concurrent access to a hash-table from multiple threads. Any accesses to hash-tables shared between threads should be protected by locks.
Warning: no pointer at 222e8008 in hash table: this indicates non-fatal corruption caused by concurrent access to a hash-table from multiple threads. Any accesses to hash-tables shared between threads should be protected by locks.
Warning: no pointer at 222e8008 in hash table: this indicates non-fatal corruption caused by concurrent access to a hash-table from multiple threads. Any accesses to hash-tables shared between threads should be protected by locks.
Warning: no pointer at 222e8008 in hash table: this indicates non-fatal corruption caused by concurrent access to a hash-table from multiple threads. Any accesses to hash-tables shared between threads should be protected by locks.
Warning: no pointer at 222e8008 in hash table: this indicates non-fatal corruption caused by concurrent access to a hash-table from multiple threads. Any accesses to hash-tables shared between threads should be protected by locks.
Warning: no pointer at 28ce0008 in hash table: this indicates non-fatal corruption caused by concurrent access to a hash-table from multiple threads. Any accesses to hash-tables shared between threads should be protected by locks.
Warning: no pointer at 222e8008 in hash table: this indicates non-fatal corruption caused by concurrent access to a hash-table from multiple threads. Any accesses to hash-tables shared between threads should be protected by locks.
Warning: no pointer at 222e8008 in hash table: this indicates non-fatal corruption caused by concurrent access to a hash-table from multiple threads. Any accesses to hash-tables shared between threads should be protected by locks.
Heap exhausted during allocation: 32456704 bytes available, 67108872 requested.
 Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB   LUB  !move  Alloc  Waste   Trig    WP  GCs Mem-age
   0:  1622     0 62018     0   160     3 12290  7459  4190 325654880 583328 193420760    0   1  0.0000
   1:     0     0     0     0 11187  3959  2583  5376  2615 376819760 1732560 481677360 12892  30  0.0000
   2:     0     0     0     0  1316   278     0     0     0 26116096     0  2000000  940   0  0.0000
   Total bytes allocated    = 728590736
   Dynamic-space-size bytes = 1048576000
GC control variables:
   *GC-INHIBIT* = true
   *GC-PENDING* = true
   *STOP-FOR-GC-PENDING* = true
CORRUPTION WARNING in SBCL pid 2958(tid 2986326928):
Signalling HEAP-EXHAUSTED in a WITHOUT-INTERRUPTS.
The integrity of this image is possibly compromised.
Continuing with fingers crossed.

debugger invoked on a SB-KERNEL::HEAP-EXHAUSTED-ERROR in thread #<THREAD
                                                                  "Hunchentoot worker (client: 127.0.0.1:54102)" RUNNING
                                                                  {B5AB059}>:
  Heap exhausted (no more space for allocation).
There are still 32456704 bytes available; the request was for 67108872 bytes.

PROCEED WITH CAUTION.


****************************************************************************************************

attach gdb:

(gdb) thread apply all ba

Thread 24 (Thread 0xb7fdeac0 (LWP 2958)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0x0, context=0xb7a035dc) at thread.h:53
#3  0x08057190 in interrupt_handle_pending (context=0xb7a035dc) at interrupt.c:861
#4  0x0805620b in low_level_handle_now_handler (signal=5, info=0xb7a0355c, void_context=0xb7a035dc)
    at interrupt.c:1713
#5  <signal handler called>
#6  0x09d047be in ?? ()
#7  0x00000000 in ?? ()

Thread 23 (Thread 0xb75b3b90 (LWP 2960)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0x0, context=0xb75b2acc) at thread.h:53
#3  0x08057190 in interrupt_handle_pending (context=0xb75b2acc) at interrupt.c:861
#4  0x0805620b in low_level_handle_now_handler (signal=5, info=0xb75b2a4c, void_context=0xb75b2acc)
    at interrupt.c:1713
#5  <signal handler called>
#6  0x0100057d in ?? ()
#7  0x0935b42a in ?? ()
#8  0x0a9b00ff in ?? ()
#9  0x00000000 in ?? ()

Thread 22 (Thread 0xb716db90 (LWP 2961)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb716cb9c, context=0xb716cc1c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb716cb9c, void_context=0xb716cc1c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0xb7fff410 in __kernel_vsyscall ()
#6  0x00b0d116 in nanosleep () from /lib/libpthread.so.0
#7  0x09b9e356 in ?? ()
#8  0xb736dff4 in ?? ()
#9  0xb736dfec in ?? ()
#10 0x0b447724 in ?? ()
#11 0x00000081 in ?? ()
#12 0x00000001 in ?? ()
#13 0x00000000 in ?? ()

Thread 21 (Thread 0xb6d27b90 (LWP 2963)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb6d26b9c, context=0xb6d26c1c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb6d26b9c, void_context=0xb6d26c1c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0xb7fff410 in __kernel_vsyscall ()
#6  0x00b0d116 in nanosleep () from /lib/libpthread.so.0
#7  0x09b9e356 in ?? ()
#8  0xb6f27ff4 in ?? ()
#9  0xb6f27fec in ?? ()
#10 0x0b49c624 in ?? ()
#11 0x000337a0 in ?? ()
#12 0x00000fa0 in ?? ()
#13 0x0c707195 in ?? ()
#14 0x0fef9fe3 in ?? ()
#15 0x009272a8 in ?? ()
#16 0xb6d26f80 in ?? ()
#17 0x00000000 in ?? ()

Thread 20 (Thread 0xb5c0fb90 (LWP 4289)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0x0, context=0xb5c0d5ac) at thread.h:53
#3  0x08057190 in interrupt_handle_pending (context=0xb5c0d5ac) at interrupt.c:861
#4  0x0805620b in low_level_handle_now_handler (signal=5, info=0xb5c0d52c, void_context=0xb5c0d5ac)
    at interrupt.c:1713
#5  <signal handler called>
#6  0x0932a63d in ?? ()
#7  0x00000000 in ?? ()

Thread 19 (Thread 0xb57c6b90 (LWP 4291)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0x0, context=0xb57c399c) at thread.h:53
#3  0x08057190 in interrupt_handle_pending (context=0xb57c399c) at interrupt.c:861
#4  0x0805620b in low_level_handle_now_handler (signal=5, info=0xb57c391c, void_context=0xb57c399c)
    at interrupt.c:1713
#5  <signal handler called>
#6  0x0fa84d4c in ?? ()
#7  0x10e0e54d in ?? ()
#8  0x011002e7 in ?? ()
#9  0x011002e7 in ?? ()
#10 0xb57c3ca3 in ?? ()
#11 0x00000038 in ?? ()
#12 0x00aae938 in ?? ()
#13 0x01100027 in ?? ()
#14 0x2fdc0007 in ?? ()
#15 0x0aa8a767 in ?? ()
#16 0x0110000b in ?? ()
#17 0x00000010 in ?? ()
#18 0x00000004 in ?? ()
#19 0x152d7a5b in ?? ()
#20 0x0aa8a167 in ?? ()
#21 0x00000008 in ?? ()
#22 0x0110000b in ?? ()
#23 0xb57c3d18 in ?? ()
#24 0x0e67fab0 in ?? ()
#25 0x0aa8a167 in ?? ()
#26 0x152d7a5b in ?? ()
#27 0xb57c3d18 in ?? ()
#28 0x00000010 in ?? ()
#29 0x0c1bfe87 in ?? ()
#30 0x0b4f3233 in ?? ()
#31 0x0b4f2d3f in ?? ()
#32 0x0c1bfe87 in ?? ()
#33 0x0b4f3233 in ?? ()
#34 0x0c1f0147 in ?? ()
#35 0x0c898a2b in ?? ()
#36 0x0f6724ad in ?? ()
#37 0x0b4f3253 in ?? ()
#38 0x0aa8a167 in ?? ()
#39 0x0cbeea81 in ?? ()
#40 0xb57c3cd8 in ?? ()
#41 0xb57c3d58 in ?? ()
#42 0x0e91aa14 in ?? ()
#43 0xb57c3d58 in ?? ()
#44 0x09f303d3 in ?? ()
#45 0x0b4f30f3 in ?? ()
#46 0x0b4f322b in ?? ()
#47 0x0110000b in ?? ()
#48 0x0b4f2d3f in ?? ()
#49 0x0aa8a167 in ?? ()
#50 0x152d7a5b in ?? ()
#51 0x0b4f324b in ?? ()
#52 0x0b4f3239 in ?? ()
#53 0x0e91aa9e in ?? ()
#54 0x0110000b in ?? ()
#55 0x0b4f2d3f in ?? ()
#56 0x0b4f324b in ?? ()
#57 0xb57c3d98 in ?? ()
#58 0x0e91aa9e in ?? ()
#59 0x0b4f304d in ?? ()
#60 0x00000010 in ?? ()
#61 0x0c1bfe87 in ?? ()
#62 0x0b4f30d3 in ?? ()
#63 0x0110000b in ?? ()
#64 0x0b4f2d3f in ?? ()
#65 0x0aa8a767 in ?? ()
#66 0x152d7a5b in ?? ()
#67 0x0b4f30f3 in ?? ()
#68 0x0b4f30e1 in ?? ()
#69 0x0b4f3163 in ?? ()
#70 0x0110000b in ?? ()
#71 0x0b4f2d3f in ?? ()
#72 0x0b4f30f3 in ?? ()
#73 0xb57c3dd8 in ?? ()
#74 0x0e91aa9e in ?? ()
#75 0xb57c3dd8 in ?? ()
#76 0x00000010 in ?? ()
#77 0x0c1bfe87 in ?? ()
#78 0x0b4f306b in ?? ()
#79 0x0110000b in ?? ()
#80 0x0b4f2d3f in ?? ()
#81 0x0aa8b8a7 in ?? ()
#82 0x152d7a5b in ?? ()
#83 0x0b4f308b in ?? ()
#84 0x0b4f3079 in ?? ()
#85 0x0b4f30cb in ?? ()
#86 0x0110000b in ?? ()
#87 0x0b4f2d3f in ?? ()
#88 0x0b4f308b in ?? ()
#89 0xb57c3e18 in ?? ()
#90 0x0e91aa9e in ?? ()
#91 0xb57c3e18 in ?? ()
#92 0x09f303d3 in ?? ()
#93 0x0b4f2e4b in ?? ()
#94 0x0b4f3003 in ?? ()
#95 0x0110000b in ?? ()
#96 0x0b4f2d3f in ?? ()
#97 0x0901a5df in ?? ()
#98 0x152d7a5b in ?? ()
#99 0x0b4f3023 in ?? ()
#100 0x0b4f3011 in ?? ()
#101 0x0b4f3063 in ?? ()
#102 0x0110000b in ?? ()
#103 0x0b4f2d3f in ?? ()
#104 0x0b4f3023 in ?? ()
#105 0xb57c3e58 in ?? ()
#106 0x0e91aa9e in ?? ()
#107 0x0b4f2d8d in ?? ()
#108 0x00000010 in ?? ()
#109 0x0c1bfe87 in ?? ()
#110 0x0b4f2e2b in ?? ()
#111 0x0110000b in ?? ()
#112 0x0b4f2d3f in ?? ()
#113 0x0901aa77 in ?? ()
#114 0x152d7a5b in ?? ()
#115 0x0b4f2e4b in ?? ()
#116 0x0b4f2e39 in ?? ()
#117 0x0b4f2eab in ?? ()
#118 0x0110000b in ?? ()
#119 0x0b4f2d3f in ?? ()
#120 0x0b4f2e4b in ?? ()
#121 0xb57c3e98 in ?? ()
#122 0x0e91aa9e in ?? ()
#123 0x0b4f2d15 in ?? ()
#124 0x00000010 in ?? ()
#125 0x0c1bfe87 in ?? ()
#126 0x0b4f2dab in ?? ()
#127 0x0110000b in ?? ()
#128 0x0b4f2d3f in ?? ()
#129 0x0aa83da7 in ?? ()
#130 0x152d7a5b in ?? ()
#131 0x0b4f2dcb in ?? ()
#132 0x0b4f2db9 in ?? ()
#133 0x0b4f2e0b in ?? ()
#134 0x0110000b in ?? ()
#135 0x0b4f2d3f in ?? ()
#136 0x0b4f2dcb in ?? ()
#137 0xb57c3ed8 in ?? ()
#138 0x0e91aa9e in ?? ()
#139 0x48b24d8c in ?? ()
#140 0x00000200 in ?? ()
#141 0x00000000 in ?? ()

Thread 18 (Thread 0xb537bb90 (LWP 4295)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0x0, context=0xb537a3ec) at thread.h:53
#3  0x08057190 in interrupt_handle_pending (context=0xb537a3ec) at interrupt.c:861
#4  0x0805620b in low_level_handle_now_handler (signal=5, info=0xb537a36c, void_context=0xb537a3ec)
    at interrupt.c:1713
#5  <signal handler called>
#6  0x090515de in ?? ()
#7  0x0c4fd17f in ?? ()
#8  0x00000014 in ?? ()
#9  0x00000008 in ?? ()
#10 0x0c4fd17f in ?? ()
#11 0x0ff9f15a in ?? ()
#12 0x00000004 in ?? ()
#13 0xb537a718 in ?? ()
#14 0x0ff9f15a in ?? ()
#15 0x0c4fce43 in ?? ()
#16 0x0110000b in ?? ()
#17 0x1a442da7 in ?? ()
#18 0x0b49bd57 in ?? ()
#19 0x0fc52ff0 in ?? ()
#20 0xb537a760 in ?? ()
#21 0xb537a760 in ?? ()
#22 0x0fc52ff0 in ?? ()
#23 0x0c4fd167 in ?? ()
#24 0x0b5b195f in ?? ()
#25 0x00000008 in ?? ()
#26 0x0c4fcf17 in ?? ()
#27 0x0c4fce43 in ?? ()
#28 0x0b4a420b in ?? ()
#29 0x00000004 in ?? ()
#30 0x0c4fd167 in ?? ()
#31 0x0c4fcf17 in ?? ()
#32 0x1a442da7 in ?? ()
#33 0x00000010 in ?? ()
#34 0x1a442da7 in ?? ()
#35 0x0c4fce43 in ?? ()
#36 0x0c4fd167 in ?? ()
#37 0x0c4fd16f in ?? ()
#38 0x0110000b in ?? ()
#39 0xb537a7a8 in ?? ()
#40 0x0fc5333a in ?? ()
#41 0x0c4fce4f in ?? ()
#42 0x0b5b195f in ?? ()
#43 0x00000004 in ?? ()
#44 0x0c4fcbd3 in ?? ()
#45 0x00000000 in ?? ()

Thread 17 (Thread 0xb2cdbb90 (LWP 4296)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0x0, context=0xb2cda3ac) at thread.h:53
#3  0x08057190 in interrupt_handle_pending (context=0xb2cda3ac) at interrupt.c:861
#4  0x0805620b in low_level_handle_now_handler (signal=5, info=0xb2cda32c, void_context=0xb2cda3ac)
    at interrupt.c:1713
#5  <signal handler called>
#6  0x09d047be in ?? ()
#7  0x01100027 in ?? ()
#8  0xb2edbf7c in ?? ()
#9  0xb2edbefc in ?? ()
#10 0xb2edbfec in ?? ()
#11 0x09ee64b5 in ?? ()
#12 0x0935b3fb in ?? ()
#13 0xb2cda718 in ?? ()
#14 0x09ee6563 in ?? ()
#15 0x01100027 in ?? ()
#16 0x00005d44 in ?? ()
#17 0x00000000 in ?? ()

Thread 16 (Thread 0xb4f30b90 (LWP 4298)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0x0, context=0xb4f2f4bc) at thread.h:53
#3  0x08057190 in interrupt_handle_pending (context=0xb4f2f4bc) at interrupt.c:861
#4  0x0805620b in low_level_handle_now_handler (signal=5, info=0xb4f2f43c, void_context=0xb4f2f4bc)
    at interrupt.c:1713
#5  <signal handler called>
#6  0x0100057d in ?? ()
#7  0x0935b42a in ?? ()
#8  0x00000000 in ?? ()

Thread 15 (Thread 0xb288eb90 (LWP 4300)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x0805d0e7 in gc_stop_the_world () at ./thread.h:53
#3  0x090bee43 in ?? ()
#4  0xb2a80372 in ?? ()
#5  0x0110000b in ?? ()
#6  0xb288d428 in ?? ()
#7  0x0110000b in ?? ()
#8  0x0110000b in ?? ()
#9  0x0901612f in ?? ()
#10 0xb288d3cc in ?? ()
#11 0xb288d428 in ?? ()
#12 0x0901612f in ?? ()
#13 0x00000000 in ?? ()

Thread 14 (Thread 0xb4ae5b90 (LWP 4301)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb4ae303c, context=0xb4ae30bc) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb4ae303c, void_context=0xb4ae30bc)
    at interrupt.c:1713
#4  <signal handler called>
#5  0xb7fff410 in __kernel_vsyscall ()
#6  0x00c78567 in syscall () from /lib/libc.so.6
#7  0x0805edad in futex_wake (lock_word=0x90bfcb0, n=1) at linux-os.c:103
#8  0x0931b7bc in ?? ()
#9  0x090bfcb0 in ?? ()
#10 0x00000001 in ?? ()
#11 0xb4ce0372 in ?? ()
#12 0x00b0c63c in __lll_unlock_wake () from /lib/libpthread.so.0
#13 0xb4ae33f8 in ?? ()
#14 0x090bfca1 in ?? ()
#15 0xb4ae3494 in ?? ()
#16 0x090c072f in ?? ()
#17 0x090162d3 in ?? ()
#18 0xb4ae3458 in ?? ()
#19 0xb4ae3458 in ?? ()
#20 0x01100527 in ?? ()
#21 0x79238fe0 in ?? ()
#22 0x0a8f8017 in ?? ()
#23 0xb4ae3458 in ?? ()
#24 0x0a8f801f in ?? ()
#25 0x00000008 in ?? ()
#26 0x00000000 in ?? ()

Thread 13 (Thread 0xb469ab90 (LWP 4302)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb469961c, context=0xb469969c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb469961c, void_context=0xb469969c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x09b3ba1b in ?? ()
#6  0x0110000b in ?? ()
#7  0x0110000b in ?? ()
#8  0x00000004 in ?? ()
#9  0x090610a7 in ?? ()
#10 0x01100027 in ?? ()
#11 0x0110000b in ?? ()
#12 0x0110000b in ?? ()
#13 0x09e924e1 in ?? ()
#14 0x0110000b in ?? ()
#15 0x0bfefad7 in ?? ()
#16 0x2ddb8007 in ?? ()
#17 0x0cbeea81 in ?? ()
#18 0x0905bd07 in ?? ()
#19 0x2edbc007 in ?? ()
#20 0xb4699a08 in ?? ()
#21 0x09b3b84b in ?? ()
#22 0x00000000 in ?? ()

Thread 12 (Thread 0xb2445b90 (LWP 4303)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0x0, context=0xb244466c) at thread.h:53
#3  0x08057190 in interrupt_handle_pending (context=0xb244466c) at interrupt.c:861
#4  0x0805620b in low_level_handle_now_handler (signal=5, info=0xb24445ec, void_context=0xb244466c)
    at interrupt.c:1713
#5  <signal handler called>
#6  0x01000b1f in ?? ()
#7  0x0e91aa4c in ?? ()
#8  0x0344a620 in ?? ()
#9  0x00000200 in ?? ()
#10 0x00000000 in ?? ()

Thread 11 (Thread 0xb4250b90 (LWP 4306)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb424f6bc, context=0xb424f73c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb424f6bc, void_context=0xb424f73c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x0924ee47 in ?? ()
#6  0x00000fa0 in ?? ()
#7  0x0935bc1f in ?? ()
#8  0x0935bc17 in ?? ()
#9  0x0935bc0f in ?? ()
#10 0x000009fc in ?? ()
#11 0x0935bc27 in ?? ()
#12 0x098b345a in ?? ()
#13 0x0913f391 in ?? ()
#14 0xb424fa6d in ?? ()
#15 0x00000000 in ?? ()

Thread 10 (Thread 0xb6055b90 (LWP 4312)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb605459c, context=0xb605461c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb605459c, void_context=0xb605461c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x0924e629 in ?? ()
#6  0x00000070 in ?? ()
#7  0x00000114 in ?? ()
#8  0x0a9615cf in ?? ()
#9  0x0a961467 in ?? ()
#10 0x00000184 in ?? ()
#11 0x00000000 in ?? ()

Thread 9 (Thread 0xb39beb90 (LWP 4313)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb39bcffc, context=0xb39bd07c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb39bcffc, void_context=0xb39bd07c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x099397cc in ?? ()
#6  0xb669c048 in ?? ()
#7  0xb39bd49c in ?? ()
#8  0xb39bd370 in ?? ()
#9  0xb39bf328 in ?? ()
#10 0xb39bde0c in ?? ()
#11 0xb3bbeffc in ?? ()
#12 0xb39bd3e4 in ?? ()
#13 0xb39bd3a8 in ?? ()
#14 0x09939a5a in ?? ()
#15 0xb39bd39c in ?? ()
#16 0xb3bbf048 in ?? ()
#17 0x0b5a9269 in ?? ()
#18 0xb39bd388 in ?? ()
#19 0x09246009 in ?? ()
#20 0xb39bd410 in ?? ()
#21 0x0936bb5c in ?? ()
#22 0x0000013a in ?? ()
#23 0x0110000b in ?? ()
#24 0x08067a24 in __dso_handle ()
#25 0xb39bd3b0 in ?? ()
#26 0xb39bd49c in ?? ()
#27 0x0004325f in ?? ()
#28 0x00000000 in ?? ()

Thread 8 (Thread 0xb1ffbb90 (LWP 4314)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00c78567 in syscall () from /lib/libc.so.6
#2  0x0805ee01 in futex_wait (lock_word=0xa8e89cc, oldval=17825803, sec=128, usec=0) at linux-os.c:103
#3  0x09ecb94e in ?? ()
#4  0x0a8e89cc in ?? ()
#5  0x0110000b in ?? ()
#6  0xffffffff in ?? ()
#7  0x00000000 in ?? ()

Thread 7 (Thread 0xb176fb90 (LWP 4316)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb176e6bc, context=0xb176e73c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb176e6bc, void_context=0xb176e73c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x0924ee47 in ?? ()
#6  0x00000000 in ?? ()

Thread 6 (Thread 0xb3573b90 (LWP 5164)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb357262c, context=0xb35726ac) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb357262c, void_context=0xb35726ac)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x09b3ba1b in ?? ()
#6  0x09060d87 in ?? ()
#7  0x0b10cb07 in ?? ()
#8  0x08065880 in alloc_overflow_ebx () at x86-assem.S:595
#9  0xb35729d0 in ?? ()
#10 0x0fed29bd in ?? ()
#11 0x092635e5 in ?? ()
#12 0x092c640d in ?? ()
#13 0x2ddb8007 in ?? ()
#14 0x0cbeea81 in ?? ()
#15 0x0905bd07 in ?? ()
#16 0x2edbc007 in ?? ()
#17 0xb3572a10 in ?? ()
#18 0x09b3b84b in ?? ()
#19 0x01100027 in ?? ()
#20 0x00000000 in ?? ()

Thread 5 (Thread 0xb0a9db90 (LWP 5183)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb0a9c5ec, context=0xb0a9c66c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb0a9c5ec, void_context=0xb0a9c66c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x09b3ba1b in ?? ()
#6  0x0b588e0f in ?? ()
#7  0x0110000b in ?? ()
#8  0x00000000 in ?? ()

Thread 4 (Thread 0xb0ee3b90 (LWP 5225)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb0ee25ec, context=0xb0ee266c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb0ee25ec, void_context=0xb0ee266c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x09b3ba11 in ?? ()
#6  0x00003288 in ?? ()
#7  0x00000000 in ?? ()

Thread 3 (Thread 0xb0657b90 (LWP 5687)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb06565dc, context=0xb065665c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb06565dc, void_context=0xb065665c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x0fe11379 in ?? ()
#6  0x0110000b in ?? ()
#7  0x0c1fc607 in ?? ()
#8  0x0a91c7a7 in ?? ()
#9  0x00000004 in ?? ()
#10 0x0a91c7a7 in ?? ()
#11 0x00000018 in ?? ()
#12 0xb0656988 in ?? ()
#13 0x10167019 in ?? ()
#14 0x15315c83 in ?? ()
#15 0x0a939d8f in ?? ()
#16 0x187065c3 in ?? ()
#17 0x0a91ca6b in ?? ()
#18 0x0110000b in ?? ()
#19 0x0110000b in ?? ()
#20 0xb06569a8 in ?? ()
#21 0x0fa60feb in ?? ()
#22 0x0923309d in ?? ()
#23 0x0a93b3d1 in ?? ()
#24 0x00000034 in ?? ()
#25 0x0110000b in ?? ()
#26 0x0a93b3d1 in ?? ()
#27 0x0a91ca3b in ?? ()
#28 0xb06569c8 in ?? ()
#29 0x0fc8cf06 in ?? ()
#30 0x0c20b507 in ?? ()
#31 0x0a939d8f in ?? ()
#32 0x0a93b3d1 in ?? ()
#33 0x0c20b507 in ?? ()
#34 0x0c844716 in ?? ()
#35 0xb06569ac in ?? ()
#36 0xb06569e8 in ?? ()
#37 0x0c844716 in ?? ()
#38 0x0110000b in ?? ()
#39 0x0110000b in ?? ()
#40 0x0c0511b1 in ?? ()
#41 0x0110000b in ?? ()
#42 0x0d791be1 in ?? ()
#43 0x0c20b507 in ?? ()
#44 0xb0656a28 in ?? ()
#45 0x0d791be1 in ?? ()
#46 0x0110000b in ?? ()
#47 0x0b41447d in ?? ()
#48 0x0c05119b in ?? ()
#49 0x0a93b3d1 in ?? ()
#50 0x0b113a27 in ?? ()
#51 0x0a93aed7 in ?? ()
#52 0x0c0511b1 in ?? ()
#53 0x0a91ca09 in ?? ()
#54 0x0110000b in ?? ()
#55 0x0c0511a3 in ?? ()
#56 0x0c0511ab in ?? ()
#57 0x01100027 in ?? ()
#58 0x0110000b in ?? ()
#59 0x0c050ee9 in ?? ()
#60 0xb0656a48 in ?? ()
#61 0x0da699d5 in ?? ()
#62 0x0c8452bb in ?? ()
#63 0x0b113a27 in ?? ()
#64 0x0a93b3d1 in ?? ()
#65 0x0a93aed7 in ?? ()
#66 0x0da42cc9 in ?? ()
#67 0x0c051163 in ?? ()
#68 0xb0656a68 in ?? ()
#69 0x0da42cc9 in ?? ()
#70 0x00000000 in ?? ()

Thread 2 (Thread 0xb68e1b90 (LWP 5757)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb68e06bc, context=0xb68e073c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb68e06bc, void_context=0xb68e073c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x0924ee47 in ?? ()
#6  0x00000fa0 in ?? ()
#7  0x0935bc1f in ?? ()
#8  0x0935bc17 in ?? ()
#9  0x0935bc0f in ?? ()
#10 0x00000988 in ?? ()
#11 0x0935bc27 in ?? ()
#12 0x098b345a in ?? ()
#13 0x0913f391 in ?? ()
#14 0xb68e0a6d in ?? ()
#15 0x00000000 in ?? ()

Thread 1 (Thread 0xb649bb90 (LWP 6250)):
#0  0xb7fff410 in __kernel_vsyscall ()
#1  0x00b099e5 in pthread_cond_wait@... () from /lib/libpthread.so.0
#2  0x08056e62 in sig_stop_for_gc_handler (signal=12, info=0xb649a2dc, context=0xb649a35c) at thread.h:53
#3  0x0805620b in low_level_handle_now_handler (signal=12, info=0xb649a2dc, void_context=0xb649a35c)
    at interrupt.c:1713
#4  <signal handler called>
#5  0x099397cc in ?? ()
#6  0x4ba28d00 in ?? ()
#7  0x0c7ef5f5 in ?? ()
#8  0xb649a650 in ?? ()
#9  0xb649c328 in ?? ()
#10 0xb649ae0c in ?? ()
#11 0xb669bffc in ?? ()
#12 0xb649a6c4 in ?? ()
#13 0xb649a688 in ?? ()
#14 0x09939a5a in ?? ()
#15 0xb66a0048 in ?? ()
#16 0x00000000 in ?? ()
#0  0xb7fff410 in __kernel_vsyscall ()

However this doesn't work:
(gdb) thread apply all call_backtrace_from_fp($ebp, 100)

Thread 24 (Thread 0xb7fdeac0 (LWP 2958)):
Undefined command: "call_backtrace_from_fp".  Try "help".

On 19 March 2010 00:00, Brett van de Sande <bvds@...> wrote:

> (gdb) thread apply all call_backtrace_from_fp($ebp, 100)
>
> Thread 24 (Thread 0xb7fdeac0 (LWP 2958)):
> Undefined command: "call_backtrace_from_fp".  Try "help".

Oops, there's a typo in the manual! It's

 thread apply all call backtrace_from_fp($ebp, 100)

not call_backtrace_from_fp. Sorry about that!

Cheers,

 -- Nikodemus

On 18 March 2010 22:49, Brett van de Sande <bvds@...> wrote:

> Actually, I don't have any large arrays or very large objects.
> The total data associated with one user session is about 500kB.

And yet something, somewhere is trying to allocate one... possibly in
SBCL (I checked a few obvious places where that could be happening,
but no dice), or in some library you are using.

Cheers,

 -- Nikodemus

>Oops, there's a typo in the manual! It's
>
> thread apply all call backtrace_from_fp($ebp, 100)
>
>not call_backtrace_from_fp. Sorry about that!

Thanks.  That one bit me!  OK, I now have the backtraces and
will be going through later today.

Also, QSHOW_SIGNAL should be QSHOW_SIGNALS in the Reporting-Bugs.html

Brett