Re: [Sbcl-help] Garbage collection for server with user sessions

[Brett v. de S. <bv...@as...>]

I would like to give a status report on what I have done.
By way of refresher/update:  I am load testing a server by running
about 200 concurrent user sessions.  Each session lasts about 15
minutes and uses about 1.5 MB memory that lasts the course of an 
entire session.  Right now, I am running sbcl-1.0.36, with the 
following modifications, on 32 bit linux.

1.  Increase sb-vm:gencgc-page-bytes to 16384.  In my case, it ends up 
    that all the memory pages eventually get protected, so 
        max_map_count * gencgc-page-bytes
    must be larger than the total heap memory.

2.  Run sbcl with "--dynamic-space-size 1500"

3.  Tune garbage collector by setting:
     (def!constant +highest-normal-generation+ 2)
     (def!constant +pseudo-static-generation+ 3)
    and 
     (let ((b (* 1024 1024)))
      ;; gc rate associated with turns (about .5 sec)
      (setf (generation-bytes-consed-between-gcs 0) (* 1 b))
      ;; gc rate associated with time between overlapped turns
      (setf (generation-bytes-consed-between-gcs 1) (* 4 b))
      ;; gc rate associated with session length (15 min).
      (setf (generation-bytes-consed-between-gcs 2) (* 100 b))))


4.  If I run the server with the above changes, it appears that 
    garbage collection is working as desired:  garbage collection occurs
    every 3 minus (generation 2), doesn't use too much cpu, and seems to 
    be recovering the memory.  However, after about a half hour, I get a 
    heap exhausted error.   So, I tried to increase the heap size:
        (def!constant dynamic-space-end         #x3ffff000)
    in src/compiler/x86/parms.lisp (to meet up with bottom of shared libs)
    This gives 922742784 bytes of heap space, instead of 500K
    However the heap exhaustion error still occurs, although a bit later:
        Heap exhausted during allocation: 234094592 bytes available, 
	268435464 requested.

5.  For monitoring, I use the function:
    (defun gen-stats (&optional (str t))
           (format str "Gen consed gcs avgage~%")
           (loop for i from 0 to SB-VM:+pseudo-static-generation+ 
	       do (format str "~A ~A ~A ~A~%" i
			  (generation-bytes-allocated i)
			  (generation-number-of-gcs i)
			  (generation-average-age i))))
    Right before the heap exhaustion error, I get something like:
      Gen consed gcs avgage
      0 5878968 1 0.0d0
      1 14762664 1 2.7070593762751765d0
      2 401231264 9 2.40689281880088d0
      3 26066944 0 0.0d0
    Which indicates everything is fine.

Any help/ideas would be appreciated.  

Brett van de Sande