Re: [Sbcl-help] Making GC more parsimoniuos of memory

[Liam H. <ln...@he...>]

Is this a stop-and-copy algorithm, which can use only half the available heap?
https://en.wikipedia.org/wiki/Cheney%27s_algorithm


Liam

On Thu, Jan 4, 2018 at 5:20 PM, Tim Josling <tim...@gm...> wrote:
> Hi
>
> I have been happily using SBCL for many years. Currently I have a very
> memory intensive application due to a large amount of data that I need
> to access at array speeds.
>
> I have set the memory limit at 30Gib. This then also required setting
> max_map_count on my linux x86_64 system. But when I get to around
> 13-16GiB used I start to get out of memory errors. These errors (see
> below) seem to occur during garbage collection.
>
> CL-USER> (SOFTWARE-VERSION)
> "4.4.0-93-generic"
> CL-USER> (SB-SYS:GET-MACHINE-VERSION)
> "Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz"
> CL-USER> (UIOP/OS:LISP-VERSION-STRING)
> "1.3.1.debian" (i.e. "Ubuntu 16.04.3 LTS")
>
> From which it appears that the GC only works when I have ~double the
> used memory available.
>
> Error message:
>
> Heap exhausted during garbage collection: 0 bytes available, 64 requested.
>  Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB   LUB  !move  Alloc
> Waste   Trig    WP  GCs Mem-age
>    0:     0     0     0     0     0     0     0     0     0        0
>   0 322122547    0   0  0.0000
>    1:     0     0     0     0     0     0     0     0     0        0
>   0 322122547    0   0  0.0000
>    2:     0     0     0     0     0     0     0     0     0        0
>   0 322122547    0   0  0.0000
>    3:     0     0     0     0     0     0     0     0     0        0
>   0 322122547    0   0  0.0000
>    4:     0     0     0     0     0     0     0     0     0        0
>   0 322122547    0   0  0.0000
>    5: 464103 469060     0     0 361072 212395   674   489    84
> 18593210400 236265440 14033953987    0   4  0.7375
>    6:     0     0     0     0  1617   226     0     0     0 60391424
>   0  2000000 1496   0  0.0000
>    Total bytes allocated    = 31920187456
>    Dynamic-space-size bytes = 32212254720
> GC control variables:
>    *GC-INHIBIT* = true
>    *GC-PENDING* = false
>    *STOP-FOR-GC-PENDING* = false
> fatal error encountered in SBCL pid 4769(tid 140735199704832):
> Heap exhausted, game over.
>
> Previous (room nil) output
>
> Breakdown for dynamic space:
>   7,476,526,208 bytes for 467,282,888 cons objects.
>   4,455,227,040 bytes for 58,096,093 simple-character-string objects.
>   1,096,192,816 bytes for 11,402,878 simple-vector objects.
>   744,423,712 bytes for 3,038,413 other objects.
>   13,772,369,776 bytes for 539,820,272 dynamic objects (space total.)
>
> Breakdown for read-only space:
>        4,944 bytes for         4 code objects.
>        4,944 bytes for         4 read-only objects (space total.)
>
> (Dynamic allowed is 32,212,254,720 bytes 30.0d0 Gib
> SBCL command from slime;
> ("sbcl" "--dynamic-space-size" "30720" "--disable-ldb" "--lose-on-corruption"))
>
> I put in reasonably frequent (gc :full t) which doesn't solve the
> problem. I could increase the memory and spill into swap but this
> makes the program run extremely slowly (10X slower) and it already
> takes too long.
>
> I could do work to compress the data and so forth to make it fit but I
> would rather not. Or buy a bigger PC.
>
> I have looked at/tried the gc parameters and nothing seems to help.
>
> So my questions are:
>
> 1. Does this sound right that the GC falls apart when the memory
> available is not roughly >= 2 times memory used?
>
> 2. Is there anything I can do to make the GC more sparing in its
> memory use or more tolerant or memory shortage, even at the cost of
> slower running?
>
> 3. Is there some problem with (mem) that causes it to burn vast
> amounts of memory?
>
> Tim Josling
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Sbcl-help mailing list
> Sbc...@li...
> https://lists.sourceforge.net/lists/listinfo/sbcl-help