Re: [Maxima-discuss] gcl memory usage

[Gunter K. <gu...@pe...>]

On 23.04.2018 23:46, Camm Maguire wrote:
> Greetings, and thanks for the dialogue on this.
> 
> Let me state upfront that I am more than open to suggestions on how this
> should be setup for our community, but I would like to avoid heuristics
> and implement a rational policy even if imperfect.
> 

That sounds good.

> 
> GCL does not allocate *anything* on startup.  Rather it probes brk() to
> determine the operating system's hard limits on the data segment size.

That would be a great plus in respect to sbcl which allocates all the
memory on startup and won't dynamically expand it lateron.

> The results of this probe are governed by the actual amount of ram+swap
> physically available, the kernel's policy/algorithms (e.g. overcommit,
> etc.), and the ambient system load as may be considered by the kernel.
> This value is then multiplied by GCL_MEM_MULTIPLE, producing a value at
> which GCL will report memory exhaustion when reached.
> 

Perhaps that is part of the problem: As soon as gcl does no more fit
into the physical RAM it gets basically useless as it will be constantly
swapping. Even worse: Already when gcl and all the other applications
that actually run no more fits into the RAM gcl is a
several-gigabyte-long chunk of memory that needs to swapped out if an
application needs a few milliseconds of CPU time and then needs to be
swapped into the memory again.

> Thus GCL will never use up memory unless the user asks for it.  The GC
> policy is then tailored to try to optimize the time/memory load tradeoff
> as best as possible.
> 
> In my experience, sessions fall into two broad categories -- 1)
> benchmark some production code scheduled for release, and 2) do real
> incremental development work.
> 
> The rate of allocation in 2) is exceedingly slow, likely resulting in
> days or weeks for a running session to begin to approach 1/2 memory.

That sounds fine. But it seems that this isn't true in maxima: If I
 - call read_matrix() on a long file or
 - run the testbench or
 - solve a longish differential equation using desolve (the solution of
the one in question is about 2 Din A4 pages long when using a 12pt font) or
 - factor a long equation (4 Din A4 pages)

if GCL_MEM_MULTIPLE is unset gcl exceeds the 3,5 Gigabytes of free
memory my system provides within seconds even if I haven't opened any
other application; unfortunately the long equations I used belong to my
working place.

When using clisp, ecl or sbcl the testbench stays below 5% of my main
memory. In clisp it even stays below 2% of my main memory.

If I set GCL_MEM_MULTIPLE to 0.2 the testbench succeeds but still uses
up astonishingly big parts of my main memory.

Something seems to be seriously wrong and maybe it is just the
combination of maxima and gcl, not one of these programs.


gunter@Marius:~$ gcl --version
GCL (GNU Common Lisp)  2.6.12 CLtL1    Fri Apr 22 15:51:11 UTC 2016
Source License: LGPL(gcl,gmp), GPL(unexec,bfd,xgcl)
Binary License:  GPL due to GPL'ed components: (XGCL READLINE UNEXEC)
Modifications of this banner must retain notice of a compatible license
Dedicated to the memory of W. Schelter

Use (help) to get some basic information on how to use GCL.
Temporary directory for compiler files:
/tmp/

In my case /tmp/ isn't a tmpfs that keeps its contents in the RAM.

gunter@Marius:~$ cat /proc/swaps
Filename				Type		Size	Used	Priority
/dev/sda5                               partition	7811068	0	-2

Currently (thunderbird is open and many applications were open which
accounts for the big "Cached" section) my memory info is:

gunter@Marius:~$ cat /proc/meminfo
MemTotal:        3855160 kB
MemFree:          385596 kB
MemAvailable:    2045568 kB
Buffers:            3212 kB
Cached:          2110116 kB
SwapCached:            0 kB
Active:          1473228 kB
Inactive:        1736004 kB
Active(anon):    1032348 kB
Inactive(anon):   344808 kB
Active(file):     440880 kB
Inactive(file):  1391196 kB
Unevictable:         204 kB
Mlocked:             204 kB
SwapTotal:       7811068 kB
SwapFree:        7811068 kB
Dirty:              1124 kB
Writeback:             0 kB
AnonPages:       1096108 kB
Mapped:           373736 kB
Shmem:            345956 kB
Slab:             155620 kB
SReclaimable:      80760 kB
SUnreclaim:        74860 kB
KernelStack:       10080 kB
PageTables:        43420 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     9738648 kB
Committed_AS:    9296560 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      154368 kB
DirectMap2M:     3854336 kB

Currently I have set GCL_MEM_MULTIPLE to default to 0.2 in src/maxima.in
which makes gcl usable on my system. But David may be right that this is
too low. On the other side not setting this variable makes maxima-gcl
basically useless to me so I am reluctant to revert my change without a
go from the mailing list.

Kind regards,

 Gunter.