Re: [Sbcl-devel] memory mappings

[Martin C. <cra...@co...>]

Scott L. Burson wrote on Tue, Oct 06, 2009 at 11:30:00AM -0700: 
> [Whoops -- sent this to Martin without CC'ing the list.]
> 
> On Tue, Oct 6, 2009 at 10:34 AM, Martin Cracauer <cra...@co...> wrote:
> > Scott L. Burson wrote on Tue, Oct 06, 2009 at 09:41:56AM -0700:
> >>  You might be interested in this:
> >>
> >>   http://bugzilla.kernel.org/show_bug.cgi?id=5493
> >
> > We don't see such effects, but we don't have Lisp heaps close to
> > approaching physical memory.
> >
> > The guys in the bug report run a 10-50 GB heap on a 4 GB box.  That's
> > gotta be unhealthy.  What do they do on a full GC?
> 
> The "guy in the bug report" is me.  

Sorry :-)

> I eventually brought that box up
> to 24GB, though I guess I hadn't done so yet at the time I first filed
> the report.  (Works okay, with three 15kRPM paging drives.  Nowadays
> one would use SSDs, if budget permits.)

But then I don't follow why you think the linear scan on the mprotect
system call is the problem.

Or are you saying that the linear scan happens when Linux is looking
for free pages or tries to free pages?

> Oh, what I didn't mention (though it's in the bug report, near the
> bottom) is that I ultimately worked around the problem by switching to
> Solaris, which has no trace of this problem -- a 35GB heap works just
> fine.

So when is this linear scan happening? On the system call or on paging
activity? 

> > I doubt that their conclusion that this has to do with the ratio of
> > RAM to heap is correct, though.  It's probably just the number of
> > mappings that counts here.
> 
> I believe that's correct.

But then why did it improve by putting in more RAM?

> >> Basically, there's code in the Linux kernel that does a linear scan
> >> through the vma table which holds all these mappings.
> >
> > Linear scan when? On the mprotect call itself?
> 
> I think it's somewhere else -- some path the kernel goes through
> regularly for some reason.  Not sure, though.

I think it is the actual paging activity then.  The kernel has memory
shortage and no reuable pages at the ready.

So it goes hunting for pages and in an attempt to drop or page out the
least important pages it look at what application does what with what
pages, hence the linear scan.

Solaris probably wouldn't have a better method of inspecting what the
process is doing, it's probably just picking pages without looking at
things in that detail, or maybe it has backpointers from other data
structures into the right place in the process memory without
scanning.

But that would mean that ITA isn't affected since we can't run with
any paging activity going on anyway.

Still sounds overall fine to me, but it is a further argument for
mainline SBCL to raise the GC page size to 32 KB.  I don't see a
downside, it's fastest and whatever problem people have with the
number of mappings is reduced to a fraction.

Martin
-- 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Martin Cracauer <cra...@co...>   http://www.cons.org/cracauer/
FreeBSD - where you want to go, today.      http://www.freebsd.org/