Re: [MLton] max-heap setting for 64-bit applications

[Wesley W. T. <we...@te...>]

On Sat, Dec 12, 2009 at 8:48 AM, Matthew Fluet <mat...@gm...>wrote:

> It isn't quite sequential access.


Sorry. You're both wrong. ;) The mremap code does a memcpy, which is 100%
sequential. That's the only transition between the two heaps on Windows ->
inside mremap. It allocates the new heap, memcopies them, then frees the old
heap. The only time the GC touches it is when it fixes up the addresses.

> I have a higher-level solution proposal: MLton.GC already has hooks that
> are
> > executed after a GC to implement finalizers. Expose these to the user. If
> a
> > user knows his application only consumes X memory on an error condition,
> he
> > can test for this after a GC and terminate with an "Out of Memory" error
> as
> > desired.
>
> You still need to get out of the GC before the GC signal handler runs.
>

Of course, but we are talking about early detection. Under normal operation
his program probably uses >much< less than the total RAM. Stopping at 1GB
with an error would certainly be earlier than thrashing set in.

Right now we are focusing in on a micro-program which just consumes memory
and we're talking about how to optimize it. It's a pathological program! We
should instead be supporting David's underlying goal: detecting when his
program enters run-away RAM consumption. This happens much much sooner than
thrashing.


> >> the mremap function is described as using the Linux page table scheme to
> >> efficiently change the mapping between virtual addresses and (physical)
> >> memory pages.  It's purpose is to be more efficient than allocating a
> new
> >> map and copying.
> >
> > If I could ...
>
> I guess my point is that the way that you indicate that you aren't
> more efficient than alloc/copy is by not providing mremap.  Everything
> else in the generic implementation with attempting the in place expand
> is more efficient; it is just the starting off with the alloc/copy
> that doesn't seem to make sense.
>

As already mentioned, the design goal for windows mremap was not efficiency,
but to use more virtual address space. I think both agree that alloc/copy as
the last step makes more sense for 64-bit systems to reduce the likelihood
of thrashing.