Re: [MLton] max-heap setting for 64-bit applications

[Wesley W. T. <we...@te...>]

On Mon, Dec 14, 2009 at 5:50 PM, Matthew Fluet <mat...@gm...> wrote:
> On Sat, Dec 12, 2009 at 8:24 AM, Wesley W. Terpstra <we...@te...> wrote:
>> The mremap code does a memcpy, which is 100% sequential.
> But, the Windows specific mremap could still fail --- note that the
> growHeap function demands significant growth from mremap.  If that
> fails, then it attempts the alloc/copy, but allowing for an alloc of a
> heap down to the minimum size.

This could only happen if you fragmentted the entire 16TB of VM.
My point was that the only swapping he (or anyone else) saw was
sequential, and should be as fast as a disk-backed copy.

>> As already mentioned, the design goal for windows mremap was not efficiency,
>> but to use more virtual address space. I think both agree that alloc/copy as
>> the last step makes more sense for 64-bit systems to reduce the likelihood
>> of thrashing.
>
> I guess the question is whether the large alloc/copy is really more
> efficient on even a 32-bit system.

I bet it's probably always faster to grow in-place. As there have been
no issues with the mremap code since I wrote it, I'm confident enough
in it to switch it to the default, committed.

It might also be worth writing an improved copy/move code-path that
allocates as it goes, thus avoiding swapping. ie:
(commit/copy/decommit)s in blocks of 4MB at a time, releasing the
source reservation as the last step. Not as amazing as a real mremap
which doesn't need to copy at all, but at least it would have the same
'trash or not thrash' behaviour.