Re: Fragmentation was RE: [GD-General] Compile times

[Thatcher U. <tu...@tu...>]

On Dec 17, 2002 at 10:04 -0600, brian hook wrote:
> > Has anyone been bitten when using a decent malloc?
> 
> To clarify, I don't think fragmentation is a problem UNLESS you have a 
> particular sequence of allocations that kick your ass and/or you have 
> extremely long up times.  I think MUDs have actually run into this type 
> of problem before, but on much smaller scales (e.g. 16MB MUDs running 
> on older boxes with maybe 32MB installed, no VM, etc. -- similar to 
> what consoles deal with today).
> 
> The average PC game simply won't have to deal with address space 
> fragmentation -- many force a reset between levels, others just don't 
> run long enough to make it a problem.  Unfortunately some stuff I've 
> worked on hasn't been "average" in terms of scale, so I'm extremely 
> attuned to this.
> 
> And if you blow off fragmentation on something like a Palm or GBA, it 
> will probably bite you in the ass one of these days.  Unless you're 
> using something that can compact behind the scenes, it's a problem, no 
> matter how good the underlying malloc implementation may be.  The 
> underlying implementation simply determines WHEN it affects you.

The reason I ask is because I *know* fragmentation is a problem if
malloc sucks, and sucky mallocs have been ubiquitous until recently,
but the only empirical study I'm aware of suggests that if malloc
doesn't suck, then fragmentation is not a problem.

http://www.cs.utexas.edu/users/wilson/papers/fragsolved.pdf

So I'm curious about any smoking guns of the form: "we used dlmalloc
(or something like it) in project X on platform Y, and observed that
our peak memory use was 3x the size of our peak live data".  Or any
updated studies.

My personal experience with this is limited; in all games I worked on
prior to starting at Oddworld, we were so fearful of malloc that we
took all kinds of measures to avoid it.  In Munch's Oddysee we pretty
much just let it rip (STL and all), with a general-purpose custom
allocator, and didn't run into significant problems.  Which scared the
crap out of me at the time, but I'm wondering if being afraid of
fragmentation is unfounded nowadays.  The other bit of personal
experience is that my chunklod demo malloc's wontonly, and I only ever
observed problems when it linked with a pre-Win2K version of
msvcrt.dll that found its way on to my hard disk.  dlmalloc and the
correct version of msvcrt were both fine.

-- 
Thatcher Ulrich
http://tulrich.com