|
From: Nicholas N. <nj...@cs...> - 2005-05-25 15:07:07
|
On Sat, 21 May 2005, Nicholas Nethercote wrote: >> Some nonscientific analysis (i.e. watching "top" while valgrind is >> running) indicates that the memory usage is initially stable, then >> quickly climbs as the buffer gets larger. > > I just looked at this. Turns out that under Valgrind, on my machine (and > presumably yours) a lot of those allocations are failing -- you should be > checking the result of malloc. > > I think the problem is this: the program allocates a 4KB block, then frees > it. Valgrind puts this block on its free list. The program then allocates > an 8KB block. There's no block on the free list which is 8KB or greater, so > Valgrind allocates new memory. And so on. So Valgrind's free list ends up > holding blocks of size 4KB, 8KB, 12KB, 16KB, etc. Huge amounts of memory > quickly get heldon the free list, but none of the blocks are ever big enough > to satisfy the requests. > > Basically it's a pathological case for Valgrind's allocator. Mark, did you > come up with your original program (the realloc one) based on behaviour you > saw with a real application, or were you just experimenting? I should think more before opening my mouth -- the problem is more subtle than I thought. I was forgetting that Valgrind's allocator merges blocks on the free list if they are adjacent. So it's certainly possible to allocate, for example, a 12KB block from the free list even if you've only previously allocated (and freed) a 4KB block and an 8KB block, because they may have been merged. Here's the real problem. Valgrind hands out heap blocks from "superblocks". Superblocks are 1MB in size, or, if you ask for a heap block larger than that, the superblock provided is that size (plus space for the administrative bytes). Unfortunately, Valgrind's allocator can only merge blocks on the free list if they are adjacent. This means they must be in the same superblock. But once your program starts asking for heap blocks bigger than 1MB, each heap block fills a whole superblock, and so there is no merging going on with free blocks larger than 1MB. That's where all the address space goes, and eventually the allocator starts failing. The fix would be for Valgrind's allocator to monitor in some way the superblocks it has allocated, and don't let too many stay hanging around if they're not being used. Eg. have a free list threshold, and if the amount of memory on the free list exceeds that, then try to deallocate superblocks when new ones get allocated. Or something like that. If indeed it's worth bothering with at all -- this is a pretty pathological case. N |