[Valgrind-developers] Strange bug with mmap

[Julian S. <js...@ac...>]

I've got the wierdest bug, which completely mystifies me, and I wondered if
either of you might be able to cast any light on it.

I'm running OpenOffice-1.0.1 on V (1.9.2, which is pretty much the current
cvs head) on SuSE 8.1 on VMWare 3.2 on Win2K.

With almost all arrangements of flags and skins to valgrind, OO runs fine.

But with --skin=addrcheck --leak-check=yes, it doesn't.  OO is a shell
script so I supply --trace-children=yes.  One of the traced children is a
run of sed.  Whilst doing the final leak-check for that run, valgrind's
low level memory allocator claims it is out of memory, and aborts.  This
is a complete lie; there is 300+ M of swap available.

Why it's claiming memory at the leak check is so as to read in symbol
tables so it can print the leak messages nicely.  This isn't relevant,
I don't think.

Anyway: V's low-level allocator works by mmaping in blocks of /dev/zero 
with read, write and execute permissions, and flags MAP_PRIVATE|MAP_ANONYMOUS.
The usual deal, in other words.  In this case it asked for a block of 1MB,
having already allocated just under 9MB.  mmap returns with -12 (-ENOMEM),
which I at first didn't believe.  However, prodding with w revealed

sewardj  29384 90.8  2.1 2096992 5456 pts/2  R    18:31   0:29 sed

with an implausibly large process size.  cat /proc/29384/maps reveals

08048000-08055000 r-xp 00000000 08:02 24022      /bin/sed
08055000-08056000 rw-p 0000c000 08:02 24022      /bin/sed
40000000-40013000 r-xp 00000000 08:02 20673      /lib/ld-2.2.5.so
40013000-40014000 rw-p 00012000 08:02 20673      /lib/ld-2.2.5.so
40014000-4001b000 r-xp 00000000 08:02 127756     
/home/sewardj/TEST/valgrind-1.9.2/Inst/lib/valgrind/vgskin_addrcheck.so
4001b000-4001c000 rw-p 00006000 08:02 127756     
/home/sewardj/TEST/valgrind-1.9.2/Inst/lib/valgrind/vgskin_addrcheck.so
4001c000-4011f000 rw-p 00000000 00:00 0
4011f000-40197000 r-xp 00000000 08:02 127516     
/home/sewardj/TEST/valgrind-1.9.2/Inst/lib/valgrind/valgrind.so
40197000-40198000 rw-p 00077000 08:02 127516     
/home/sewardj/TEST/valgrind-1.9.2/Inst/lib/valgrind/valgrind.so
40198000-401e6000 rw-p 00000000 00:00 0
401e6000-401f0000 rwxp 00000000 00:00 0
401f1000-40307000 r-xp 00000000 08:02 20679      /lib/libc.so.6
40307000-4030d000 rw-p 00115000 08:02 20679      /lib/libc.so.6
4030d000-40311000 rw-p 00000000 00:00 0
40311000-40aeb000 rwxp 00000000 00:00 0
40aec000-40aee000 rwxp 00000000 00:00 0
40af1000-40af3000 rwxp 00000000 00:00 0
40b23000-40b29000 rwxp 00000000 00:00 0
40b29000-40b2a000 rw-p 00000000 00:00 0
40b2b000-c0000000 rwxp 80b2c000 00:00 0     (!!!!!!!!!!!!!)

so there is an enormous /dev/zero segment all the way from 40b2b000 up to
the stack base (!).  A bit of debug printing shows accounts for the various
other segments (40b29000-40b2a000) getting allocated, but nowhere does it
ask for more than 2GB of space.  I have no idea why the kernel thinks that
space is allocated.

I wondered if this could be some conflict between parent and child.  But 
the mmaps are private anonymous mappings, which AIUI gives shared-copy-on-
write semantics.

Then I wondered if it's something strange about running on vmware.  But
that's proved unfailingly reliable in the past.

So I'm completely lost.  Any idea why the kernel could come to believe
such a huge lump of address space is allocated?  This is SuSE's "2.4.18-4GB"
kernel, whatever that might entail.

Just an idea: it wouldn't be some wierd thing with the kernel extending
the stack segment down way too far ???

J