|
From: Konstantin S. <kon...@gm...> - 2011-01-03 20:23:59
|
Hi, We are seeing more and more of this error: valgrind: mmap(0x38000000, 2375680) failed in UME with error 22 (Invalid argument). valgrind: this can be caused by executables with very large text, data or bss segments. (known as http://bugs.kde.org/show_bug.cgi?id=138424, http://bugs.kde.org/show_bug.cgi?id=184981, http://code.google.com/p/chromium/issues/detail?id=28439). Any chance to fix it? Do you have a reliable reproducer? (If no, we could send a pre-built binary for Ubuntu 10.04)... Thanks, and happy new year! --kcc |
|
From: Tom H. <to...@co...> - 2011-01-03 20:33:44
|
On 03/01/11 20:23, Konstantin Serebryany wrote: > We are seeing more and more of this error: > valgrind: mmap(0x38000000, 2375680) failed in UME with error 22 (Invalid > argument). > valgrind: this can be caused by executables with very large text, data > or bss segments. > > (known as > http://bugs.kde.org/show_bug.cgi?id=138424, > http://bugs.kde.org/show_bug.cgi?id=184981, > http://code.google.com/p/chromium/issues/detail?id=28439). > > Any chance to fix it? Do you have a reliable reproducer? (If no, we > could send a pre-built binary for Ubuntu 10.04)... What makes you think it can be fixed? It's a basic limitation of valgrind that it needs to reserve some memory for itself and hence there will be less available for the client program than when it is not running valgrind. That is, I believe, the basic cause of this error, that after valgrind has reserved memory for itself there is not enough room to load the executable. Tom -- Tom Hughes (to...@co...) http://compton.nu/ |
|
From: Julian S. <js...@ac...> - 2011-01-04 13:59:49
|
On Monday, January 03, 2011, Tom Hughes wrote: > On 03/01/11 20:23, Konstantin Serebryany wrote: > > We are seeing more and more of this error: > > valgrind: mmap(0x38000000, 2375680) failed in UME with error 22 (Invalid > > argument). > > valgrind: this can be caused by executables with very large text, data > > or bss segments. > > > > (known as > > http://bugs.kde.org/show_bug.cgi?id=138424, > > http://bugs.kde.org/show_bug.cgi?id=184981, > > http://code.google.com/p/chromium/issues/detail?id=28439). > > > > Any chance to fix it? Do you have a reliable reproducer? (If no, we > > could send a pre-built binary for Ubuntu 10.04)... > > What makes you think it can be fixed? It's a basic limitation of > valgrind that it needs to reserve some memory for itself and hence there > will be less available for the client program than when it is not > running valgrind. > > That is, I believe, the basic cause of this error, that after valgrind > has reserved memory for itself there is not enough room to load the > executable. It could also really be caused by executables with very large text etc segments. Since the text segment will get mapped to 0x8048000 (roughly 134MB), if its size exceeds about 800MB then the top part will overlap valgrind's load point of 0x38000000 (939MB) and so the UME will fail. (Actually .. to be more precise .. won't it fail if the size of text + data + bss exceeds about 800MB ?) Konstantin, what happens if you edit configure.in and change all the 0x38000000 to (eg) 0x58000000, to select a higher load address for V? J |
|
From: Konstantin S. <kon...@gm...> - 2011-01-06 09:05:40
|
On Tue, Jan 4, 2011 at 4:59 PM, Julian Seward <js...@ac...> wrote: > On Monday, January 03, 2011, Tom Hughes wrote: > > On 03/01/11 20:23, Konstantin Serebryany wrote: > > > We are seeing more and more of this error: > > > valgrind: mmap(0x38000000, 2375680) failed in UME with error 22 > (Invalid > > > argument). > > > valgrind: this can be caused by executables with very large text, data > > > or bss segments. > > > > > > (known as > > > http://bugs.kde.org/show_bug.cgi?id=138424, > > > http://bugs.kde.org/show_bug.cgi?id=184981, > > > http://code.google.com/p/chromium/issues/detail?id=28439). > > > > > > Any chance to fix it? Do you have a reliable reproducer? (If no, we > > > could send a pre-built binary for Ubuntu 10.04)... > > > > What makes you think it can be fixed? It's a basic limitation of > > valgrind that it needs to reserve some memory for itself and hence there > > will be less available for the client program than when it is not > > running valgrind. > > > > That is, I believe, the basic cause of this error, that after valgrind > > has reserved memory for itself there is not enough room to load the > > executable. > > It could also really be caused by executables with very large > text etc segments. Since the text segment will get mapped to > 0x8048000 (roughly 134MB), if its size exceeds about 800MB then > the top part will overlap valgrind's load point of 0x38000000 > (939MB) and so the UME will fail. > > (Actually .. to be more precise .. won't it fail if the size of > text + data + bss exceeds about 800MB ?) > > Konstantin, what happens if you edit configure.in and change all the > 0x38000000 to (eg) 0x58000000, to select a higher load address > for V? > This makes the warning look like this: valgrind: mmap(0x58000000, 2375680) failed in UME with error 22 (Invalid argument). :( --kcc > > J > |
|
From: Julian S. <js...@ac...> - 2011-01-06 11:18:32
|
> > (Actually .. to be more precise .. won't it fail if the size of > > text + data + bss exceeds about 800MB ?) > > > > Konstantin, what happens if you edit configure.in and change all the > > 0x38000000 to (eg) 0x58000000, to select a higher load address > > for V? > > This makes the warning look like this: > valgrind: mmap(0x58000000, 2375680) failed in UME with error 22 (Invalid > argument). Hmm, that's odd. But it depends on the text/bss/data sizes for your executable. Did you check that they will fit in (0x58000000 - 0x400000) bytes? J |
|
From: Konstantin S. <kon...@gm...> - 2011-01-11 09:30:07
|
2011/1/6 Julian Seward <js...@ac...> > > > > (Actually .. to be more precise .. won't it fail if the size of > > > text + data + bss exceeds about 800MB ?) > > > > > > Konstantin, what happens if you edit configure.in and change all the > > > 0x38000000 to (eg) 0x58000000, to select a higher load address > > > for V? > > > > This makes the warning look like this: > > valgrind: mmap(0x58000000, 2375680) failed in UME with error 22 (Invalid > > argument). > > Hmm, that's odd. But it depends on the text/bss/data sizes for your > executable. Did you check that they will fit in (0x58000000 - 0x400000) > bytes? > Yes. This binary is large, but not enormous. % objdump -h out/Release/chrome | grep '\.text\|\.bss\|\.data' 12 .text 01d80846 0000000000439da0 0000000000439da0 00039da0 2**4 24 .data.rel.ro 001c32d8 0000000003e4a760 0000000003e4a760 0384a760 2**5 28 .data 00085c70 000000000401b580 000000000401b580 03a1b580 2**5 29 .bss 000a0960 00000000040a1200 00000000040a1200 03aa11f0 2**5 --kcc > > J > |
|
From: Konstantin S. <kon...@gm...> - 2011-01-11 11:45:08
|
I've added some debug prints and here is what I see:
--24984:0:ume mmap_file_fixed_client #1
--24984:0:aspacem <<< SHOW_SEGMENTS: after #1 (17 segments, 1 segnames)
--24984:0:aspacem ( 0)
/home/kcc/valgrind/valgrind-variant/valgrind/inst/lib/valgrind/memcheck-amd64-linux
--24984:0:aspacem 0: RSVN 0000000000-0003ffffff 64m ----- SmFixed
--24984:0:aspacem 1: 0004000000-0037ffffff 832m
--24984:0:aspacem 2: FILE 0038000000-0038243fff 2375680 r-x-- d=0x01d
i=1221334 o=2097152 (0)
--24984:0:aspacem 3: 0038244000-0038442fff 2093056
--24984:0:aspacem 4: FILE 0038443000-0038445fff 12288 rw--- d=0x01d
i=1221334 o=4468736 (0)
--24984:0:aspacem 5: ANON 0038446000-004500afff 203m rw---
--24984:0:aspacem 6: 004500b000-4001ffffff 261071m
--24984:0:aspacem 7: RSVN 4002000000-4002000fff 4096 ----- SmFixed
--24984:0:aspacem 8: ANON 4002001000-4002400fff 4194304 rwx--
--24984:0:aspacem 9: 4002401000-7fffffffff 262107m
--24984:0:aspacem 10: RSVN 8000000000-7fff56bd6fff 130557g ----- SmFixed
--24984:0:aspacem 11: ANON 7fff56bd7000-7fff56bebfff 86016 rw---
--24984:0:aspacem 12: RSVN 7fff56bec000-7fff56bfefff 77824 ----- SmFixed
--24984:0:aspacem 13: ANON 7fff56bff000-7fff56bfffff 4096 r-x--
--24984:0:aspacem 14: RSVN 7fff56c00000-ffffffffff5fffff 16383e -----
SmFixed
--24984:0:aspacem 15: ANON ffffffffff600000-ffffffffff600fff 4096 r-x--
--24984:0:aspacem 16: RSVN ffffffffff601000-ffffffffffffffff 9m -----
SmFixed
--24984:0:aspacem >>>
pid=24984 valgrind: mmap(0x38000000, 2375680) failed in UME with error 22
(Invalid argument).
valgrind: this can be caused by executables with very large text, data or
bss segments.
The function am_mmap_named_file_fixed_client boils out here:
advised = VG_(am_get_advisory)( &req, True/*client*/, &ok );
if (!ok || advised != start)
return VG_(mk_SysRes_Error)( VKI_EINVAL );
This is strange: looks like something is already mapped at 0038000000 with
the same size.
The error happens only when I run a tree of processes under 'valgrind
--trace-children=yes'.
Any suggestion how to debug it further?
--kcc
On Tue, Jan 11, 2011 at 12:29 PM, Konstantin Serebryany <
kon...@gm...> wrote:
>
>
> 2011/1/6 Julian Seward <js...@ac...>
>
>
>> > > (Actually .. to be more precise .. won't it fail if the size of
>> > > text + data + bss exceeds about 800MB ?)
>> > >
>> > > Konstantin, what happens if you edit configure.in and change all the
>> > > 0x38000000 to (eg) 0x58000000, to select a higher load address
>> > > for V?
>> >
>> > This makes the warning look like this:
>> > valgrind: mmap(0x58000000, 2375680) failed in UME with error 22 (Invalid
>> > argument).
>>
>> Hmm, that's odd. But it depends on the text/bss/data sizes for your
>> executable. Did you check that they will fit in (0x58000000 - 0x400000)
>> bytes?
>>
>
> Yes. This binary is large, but not enormous.
> % objdump -h out/Release/chrome | grep '\.text\|\.bss\|\.data'
> 12 .text 01d80846 0000000000439da0 0000000000439da0 00039da0
> 2**4
> 24 .data.rel.ro 001c32d8 0000000003e4a760 0000000003e4a760 0384a760
> 2**5
> 28 .data 00085c70 000000000401b580 000000000401b580 03a1b580
> 2**5
> 29 .bss 000a0960 00000000040a1200 00000000040a1200 03aa11f0
> 2**5
>
> --kcc
>
>
>>
>> J
>>
>
>
|
|
From: John R. <jr...@bi...> - 2011-01-11 15:14:18
|
> The function am_mmap_named_file_fixed_client boils out here:
> advised = VG_(am_get_advisory)( &req, True/*client*/, &ok );
>
>
> if (!ok || advised != start)
{
system("cat /proc/self/maps");
>
>
> return VG_(mk_SysRes_Error)( VKI_EINVAL );
}
> Any suggestion how to debug it further?
/* as above; or the really grubby:
int const fd= open("/proc/self/maps", O_RDONLY);
for (;;) {
char buf[100];
ssize_t const len = read(fd, buf, sizeof(buf));
write(2, buf, len);
if (len<=0) {
close(fd);
break;
}
}
*/
--
|
|
From: John R. <jr...@bi...> - 2011-01-11 15:37:33
|
> The error happens only when I run a tree of processes under 'valgrind > --trace-children=yes'. > > Any suggestion how to debug it further? Run the whole thing under strace: strace -o strace.out -f valgrind ... then look at all the mmap() calls, paying particular attention to the EINVAL paragraphs displayed by "$ man 2 mmap". -- |
|
From: John R. <jr...@bi...> - 2011-01-11 15:50:52
|
> The error happens only when I run a tree of processes under 'valgrind > --trace-children=yes'. Is the system running out of memory resources? When the mmap() fails then copy the contents of /proc/meminfo onto stderr. (Make a subroutine and use it to print out both /proc/self/maps and /proc/meminfo.) -- |
|
From: Konstantin S. <kon...@gm...> - 2011-01-12 05:23:39
|
On Tue, Jan 11, 2011 at 6:50 PM, John Reiser <jr...@bi...> wrote: > > The error happens only when I run a tree of processes under 'valgrind > > --trace-children=yes'. > > Is the system running out of memory resources? No. The set of processes doesn't take more than half of available RAM. > When the mmap() fails then > mmap doesn't fail -- it does not get called. am_mmap_named_file_fixed_client exits before calling mmap --kcc > copy the contents of /proc/meminfo onto stderr. (Make a subroutine > and use it to print out both /proc/self/maps and /proc/meminfo.) > > -- > > > ------------------------------------------------------------------------------ > Gaining the trust of online customers is vital for the success of any > company > that requires sensitive data to be transmitted over the Web. Learn how to > best implement a security strategy that keeps consumers' information secure > and instills the confidence they need to proceed with transactions. > http://p.sf.net/sfu/oracle-sfdevnl > _______________________________________________ > Valgrind-developers mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-developers > |
|
From: Konstantin S. <kon...@gm...> - 2011-01-12 09:05:42
|
The failure happens when chrome tries to fork/exec itself via /proc/self/exe link. This sounds unfriendly to valgrind, since /proc/self/exe points to memcheck instead of chrome. Right? Any suggestion how to workaround this (other than removing /proc/self/exe from the chrome source code)? Thanks! --kcc On Wed, Jan 12, 2011 at 8:23 AM, Konstantin Serebryany < kon...@gm...> wrote: > > > On Tue, Jan 11, 2011 at 6:50 PM, John Reiser <jr...@bi...> wrote: > >> > The error happens only when I run a tree of processes under 'valgrind >> > --trace-children=yes'. >> >> Is the system running out of memory resources? > > No. The set of processes doesn't take more than half of available RAM. > >> When the mmap() fails then >> > > mmap doesn't fail -- it does not get called. > am_mmap_named_file_fixed_client exits before calling mmap > > > --kcc > > >> copy the contents of /proc/meminfo onto stderr. (Make a subroutine >> and use it to print out both /proc/self/maps and /proc/meminfo.) >> >> -- >> >> >> ------------------------------------------------------------------------------ >> Gaining the trust of online customers is vital for the success of any >> company >> that requires sensitive data to be transmitted over the Web. Learn how >> to >> best implement a security strategy that keeps consumers' information >> secure >> and instills the confidence they need to proceed with transactions. >> http://p.sf.net/sfu/oracle-sfdevnl >> _______________________________________________ >> Valgrind-developers mailing list >> Val...@li... >> https://lists.sourceforge.net/lists/listinfo/valgrind-developers >> > > |
|
From: Tom H. <to...@co...> - 2011-01-12 09:19:45
|
On 12/01/11 09:05, Konstantin Serebryany wrote: > The failure happens when chrome tries to fork/exec itself via > /proc/self/exe link. > This sounds unfriendly to valgrind, since /proc/self/exe points to > memcheck instead of chrome. Right? > Any suggestion how to workaround this (other than > removing /proc/self/exe from the chrome source code)? If it did readlink(/proc/self/exe) and then exec'ed the result it would work as valgrind traps the readlink and returns a path to the real executable. Tom -- Tom Hughes (to...@co...) http://compton.nu/ |
|
From: Christian B. <bor...@de...> - 2011-01-12 09:31:44
|
Am 12.01.2011 10:19, schrieb Tom Hughes: > On 12/01/11 09:05, Konstantin Serebryany wrote: > >> The failure happens when chrome tries to fork/exec itself via >> /proc/self/exe link. >> This sounds unfriendly to valgrind, since /proc/self/exe points to >> memcheck instead of chrome. Right? >> Any suggestion how to workaround this (other than >> removing /proc/self/exe from the chrome source code)? > > If it did readlink(/proc/self/exe) and then exec'ed the result it would > work as valgrind traps the readlink and returns a path to the real > executable. This is pretty hacky, but cant we do the same thing in PRE(sys_execve)? |
|
From: Tom H. <to...@co...> - 2011-01-12 09:33:14
|
On 12/01/11 09:31, Christian Borntraeger wrote: > Am 12.01.2011 10:19, schrieb Tom Hughes: >> On 12/01/11 09:05, Konstantin Serebryany wrote: >> >>> The failure happens when chrome tries to fork/exec itself via >>> /proc/self/exe link. >>> This sounds unfriendly to valgrind, since /proc/self/exe points to >>> memcheck instead of chrome. Right? >>> Any suggestion how to workaround this (other than >>> removing /proc/self/exe from the chrome source code)? >> >> If it did readlink(/proc/self/exe) and then exec'ed the result it would >> work as valgrind traps the readlink and returns a path to the real >> executable. > > This is pretty hacky, but cant we do the same thing in PRE(sys_execve)? Well sure, I was just trying to give me a workaround. There is a question here about how far we should go though - it will very quickly get out of control if we try and make every call that deals with filenames cope with this. Tom -- Tom Hughes (to...@co...) http://compton.nu/ |
|
From: Konstantin S. <kon...@gm...> - 2011-01-12 15:29:23
|
Making chrome not use /proc/self/exe for forking eliminated this problem. Thanks everyone for suggestions! --kcc On Wed, Jan 12, 2011 at 12:33 PM, Tom Hughes <to...@co...> wrote: > On 12/01/11 09:31, Christian Borntraeger wrote: > > Am 12.01.2011 10:19, schrieb Tom Hughes: > >> On 12/01/11 09:05, Konstantin Serebryany wrote: > >> > >>> The failure happens when chrome tries to fork/exec itself via > >>> /proc/self/exe link. > >>> This sounds unfriendly to valgrind, since /proc/self/exe points to > >>> memcheck instead of chrome. Right? > >>> Any suggestion how to workaround this (other than > >>> removing /proc/self/exe from the chrome source code)? > >> > >> If it did readlink(/proc/self/exe) and then exec'ed the result it would > >> work as valgrind traps the readlink and returns a path to the real > >> executable. > > > > This is pretty hacky, but cant we do the same thing in PRE(sys_execve)? > > Well sure, I was just trying to give me a workaround. > > There is a question here about how far we should go though - it will > very quickly get out of control if we try and make every call that deals > with filenames cope with this. > > Tom > > -- > Tom Hughes (to...@co...) > http://compton.nu/ > |