|
From: Mario E. <ma...@em...> - 2020-09-08 10:40:07
Attachments:
log.txt
|
Dear All,
many years ago, I've been using valgrind frequently and successfully,
admittedly without ever giving it much thought! Thanks for the awesome
tool.
Now I'm setting up a larger CI system and want automatic memcheck for
our tests. However, in the whole past year, I could not get a single
successful run. So I must be doing something very wrong. Help would be
greatly appreciated :-(
The error I get most frequently is (full output attached in log.txt)
==32== Valgrind's memory management: out of memory:
==32== newSuperblock's request for 6864695621860790272 bytes failed.
==32== 114,106,368 bytes have already been mmap-ed ANONYMOUS.
Here is what I tried so far:
- Versions valgrind-3.13.0 from Ubuntu 18.04 and valgrind-3.16.1
compiled from source
- Executed valgrind in a docker container running Ubuntu 18.04 x86_64
and Ubuntu 20.04 x86_64
- Checked `ulimit -a` in Docker, there are no tight limits
- Tried valgrind with some 50++ different executables, all lead to
the same error message
- Tried valgrind outside Docker, leads to the same error message
- Checked `ulimit -a` outside Docker, there are no tight limits
- Tried the tests work successfully when _not_ using valgrind
I have also tried valgrind on other executables than our debug builds,
and it seems to work there without problems. So maybe the errors are
related to how we create debug builds?
We make pretty standard debug builds (I assume), with flags
-ggdb3 -fno-omit-frame-pointer -O1 -m64 -march=nehalem -mtune=haswell.
Are some of these suspicious?
The host machines I have tried are relatively modern desktop computers
with 64GB of RAM, and modern Skylake or Ryzen processors. The OS is
typically Ubuntu 18.04 or 20.04. I have not set up any tight permission
restrictions like selinux (unless it would be the default for Ubuntu).
And ideas for what I can try are more than appreciated!
All the best,
Mario Emmenlauer
|
|
From: Mario E. <ma...@em...> - 2020-09-08 10:25:36
|
On 08.09.20 12:04, Mario Emmenlauer wrote:
> The error I get most frequently is (full output attached in log.txt)
> ==32== Valgrind's memory management: out of memory:
> ==32== newSuperblock's request for 6864695621860790272 bytes failed.
> ==32== 114,106,368 bytes have already been mmap-ed ANONYMOUS.
Argh! After sending the email, I went through the stack trace for
the hundredth time, and spotted the use of "zlib". And indeed, when
replacing my own zlib 1.2.11 with the system zlib 1.2.11, valgrind
works as expected!
Does that make sense? Is zlib used by valgrind itself? And why could
my debug build differ (so much) from the system zlib that it breaks
valgrind? I double-checked and its the identical source code from
Ubuntu, just missing two or three patches.
All the best,
Mario Emmenlauer
--
BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203
Balanstr. 43 mailto: memmenlauer * biodataanalysis.de
D-81669 München http://www.biodataanalysis.de/
|
|
From: Mario E. <ma...@em...> - 2020-09-08 12:10:19
|
On 08.09.20 12:25, Mario Emmenlauer wrote:
>
> On 08.09.20 12:04, Mario Emmenlauer wrote:
>> The error I get most frequently is (full output attached in log.txt)
>> ==32== Valgrind's memory management: out of memory:
>> ==32== newSuperblock's request for 6864695621860790272 bytes failed.
>> ==32== 114,106,368 bytes have already been mmap-ed ANONYMOUS.
>
> Argh! After sending the email, I went through the stack trace for
> the hundredth time, and spotted the use of "zlib". And indeed, when
> replacing my own zlib 1.2.11 with the system zlib 1.2.11, valgrind
> works as expected!
>
> Does that make sense? Is zlib used by valgrind itself? And why could
> my debug build differ (so much) from the system zlib that it breaks
> valgrind? I double-checked and its the identical source code from
> Ubuntu, just missing two or three patches.
So it seems I can (partially) answer my own question: when valgrind
is used on an executable that links zlib built with -ggdb3, then it
does not work (due to aforementioned error). Keeping all other debug-
settings except -ggdb3 works still fine.
I have no clue as to _why_ this may happen, but I hope it can be
helpful to other people running into the same issue.
All the best,
Mario Emmenlauer
|
|
From: Philippe W. <phi...@sk...> - 2020-09-12 14:18:41
|
On Tue, 2020-09-08 at 14:09 +0200, Mario Emmenlauer wrote: > On 08.09.20 12:25, Mario Emmenlauer wrote: > > On 08.09.20 12:04, Mario Emmenlauer wrote: > > > The error I get most frequently is (full output attached in log.txt) > > > ==32== Valgrind's memory management: out of memory: > > > ==32== newSuperblock's request for 6864695621860790272 bytes failed. > > > ==32== 114,106,368 bytes have already been mmap-ed ANONYMOUS. > > > > Argh! After sending the email, I went through the stack trace for > > the hundredth time, and spotted the use of "zlib". And indeed, when > > replacing my own zlib 1.2.11 with the system zlib 1.2.11, valgrind > > works as expected! > > > > Does that make sense? Is zlib used by valgrind itself? And why could > > my debug build differ (so much) from the system zlib that it breaks > > valgrind? I double-checked and its the identical source code from > > Ubuntu, just missing two or three patches. > > So it seems I can (partially) answer my own question: when valgrind > is used on an executable that links zlib built with -ggdb3, then it > does not work (due to aforementioned error). Keeping all other debug- > settings except -ggdb3 works still fine. > > I have no clue as to _why_ this may happen, but I hope it can be > helpful to other people running into the same issue. zlib is not used by the valgrind tools. In fact, valgrind tools do not use any library (even not libc). The above newSuperblock trace shows that a *huge* block is requested. As this bug only happens when you use -ggdb3, this is likely a problem in the debuginfo reader of valgrind: some debug info generated by -ggdb3 is very probably not handled properly. I have recompiled libz with -ggdb3, but no problem when running this lib under valgrind. We might have a more clear idea of what happens on your side by adding some trace. The best is to file a bug on bugzilla, and attach the output of running valgrind with -d -d -d -v -v -v. That might give some information about what is wrong and possibly some more detailed trace can then be activated. Thanks Philippe |