[Valgrind-users] Valgrind taking a lot of time trying to determine if there are memory leaks before it prints "leak summary".

[Wojciech B. <woj...@gm...>]

Hello,

I have a problem with Valgrind taking a lot of time trying to
determine if there are memory leaks before it exits.

I'm running Valgrind 3.24.0 with a multi-process 32-bit application in
Linux on x86_64.

This is how I run it:
valgrind --log-file=/tmp/valgrind/memcheck-%p.log --tool=memcheck
--leak-check=summary --time-stamp=yes --track-fds=no
--trace-signals=no --track-origins=no --trace-children=yes
--error-limit=no --verbose my_program

I also use "--trace-children-skip=..." to not trace processes I'm not
interested in, omitted above for readability.

Host: Fedora 41.
VM: Fedora 39. Valgrind has been compiled from source and installed in the VM.
Docker container: Fedora 41, run with docker-compose in privileged
mode. Valgrind has been compiled from source and installed in the
container.

Scenarios:
1. Host -> VM -> I run Valgrind with "leak-check=summary" -> OK.
2. Host -> VM -> Docker Container -> I run Valgrind with "leak-check=no" -> OK.
3. Host -> VM -> Docker Container -> I run Valgrind with
"leak-check=summary" -> NOT OK.

"OK" above means that it took Valgrind "a moment" to analyze the
memory and produce a leak report.
"NOT OK" above means that it took a lot more (10 minutes per process or so).

In order to produce the report I send SIGTERM to my program. All the
child processes call exit(). The main process calls waitpid() and
waits for all children processes. Main process then calls exit().

Scenario 1:
All is fine when I run Valgrind in the VM.

==00:00:00:38.974 223162==
==00:00:00:38.974 223162== Searching for pointers to 472 not-freed blocks
==00:00:00:40.769 223162== Checked 265,221,516 bytes
==00:00:00:40.769 223162==

Scenario 2:
I run Valgrind in a container in a VM.
When running with "leak-check=no" Valgrind also finishes quickly.
Obviously I'm missing the "LEAK SUMMARY" which I need.

Scenario 3:
The problem only appears here, when:
If I run my program and Valgrind in a docker container.
If I enable "leak-check".

==00:00:10:37.904 769== Searching for pointers to 379 not-freed blocks
==00:00:36:34.041 769== Checked 265,208,940 bytes
==00:00:36:34.041 769== Skipped 195,035,136 bytes due to read errors

Notice the timestamps difference between (1) and (3).
The time it takes for Valgrind to search does not seem to depend on
"leak-resolution" or "leak-check-heuristics" settings.

Suspicious:
Obviously "skipped 195,035,136 bytes due to read errors" does not look good.

If I add "--trace-signals=yes", I have to kill valgrind, as it
produces enormous log files for some of the processes.
These log files contain repeated lines, similar to the one below, and
that's the reason for their size (megabytes of text).

==00:00:01:52.043 12423== Searching for pointers to 378 not-freed blocks
--00:00:01:52.237 12423-- sync signal handler: signal=7, si_code=2,
EIP=0x0, eip=0x58002d1d, from kernel
--00:00:01:52.237 12423-- sync signal handler: signal=7, si_code=2,
EIP=0x0, eip=0x58002d1d, from kernel
--00:00:01:52.237 12423-- sync signal handler: signal=7, si_code=2,
EIP=0x0, eip=0x58002d1d, from kernel
...

==00:00:01:43.214 12425== Searching for pointers to 378 not-freed blocks
--00:00:01:43.379 12425-- sync signal handler: signal=7, si_code=2,
EIP=0x0, eip=0x58002d1d, from kernel
--00:00:01:43.379 12425-- sync signal handler: signal=7, si_code=2,
EIP=0x0, eip=0x58002d1d, from kernel
...

==00:00:02:02.470 12443== Searching for pointers to 576 not-freed blocks
--00:00:02:02.677 12443-- sync signal handler: signal=7, si_code=2,
EIP=0x0, eip=0x58002d1d, from kernel
--00:00:02:02.677 12443-- sync signal handler: signal=7, si_code=2,
EIP=0x0, eip=0x58002d1d, from kernel
--00:00:02:02.677 12443-- sync signal handler: signal=7, si_code=2,
EIP=0x0, eip=0x58002d1d, from kernel
...

I was hoping you may offer some advice on what would be the next step
in trying to figure out the root cause of the problem.

Kind regards,
Wojciech