valgrind-users — General discussion list for Valgrind users

Re: [Valgrind-users] Facing Issues with 32 bit Legacy Apps on ARM64 bit

[Paul F. <pj...@wa...>]

On 10-09-24 06:59, Tech info wrote:
> We are debugging memory leaks in our legacy 32-bit multithreaded 
> applications on ARM64-bit processors, but we're facing performance 
> issues with Valgrind:
> 
>     *Application Crash*: After 9-10 minutes, the application crashes,
>     and |top commands|showsno activity for particularlegacy app.
> 

We need much more information.

Can you use Valgrind and vgdb to see where it is hanging?

I don't think that anyone is actively working on the 32bit arm port at 
the moment so unless there is an easy fix you might be on your own.

A+
Paul

[Valgrind-users] Facing Issues with 32 bit Legacy Apps on ARM64 bit

[Tech i. <tec...@gm...>]

We are debugging memory leaks in our legacy 32-bit multithreaded
applications on ARM64-bit processors, but we're facing performance issues
with Valgrind:

*Application Crash*: After 9-10 minutes, the application crashes, and top
commands shows no activity for particular legacy app.


Thanks for your help.

Re: [Valgrind-users] Subject: Valgrind Performance Issues with Legacy Apps on ARM64

[Tech i. <tec...@gm...>]

Dear Valgrind Support Team,

We are debugging memory leaks in our legacy 32-bit multithreaded
applications on ARM64-bit processors, but we're facing performance issues
with Valgrind:

   - *High CPU Usage*: Valgrind causes the application to use 100% CPU,
   disrupting other system processes.
   - *Application Crash*: After 9-10 minutes, the application crashes, and
   top shows no activity.
   - *No Improvement with Nice Value*: Adjusting the nice value to 10 did
   not alleviate the issue.
    We seek advice on optimizing Valgrind's performance or alternative
   debugging methods under these conditions.

Thank you for your help.

On Mon, Sep 9, 2024 at 9:53 AM Tech info <tec...@gm...> wrote:

>
> Dear Valgrind Support Team,
>
> We are debugging memory leaks in our legacy 32-bit multithreaded
> applications on ARM64-bit processors, but we're facing performance issues
> with Valgrind:
>
>    - *High CPU Usage*: Valgrind causes the application to use 100% CPU,
>    disrupting other system processes.
>    - *Application Crash*: After 9-10 minutes, the application crashes,
>    and top shows no activity.
>    - *No Improvement with Nice Value*: Adjusting the nice value to 10 did
>    not alleviate the issue.
>
> Attached are snapshots of the system’s behavior. We seek advice on
> optimizing Valgrind's performance or alternative debugging methods under
> these conditions.
>
> Thank you for your help.
>

Re: [Valgrind-users] Query regarding Valgrind mailer list

[Paul F. <pj...@wa...>]

> On 5 Sep 2024, at 11:18, Isharat Mahmood <ish...@gm...> wrote:
> 
> Dear Team,
> 
> We have a query regarding Valgrind working model.
> Is it the right mailer list to ask?



If it is a Valgrind User question then this is the right list.

Otherwise there is a mailing list for Valgrind developers.

A+
Paul

[Valgrind-users] Query regarding Valgrind mailer list

[Isharat M. <ish...@gm...>]

Dear Team,

We have a query regarding Valgrind working model.
Is it the right mailer list to ask?


Thanks,
Md Isharat

Re: [Valgrind-users] Simple list of functions ever called

[Paul F. <pj...@wa...>]

On 06-06-24 15:43, Byron Hawkins wrote:
> For the purposes of studying dead-code elimination in LLVM, I'd like to 
> generate a simple list of all the functions that are ever called by the 
> target program. There's no need for any timing or frequency reports or 
> backtraces or any other details. So far, the best solution I can find is 
> to filter callgrind output like this:
> 
>      grep -E "^c?fn" callgrind/output.raw | cut -d " " -f 2- | grep -Ev 
> "^0x|^c?fn"
> 
> It doesn't seem highly reliable, since I'm just picking out entries by 
> sort of guessing, then checking the results with gdb (setting a 
> breakpoint on every symbol not encountered by callgrind). All target 
> programs are trivial unit tests with rigid, deterministic behavior, but 
> it would still be nicer to have a more formally defined approach for 
> extracting a precise set of function symbols for which invocation was 
> observed during the execution. Thanks in advance for any suggestions.

There are various ways to get the function names. I usually use a train 
of commands like

awk '/^fn=.* .*/{print $2}' callgrind.out.3727 | sort | uniq -c | sort -n

(only printing the first field with the fn name from awk so not getting 
the full arg list).

A+
Paul

Re: [Valgrind-users] Can you run two instances of valgrind on separate nodes in a multi-core NUMA environment?

[Philippe W. <phi...@sk...>]

On Sat, 2024-07-20 at 10:41 -0700, Tsiang Elaine Reisler wrote:
> > 
> I agree there is no interaction in how I am using valgrind (see last
> post), but not knowing this before prompted me to ask the question on
> this list. I did not use --vgd-stop-at or setup --multi mode of
> interacting with gdbserver via gdb. Nevertheless, 'target remote |
> vgdb' is equivalent to attaching to the process in gdb, allowing full
> debugging, though without the vgdb commands (v.clo, v.info etc). These
> queries elicit no response presumably because valgrind-monitor.py is
> missing from gdb.
The gdb python commands only provide an easier way to use the valgrind
monitor commands.
But if the python code is not loaded, monitor commands can still be used
(but of course prefixed then with the "monitor" gdb command
and not using any of the functionalities that the python layer provides).

Note that monitor command output can be either output to the gdb console
or to the output of the process.
So, you might check both to be sure.

> 
> If you vgdb v.kill the valgrind process, can you and how, get a final
> detail report of all the errors which is generated when the process
> terminates "normally"?
Kill the process really means kill the process, without giving any chance
to let it run more (including to process errors).
If you really need to kill your process but you want to see the errors,
then in memcheck, you should launch a leak search.
And then in all other tools, you can use a monitor command to list
the already found errors.

Philippe

Re: [Valgrind-users] Can you run two instances of valgrind on separate nodes in a multi-core NUMA environment?

[Tsiang E. R. <te...@ih...>]

On Sat, 2024-07-20 at 11:01 +0200, Philippe Waroquiers via Valgrind-
users wrote:
> On Wed, 2024-07-17 at 20:47 +0200, Julian Seward wrote:
> > Also, there is no gdbserver involved unless you start it with
> > specific
> > flags to invoke GDB support.  But that is not the default.
> 
> Note that the option to activate or not the gdbserver in valgrind is:
>     --vgdb=no|yes|full        activate gdbserver? [yes]
> Default option is yes, which means that the gdbserver is activated
> but has
> no impact on the generated code. With full, the generated code is
> slower but allows
> more precise breakpoints and watchpoints.
I am running with the default 'yes' - so I can query it with shell
vgdb.

> 
> Unless you also use e.g. --vgdb-stop-at= or --vgdb-error=XXXX, 
> -vgdb=yes should have no visible impact and cause no interaction
> between
> different valgrind processes.
> 
I agree there is no interaction in how I am using valgrind (see last
post), but not knowing this before prompted me to ask the question on
this list. I did not use --vgd-stop-at or setup --multi mode of
interacting with gdbserver via gdb. Nevertheless, 'target remote |
vgdb' is equivalent to attaching to the process in gdb, allowing full
debugging, though without the vgdb commands (v.clo, v.info etc). These
queries elicit no response presumably because valgrind-monitor.py is
missing from gdb.

> Note that if --vgdb=no is specified, a set of features (such as
> external control
> from the shell, callgrind_control, ...) will not be available (in
> addition to no 
> no debugging).
Once I attached to the valgrind process from gdb, any further query
from shell vgdb elicits a response I'm busy talking to another client
already, can't talk to you. Detaching in gdb then allows further
queries with shell vgdb. This leads me to another question:

If you vgdb v.kill the valgrind process, can you and how, get a final
detail report of all the errors which is generated when the process
terminates "normally"?

Thanks,
Elaine

> 
> Thanks
> Philippe
> 
> 
> 
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users

Re: [Valgrind-users] Can you run two instances of valgrind on separate nodes in a multi-core NUMA environment?

[Philippe W. <phi...@sk...>]

On Wed, 2024-07-17 at 20:47 +0200, Julian Seward wrote:
> Also, there is no gdbserver involved unless you start it with specific
> flags to invoke GDB support.  But that is not the default.

Note that the option to activate or not the gdbserver in valgrind is:
    --vgdb=no|yes|full        activate gdbserver? [yes]
Default option is yes, which means that the gdbserver is activated but has
no impact on the generated code. With full, the generated code is slower but allows
more precise breakpoints and watchpoints.

Unless you also use e.g. --vgdb-stop-at= or --vgdb-error=XXXX, 
-vgdb=yes should have no visible impact and cause no interaction between
different valgrind processes.

Note that if --vgdb=no is specified, a set of features (such as external control
from the shell, callgrind_control, ...) will not be available (in addition to no 
no debugging).

Thanks
Philippe

Re: [Valgrind-users] Can you run two instances of valgrind on separate nodes in a multi-core NUMA environment?

[Tsiang E. R. <te...@ih...>]

On Thu, 2024-07-18 at 07:17 +0200, Julian Seward wrote:
> On 18/07/2024 00:00, Tsiang Elaine Reisler wrote:
> > Yes, there is cross-thread synchronization via shared memory, but
> > not
> > cross-process. I am just running helgrind and drd on exactly the
> > same
> > stand-alone program.
> 
> What CPU are you running this on?
> 
> J
Epyc 7513.

I have done some more experiments. It seems the problem is not in
valgrind. Ordinarily I don't pay much attention to NUMA, just run with
whatever defaults. I can succeed in binding the separate valgrinds to
different nodes, but only by chance, if I start them from various
virtual terminals (same with system consoles). As far as what I want to
do at this point, that is a satisfactory solution.

However, trying to bind valgrind explicitly by numactl --cpunodebind
does not work. It starts and runs valgrind through the first segment.
Everyone sounds happy. But afterwards, that process never gets another
robin round. I have no idea - no search results.

Thanks,
Elaine

PS - apologies for the previous redundant post. I got a rejection
notice by sourceforge because I used a different reply-to address than
my send address. Without checking the list, I made another post.

Re: [Valgrind-users] Can you run two instances of valgrind on separate nodes in a multi-core NUMA environment?

[Julian S. <jse...@gm...>]

On 18/07/2024 00:00, Tsiang Elaine Reisler wrote:
> Yes, there is cross-thread synchronization via shared memory, but not
> cross-process. I am just running helgrind and drd on exactly the same
> stand-alone program.

What CPU are you running this on?

J

Re: [Valgrind-users] Can you run two instances of valgrind on separate nodes in a multi-core NUMA environment?

[Tsiang E. R. <te...@ih...>]

On Wed, 2024-07-17 at 20:47 +0200, Julian Seward wrote:
> 
> I don't follow the details exactly, but FWIW .. valgrind running an
> application is "just another normal process".  It has no
> understanding
> of or special-casing relating to NUMA, or particular cores/nodes in a
> multiprocessor machine.
That was my understanding also, that each valgrind instance is stand-
alone - hence two should run on separate nodes the same as on the same
node, but with better access to memory/cpu because they do not have to
contend for them on the same node
> 
> > My conjecture is that the valgrind core is one instance of the
> > gdbserver, which then spawns the tools, and hence one should not
> > force
> 
> Also, there is no gdbserver involved unless you start it with
> specific
> flags to invoke GDB support.  But that is not the default.
> 
> It might be that if you are doing cross-process synchronisation via
> accesses to shared memory, that depend on specific details of the
> machine's
> memory coherence model, that you could wind up with problems. 
> Something
> like that I could believe.
> 
> J
> 
> 
Yes, there is cross-thread synchronization via shared memory, but not
cross-process. I am just running helgrind and drd on exactly the same
stand-alone program.

Thanks,
Elaine
> 

> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users

Re: [Valgrind-users] Can you run two instances of valgrind on separate nodes in a multi-core NUMA environment?

[Tsiang E. R. <te...@ih...>]

On Wed, 2024-07-17 at 20:47 +0200, Julian Seward wrote:
> 
> I don't follow the details exactly, but FWIW .. valgrind running an
> application is "just another normal process".  It has no
> understanding
> of or special-casing relating to NUMA, or particular cores/nodes in a
> multiprocessor machine.
That was my understanding also, that each valgrind instance is stand-
alone - hence two should run on separate nodes the same as on the same
node, but with better access to memory/cpu because they do not have to
contend for them on the same node.
> 
> > My conjecture is that the valgrind core is one instance of the
> > gdbserver, which then spawns the tools, and hence one should not
> > force
> 
> Also, there is no gdbserver involved unless you start it with
> specific
> flags to invoke GDB support.  But that is not the default.
> 
> It might be that if you are doing cross-process synchronisation via
> accesses to shared memory, that depend on specific details of the
> machine's
> memory coherence model, that you could wind up with problems. 
> Something
> like that I could believe.
> 
> J
Yes, there is cross-thread synchronization via shared memory, but not
cross-process. I am just running helgrind and drd on exactly the same
stand-alone program.

Thanks,
Elaine
> 
> 
> 
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users

Re: [Valgrind-users] Can you run two instances of valgrind on separate nodes in a multi-core NUMA environment?

[Julian S. <jse...@gm...>]

I don't follow the details exactly, but FWIW .. valgrind running an
application is "just another normal process".  It has no understanding
of or special-casing relating to NUMA, or particular cores/nodes in a
multiprocessor machine.

> My conjecture is that the valgrind core is one instance of the
> gdbserver, which then spawns the tools, and hence one should not force

Also, there is no gdbserver involved unless you start it with specific
flags to invoke GDB support.  But that is not the default.

It might be that if you are doing cross-process synchronisation via
accesses to shared memory, that depend on specific details of the machine's
memory coherence model, that you could wind up with problems.  Something
like that I could believe.

J

[Valgrind-users] Can you run two instances of valgrind on separate nodes in a multi-core NUMA environment?

[Tsiang E. R. <te...@ih...>]

Hi,

It appears that when the default policy is preferred local, the two
instances (one helgrind, one drd) are run on the same node.

I thought I could afford more processor/memory access by forcing the
two instances to run on separate nodes. When I succeeded in doing that,
one process starts up fine, but is apparently not being run, while the
other process proceeds normally. BTW, I am running from /dev/pts in
gnome.

My conjecture is that the valgrind core is one instance of the
gdbserver, which then spawns the tools, and hence one should not force
the tools to run on separate nodes. However, once that is done, it
becomes difficult to correct without restarting the process that is
already running, or maybe even rebooting - too much sunk cost.

So I thought to ask this question while I am waiting.

Thanks,
Elaine

Re: [Valgrind-users] alternative ways of communication

[Paul F. <pj...@wa...>]

On 04-07-24 14:26, Thomas Wollenzin wrote:
> Hi,
> 
> I was wondering whether there are other ways of communication for the 
> Valgrind community. In particular I'm thinking Slack or Discord?

The main problem that I see with these platforms is that the perceived 
ease of access means that there are a lot of low quality questions 
without any follow-up from the OP.

We use Slack at work. It's OK, but then it's not a public forum.

I recently tried using the ARM Discord server. There are many channels 
and the busy ones are flooded with noise. I don't find it to be a good 
way to communicate. There is a Discord server for the macOS Valgrind 
port, which is even quieter than IRC.

I also use Mattermost which is fairly good. It's used by WG21 for C++ 
standardization.

It seems like what I'm saying is the better ones have restricted access. 
That wouldn't be much use for Valgrind.

A+
Paul

Re: [Valgrind-users] alternative ways of communication

[Mark W. <ma...@kl...>]

Hi Thomas,

On Thu, 2024-07-04 at 14:26 +0000, Thomas Wollenzin wrote:
> I was wondering whether there are other ways of communication for the Valgrind community.

We do have mailinglists. And there is an IRC channel for Valgrind
developers: #valgrind-dev at irc.libera.chat but users are also more
than welcome. See https://valgrind.org/support/mailing_lists.html

And of course bugzilla:
https://valgrind.org/support/bug_reports.html

> In particular I'm thinking Slack or Discord?

Slack and Discord are proprietary platforms.
Please don't use them for free software projects.
Good background, plus alternatives, is here:
https://drewdevault.com/2021/12/28/Dont-use-Discord-for-FOSS.html

Cheers,

Mark

[Valgrind-users] alternative ways of communication

[Thomas W. <wol...@ms...>]

Hi,

I was wondering whether there are other ways of communication for the Valgrind community. In particular I'm thinking Slack or Discord?

Cheers,
Thomas

Re: [Valgrind-users] helgrind question regarding 'Possible data race during write of size 8'

[Thomas W. <wol...@ms...>]

Thanks for the reply, Julian.
I indeed might have looked at an incomplete report.
After having worked with helgrind a few more days things make more sense now.

Cheers,
Thomas
________________________________
From: Julian Seward <jse...@gm...>
Sent: Friday, June 28, 2024 12:44 PM
To: Thomas Wollenzin <wol...@ms...>; val...@li... <val...@li...>
Subject: Re: [Valgrind-users] helgrind question regarding 'Possible data race during write of size 8'


At least in the text you included, there's nothing that indicates what
the the other thread/access is.  We have:

> Possible data race during write of size 8 at 0x58EEC60 by thread #1
> Locks held: none

and these, but they just tell you about the data address involved in
the race:

>   Address 0x58eec60 is 160 bytes inside a block of size 408 alloc'd
>     at 0x484BF58: operator new(unsigned long) (vg_replace_malloc.c:487)

>   Block was alloc'd by thread #1

There should be yet another stack which shows where the conflicting access
in the "other" thread is.

J

Re: [Valgrind-users] helgrind question regarding 'Possible data race during write of size 8'

[Julian S. <jse...@gm...>]

At least in the text you included, there's nothing that indicates what
the the other thread/access is.  We have:

> Possible data race during write of size 8 at 0x58EEC60 by thread #1
> Locks held: none

and these, but they just tell you about the data address involved in
the race:

>   Address 0x58eec60 is 160 bytes inside a block of size 408 alloc'd
>     at 0x484BF58: operator new(unsigned long) (vg_replace_malloc.c:487)

>   Block was alloc'd by thread #1

There should be yet another stack which shows where the conflicting access
in the "other" thread is.

J

[Valgrind-users] helgrind question regarding 'Possible data race during write of size 8'

[Thomas W. <wol...@ms...>]

Hi,

I'm running helgrind against our code base and see this report. I once again cannot include all the code due to its proprietary and disclosed nature.

Possible data race during write of size 8 at 0x58EEC60 by thread #1
Locks held: none
   at 0x4A87603: ...
   by 0xE818EB4: ...
   by 0xE816B04: ...
   by 0xE816ABE: ...
   by 0xDD80701: ...
   by 0xDD80701: ...
   by 0xDD1FF8C: ...
   by 0x42503E: ...
   by 0x4263B0: ...
   by 0x4267E6: ...
 Address 0x58eec60 is 160 bytes inside a block of size 408 alloc'd
   at 0x484BF58: operator new(unsigned long) (vg_replace_malloc.c:487)
   by 0xE816395: ...
   by 0xE8143DC: ...
   by 0xE8143DC: ...
   by 0x4A64C49: ...
   by 0xDD7427A: ...
   by 0xDD7E472: ...
   by 0xDD7E58E: ...
   by 0xDD1FF8C: ...
   by 0x42503E: ...
   by 0x4263B0: ...
   by 0x4267E6: ...
 Block was alloc'd by thread #1
I'm a bit confused as to how this can be a data race if there's only the main thread involved. Could this be a false positive or how do I read this report correctly?

Thanks,
Thomas

Re: [Valgrind-users] [Valgrind-developers] A thousand bugs

[Paul F. <pj...@wa...>]

On 23-06-24 15:43, Mark Wielaard wrote:
> Hi all,
> 
> On Thu, Nov 16, 2023 at 08:22:33PM +0100, Mark Wielaard wrote:
>> Valgrind is more than 20 years old and we have been collecting
>> bugs slightly faster than we have been able to close them. Which means
>> we now have around a thousand bugs open. This is a slightly
>> intimidating number.
> 
> I am happy to say that we have been closing bugs faster than they are
> being filed in the last 6 months. There have been 85 new bugs filed
> and 110 bugs closed. There are now "only" 975 bugs open.

So we should clear the backlog in about 20 years time.

A+
Paul

Re: [Valgrind-users] [Valgrind-developers] A thousand bugs

[Mark W. <ma...@kl...>]

Hi all,

On Thu, Nov 16, 2023 at 08:22:33PM +0100, Mark Wielaard wrote:
> Valgrind is more than 20 years old and we have been collecting
> bugs slightly faster than we have been able to close them. Which means
> we now have around a thousand bugs open. This is a slightly
> intimidating number.

I am happy to say that we have been closing bugs faster than they are
being filed in the last 6 months. There have been 85 new bugs filed
and 110 bugs closed. There are now "only" 975 bugs open.

Great progress, but still slightly intimidating. So if people could
take a look and confirm or close issues that would be appreciated.

> But some of the bugs are more than 10 years old (the oldest bugs are
> from 2004). So some of them are likely not really relevant anymore.
> 
> If people could do some quick spot checks of some of these bugs that
> would be really appreciated.
> 
> The bugs can be found here:
> https://bugs.kde.org/buglist.cgi?product=valgrind&resolution=---
> Or per component here:
> https://bugs.kde.org/describecomponents.cgi?product=valgrind
> 
> If you filed a bug yourself please look if the issue is still
> relevant. Please add a comment saying so if it is, or close it if it
> isn't. Same if it has a simple reproducer. If it has a patch attached
> please check if it still applies.
> 
> Please don't go out of your way to close bugs, even old bugs can still
> be relevant. But it would be good to know if they really still are and
> we need to take another look at them.
> 
> Thanks,
> 
> Mark

Re: [Valgrind-users] Question regarding 'Conditional jump or move depends on uninitialised value(s)'

[Thomas W. <wol...@ms...>]

That's indeed very interesting Paul. Thanks for making me aware.

Thomas

________________________________
From: Paul Floyd via Valgrind-users <val...@li...>
Sent: Wednesday, June 19, 2024 10:07 PM
To: val...@li... <val...@li...>
Subject: Re: [Valgrind-users] Question regarding 'Conditional jump or move depends on uninitialised value(s)'



On 19-06-24 08:15, Thomas Wollenzin wrote:
> Thanks for the hint, Sean.
> While that might work perfectly fine, I'm personally not a big fan of
> these types of tools. They're fine for proofing a theory but shouldn't
> be used to 'cover up' developer mistakes. Code should be written as
> solidly as possible.


Read more about the proposal here

https://www.open-std.org/JTC1/SC22/WG21/docs/papers/2022/p2723r0.html

I'm fairly certain that things like this will be part of C++ in the next
5 years or so. That won't be using this option directly, more likely
part of a 'profiles' feature.


A+
Paul


_______________________________________________
Valgrind-users mailing list
Val...@li...
https://lists.sourceforge.net/lists/listinfo/valgrind-users

Re: [Valgrind-users] Question regarding 'Conditional jump or move depends on uninitialised value(s)'

[John R. <jr...@bi...>]

> It seems optimizatin of the code will make
> it indistinquisable from explicit initialization.

So what you want is  -ftrivial-auto-var-init=$POISON
where POISON is something like 0xA5 or 'deaddead'
and memcheck is told the value used for POISON.
The programmer should specify a value that
is highly unlikely to be a legitimate value.