valgrind-users — General discussion list for Valgrind users

Re: [Valgrind-users] Malloc/free in different threads

[Saurabh T <sa...@ho...>]

Sorry I forgot to say the false positives are for memory leaks ("definitely lost").


From: Saurabh T <sa...@ho...>
Sent: Friday, December 18, 2020 9:44 AM
To: Valgrind <val...@li...>
Subject: [Valgrind-users] Malloc/free in different threads 
 
Hi, I believe I am seeing lots of false positives in valgrind if I call free() on a different thread from the one that called malloc() - the pointers are exchanged between threads safely in between. Is this a known issue, or am I doing something wrong? Thank you.

[Valgrind-users] Malloc/free in different threads

[Saurabh T <sa...@ho...>]

Hi, I believe I am seeing lots of false positives in valgrind if I call free() on a different thread from the one that called malloc() - the pointers are exchanged between threads safely in between. Is this a known issue, or am I doing something wrong? Thank you.

Re: [Valgrind-users] Valgrind installation on AIX environment

[Philippe W. <phi...@sk...>]

Hello

Support for AIX was removed I believe in 2011.

Philippe


On Thu, 2020-12-17 at 12:31 +0000, SHAIKH Mohammadshaeeque wrote:
> Hello Valgrind Team,
>  
> I am interested to use Valgrind tool to detect memory leak issue on AIX for C++ programs,
> I want to know whether Valgrind tools supports for below type of specs environment?
>  
> AIX 7.1
> System Model: IBM,8284-22A
> Processor Type: PowerPC_POWER8
> Processor Implementation Mode: POWER 7
> CPU Type: 64-bit
> Kernel Type: 64-bit
> Xlc compiler
> dbx debugger
>  
> If yes then, please suggest to install on AIX which modification is needed on configure.sh file or other modifications.
>  
> Please provide your input.
>  
> Thanks,
> Mohammadshaeeque SHAIKH.
>  
>  
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users

[Valgrind-users] Valgrind installation on AIX environment

[SHAIKH M. <m.s...@so...>]

Hello Valgrind Team,

I am interested to use Valgrind tool to detect memory leak issue on AIX for C++ programs,
I want to know whether Valgrind tools supports for below type of specs environment?

AIX 7.1
System Model: IBM,8284-22A
Processor Type: PowerPC_POWER8
Processor Implementation Mode: POWER 7
CPU Type: 64-bit
Kernel Type: 64-bit
Xlc compiler
dbx debugger

If yes then, please suggest to install on AIX which modification is needed on configure.sh file or other modifications.

Please provide your input.

Thanks,
Mohammadshaeeque SHAIKH.

[Valgrind-users] Getting "Serious error when reading debug info" when using mmap

[Oren E. (A. Rahien) <ay...@ay...>]

Hi,
I'm getting a strange error when running Valgrind.

--23266-- WARNING: Serious error when reading debug info
--23266-- When reading debug info from /tmp/db/test.txt:
--23266-- can't read file to inspect ELF header

Note that the file in question is a data file, which I'm mmaping

I can reproduce this using the following code:
https://gist.github.com/ayende/d65a7a3c06556d62c8fa0f88b04ea449

valgrind-3.16.1 & clang version 10.0.0-4ubuntu1

This seems to be related to mmaping a file, but I can't figure out exactly
why

Running Valgrind under `strace` gives:

    mmap(0x4e4d000, 65536, PROT_READ, MAP_SHARED|MAP_FIXED, 3, 0) =
0x4e4d000
    statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL,
{stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFREG|0600,
stx_size=262144, ...}) = 0
    readlink("/proc/self/fd/3", "/tmp/db/test.txt", 4096) = 16
    statx(AT_FDCWD, "/tmp/db/test.txt", AT_STATX_SYNC_AS_STAT, STATX_ALL,
{stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFREG|0600,
stx_size=262144, ...}) = 0
    pread64(3, 0x10032c68a0, 1024, 0)       = -1 EINVAL (Invalid argument)
    getpid()                                = 23260
    write(1027, "--23260-- WARNING: Serious error"..., 57--23260-- WARNING:
Serious error when reading debug info
    ) = 57
    getpid()                                = 23260
    write(1027, "--23260-- When reading debug inf"..., 57--23260-- When
reading debug info from /tmp/db/test.txt:

So it looks like the `pread64()` code is failing, but I can't figure out
why that is the case, or why it would cause Valgrind to give this error.
Any ideas?

Re: [Valgrind-users] Getting "Serious error when reading debug info" when using mmap

[Oren E. (A. Rahien) <ay...@ay...>]

FWIW, I'm pretty sure that the issue is around the use of O_DIRECT and
alignment of memory.
Note that in the call to pread64() , the buffer isn't 4KB aligned, which is
usually required for O_DIRECT.
I assume that this generates the error and the rest of the issue.

On Wed, Dec 16, 2020 at 11:17 PM Oren Eini (Ayende Rahien) <
ay...@ay...> wrote:

> Hi,
> I'm getting a strange error when running Valgrind.
>
> --23266-- WARNING: Serious error when reading debug info
> --23266-- When reading debug info from /tmp/db/test.txt:
> --23266-- can't read file to inspect ELF header
>
> Note that the file in question is a data file, which I'm mmaping
>
> I can reproduce this using the following code:
> https://gist.github.com/ayende/d65a7a3c06556d62c8fa0f88b04ea449
>
> valgrind-3.16.1 & clang version 10.0.0-4ubuntu1
>
> This seems to be related to mmaping a file, but I can't figure out exactly
> why
>
> Running Valgrind under `strace` gives:
>
>     mmap(0x4e4d000, 65536, PROT_READ, MAP_SHARED|MAP_FIXED, 3, 0) =
> 0x4e4d000
>     statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL,
> {stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFREG|0600,
> stx_size=262144, ...}) = 0
>     readlink("/proc/self/fd/3", "/tmp/db/test.txt", 4096) = 16
>     statx(AT_FDCWD, "/tmp/db/test.txt", AT_STATX_SYNC_AS_STAT, STATX_ALL,
> {stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFREG|0600,
> stx_size=262144, ...}) = 0
>     pread64(3, 0x10032c68a0, 1024, 0)       = -1 EINVAL (Invalid argument)
>     getpid()                                = 23260
>     write(1027, "--23260-- WARNING: Serious error"..., 57--23260--
> WARNING: Serious error when reading debug info
>     ) = 57
>     getpid()                                = 23260
>     write(1027, "--23260-- When reading debug inf"..., 57--23260-- When
> reading debug info from /tmp/db/test.txt:
>
> So it looks like the `pread64()` code is failing, but I can't figure out
> why that is the case, or why it would cause Valgrind to give this error.
> Any ideas?
>


-- 
<https://ravendb.net/> *Oren Eini*
*CEO   /   Hibernating Rhinos LTD <https://hibernatingrhinos.com/>*
Mobile:  972-52-548-6969
Sales:  sa...@ra...
Skype:  ayenderahien
Support:  su...@ra...
<https://www.facebook.com/pages/RavenDB/265907650186374>
<https://twitter.com/ravendb>
<https://www.linkedin.com/company/hibernating-rhinos-ltd-/>
<https://ravendb.net/emailsignature/displayeventpage>

Re: [Valgrind-users] Why valgrind tool massif does not print xtree callstack? (arm)

[Philippe W. <phi...@sk...>]

Strange ... 'v.info scheduler' shows that valgrind can compute a correct guest
stack trace, but the output file is not correct.

It looks like with gdb+vgdb, you cannot put a breakpoint on a redirected function:
 valgrind/massif redirects malloc function to its own implementation, and so
 the breakpoint is not reached (at least for me, on amd64 platform).

Also, the host stack trace is empty in the below 'v.info scheduler' output.


Does memcheck give the same unexpected behaviour for its (leak) reports ?

What does memcheck give if you use --xtree-memory=full ?

Can you try the same gdb+vgdb+ break malloc experiment using --tool=none ?

Thanks

Philippe



On Mon, 2020-12-14 at 10:26 +0100, Łukasz Bolda wrote:
> I'm using valgrind 3.14.0 from http://ftp.debian.org/debian/pool/main/v/valgrind/
> Cause it's stable version.
> 
> I did not compile this application.
> 
> Right now i'm trying latest valgrind 3.16.1 and it changed from this:
> 99.87% (20,675,133B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
> ->99.87% (20,675,133B) 0xFFFFFFFF: ???
> 
> to this (different app, but the only change is from 0xFFFFFFFF to 0x0):
> 99.82% (46,610B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
> ->99.82% (46,610B) 0x0: ???
> 
> 
> Using "(gdb) break malloc" and then "(gdb) bt"produces regular callstack
> 
> 
> Also using  gdb+vgdb is producing correct backtrace:
> (gdb) monitor v.info scheduler
> 
> Host stacktrace:
> ==1543== at 0x5800B148: ??? (in /home/root/lbolda/vg16/usr/lib/arm-linux-gnueabihf/valgrind/massif-arm-linux)
> 
> sched status:
> 	running_tid=1
> 
> Thread 1: status = VgTs_Runnable (lwpid 1543)
> ==1543==	at 0x42826: service_main(int, char const**, bool) (serviceMain.cpp:155)
> ==1543==	by 0x41D1F: main (serviceMain.cpp:17)
> client stack range: [0x7DDE5000 0xDDE8FFF] client SP: 0x7DDE86A8
> valgrind stack range: [0x41E82000 0x41F81FFF] top usage: 25600 of 1048576
> 
> Thread 2: status = VgTs_WaitSys syscall 285 (lwpid 1560)
> (...)
> Thread 3: status = VgTs_WaitSys syscall 285 (lwpid 1561)
> (...)
> 
> 
> Thank you for helping me!
> 
> Best regards,
> Łukasz Bolda
> 
> 
> -----Original Message-----
> From: Philippe Waroquiers <phi...@sk...> 
> Sent: Friday, December 11, 2020 4:31 PM
> To: Łukasz Bolda <Luk...@ra...>; val...@li...
> Subject: Re: [Valgrind-users] Why valgrind tool massif does not print xtree callstack? (arm)
> 
> Which version of valgrind are you using ?
> You much better should use the last released version (or a more recent GIT version).
> 
> Have you compiled your application with -g ?
> 
> In case you debug your native executable with gdb and put a break on malloc, is gdb backtrace command giving a good stack trace ?
> 
> If you then debug using  gdb+vgdb your executable running under valgrind, is then the gdb backtrace correct ?
> And when positioned on the break point, what is the output of
>    (gdb) monitor v.info scheduler
> 
> Thanks
> Philippe
> 
> On Fri, 2020-12-11 at 15:02 +0100, Łukasz Bolda wrote:
> > Hello!
> > This is my first message on this list, so I'd like to say hi to everyone!
> > 
> > I'm profiling my software using valgrind tool massif  on arm hardware 
> > with parameters like this:
> > valgrind --tool=massif --massif-out-file=massif.out.%p 
> > --xtree-memory=full --verbose MY_BIN
> > 
> > Unfortuatelly i do not receive any callstack in the results:
> > ms_print massif.out.1234
> > (...)
> > 99.87% (20,675,133B) (heap allocation functions) malloc/new/new[], 
> > --alloc-fns, etc.
> > ->99.87% (20,675,133B) 0xFFFFFFFF: ???
> > 
> > 
> > What sould i do to receive full callstacks from massif?
> > 
> > Same thing happens when i'm using this tool on system binary for eg. ls.
> > Only 0xFFFFFFFF is present.
> > 
> > On my x86 machine everything runs fine.
> > 
> > Greetings,
> > Łukasz Bolda
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Valgrind-users mailing list
> > Val...@li...
> > https://lists.sourceforge.net/lists/listinfo/valgrind-users
> 
> 
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users

Re: [Valgrind-users] Why valgrind tool massif does not print xtree callstack? (arm)

[Łukasz B. <Luk...@ra...>]

I'm using valgrind 3.14.0 from http://ftp.debian.org/debian/pool/main/v/valgrind/
Cause it's stable version.

I did not compile this application.

Right now i'm trying latest valgrind 3.16.1 and it changed from this:
99.87% (20,675,133B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.87% (20,675,133B) 0xFFFFFFFF: ???

to this (different app, but the only change is from 0xFFFFFFFF to 0x0):
99.82% (46,610B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.82% (46,610B) 0x0: ???


Using "(gdb) break malloc" and then "(gdb) bt"produces regular callstack


Also using  gdb+vgdb is producing correct backtrace:
(gdb) monitor v.info scheduler

Host stacktrace:
==1543== at 0x5800B148: ??? (in /home/root/lbolda/vg16/usr/lib/arm-linux-gnueabihf/valgrind/massif-arm-linux)

sched status:
	running_tid=1

Thread 1: status = VgTs_Runnable (lwpid 1543)
==1543==	at 0x42826: service_main(int, char const**, bool) (serviceMain.cpp:155)
==1543==	by 0x41D1F: main (serviceMain.cpp:17)
client stack range: [0x7DDE5000 0xDDE8FFF] client SP: 0x7DDE86A8
valgrind stack range: [0x41E82000 0x41F81FFF] top usage: 25600 of 1048576

Thread 2: status = VgTs_WaitSys syscall 285 (lwpid 1560)
(...)
Thread 3: status = VgTs_WaitSys syscall 285 (lwpid 1561)
(...)


Thank you for helping me!

Best regards,
Łukasz Bolda


-----Original Message-----
From: Philippe Waroquiers <phi...@sk...> 
Sent: Friday, December 11, 2020 4:31 PM
To: Łukasz Bolda <Luk...@ra...>; val...@li...
Subject: Re: [Valgrind-users] Why valgrind tool massif does not print xtree callstack? (arm)

Which version of valgrind are you using ?
You much better should use the last released version (or a more recent GIT version).

Have you compiled your application with -g ?

In case you debug your native executable with gdb and put a break on malloc, is gdb backtrace command giving a good stack trace ?

If you then debug using  gdb+vgdb your executable running under valgrind, is then the gdb backtrace correct ?
And when positioned on the break point, what is the output of
   (gdb) monitor v.info scheduler

Thanks
Philippe

On Fri, 2020-12-11 at 15:02 +0100, Łukasz Bolda wrote:
> Hello!
> This is my first message on this list, so I'd like to say hi to everyone!
> 
> I'm profiling my software using valgrind tool massif  on arm hardware 
> with parameters like this:
> valgrind --tool=massif --massif-out-file=massif.out.%p 
> --xtree-memory=full --verbose MY_BIN
> 
> Unfortuatelly i do not receive any callstack in the results:
> ms_print massif.out.1234
> (...)
> 99.87% (20,675,133B) (heap allocation functions) malloc/new/new[], 
> --alloc-fns, etc.
> ->99.87% (20,675,133B) 0xFFFFFFFF: ???
> 
> 
> What sould i do to receive full callstacks from massif?
> 
> Same thing happens when i'm using this tool on system binary for eg. ls.
> Only 0xFFFFFFFF is present.
> 
> On my x86 machine everything runs fine.
> 
> Greetings,
> Łukasz Bolda
> 
> 
> 
> 
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users

Re: [Valgrind-users] Why valgrind tool massif does not print xtree callstack? (arm)

[Philippe W. <phi...@sk...>]

Which version of valgrind are you using ?
You much better should use the last released version (or a more recent GIT version).

Have you compiled your application with -g ?

In case you debug your native executable with gdb and put a break on malloc,
is gdb backtrace command giving a good stack trace ?

If you then debug using  gdb+vgdb your executable running under valgrind,
is then the gdb backtrace correct ?
And when positioned on the break point, what is the output of
   (gdb) monitor v.info scheduler

Thanks
Philippe

On Fri, 2020-12-11 at 15:02 +0100, Łukasz Bolda wrote:
> Hello!
> This is my first message on this list, so I'd like to say hi to everyone!
> 
> I'm profiling my software using valgrind tool massif  on arm hardware with
> parameters like this:
> valgrind --tool=massif --massif-out-file=massif.out.%p --xtree-memory=full
> --verbose MY_BIN
> 
> Unfortuatelly i do not receive any callstack in the results:
> ms_print massif.out.1234
> (...)
> 99.87% (20,675,133B) (heap allocation functions) malloc/new/new[],
> --alloc-fns, etc.
> ->99.87% (20,675,133B) 0xFFFFFFFF: ???
> 
> 
> What sould i do to receive full callstacks from massif?
> 
> Same thing happens when i'm using this tool on system binary for eg. ls.
> Only 0xFFFFFFFF is present.
> 
> On my x86 machine everything runs fine.
> 
> Greetings,
> Łukasz Bolda
> 
> 
> 
> 
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users

[Valgrind-users] Why valgrind tool massif does not print xtree callstack? (arm)

[Łukasz B. <Luk...@ra...>]

Hello!
This is my first message on this list, so I'd like to say hi to everyone!

I'm profiling my software using valgrind tool massif  on arm hardware with
parameters like this:
valgrind --tool=massif --massif-out-file=massif.out.%p --xtree-memory=full
--verbose MY_BIN

Unfortuatelly i do not receive any callstack in the results:
ms_print massif.out.1234
(...)
99.87% (20,675,133B) (heap allocation functions) malloc/new/new[],
--alloc-fns, etc.
->99.87% (20,675,133B) 0xFFFFFFFF: ???


What sould i do to receive full callstacks from massif?

Same thing happens when i'm using this tool on system binary for eg. ls.
Only 0xFFFFFFFF is present.

On my x86 machine everything runs fine.

Greetings,
Łukasz Bolda

Re: [Valgrind-users] 3.16.1 RPi 3b+

[John R. <jr...@bi...>]

On 2020-12-07 20:22 UTC, John A wrote:
  [[attachment smime.p7m]]

Send the body of the message in plain text,
perhaps crypto-signed with the signature in an attachment.
Nobody pays any attention to a .p7m body.

[Valgrind-users] 3.16.1 RPi 3b+

[John A <jo...@ar...>]

0€	*†H†÷

 €0€

10	+

[Valgrind-users] Obtaining Use-Def information of temporaries in VEX-IR

[Yashas A. <es1...@ii...>]

Hello

I'm trying to write a Valgrind tool that will obtain Use-Def information of
temporary values in VEX-IR, i.e, the definition of a temporary value and
its uses in the IR. Does an implementation of this already exist in
Valgrind which I can use, or should I start from scratch?

Thanks
Yashas

-- 


Disclaimer:- The information contained in this electronic message and any 
attachments to this message are intended for the exclusive use of the 
addressee(s) and may contain proprietary, confidential or privileged 
information. If you are not the intended recipient, you should not 
disseminate, distribute or copy this email. Please notify the sender 
immediately and destroy all copies of this message and any attachments. The 
views expressed in this E-mail message (including the enclosure/(s) or 
attachment/(s) if any) are those of the individual sender, except where the 
sender expressly, and with authority, states them to be the views of IIT 
Hyderabad. Before opening any mail and attachments please check them for 
viruses, malware, and the sender email address is indeed from IITH domain. 
IITH does not accept any liability for malware infected mails.

Re: [Valgrind-users] Automated report for valgrind

[John R. <jr...@bi...>]

> I would like to ask if there is any automated reporting tool which processes the output of the valgrind tool and also scans the code in order to do a "blame" or "annotate" per developer.

The real error (the place that needs fixing) can be several levels up the call chain,
and in C++ is not always the most-recent non-STL code.  Identifying the root cause
can require an experienced human in many cases.

[Valgrind-users] Automated report for valgrind

[Dimitris K. <kat...@gm...>]

Hi everyone,

I would like to ask if there is any automated reporting tool which
processes the output of the valgrind tool and also scans the code in order
to do a "blame" or "annotate" per developer.

Thanks

Re: [Valgrind-users] License Question

[Mark W. <ma...@kl...>]

Hi,

On Sun, Nov 08, 2020 at 01:06:39PM -0500, Rayce West wrote:
> After researching GGPL some, I'm wondering how it's possible that angr (and
> pyvex) are allowed to use BSD licenses when they integrate VEX as their
> lifter and IR, especially since it's the General and not the Lesser?

It is fine to combine code under a lax-permissive license like the BSD
and code under the GPL, like valgrind. As long as the licenses don't
conflict and the program as a whole is distributed under the GPL. See
GPLv2, section 2 (b).

Cheers,

Mark

[Valgrind-users] License Question

[Rayce W. <rwest@g.clemson.edu>]

Hello,

After researching GGPL some, I'm wondering how it's possible that angr (and
pyvex) are allowed to use BSD licenses when they integrate VEX as their
lifter and IR, especially since it's the General and not the Lesser?  It's
not that I wish to cause them any trouble, I've just been considering using
VEX for other work myself and I (frankly) wouldn't want to GPL it if I
did.

Thanks for any insight!

Rayce

Re: [Valgrind-users] Partial heap profiling with Massif

[LEPAREUR L. <loi...@ce...>]

Le 29/09/2020 à 15:19, Milian Wolff a écrit :
> On Dienstag, 29. September 2020 12:19:24 CEST LEPAREUR Loic wrote:
>> Le 21/09/2020 à 17:04, Milian Wolff a écrit :
>>> On Donnerstag, 17. September 2020 15:32:04 CEST John Reiser wrote:
>>>> On 2020-09-17 LEPAREUR Loic wrote:
>>>>> Several years ago, I developed a Massif patch to let the client code
>>>>> select which part of the code should be profiled with Massif.
>>>> Today this is much less interesting unless you compare and contrast with
>>>> https://github.com/KDE/heaptrack .  Do a web search for "heaptrack vs
>>>> massif". Consider particularly "A Faster Massif" in
>>>> https://milianw.de/tag/heaptrack , and
>>>> https://milianw.de/tag/massif-visualizer .
>>> In other news: I just pushed a bunch of patches to heaptrack git which
>>> allow you to do time-diffing. I.e. you can now select a time range in the
>>> charts and then request filtering to show the delta costs between the two
>>> time points.
>>>
>>> This should allow you to solve your problem nicely.
>>>
>>> Alternatively, you can also runtime-attach heaptrack after you finished
>>> your initialization phase to only record the main computational loop. The
>>> overhead should be minimal or actually close to zero when your main loop
>>> isn't allocating anything.
>>>
>>> Cheers
>> OK, thanks for pointing that out. I didn't know about it, I will give it
>> a try.
>>
>> Heaptrack requires recent versions of KChart and Qt for the GUI and it
>> isn't easy to upgrade them here.
> What do you mean by "recent" - to my knowledge it can build with some-years-
> old versions of both ;-)
>
> But even then, you can just download the AppImage and don't need to install
> anything then:
>
> https://download.kde.org/stable/heaptrack/1.2.0/
>
>> And since you LD_PRELOAD Heaptrack, it
>> would be very easy to add client requests mechanism, but I will try
>> vanilla version first.
> Yes, there's already `heaptrack_api.h`, which could be expanded to add support
> for client markers or similar.
>
> Cheers
>
Sorry, I only meant : "more recent that what I have here".

OK I will try the AppImage, it's nice it exists one.

Loïc

Re: [Valgrind-users] running valgrind on dpdk

[John R. <jr...@bi...>]

On 2020-09-29  Prujan, Michael wrote:

> I’m running valgrind-3.16.1 on application with DPDK.
> 
> I got the following errors:
> 
> ERROR: This system does not support "FSGSBASE".

That's correct: the virtual machine that valgrind exports to the app,
does not support FSGSBASE.  The valgrind source says:
     ./VEX/priv/guest_amd64_helpers.c:            /* Don't advertise FSGSBASE support, bit 0 in EBX.  */

Perhaps you could develop the support for fsgsbase, and submit the implementation patch?

[Valgrind-users] running valgrind on dpdk

[Prujan, M. <Mic...@ve...>]

Hi,
I'm running valgrind-3.16.1 on application with DPDK.
I got the following errors:

ERROR: This system does not support "FSGSBASE".
Please check that RTE_MACHINE is set correctly.
EAL: FATAL: unsupported cpu type.
EAL: unsupported cpu type.

However, FSGSBASE exists.
Cat /proc/cpuinfo:

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_pt mba tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local ibpb ibrs stibp dtherm arat pln pts pku ospke spec_ctrl intel_stibp


I compiled DPDK, without that flag,... application is not running!!!!

Thanks,
Michael.



This electronic message may contain proprietary and confidential information of Verint Systems Inc., its affiliates and/or subsidiaries. The information is intended to be for the use of the individual(s) or entity(ies) named above. If you are not the intended recipient (or authorized to receive this e-mail for the intended recipient), you may not use, copy, disclose or distribute to anyone this message or any information contained in this message. If you have received this electronic message in error, please notify us by replying to this e-mail.

Re: [Valgrind-users] Partial heap profiling with Massif

[Milian W. <ma...@mi...>]

On Dienstag, 29. September 2020 12:19:24 CEST LEPAREUR Loic wrote:
> Le 21/09/2020 à 17:04, Milian Wolff a écrit :
> > On Donnerstag, 17. September 2020 15:32:04 CEST John Reiser wrote:
> >> On 2020-09-17 LEPAREUR Loic wrote:
> >>> Several years ago, I developed a Massif patch to let the client code
> >>> select which part of the code should be profiled with Massif.
> >> 
> >> Today this is much less interesting unless you compare and contrast with
> >> https://github.com/KDE/heaptrack .  Do a web search for "heaptrack vs
> >> massif". Consider particularly "A Faster Massif" in
> >> https://milianw.de/tag/heaptrack , and
> >> https://milianw.de/tag/massif-visualizer .
> > 
> > In other news: I just pushed a bunch of patches to heaptrack git which
> > allow you to do time-diffing. I.e. you can now select a time range in the
> > charts and then request filtering to show the delta costs between the two
> > time points.
> > 
> > This should allow you to solve your problem nicely.
> > 
> > Alternatively, you can also runtime-attach heaptrack after you finished
> > your initialization phase to only record the main computational loop. The
> > overhead should be minimal or actually close to zero when your main loop
> > isn't allocating anything.
> > 
> > Cheers
> 
> OK, thanks for pointing that out. I didn't know about it, I will give it
> a try.
> 
> Heaptrack requires recent versions of KChart and Qt for the GUI and it
> isn't easy to upgrade them here.

What do you mean by "recent" - to my knowledge it can build with some-years-
old versions of both ;-)

But even then, you can just download the AppImage and don't need to install 
anything then:

https://download.kde.org/stable/heaptrack/1.2.0/

> And since you LD_PRELOAD Heaptrack, it
> would be very easy to add client requests mechanism, but I will try
> vanilla version first.

Yes, there's already `heaptrack_api.h`, which could be expanded to add support 
for client markers or similar.

Cheers

-- 
Milian Wolff
ma...@mi...
http://milianw.de

Re: [Valgrind-users] Partial heap profiling with Massif

[LEPAREUR L. <loi...@ce...>]

Le 21/09/2020 à 17:04, Milian Wolff a écrit :
> On Donnerstag, 17. September 2020 15:32:04 CEST John Reiser wrote:
>> On 2020-09-17 LEPAREUR Loic wrote:
>>> Several years ago, I developed a Massif patch to let the client code
>>> select which part of the code should be profiled with Massif.
>> Today this is much less interesting unless you compare and contrast with
>> https://github.com/KDE/heaptrack .  Do a web search for "heaptrack vs
>> massif". Consider particularly "A Faster Massif" in
>> https://milianw.de/tag/heaptrack , and
>> https://milianw.de/tag/massif-visualizer .
> In other news: I just pushed a bunch of patches to heaptrack git which allow
> you to do time-diffing. I.e. you can now select a time range in the charts and
> then request filtering to show the delta costs between the two time points.
>
> This should allow you to solve your problem nicely.
>
> Alternatively, you can also runtime-attach heaptrack after you finished your
> initialization phase to only record the main computational loop. The overhead
> should be minimal or actually close to zero when your main loop isn't
> allocating anything.
>
> Cheers
>
OK, thanks for pointing that out. I didn't know about it, I will give it 
a try.

Heaptrack requires recent versions of KChart and Qt for the GUI and it 
isn't easy to upgrade them here. And since you LD_PRELOAD Heaptrack, it 
would be very easy to add client requests mechanism, but I will try 
vanilla version first.

Cheers

Re: [Valgrind-users] Partial heap profiling with Massif

[Milian W. <ma...@mi...>]

On Donnerstag, 17. September 2020 15:32:04 CEST John Reiser wrote:
> On 2020-09-17 LEPAREUR Loic wrote:
> > Several years ago, I developed a Massif patch to let the client code
> > select which part of the code should be profiled with Massif.

> Today this is much less interesting unless you compare and contrast with
> https://github.com/KDE/heaptrack .  Do a web search for "heaptrack vs
> massif". Consider particularly "A Faster Massif" in 
> https://milianw.de/tag/heaptrack , and 
> https://milianw.de/tag/massif-visualizer .

In other news: I just pushed a bunch of patches to heaptrack git which allow 
you to do time-diffing. I.e. you can now select a time range in the charts and 
then request filtering to show the delta costs between the two time points.

This should allow you to solve your problem nicely.

Alternatively, you can also runtime-attach heaptrack after you finished your 
initialization phase to only record the main computational loop. The overhead 
should be minimal or actually close to zero when your main loop isn't 
allocating anything.

Cheers

-- 
Milian Wolff
ma...@mi...
http://milianw.de

Re: [Valgrind-users] Partial heap profiling with Massif

[John R. <jr...@bi...>]

On 2020-09-17 LEPAREUR Loic wrote:

> Several years ago, I developed a Massif patch to let the client code select which part of the code should be profiled with Massif.

Today this is much less interesting unless you compare and contrast with
https://github.com/KDE/heaptrack .  Do a web search for "heaptrack vs massif".
Consider particularly "A Faster Massif" in  https://milianw.de/tag/heaptrack ,
and  https://milianw.de/tag/massif-visualizer .

[Valgrind-users] Partial heap profiling with Massif

[LEPAREUR L. <loi...@ce...>]

Hi,

Several years ago, I developed a Massif patch to let the client code 
select which part of the code should be profiled with Massif. This is 
particulary useful for scientific simulation codes which often have two 
phases during the run :

   1) initialisation's phase
   2) main computational loop (heavy CPU load)

During phase 1, one allocates a lot of memory (reading meshes, creating 
variables, ...) but the critical part is the main loop : bogus codes 
slightly increase memory consumption but the increase due to one 
iteration of the loop is tiny compared to the total amount of memory 
allocated before the main loop. Since the memory is freed after the 
loop, this is not a leak, but the code can run out of memory because 
there are thousands and thousands of iterations.

To detect the problem with Massif, one has to let the code run many 
iterations of the loop under valgrind ... and it's way too slow. The 
patch I've made ease the detection of such a problem within just a few 
iterations. It consists in two new command line options and three 
client's requests :

--record-from-start=yes/no : this disables heap profiling until Massif 
meets the client's request which tells it to start profiling the heap 
(see below).

--disable-auto-snapshots=yes/no : this disables all the snapshots except 
the ones that are explicitly asked in a client's request.

VALGRIND_START_MEM_RECORDING : this request tells Massif to start heap 
profiling.

VALGRIND_STOP_MEM_RECORDING : this request tells Massif to stop heap 
profiling.

VALGRIND_TAKE_DETAILED_SNAPSHOT : this request tells Massif to take a 
detailed snapshot.

So using VALGRIND_START_MEM_RECORDING just before the main loop and 
VALGRIND_TAKE_DETAILED_SNAPSHOT at the beginning of the loop, Massif can 
report exactly what is going on during the main loop. As an example, I 
made a regression test which simulates that problem (big initial 
allocation and then tiny allocations in a loop).

Massif report without the patch:

     MB
1.001^ :
      | #::::::::::::::::::::::::::::::::::::::::::::::::::@:::::::::
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
      |          #:::::::::::: :::: :::: ::: ::::::::::: :::: ::: ::@
    0 
+----------------------------------------------------------------------->MB
0 7.056

Massif report with the patch:

      B
216^ @
| @
| @
| @@@@@@@@
| @      @
| @@@@@@@@@      @
|                                                        @ @      @
      | @@@@@@@@@       @      @
      | @       @       @      @
      | @@@@@@@@@       @       @      @
      |                                        @ @       @       @      @
      |                                        @ @       @       @      @
      |                                @@@@@@@@@ @       @       @      @
      |                                @       @ @       @       @      @
      |                        @@@@@@@@@       @ @       @       @      @
      |                        @       @       @ @       @       @      @
      |                @@@@@@@@@       @       @ @       @       @      @
      |                @       @       @       @ @       @       @      @
      |        @@@@@@@@@       @       @       @ @       @       @      @
      |        @       @       @       @       @ @       @       @      @
    0 
+----------------------------------------------------------------------->MB
0 4.550

I've developed the patch in a GIT branch that I've just rebased on the 
master. I've updated the docs (NEWS, manual, ...) and created a 
regression test for it in massif/tests. I think this feature could 
interest other developers of scientific codes (even if, I think, DHAT 
can now help with such issue), so if you (valgrind developers) think it 
could be interesting to take a look at it, let me know : I can send you 
the patch or do a pull request.

Thanks,
Loïc