valgrind-users — General discussion list for Valgrind users

[Valgrind-users] MySQL related messages

[James R. <jam...@gm...>]

I have a bunch of what seem to be MySQL related messages for my crawler at
https://github.com/JamesRead5737/webcrawler

==371932== 3,416 bytes in 61 blocks are still reachable in loss record 406
of 408
==371932==    at 0x483BE63: operator new(unsigned long) (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==371932==    by 0x49CF339: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CF5FD: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49D02B8: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49D0CEE: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CE5AE: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x55C347E: __pthread_once_slow (pthread_once.c:116)
==371932==    by 0x49CE07D: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CF022: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CF10E: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x497D21B: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x497D41A: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==
==371932== 4,328 bytes in 1 blocks are still reachable in loss record 407
of 408
==371932==    at 0x483BE63: operator new(unsigned long) (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==371932==    by 0x49CD678: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CF1BB: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CF3C1: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CF5FD: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49D02B8: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49D0CEE: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CE5AE: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x55C347E: __pthread_once_slow (pthread_once.c:116)
==371932==    by 0x49CE07D: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CF022: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CF10E: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==
==371932== 16,528 bytes in 1 blocks are possibly lost in loss record 408 of
408
==371932==    at 0x483B7F3: malloc (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==371932==    by 0x49D46F7: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49E3199: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49D4CBB: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CD954: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CE5E1: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x55C347E: __pthread_once_slow (pthread_once.c:116)
==371932==    by 0x49CE07D: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CF022: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x49CF10E: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x497D21B: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)
==371932==    by 0x497D41A: ??? (in
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.1.20)

I'm guessing I'm going to have to compile MySQL with -g flag in order to
decipher this stuff?

James Read

Re: [Valgrind-users] Help understanding some valgrind output

[David C. <dcc...@ac...>]

On 6/4/2020 7:26 PM, James Read wrote:
> Here is my valgrind output that I don't understand:
>
> ==319842== Invalid write of size 1
> ==319842==    at 0x48436E4: mempcpy (in 
> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==319842==    by 0x50CD1D8: _IO_default_xsputn (genops.c:386)
> ==319842==    by 0x50CD1D8: _IO_default_xsputn (genops.c:370)
> ==319842==    by 0x50B227B: __vfprintf_internal (vfprintf-internal.c:1688)
> ==319842==    by 0x50C0278: __vsprintf_internal (iovsprintf.c:95)
> ==319842==    by 0x509D047: sprintf (sprintf.c:30)
> ==319842==    by 0x10B88F: html_link_find (crawler.c:452)
> ==319842==    by 0x10BD6F: html_parse (crawler.c:536)
> ==319842==    by 0x10C2CB: check_multi_info (crawler.c:678)
> ==319842==    by 0x10C3DA: event_cb (crawler.c:706)
> ==319842==    by 0x10D828: crawler_init (crawler.c:1154)
> ==319842==    by 0x10DAE8: main (crawler.c:1207)
> ==319842==  Address 0xf107d18 is 0 bytes after a block of size 8,200 
> alloc'd
> ==319842==    at 0x483B7F3: malloc (in 
> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==319842==    by 0x10B736: html_link_find (crawler.c:440)
> ==319842==    by 0x10BD6F: html_parse (crawler.c:536)
> ==319842==    by 0x10C2CB: check_multi_info (crawler.c:678)
> ==319842==    by 0x10C3DA: event_cb (crawler.c:706)
> ==319842==    by 0x10D828: crawler_init (crawler.c:1154)
> ==319842==    by 0x10DAE8: main (crawler.c:1207)
> ==319842==
>
> valgrind: m_mallocfree.c:305 (get_bszB_as_is): Assertion 'bszB_lo == 
> bszB_hi' failed.
> valgrind: Heap block lo/hi size mismatch: lo = 8272, hi = 
> 3625731377157460067.
> This is probably caused by your program erroneously writing past the
> end of a heap block and corrupting heap metadata.  If you fix any
> invalid writes reported by Memcheck, this assertion failure will
> probably go away.  Please try that before reporting this as a bug.
>
> The code this pertains to can be found at 
> https://github.com/JamesRead5737/webcrawler
>
> Any help in understanding what this error means would be greatly 
> appreciated.
>
> James Read
>

Line 452 of crawler.c (routine html_link_find()) is calling sprintf(), 
which is then writing past the end of an allocated buffer of 8200 
bytes.  My first guess is that you are not resizing the buffer enough to 
fit the text being written by sprintf().

Find out why this is occurring in html_link_find() (it should happen 
even when your code is not running under valgrind), fix it, then rerun 
valgrind.  If your program is writing outside of buffers, it is possible 
that valgrind won't catch it and thus the assertion failure later.

-- 
     David Chapman      dcc...@ac...
     Chapman Consulting -- San Jose, CA
     EDA Software Developer, Expert Witness
     www.chapman-consulting-sj.com
     2018-2019 Chair, IEEE Consultants' Network of Silicon Valley

Re: [Valgrind-users] Help understanding some valgrind output

[Derrick M. <der...@gm...>]

Oh sorry, you're trying to write 8201 bytes into an 8200 byte buffer.

On Thu, Jun 4, 2020, 22:36 Derrick McKee <der...@gm...> wrote:

> It looks like html_link_find is allocating a buffer of size 0, and then
> you are trying to write 1 byte to it.
>
> On Thu, Jun 4, 2020, 22:28 James Read <jam...@gm...> wrote:
>
>> Here is my valgrind output that I don't understand:
>>
>> ==319842== Invalid write of size 1
>> ==319842==    at 0x48436E4: mempcpy (in
>> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
>> ==319842==    by 0x50CD1D8: _IO_default_xsputn (genops.c:386)
>> ==319842==    by 0x50CD1D8: _IO_default_xsputn (genops.c:370)
>> ==319842==    by 0x50B227B: __vfprintf_internal (vfprintf-internal.c:1688)
>> ==319842==    by 0x50C0278: __vsprintf_internal (iovsprintf.c:95)
>> ==319842==    by 0x509D047: sprintf (sprintf.c:30)
>> ==319842==    by 0x10B88F: html_link_find (crawler.c:452)
>> ==319842==    by 0x10BD6F: html_parse (crawler.c:536)
>> ==319842==    by 0x10C2CB: check_multi_info (crawler.c:678)
>> ==319842==    by 0x10C3DA: event_cb (crawler.c:706)
>> ==319842==    by 0x10D828: crawler_init (crawler.c:1154)
>> ==319842==    by 0x10DAE8: main (crawler.c:1207)
>> ==319842==  Address 0xf107d18 is 0 bytes after a block of size 8,200
>> alloc'd
>> ==319842==    at 0x483B7F3: malloc (in
>> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
>> ==319842==    by 0x10B736: html_link_find (crawler.c:440)
>> ==319842==    by 0x10BD6F: html_parse (crawler.c:536)
>> ==319842==    by 0x10C2CB: check_multi_info (crawler.c:678)
>> ==319842==    by 0x10C3DA: event_cb (crawler.c:706)
>> ==319842==    by 0x10D828: crawler_init (crawler.c:1154)
>> ==319842==    by 0x10DAE8: main (crawler.c:1207)
>> ==319842==
>>
>> valgrind: m_mallocfree.c:305 (get_bszB_as_is): Assertion 'bszB_lo ==
>> bszB_hi' failed.
>> valgrind: Heap block lo/hi size mismatch: lo = 8272, hi =
>> 3625731377157460067.
>> This is probably caused by your program erroneously writing past the
>> end of a heap block and corrupting heap metadata.  If you fix any
>> invalid writes reported by Memcheck, this assertion failure will
>> probably go away.  Please try that before reporting this as a bug.
>>
>> The code this pertains to can be found at
>> https://github.com/JamesRead5737/webcrawler
>>
>> Any help in understanding what this error means would be greatly
>> appreciated.
>>
>> James Read
>> _______________________________________________
>> Valgrind-users mailing list
>> Val...@li...
>> https://lists.sourceforge.net/lists/listinfo/valgrind-users
>>
>

Re: [Valgrind-users] Help understanding some valgrind output

[Derrick M. <der...@gm...>]

It looks like html_link_find is allocating a buffer of size 0, and then you
are trying to write 1 byte to it.

On Thu, Jun 4, 2020, 22:28 James Read <jam...@gm...> wrote:

> Here is my valgrind output that I don't understand:
>
> ==319842== Invalid write of size 1
> ==319842==    at 0x48436E4: mempcpy (in
> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==319842==    by 0x50CD1D8: _IO_default_xsputn (genops.c:386)
> ==319842==    by 0x50CD1D8: _IO_default_xsputn (genops.c:370)
> ==319842==    by 0x50B227B: __vfprintf_internal (vfprintf-internal.c:1688)
> ==319842==    by 0x50C0278: __vsprintf_internal (iovsprintf.c:95)
> ==319842==    by 0x509D047: sprintf (sprintf.c:30)
> ==319842==    by 0x10B88F: html_link_find (crawler.c:452)
> ==319842==    by 0x10BD6F: html_parse (crawler.c:536)
> ==319842==    by 0x10C2CB: check_multi_info (crawler.c:678)
> ==319842==    by 0x10C3DA: event_cb (crawler.c:706)
> ==319842==    by 0x10D828: crawler_init (crawler.c:1154)
> ==319842==    by 0x10DAE8: main (crawler.c:1207)
> ==319842==  Address 0xf107d18 is 0 bytes after a block of size 8,200
> alloc'd
> ==319842==    at 0x483B7F3: malloc (in
> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==319842==    by 0x10B736: html_link_find (crawler.c:440)
> ==319842==    by 0x10BD6F: html_parse (crawler.c:536)
> ==319842==    by 0x10C2CB: check_multi_info (crawler.c:678)
> ==319842==    by 0x10C3DA: event_cb (crawler.c:706)
> ==319842==    by 0x10D828: crawler_init (crawler.c:1154)
> ==319842==    by 0x10DAE8: main (crawler.c:1207)
> ==319842==
>
> valgrind: m_mallocfree.c:305 (get_bszB_as_is): Assertion 'bszB_lo ==
> bszB_hi' failed.
> valgrind: Heap block lo/hi size mismatch: lo = 8272, hi =
> 3625731377157460067.
> This is probably caused by your program erroneously writing past the
> end of a heap block and corrupting heap metadata.  If you fix any
> invalid writes reported by Memcheck, this assertion failure will
> probably go away.  Please try that before reporting this as a bug.
>
> The code this pertains to can be found at
> https://github.com/JamesRead5737/webcrawler
>
> Any help in understanding what this error means would be greatly
> appreciated.
>
> James Read
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users
>

[Valgrind-users] Help understanding some valgrind output

[James R. <jam...@gm...>]

Here is my valgrind output that I don't understand:

==319842== Invalid write of size 1
==319842==    at 0x48436E4: mempcpy (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==319842==    by 0x50CD1D8: _IO_default_xsputn (genops.c:386)
==319842==    by 0x50CD1D8: _IO_default_xsputn (genops.c:370)
==319842==    by 0x50B227B: __vfprintf_internal (vfprintf-internal.c:1688)
==319842==    by 0x50C0278: __vsprintf_internal (iovsprintf.c:95)
==319842==    by 0x509D047: sprintf (sprintf.c:30)
==319842==    by 0x10B88F: html_link_find (crawler.c:452)
==319842==    by 0x10BD6F: html_parse (crawler.c:536)
==319842==    by 0x10C2CB: check_multi_info (crawler.c:678)
==319842==    by 0x10C3DA: event_cb (crawler.c:706)
==319842==    by 0x10D828: crawler_init (crawler.c:1154)
==319842==    by 0x10DAE8: main (crawler.c:1207)
==319842==  Address 0xf107d18 is 0 bytes after a block of size 8,200 alloc'd
==319842==    at 0x483B7F3: malloc (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==319842==    by 0x10B736: html_link_find (crawler.c:440)
==319842==    by 0x10BD6F: html_parse (crawler.c:536)
==319842==    by 0x10C2CB: check_multi_info (crawler.c:678)
==319842==    by 0x10C3DA: event_cb (crawler.c:706)
==319842==    by 0x10D828: crawler_init (crawler.c:1154)
==319842==    by 0x10DAE8: main (crawler.c:1207)
==319842==

valgrind: m_mallocfree.c:305 (get_bszB_as_is): Assertion 'bszB_lo ==
bszB_hi' failed.
valgrind: Heap block lo/hi size mismatch: lo = 8272, hi =
3625731377157460067.
This is probably caused by your program erroneously writing past the
end of a heap block and corrupting heap metadata.  If you fix any
invalid writes reported by Memcheck, this assertion failure will
probably go away.  Please try that before reporting this as a bug.

The code this pertains to can be found at
https://github.com/JamesRead5737/webcrawler

Any help in understanding what this error means would be greatly
appreciated.

James Read

Re: [Valgrind-users] DHAT speed

[John R. <jr...@bi...>]

On 5/27/20 Paul FLOYD wrote:

> Well, no real surprises. This is with a testcase that runs standalone in about 5 seconds and under DHAT in about 200 seconds (so a reasonable slowdown of 40x).
> 
> # Overhead          Command       Shared Object
> Symbol
> # ........  ...............  ..................  ................................................................................................................................................................................................................
> #
>     29.11%  dhat-amd64-linu  dhat-amd64-linux    [.] interval_tree_Cmp
>     21.13%  dhat-amd64-linu  perf-26905.map      [.] 0x00000010057a25f8
>     13.32%  dhat-amd64-linu  dhat-amd64-linux    [.] vgPlain_lookupFM
>      9.56%  dhat-amd64-linu  dhat-amd64-linux    [.] dh_handle_read
>      8.83%  dhat-amd64-linu  dhat-amd64-linux    [.] vgPlain_nextIterFM
>      4.66%  dhat-amd64-linu  dhat-amd64-linux    [.] check_for_peak
>      1.85%  dhat-amd64-linu  dhat-amd64-linux    [.] vgPlain_disp_cp_xindir
>      1.32%  dhat-amd64-linu  [kernel.kallsyms]   [k] 0xffffffff8103ec0a
>      1.00%  dhat-amd64-linu  dhat-amd64-linux    [.] dh_handle_write

To me this suggests two things:

1) investigate the coding of the 4 or 5 highest-use subroutines (interval_tree_Cmp,
vgPlain_lookupFM, dh_handle_read, vgPlain_nextIterFM)

2) see whether DHAT might recognize and use higher-level abstractions
than MemoryRead and MemoryWrite of individual addresses.  Similar to memcheck
intercepting and analyzing strlen (etc.) as a complete concept instead of as its
individual Reads and Writes, perhaps DHAT could intercept (and/or recognize)
vector linear search, vector addition, vector partial sum, other BLAS routines, etc.,
and then analyze the algorithm as a whole.

Re: [Valgrind-users] DHAT speed

[Paul F. <pj...@wa...>]

[snip - perf]

Well, no real surprises. This is with a testcase that runs standalone in about 5 seconds and under DHAT in about 200 seconds (so a reasonable slowdown of 40x).

# Overhead          Command       Shared Object                                                                                                                                                                                                            
Symbol
# ........  ...............  ..................  ................................................................................................................................................................................................................
#
   29.11%  dhat-amd64-linu  dhat-amd64-linux    [.] interval_tree_Cmp                                                                                                                                                                                           
   21.13%  dhat-amd64-linu  perf-26905.map      [.] 0x00000010057a25f8                                                                                                                                                                                          
   13.32%  dhat-amd64-linu  dhat-amd64-linux    [.] vgPlain_lookupFM                                                                                                                                                                                            
    9.56%  dhat-amd64-linu  dhat-amd64-linux    [.] dh_handle_read                                                                                                                                                                                              
    8.83%  dhat-amd64-linu  dhat-amd64-linux    [.] vgPlain_nextIterFM                                                                                                                                                                                          
    4.66%  dhat-amd64-linu  dhat-amd64-linux    [.] check_for_peak                                                                                                                                                                                              
    1.85%  dhat-amd64-linu  dhat-amd64-linux    [.] vgPlain_disp_cp_xindir                                                                                                                                                                                      
    1.32%  dhat-amd64-linu  [kernel.kallsyms]   [k] 0xffffffff8103ec0a                                                                                                                                                                                          
    1.00%  dhat-amd64-linu  dhat-amd64-linux    [.] dh_handle_write  


A+
Paul

Re: [Valgrind-users] DHAT speed

[John R. <jr...@bi...>]

>  > So (4 * 5 * 10 * 3) is a slowdown of 600X, which turns 10 minutes into 100 hours.
> 
> What I'm seeing is a DHAT-only slowdown that is much more than that.

Running 'perf' is likely to give data and strong hints about what is going on.
The overhead of 'perf' is only a few percent.

     perf record valgrind --tool=DHAT <<valgrind_args>> ./my_app <<my_app_args>>
     perf report  > perf_output.txt

The "perf record ..." will stop shortly after the valgrind sub-process terminates.
You don't have to wait for DHAT to finish; just 'kill' it after a while.

Re: [Valgrind-users] DHAT speed

[Paul F. <pj...@wa...>]

> Message du 26/05/20 13:19> De : "John Reiser"  the ratio is about 1:50. So right away, that's a hardware slowdown of 4X.Maybe more. The machine has 12Mbyte of cache according to cpuinfo.> Valgrind runs every tool single-threaded. So if your app averages 5 active threads,> then that is a slowdown of 5X.I was running the application in single thread mode.> Valgrind's JIT (Just-In-Time) instruction emulator has a slowdown. Assume 10X (or measure nulgrind.)Yes, this is what I see with nulgrind, about a 11x factor slowdown. However this will also account for a large part of the cache overhead.> Finally we get to "useful work": the slowdown of the tool DHAT. Assume 3X.> So (4 * 5 * 10 * 3) is a slowdown of 600X, which turns 10 minutes into 100 hours.What I'm seeing is a DHAT-only slowdown that is much more than that.A+Paul

Re: [Valgrind-users] DHAT speed

[Paul F. <pj...@wa...>]

 > That doesn't sound right. I use DHAT extensively and expect a slowdown of> perhaps 50:1, maybe less. What you're describing is a slowdown factor of> at least several thousand.>> Bear in mind though that (1) V sequentialises thread execution, which wil> make a big difference if the program is heavily multithreaded, and (2)> I suspect dhat's scheme of looking up all memory accesses in an AVL tree> (of malloc'd blocks) doesn't scale all that well if you have tens of> millions of blocks.>> Can you run it on a smaller workload?  HiI'll try on something smaller and also get some info on the number of blocks of memory allocated.A+Paul

Re: [Valgrind-users] DHAT speed

[John R. <jr...@bi...>]

> The server has 48Gbytes of RAM and only about 6Gbytes is being used.
> 
> The executable is quite big
> 
> text data bss dec hex filename
> 57369168 417156 20903108 78689432 4b0b498 [snip]
> 
> The run under DHAT is using about 2Gbytes virtual and 1.5Gbytes resident according to htop. Running standalone those are about 750M and 350M respectively.

Some hardware cache+memory delays:
    L1 hit    3 cycles  ( 32KB size)
    L2 hit   11 cycles  (256KB size)
    L3 hit   25 cycles  (4MB to 40MB size)
    miss    180 cycles
The dynamic RAM chips commonly used for main memory have stayed the same speed
for over 25 years: 60 nanoseconds from CAS (Column Address Strobe) to DataOut.
If the CPU runs at 3GHz, then a cache miss costs at least 180 cycles.
A quick estimate of (largest and slowest) cache size is given by the "cache size"
line from /proc/cpuinfo:
-----
$ sed 9q  < /proc/cpuinfo  # on a 8-year old consumer-grade machine
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 42
model name	: Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz
stepping	: 7
microcode	: 0x2f
cpu MHz		: 1599.982
cache size	: 6144 KB
-----

Assume that your "big" server has 30MB of L3 cache.  For a resident set size of
350MB then that is a ratio of about 1:12.  For a resident set size of 1.5GB then
the ratio is about 1:50.  So right away, that's a hardware slowdown of 4X.

Valgrind runs every tool single-threaded.  So if your app averages 5 active threads,
then that is a slowdown of 5X.

Valgrind's JIT (Just-In-Time) instruction emulator has a slowdown.  Assume 10X (or measure nulgrind.)

Finally we get to "useful work": the slowdown of the tool DHAT.  Assume 3X.

So (4 * 5 * 10 * 3) is a slowdown of 600X, which turns 10 minutes into 100 hours.

Re: [Valgrind-users] DHAT speed

[Paul F. <pj...@wa...>]

On 5/26/20, Paul FLOYD wrote:

> > I'm running DHAT on what I consider to be a relatively small example. Standalone the executable runs in a bit under 10 minutes. Based on the CPU time that we print after every 10% of 
progress, under DHAT the same executable is going to take about 422 hours - about two and a half weeks.
> >
> > Does anyone have any ideas what could be causing it to be so slow?

> Please tell us about your environment.
> (EVERY query should supply this information!)

I just wanted to have an idea of the general expectation for DHAT overhead.

> Which version of DHAT? Which hardware? Which operating system and version?
> Which compiler and version built DHAT? Which compiler(s) built the program under test?

Red Hat Enterprise Linux Workstation release 6.5 (Santiago)

gcc (GCC) 5.3.0 for the test executable.
gcc version 9.2.0 (GCC) for building Valgrind, built from git HEAD.

The server has 48Gbytes of RAM and only about 6Gbytes is being used.

The executable is quite big

text data bss dec hex filename
57369168 417156 20903108 78689432 4b0b498 [snip]

The run under DHAT is using about 2Gbytes virtual and 1.5Gbytes resident according to htop. Running standalone those are about 750M and 350M respectively.

I'll have to find a much smaller test before I consider doing any profiling of DHAT. I'll also try nullgrind to see if it really is the DHAT instrumentation overhead that is slowing things down.

A+
Paul

Re: [Valgrind-users] DHAT speed

[Julian S. <js...@ac...>]

That doesn't sound right.  I use DHAT extensively and expect a slowdown of
perhaps 50:1, maybe less.  What you're describing is a slowdown factor of
at least several thousand.

Bear in mind though that (1) V sequentialises thread execution, which wil
make a big difference if the program is heavily multithreaded, and (2)
I suspect dhat's scheme of looking up all memory accesses in an AVL tree
(of malloc'd blocks) doesn't scale all that well if you have tens of
millions of blocks.

Can you run it on a smaller workload?

J

On 26/05/2020 09:21, Paul FLOYD wrote:
> Hi
>   
> I'm running DHAT on what I consider to be a relatively small example. Standalone the executable runs in a bit under 10 minutes. Based on the CPU time that we print after every 10% of progress, under DHAT the same executable is going to take about 422 hours - about two and a half weeks.
>   
> Does anyone have any ideas what could be causing it to be so slow? Indeed, is this the sort of slowdown that I should be expecting with DHAT? The executable is intensive in both memory and floating point. Probably not helping matters, the data structures that I want to look at are over 1kB in size so I tweaked the HISTOGRAM_SIZE_LIMIT to bump it up to 2kb.
>   
> On the DHAT side, I have thought of trying to use some macro hackery to try to inline the avl comparator function calls. Otherwise I don't have much in the way of other ideas, and DHAT doesn't have any cli options to tweak things.
>   
> A+
> Paul
>   
> 
> 
> 
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users
> 

Re: [Valgrind-users] Valgrind-3.16.0.RC2 is available for testing

[Tom H. <to...@co...>]

That's correct as AVX512 is not currently supported in valgrind so
you will need a version that doesn't use that for valgrind use.

Progress on adding AVX512 support is being tracked here:

https://bugs.kde.org/show_bug.cgi?id=383010

Tom

On 26/05/2020 10:07, Patrick Bégou wrote:
> Thanks all for these precisions.
> I have deployed OpenMPI myself. So I have to build it again, disabling
> AVX512 optimizations.  This level of optimization is used in all our CFD
> codes and libraries as it improve the global code performances.
> 
> Patrick
> 
> Le 26/05/2020 à 10:45, Tom Hughes a écrit :
>> Sorry, I misunderstood what you meant by "as previous version" there.
>>
>> I thought you meant the previous version worked but you actually
>> meant that it failed.
>>
>> As Julian says there is no easy fix - you have a library installed
>> that has been compiled to assume certain instructions are available
>> that are not in fact available under valgrind at the moment.
>>
>> Tom
>>
>> On 26/05/2020 09:26, Patrick Bégou wrote:
>>> Hi Tom,
>>>
>>> I'm a new user of Valgrind. I was needing it to check a large mpi code.
>>> So I downloaded 3.15 version but even if hardware and software are 2 to
>>> 3 years old, valgrind does'nt work for me.
>>> Nor gcc7, nor OpenMPI, nor my application (even the small test) used
>>> specific option when they were built.
>>>
>>> If this unsupported instruction (I do not know what is an EVEX prefix,
>>> sorry) is the problem, how can I avoid it to use valgrind ?
>>>
>>> I was just thinking that 3.16 could solve my problem....
>>>
>>> Patrick
>>>
>>>
>>>
>>> Le 26/05/2020 à 10:12, Tom Hughes a écrit :
>>>> On 26/05/2020 09:06, Patrick Bégou wrote:
>>>>
>>>>> valgrind-3.16.0.RC2 doesn't work for me (as previous version on this
>>>>> server).
>>>>
>>>> Are you saying that it fails on a binary that worked before?
>>>>
>>>>> vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F
>>>>> 0x5 0x25 0xA8 0x18 0x0
>>>>> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
>>>>> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
>>>>> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
>>>>> ==306850== valgrind: Unrecognised instruction at address 0x6ddf581.
>>>>> vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F
>>>>> 0x5 0x25 0xA8 0x18 0x0
>>>>> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
>>>>> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
>>>>> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
>>>>
>>>> Because this is an instruction with an EVEX prefix that is not
>>>> supported by any version of valgrind ever so I don't see how this
>>>> binary can have worked with the previous version of valgrind.
>>>>
>>>> I suspect that you have in fact recompiled the program with
>>>> a different compiler or different optimization settings since
>>>> the time when it worked?
>>>>
>>>> Tom
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Valgrind-users mailing list
>>> Val...@li...
>>> https://lists.sourceforge.net/lists/listinfo/valgrind-users
>>>
>>
>>
> 
> 
> 
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users
> 


-- 
Tom Hughes (to...@co...)
http://compton.nu/

Re: [Valgrind-users] DHAT speed

[John R. <jr...@bi...>]

On 5/26/20, Paul FLOYD wrote:

> I'm running DHAT on what I consider to be a relatively small example. Standalone the executable runs in a bit under 10 minutes. Based on the CPU time that we print after every 10% of progress, under DHAT the same executable is going to take about 422 hours - about two and a half weeks.
> 
> Does anyone have any ideas what could be causing it to be so slow? 

Please tell us about your environment.
(EVERY query should supply this information!)

Which version of DHAT?  Which hardware?  Which operating system and version?
Which compiler and version built DHAT?  Which compiler(s) built the program under test?

How big is the program?  On a Linux system, run "size ./my_app; ldd ./my_app"
for static size, and "ps alx | grep my_app" for total VIRTual and RESident size.
How much RAM is available for use?  How much demand paging is occurring?

There are standard performance tools that apply to EVERY executable program,
not just DHAT.  On a Linux system with a DHAT that was built with symbol
information retained ("-g" command-line option to gcc or clang, and
no symbol stripping during static binding by /usr/bin/ld or /usr/bin/strip),
what does 'perf' say?
     perf record valgrind --tool=DHAT <<valgrind_args>> ./my_app <<my_app_args>>
     perf report  > perf_output.txt

A smart user would supply such information in the first post.

Re: [Valgrind-users] Valgrind-3.16.0.RC2 is available for testing

[Patrick B. <Pat...@le...>]

Thanks all for these precisions.
I have deployed OpenMPI myself. So I have to build it again, disabling
AVX512 optimizations.  This level of optimization is used in all our CFD
codes and libraries as it improve the global code performances.

Patrick

Le 26/05/2020 à 10:45, Tom Hughes a écrit :
> Sorry, I misunderstood what you meant by "as previous version" there.
>
> I thought you meant the previous version worked but you actually
> meant that it failed.
>
> As Julian says there is no easy fix - you have a library installed
> that has been compiled to assume certain instructions are available
> that are not in fact available under valgrind at the moment.
>
> Tom
>
> On 26/05/2020 09:26, Patrick Bégou wrote:
>> Hi Tom,
>>
>> I'm a new user of Valgrind. I was needing it to check a large mpi code.
>> So I downloaded 3.15 version but even if hardware and software are 2 to
>> 3 years old, valgrind does'nt work for me.
>> Nor gcc7, nor OpenMPI, nor my application (even the small test) used
>> specific option when they were built.
>>
>> If this unsupported instruction (I do not know what is an EVEX prefix,
>> sorry) is the problem, how can I avoid it to use valgrind ?
>>
>> I was just thinking that 3.16 could solve my problem....
>>
>> Patrick
>>
>>
>>
>> Le 26/05/2020 à 10:12, Tom Hughes a écrit :
>>> On 26/05/2020 09:06, Patrick Bégou wrote:
>>>
>>>> valgrind-3.16.0.RC2 doesn't work for me (as previous version on this
>>>> server).
>>>
>>> Are you saying that it fails on a binary that worked before?
>>>
>>>> vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F
>>>> 0x5 0x25 0xA8 0x18 0x0
>>>> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
>>>> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
>>>> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
>>>> ==306850== valgrind: Unrecognised instruction at address 0x6ddf581.
>>>> vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F
>>>> 0x5 0x25 0xA8 0x18 0x0
>>>> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
>>>> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
>>>> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
>>>
>>> Because this is an instruction with an EVEX prefix that is not
>>> supported by any version of valgrind ever so I don't see how this
>>> binary can have worked with the previous version of valgrind.
>>>
>>> I suspect that you have in fact recompiled the program with
>>> a different compiler or different optimization settings since
>>> the time when it worked?
>>>
>>> Tom
>>>
>>
>>
>>
>> _______________________________________________
>> Valgrind-users mailing list
>> Val...@li...
>> https://lists.sourceforge.net/lists/listinfo/valgrind-users
>>
>
>

Re: [Valgrind-users] Valgrind-3.16.0.RC2 is available for testing

[Tom H. <to...@co...>]

Sorry, I misunderstood what you meant by "as previous version" there.

I thought you meant the previous version worked but you actually
meant that it failed.

As Julian says there is no easy fix - you have a library installed
that has been compiled to assume certain instructions are available
that are not in fact available under valgrind at the moment.

Tom

On 26/05/2020 09:26, Patrick Bégou wrote:
> Hi Tom,
> 
> I'm a new user of Valgrind. I was needing it to check a large mpi code.
> So I downloaded 3.15 version but even if hardware and software are 2 to
> 3 years old, valgrind does'nt work for me.
> Nor gcc7, nor OpenMPI, nor my application (even the small test) used
> specific option when they were built.
> 
> If this unsupported instruction (I do not know what is an EVEX prefix,
> sorry) is the problem, how can I avoid it to use valgrind ?
> 
> I was just thinking that 3.16 could solve my problem....
> 
> Patrick
> 
> 
> 
> Le 26/05/2020 à 10:12, Tom Hughes a écrit :
>> On 26/05/2020 09:06, Patrick Bégou wrote:
>>
>>> valgrind-3.16.0.RC2 doesn't work for me (as previous version on this
>>> server).
>>
>> Are you saying that it fails on a binary that worked before?
>>
>>> vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F
>>> 0x5 0x25 0xA8 0x18 0x0
>>> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
>>> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
>>> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
>>> ==306850== valgrind: Unrecognised instruction at address 0x6ddf581.
>>> vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F
>>> 0x5 0x25 0xA8 0x18 0x0
>>> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
>>> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
>>> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
>>
>> Because this is an instruction with an EVEX prefix that is not
>> supported by any version of valgrind ever so I don't see how this
>> binary can have worked with the previous version of valgrind.
>>
>> I suspect that you have in fact recompiled the program with
>> a different compiler or different optimization settings since
>> the time when it worked?
>>
>> Tom
>>
> 
> 
> 
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users
> 


-- 
Tom Hughes (to...@co...)
http://compton.nu/

Re: [Valgrind-users] Valgrind-3.16.0.RC2 is available for testing

[Julian S. <js...@ac...>]

You can't easily avoid this problem, because it occurs in a system library,
not in your own code:

==306851== valgrind: Unrecognised instruction at address 0x6ddf581.
==306851==    at 0x6DDF581: opal_pointer_array_construct (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)

Possibly your least-worst option is to talk with the people who built/
installed OpenMPI on the machine, to see if you can get a build that
doesn't use AVX512 instructions.

J

Re: [Valgrind-users] Valgrind-3.16.0.RC2 is available for testing

[Patrick B. <Pat...@le...>]

Hi Tom,

I'm a new user of Valgrind. I was needing it to check a large mpi code.
So I downloaded 3.15 version but even if hardware and software are 2 to
3 years old, valgrind does'nt work for me.
Nor gcc7, nor OpenMPI, nor my application (even the small test) used
specific option when they were built.

If this unsupported instruction (I do not know what is an EVEX prefix,
sorry) is the problem, how can I avoid it to use valgrind ?

I was just thinking that 3.16 could solve my problem....

Patrick



Le 26/05/2020 à 10:12, Tom Hughes a écrit :
> On 26/05/2020 09:06, Patrick Bégou wrote:
>
>> valgrind-3.16.0.RC2 doesn't work for me (as previous version on this
>> server).
>
> Are you saying that it fails on a binary that worked before?
>
>> vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F
>> 0x5 0x25 0xA8 0x18 0x0
>> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
>> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
>> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
>> ==306850== valgrind: Unrecognised instruction at address 0x6ddf581.
>> vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F
>> 0x5 0x25 0xA8 0x18 0x0
>> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
>> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
>> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
>
> Because this is an instruction with an EVEX prefix that is not
> supported by any version of valgrind ever so I don't see how this
> binary can have worked with the previous version of valgrind.
>
> I suspect that you have in fact recompiled the program with
> a different compiler or different optimization settings since
> the time when it worked?
>
> Tom
>

Re: [Valgrind-users] Valgrind-3.16.0.RC2 is available for testing

[Tom H. <to...@co...>]

On 26/05/2020 09:06, Patrick Bégou wrote:

> valgrind-3.16.0.RC2 doesn't work for me (as previous version on this 
> server).

Are you saying that it fails on a binary that worked before?

> vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F 0x5 
> 0x25 0xA8 0x18 0x0
> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
> ==306850== valgrind: Unrecognised instruction at address 0x6ddf581.
> vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F 0x5 
> 0x25 0xA8 0x18 0x0
> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0

Because this is an instruction with an EVEX prefix that is not
supported by any version of valgrind ever so I don't see how this
binary can have worked with the previous version of valgrind.

I suspect that you have in fact recompiled the program with
a different compiler or different optimization settings since
the time when it worked?

Tom

-- 
Tom Hughes (to...@co...)
http://compton.nu/

Re: [Valgrind-users] Valgrind-3.16.0.RC2 is available for testing

[Patrick B. <Pat...@le...>]

Le 19/05/2020 à 14:32, Julian Seward a écrit :
>
> Greetings.
>
> A first release candidate for 3.16.0 is available at
> https://sourceware.org/pub/valgrind/valgrind-3.16.0.RC2.tar.bz2
> (md5 = 21ac87434ed32bcfe5ea86a0978440ba)
>
> Please give it a try on platforms that are important for you.  If no
> serious
> issues are reported, the 3.16.0 final release will happen on 25 May,
> that is,
> next Monday.
>
> J

Hi all,

valgrind-3.16.0.RC2 doesn't work for me (as previous version on this
server). 

_*My fortran test program (error prune I think) is as simple as:*_

    PROGRAM reduce
    USE mpi
    IMPLICIT NONE

    INTEGER :: me,  ncpus, ierr
    REAL :: buff,  resu=0

    CALL MPI_INIT(ierr)
    CALL MPI_COMM_RANK(MPI_COMM_WORLD,me,ierr)
    CALL MPI_COMM_SIZE(MPI_COMM_WORLD,ncpus,ierr)
    buff=1

    CALL MPI_ALLREDUCE(buff,resu,1,MPI_REAL,MPI_SUM,MPI_COMM_WORLD,ierr)
    if (me == 0 ) WRITE(6,'(a,i0,2(a,f14.6))') 'On ',me,' I have
    ',buff,' and got ',resu

    CALL MPI_FINALIZE(ierr)

    END PROGRAM reduce

_*Compilation with:*_

mpifort reduce.F90 -o reduce
mpifort --show
/opt/GCC73/bin/gfortran -I/opt/openmpi-GCC73/v3.1.x-20181010/include
-pthread -I/opt/openmpi-GCC73/v3.1.x-20181010/lib -Wl,-rpath
-Wl,/opt/openmpi-GCC73/v3.1.x-20181010/lib -Wl,--enable-new-dtags
-L/opt/openmpi-GCC73/v3.1.x-20181010/lib -lmpi_usempif08
-lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi

_*OS is *_CentOS Linux release 7.7.1908 (Core)

_*Valgrind compiled with gcc7.3, configure options are:*_

    ./configure --enable-only64bit --with-mpicc=$(which mpicc)
    --prefix=/robin/data/begou/VALGRIND/valgrind-binaries

*Hardware is:*

    Dell Poweredge R940

    4 x Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz (laubnched Q3 2017)

    1.5 TB of RAM

_*Compiler*_ is gcc (GCC) 7.3.0 (january 2018)

_*OpenMPI *_is

    v3.1. from the git repo 2018/10/10 (because a patch was needed at
    this time) compiled with gcc7.3.0

    *configure options are:*
    --prefix=/opt/openmpi-GCC73/v3.1.x-20181010'
    '--enable-mpirun-prefix-by-default' '--disable-dlopen'
    '--enable-mca-no-build=openib' '-
    -without-verbs' '--enable-mpi-cxx' '--without-slurm'
    '--enable-mpi-thread-multiple

_*Error is:*_

[begou@grivola TESTS]$valgrind --version
valgrind-3.16.0.RC2

[begou@grivola TESTS]$mpirun -np 2 valgrind ./reduce
==306850== Memcheck, a memory error detector
==306850== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==306850== Using Valgrind-3.16.0.RC2 and LibVEX; rerun with -h for
copyright info
==306850== Command: ./reduce
==306850==
==306851== Memcheck, a memory error detector
==306851== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==306851== Using Valgrind-3.16.0.RC2 and LibVEX; rerun with -h for
copyright info
==306851== Command: ./reduce
==306851==
vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F 0x5
0x25 0xA8 0x18 0x0
vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
==306850== valgrind: Unrecognised instruction at address 0x6ddf581.
vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0xFD 0x8 0x6F 0x5
0x25 0xA8 0x18 0x0
vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
==306851== valgrind: Unrecognised instruction at address 0x6ddf581.
==306851==    at 0x6DDF581: opal_pointer_array_construct (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306851==    by 0x6E01A78: mca_base_var_init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306851==    by 0x6DE3E39: opal_init_util (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306851==    by 0x552ED60: ompi_mpi_init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi.so.40.10.3)
==306851==    by 0x555F9ED: PMPI_Init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi.so.40.10.3)
==306851==    by 0x52ABBB7: PMPI_INIT (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi_mpifh.so.40.11.2)
==306851==    by 0x400CDD: MAIN__ (in /HA/sources/begou/TESTS/reduce)
==306851==    by 0x400E8C: main (in /HA/sources/begou/TESTS/reduce)
==306851== Your program just tried to execute an instruction that Valgrind
==306851== did not recognise.  There are two possible reasons for this.
==306851== 1. Your program has a bug and erroneously jumped to a non-code
==306851==    location.  If you are running Memcheck and you just saw a
==306851==    warning about a bad jump, it's probably your program's fault.
==306851== 2. The instruction is legitimate but Valgrind doesn't handle it,
==306851==    i.e. it's Valgrind's fault.  If you think this is the case or
==306851==    you are not sure, please let us know and we'll try to fix it.
==306851== Either way, Valgrind will now raise a SIGILL signal which will
==306851== probably kill your program.
==306850==    at 0x6DDF581: opal_pointer_array_construct (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306850==    by 0x6E01A78: mca_base_var_init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306850==    by 0x6DE3E39: opal_init_util (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306850==    by 0x552ED60: ompi_mpi_init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi.so.40.10.3)
==306850==    by 0x555F9ED: PMPI_Init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi.so.40.10.3)
==306850==    by 0x52ABBB7: PMPI_INIT (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi_mpifh.so.40.11.2)
==306850==    by 0x400CDD: MAIN__ (in /HA/sources/begou/TESTS/reduce)
==306850==    by 0x400E8C: main (in /HA/sources/begou/TESTS/reduce)
==306850== Your program just tried to execute an instruction that Valgrind
==306850== did not recognise.  There are two possible reasons for this.
==306850== 1. Your program has a bug and erroneously jumped to a non-code
==306850==    location.  If you are running Memcheck and you just saw a
==306850==    warning about a bad jump, it's probably your program's fault.
==306850== 2. The instruction is legitimate but Valgrind doesn't handle it,
==306850==    i.e. it's Valgrind's fault.  If you think this is the case or
==306850==    you are not sure, please let us know and we'll try to fix it.
==306850== Either way, Valgrind will now raise a SIGILL signal which will
==306850== probably kill your program.

Program received signal SIGILL: Illegal instruction.

Program received signal SIGILL: Illegal instruction.

Backtrace for this error:

Backtrace for this error:
#0  0x66ce3af in ???
#0  0x66ce3af in ???
#1  0x6ddf581 in ???
#2  0x6e01a78 in ???
#3  0x6de3e39 in ???
#1  0x6ddf581 in ???
#4  0x552ed60 in ???
#5  0x555f9ed in ???
#6  0x52abbb7 in ???
#2  0x6e01a78 in ???
#7  0x400cdd in ???
#8  0x400e8c in ???
#9  0x66ba504 in ???
#10  0x400c18 in ???
#3  0x6de3e39 in ???
#11  0xffffffffffffffff in ???
#4  0x552ed60 in ???
#5  0x555f9ed in ???
#6  0x52abbb7 in ???
#7  0x400cdd in ???
#8  0x400e8c in ???
==306851==
==306851== Process terminating with default action of signal 4 (SIGILL)
#9  0x66ba504 in ???
==306851==    at 0x648B4BB: raise (in /usr/lib64/libpthread-2.17.so)
==306851==    by 0x66CE3AF: ??? (in /usr/lib64/libc-2.17.so)
==306851==    by 0x6DDF580: opal_pointer_array_construct (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306851==    by 0x6E01A78: mca_base_var_init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306851==    by 0x6DE3E39: opal_init_util (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306851==    by 0x552ED60: ompi_mpi_init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi.so.40.10.3)
==306851==    by 0x555F9ED: PMPI_Init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi.so.40.10.3)
==306851==    by 0x52ABBB7: PMPI_INIT (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi_mpifh.so.40.11.2)
==306851==    by 0x400CDD: MAIN__ (in /HA/sources/begou/TESTS/reduce)
==306851==    by 0x400E8C: main (in /HA/sources/begou/TESTS/reduce)
#10  0x400c18 in ???
#11  0xffffffffffffffff in ???
==306850==
==306850== Process terminating with default action of signal 4 (SIGILL)
==306850==    at 0x648B4BB: raise (in /usr/lib64/libpthread-2.17.so)
==306850==    by 0x66CE3AF: ??? (in /usr/lib64/libc-2.17.so)
==306850==    by 0x6DDF580: opal_pointer_array_construct (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306850==    by 0x6E01A78: mca_base_var_init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306850==    by 0x6DE3E39: opal_init_util (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libopen-pal.so.40.10.3)
==306850==    by 0x552ED60: ompi_mpi_init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi.so.40.10.3)
==306850==    by 0x555F9ED: PMPI_Init (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi.so.40.10.3)
==306850==    by 0x52ABBB7: PMPI_INIT (in
/opt/openmpi-GCC73/v3.1.x-20181010/lib/libmpi_mpifh.so.40.11.2)
==306850==    by 0x400CDD: MAIN__ (in /HA/sources/begou/TESTS/reduce)
==306850==    by 0x400E8C: main (in /HA/sources/begou/TESTS/reduce)
==306851==
==306851== HEAP SUMMARY:
==306851==     in use at exit: 8,830 bytes in 65 blocks
==306851==   total heap usage: 123 allocs, 58 frees, 90,778 bytes allocated
==306851==
==306850==
==306850== HEAP SUMMARY:
==306850==     in use at exit: 8,830 bytes in 65 blocks
==306850==   total heap usage: 123 allocs, 58 frees, 90,778 bytes allocated
==306850==
==306851== LEAK SUMMARY:
==306851==    definitely lost: 0 bytes in 0 blocks
==306851==    indirectly lost: 0 bytes in 0 blocks
==306851==      possibly lost: 0 bytes in 0 blocks
==306851==    still reachable: 8,830 bytes in 65 blocks
==306851==         suppressed: 0 bytes in 0 blocks
==306851== Rerun with --leak-check=full to see details of leaked memory
==306851==
==306851== For lists of detected and suppressed errors, rerun with: -s
==306851== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==306850== LEAK SUMMARY:
==306850==    definitely lost: 0 bytes in 0 blocks
==306850==    indirectly lost: 0 bytes in 0 blocks
==306850==      possibly lost: 0 bytes in 0 blocks
==306850==    still reachable: 8,830 bytes in 65 blocks
==306850==         suppressed: 0 bytes in 0 blocks
==306850== Rerun with --leak-check=full to see details of leaked memory
==306850==
==306850== For lists of detected and suppressed errors, rerun with: -s
==306850== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node grivola exited on
signal 4 (Illegal instruction).


Thanks

Patrick

[Valgrind-users] DHAT speed

[Paul F. <pj...@wa...>]

Hi
 
I'm running DHAT on what I consider to be a relatively small example. Standalone the executable runs in a bit under 10 minutes. Based on the CPU time that we print after every 10% of progress, under DHAT the same executable is going to take about 422 hours - about two and a half weeks.
 
Does anyone have any ideas what could be causing it to be so slow? Indeed, is this the sort of slowdown that I should be expecting with DHAT? The executable is intensive in both memory and floating point. Probably not helping matters, the data structures that I want to look at are over 1kB in size so I tweaked the HISTOGRAM_SIZE_LIMIT to bump it up to 2kb.
 
On the DHAT side, I have thought of trying to use some macro hackery to try to inline the avl comparator function calls. Otherwise I don't have much in the way of other ideas, and DHAT doesn't have any cli options to tweak things.
 
A+
Paul
 

Re: [Valgrind-users] How to disable leak check from inside a program

[John R. <jr...@bi...>]

> What would help a lot is to have a VALGRIND request, like
> VALGRIND_DO_CLIENT_REQUEST_STMT, that we could use in our signal
> handler to turn off leak checking.

In valgrind-3.16 (to be released in 3 days; RC2 available today) see
     valgrind --help-dyn-options

Re: [Valgrind-users] How to disable leak check from inside a program

[Philippe W. <phi...@sk...>]

On Fri, 2020-05-22 at 15:22 +0300, Michael Widenius wrote:
> Hi!
> I have searched documentation, internet and header files like
> memcheck.h, but not found a solution:
> 
> When running the MariaDB test suite under valgrind, we sometimes may
> get a core dump.  In this case, the leaked memory report can be very
> long and will be totally useless.
> 
> What would help a lot is to have a VALGRIND request, like
> VALGRIND_DO_CLIENT_REQUEST_STMT, that we could use in our signal
> handler to turn off leak checking.
> 
> Is that possible and if not, is that something that could get
> implemented in the future?
> Is this something that anyone else has ever requested ?
> 
> Regards,
> Monty

Hello,

The next version of valgrind is almost ready (Release Candidate was produced
a few days ago).

This release contains a feature to dynamically change many options.

You can obtain the list of dynamically changeable options doing:

 valgrind --help-dyn-options

For memcheck, this gives the below help.
Based on this, you should be able to obtain what you need.

Hope this helps

Philippe


valgrind --help-dyn-options
Some command line settings are "dynamic", meaning they can be changed
while Valgrind is running, like this:
    From the shell, using vgdb. Example:
      $ vgdb "v.clo --trace-children=yes --child-silent-after-fork=no"
    From a gdb attached to the valgrind gdbserver. Example:
      (gdb) monitor v.clo --trace-children=yes --child-silent-after-fork=no"
    From your program, using a client request. Example:
      #include <valgrind/valgrind.h>
      VALGRIND_CLO_CHANGE("--trace-children=yes");
      VALGRIND_CLO_CHANGE("--child-silent-after-fork=no");

  dynamically changeable options:
    -v --verbose -q --quiet -d --stats --vgdb=no --vgdb=yes --vgdb=full
    --vgdb-poll --vgdb-error --vgdb-stop-at --error-markers --show-error-list -s
    --show-below-main --time-stamp --trace-children --child-silent-after-fork
    --trace-sched --trace-signals --trace-symtab --trace-cfi --debug-dump=syms
    --debug-dump=line --debug-dump=frames --trace-redir --trace-syscalls
    --sym-offsets --progress-interval --merge-recursive-frames
    --vex-iropt-verbosity --suppressions --trace-flags --trace-notbelow
    --trace-notabove --profile-flags --gen-suppressions=no
    --gen-suppressions=yes --gen-suppressions=all --errors-for-leak-kinds
    --show-leak-kinds --leak-check-heuristics --show-reachable
    --show-possibly-lost --freelist-vol --freelist-big-blocks --leak-check=no
    --leak-check=summary --leak-check=yes --leak-check=full --ignore-ranges
    --ignore-range-below-sp --show-mismatched-frees
valgrind: Use --help for more information.

Re: [Valgrind-users] How to disable leak check from inside a program

[Derrick M. <der...@gm...>]

I am not too familiar with memcheck, but there are client requests to
enable/disable checks within a memory range using
VG_USERREQ__ENABLE_ADDR_ERROR_REPORTING_IN_RANGE and
VG_USERREQ__DISABLE_ADDR_ERROR_REPORTING_IN_RANGE.


On Fri, May 22, 2020 at 8:24 AM Michael Widenius
<mic...@gm...> wrote:
>
> Hi!
> I have searched documentation, internet and header files like
> memcheck.h, but not found a solution:
>
> When running the MariaDB test suite under valgrind, we sometimes may
> get a core dump.  In this case, the leaked memory report can be very
> long and will be totally useless.
>
> What would help a lot is to have a VALGRIND request, like
> VALGRIND_DO_CLIENT_REQUEST_STMT, that we could use in our signal
> handler to turn off leak checking.
>
> Is that possible and if not, is that something that could get
> implemented in the future?
> Is this something that anyone else has ever requested ?
>
> Regards,
> Monty
>
>
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users



--
Derrick McKee
Phone: (703) 957-9362
Email: der...@gm...