valgrind-users — General discussion list for Valgrind users

[Valgrind-users] Valgrind-3.21.0 is available

[Mark W. <ma...@kl...>]

We are pleased to announce a new release of Valgrind, version 3.21.0,
available from https://valgrind.org/downloads/current.html.

See the release notes below for details of changes.

Our thanks to all those who contribute to Valgrind's development.  This
release represents a great deal of time, energy and effort on the part
of many people.

Happy and productive debugging and profiling,

-- The Valgrind Developers

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This release supports X86/Linux, AMD64/Linux, ARM32/Linux, ARM64/Linux,
PPC32/Linux, PPC64BE/Linux, PPC64LE/Linux, S390X/Linux, MIPS32/Linux,
MIPS64/Linux, ARM/Android, ARM64/Android, MIPS32/Android, X86/Android,
X86/Solaris, AMD64/Solaris, AMD64/MacOSX 10.12, X86/FreeBSD and
AMD64/FreeBSD.  There is also preliminary support for X86/macOS 10.13,
AMD64/macOS 10.13 and nanoMIPS/Linux.

* ==================== CORE CHANGES ===================

* When GDB is used to debug a program running under valgrind using
  the valgrind gdbserver, GDB will automatically load some
  python code provided in valgrind defining GDB front end commands
  corresponding to the valgrind monitor commands.
  These GDB front end commands accept the same format as
  the monitor commands directly sent to the Valgrind gdbserver.
  These GDB front end commands provide a better integration
  in the GDB command line interface, so as to use for example
  GDB auto-completion, command specific help, searching for
  a command or command help matching a regexp, ...
  For relevant monitor commands, GDB will evaluate arguments
  to make the use of monitor commands easier.
  For example, instead of having to print the address of a variable
  to pass it to a subsequent monitor command, the GDB front end
  command will evaluate the address argument. It is for example
  possible to do:
    (gdb) memcheck who_points_at &some_struct sizeof(some_struct)
  instead of:
    (gdb) p &some_struct
    $2 = (some_struct_type *) 0x1130a0 <some_struct>
    (gdb) p sizeof(some_struct)
    $3 = 40
    (gdb) monitor who_point_at 0x1130a0 40

* The vgdb utility now supports extended-remote protocol when
  invoked with --multi. In this mode the GDB run command is
  supported. Which means you don't need to run gdb and valgrind
  from different terminals. So for example to start your program
  in gdb and run it under valgrind you can do:
  $ gdb prog
  (gdb) set remote exec-file prog
  (gdb) set sysroot /
  (gdb) target extended-remote | vgdb --multi
  (gdb) start

* The behaviour of realloc with a size of zero can now
  be changed for tools that intercept malloc. Those
  tools are memcheck, helgrind, drd, massif and dhat.
  Realloc implementations generally do one of two things
     - free the memory like free() and return NULL
       (GNU libc and ptmalloc).
     - either free the memory and then allocate a
       minimum sized block or just return the
       original pointer. Return NULL if the
       allocation of the minimum sized block fails
       (jemalloc, musl, snmalloc, Solaris, macOS).
  When Valgrind is configured and built it will
  try to match the OS and libc behaviour. However
  if you are using a non-default library to replace
  malloc and family (e.g., musl on a glibc Linux or
  tcmalloc on FreeBSD) then you can use a command line
  option to change the behaviour of Valgrind:
    --realloc-zero-bytes-frees=yes|no [yes on Linux glibc, no otherwise]

* ================== PLATFORM CHANGES =================

* Make the address space limit on FreeBSD amd64 128Gbytes
  (the same as Linux and Solaris, it was 32Gbytes)

* ==================== TOOL CHANGES ===================

* Memcheck:
  - When doing a delta leak_search, it is now possible to only
    output the new loss records compared to the previous leak search.
    This is available in the memcheck monitor command 'leak_search'
    by specifying the "new" keyword or in your program by using
    the client request VALGRIND_DO_NEW_LEAK_CHECK.
    Whenever a "delta" leak search is done (i.e. when specifying
    "new" or "increased" or "changed" in the monitor command),
    the new loss records have a "new" marker.
  - Valgrind now contains python code that defines GDB memcheck
    front end monitor commands. See CORE CHANGES.
  - Performs checks for the use of realloc with a size of zero.
    This is non-portable and a source of errors. If memcheck
    detects such a usage it will generate an error
      realloc() with size 0
    followed by the usual callstacks.
    A switch has been added to allow this to be turned off:
      --show-realloc-size-zero=yes|no [yes]

* Helgrind:
  - The option ---history-backtrace-size=<number> allows to configure
    the number of entries to record in the stack traces of "old"
    accesses. Previously, this number was hardcoded to 8.
  - Valgrind now contains python code that defines GDB helgrind
    front end monitor commands. See CORE CHANGES.

* Cachegrind:
  - `--cache-sim=no` is now the default. The cache simulation is old and
    unlikely to match any real modern machine. This means only the `Ir`
    event are gathered by default, but that is by far the most useful
    event.
  - `cg_annotate`, `cg_diff`, and `cg_merge` have been rewritten in
    Python. As a result, they all have more flexible command line
    argument handling, e.g. supporting `--show-percs` and
    `--no-show-percs` forms as well as the existing `--show-percs=yes`
    and `--show-percs=no`.
  - `cg_annotate` has some functional changes.
    - It's much faster, e.g. 3-4x on common cases.
    - It now supports diffing (with `--diff`, `--mod-filename`, and
      `--mod-funcname`) and merging (by passing multiple data files).
    - It now provides more information at the file and function level.
      There are now "File:function" and "Function:file" sections. These
      are very useful for programs that use inlining a lot.
    - Support for user-annotated files and the `-I`/`--include` option
      has been removed, because it was of little use and blocked other
      improvements.
    - The `--auto` option is renamed `--annotate`, though the old
      `--auto=yes`/`--auto=no` forms are still supported.
  - `cg_diff` and `cg_merge` are now deprecated, because `cg_annotate`
    now does a better job of diffing and merging.
  - The Cachegrind output file format has changed very slightly, but in
    ways nobody is likely to notice.

* Callgrind:
  - Valgrind now contains python code that defines GDB callgrind
    front end monitor commands. See CORE CHANGES.

* Massif:
  - Valgrind now contains python code that defines GDB massif
    front end monitor commands. See CORE CHANGES.

* DHAT:
  - A new kind of user request has been added which allows you to
    override the 1024 byte limit on access count histograms for blocks
    of memory. The client request is DHAT_HISTOGRAM_MEMORY.

* ==================== FIXED BUGS ====================

The following bugs have been fixed or resolved.  Note that "n-i-bz"
stands for "not in bugzilla" -- that is, a bug that was reported to us
but never got a bugzilla entry.  We encourage you to file bugs in
bugzilla (https://bugs.kde.org/enter_bug.cgi?product=valgrind) rather
than mailing the developers (or mailing lists) directly -- bugs that
are not entered into bugzilla tend to get forgotten about or ignored.

170510  Don't warn about ioctl of size 0 without direction hint
241072  List tools in --help output
327548  false positive while destroying mutex
382034  Testcases build fixes for musl
351857  confusing error message about valid command line option
374596  inconsistent RDTSCP support on x86_64
392331  Spurious lock not held error from inside pthread_cond_timedwait
397083  Likely false positive "uninitialised value(s)" for __wmemchr_avx2 and __wmemcmp_avx2_movbe
400793  pthread_rwlock_timedwrlock false positive
419054  Unhandled syscall getcpu on arm32
433873  openat2 syscall unimplemented on Linux
434057  Add stdio mode to valgrind's gdbserver
435441  valgrind fails to interpose malloc on musl 1.2.2 due to weak symbol name and no libc soname
436413  Warn about realloc of size zero
439685  compiler warning in callgrind/main.c
444110  priv/guest_ppc_toIR.c:36198:31: warning: duplicated 'if' condition.
444487  hginfo test detects an extra lock inside data symbol "_rtld_local"
444488  Use glibc.pthread.stack_cache_size tunable
444568  drd/tests/pth_barrier_thr_cr fails on Fedora 38
445743  "The impossible happened: mutex is locked simultaneously by two threads"
        while using mutexes with priority inheritance and signals
449309  Missing loopback device ioctl(s) 
459476  vgdb: allow address reuse to avoid "address already in use" errorsuse" errors
460356  s390: Sqrt32Fx4 -- cannot reduce tree
462830  WARNING: unhandled amd64-freebsd syscall: 474
463027  broken check for MPX instruction support in assembler
464103  Enhancement: add a client request to DHAT to mark memory to be histogrammed
464476  Firefox fails to start under Valgrind
464609  Valgrind memcheck should support Linux pidfd_open
464680  Show issues caused by memory policies like selinux deny_execmem
464859  Build failures with GCC-13 (drd tsan_unittest)
464969  D language demangling
465435  m_libcfile.c:66 (vgPlain_safe_fd): Assertion 'newfd >= VG_(fd_hard_limit)' failed.
466104  aligned_alloc problems, part 1
467036  Add time cost statistics for Regtest
467482  Build failure on aarch64 Alpine
467714  fdleak_* and rlimit tests fail when parent process has more than
        64 descriptors opened
467839  Gdbserver: Improve compatibility of library directory name
468401  [PATCH] Add a style file for clang-format
468556  Build failure for vgdb
468606  build: remove "Valgrind relies on GCC" check/output
469097  ppc64(be) doesn't support SCV syscall instruction
n-i-bz  FreeBSD rfork syscall fail with EINVAL or ENOSYS rather than VG_(unimplemented)

To see details of a given bug, visit
  https://bugs.kde.org/show_bug.cgi?id=XXXXXX
where XXXXXX is the bug number as listed above.

* ==================== KNOWN ISSUES ===================

* configure --enable-lto=yes is know to not work in all setups.
  See bug 469049. Workaround: Build without LTO.

Re: [Valgrind-users] tool=cachegrind always reports LL misses, even if the cache is bigger than the used memory

[John R. <jr...@bi...>]

> I would think that, regardless of cache size, the *first* access to a line
> causes a miss.

The PowerPC and relatives have an instruction which force-allocates
(if necessary) AND zeroes an entire cache line, so in some ways the
"first" access is a Write which succeeds with no miss.

Re: [Valgrind-users] tool=cachegrind always reports LL misses, even if the cache is bigger than the used memory

[Eliot M. <mo...@cs...>]

On 4/25/2023 3:56 PM, Volker Dirr wrote:
> Hallo,
> 
> maybe I misunderstood, but it look like I don't understand tool=cachegrind correct (or there is a bug).
> 
> I have a software. If it runs, then the task manager tells me that is use only 38 MB memory.
> 
> Now i used cachegrind like this:
> valgrind --tool=cachegrind --LL=2097152,16,64 ./fet-cl --inputfile=German-100_and_0.fet  
> --randomseeds10=10 --randomseeds11=11 --randomseeds12=12 --randomseeds20=20 --randomseeds21=21 
> --randomseeds22=22
> 
> i got this report:
> [...]
> ==43032== LL misses:           8,283,261  (      7,840,142 rd   + 443,119 wr)
> 
> 
> i doubled the LL cache by this:
> valgrind --tool=cachegrind --LL=4194304,16,64 ./fet-cl --inputfile=German-100_and_0.fet  
> --randomseeds10=10 --randomseeds11=11 --randomseeds12=12 --randomseeds20=20 --randomseeds21=21 
> --randomseeds22=22
> 
> and got this:
> [...]
> ==48663== LL misses:             230,426  (         89,082 rd   + 141,344 wr)
> 
> 
> That sound ok (larger cache should reduce the misses).
> 
> But I continued to double the cache again and again. Up to:
> valgrind --tool=cachegrind --LL=268435456,16,64 ./fet-cl --inputfile=German-100_and_0.fet  
> --randomseeds10=10 --randomseeds11=11 --randomseeds12=12 --randomseeds20=20 --randomseeds21=21 
> --randomseeds22=22
> 
> So much more memory then my program use at all.
> But the misses never drop down to 0. They stay at:
> [...]
> ==6637== LL misses:             180,120  (         41,252 rd   + 138,868 wr)
> 
> 
> I don't understand that. Shouldn't the misses drop down to 0 as soon as LL is >64MB (since my 
> software use only 38 MB)? (But i tried up to 256MB and it doesn't drop).
> Is that a bug in valgrind or is there a bug in my logic in understanding the LL misses?
> 
> Please let me know.

I would think that, regardless of cache size, the *first* access to a line
causes a miss.  These are sometimes called mandatory misses.  A large cache
*will* eliminate *capacity* misses.

HTH -- Eliot Moss

[Valgrind-users] tool=cachegrind always reports LL misses, even if the cache is bigger than the used memory

[Volker D. <u6...@ti...>]

Hallo,

maybe I misunderstood, but it look like I don't understand 
tool=cachegrind correct (or there is a bug).

I have a software. If it runs, then the task manager tells me that is 
use only 38 MB memory.

Now i used cachegrind like this:
valgrind --tool=cachegrind --LL=2097152,16,64 ./fet-cl 
--inputfile=German-100_and_0.fet  --randomseeds10=10 --randomseeds11=11 
--randomseeds12=12 --randomseeds20=20 --randomseeds21=21 
--randomseeds22=22

i got this report:
[...]
==43032== LL misses:           8,283,261  (      7,840,142 rd   +        
443,119 wr)


i doubled the LL cache by this:
valgrind --tool=cachegrind --LL=4194304,16,64 ./fet-cl 
--inputfile=German-100_and_0.fet  --randomseeds10=10 --randomseeds11=11 
--randomseeds12=12 --randomseeds20=20 --randomseeds21=21 
--randomseeds22=22

and got this:
[...]
==48663== LL misses:             230,426  (         89,082 rd   +        
141,344 wr)


That sound ok (larger cache should reduce the misses).

But I continued to double the cache again and again. Up to:
valgrind --tool=cachegrind --LL=268435456,16,64 ./fet-cl 
--inputfile=German-100_and_0.fet  --randomseeds10=10 --randomseeds11=11 
--randomseeds12=12 --randomseeds20=20 --randomseeds21=21 
--randomseeds22=22

So much more memory then my program use at all.
But the misses never drop down to 0. They stay at:
[...]
==6637== LL misses:             180,120  (         41,252 rd   +        
138,868 wr)


I don't understand that. Shouldn't the misses drop down to 0 as soon as 
LL is >64MB (since my software use only 38 MB)? (But i tried up to 256MB 
and it doesn't drop).
Is that a bug in valgrind or is there a bug in my logic in understanding 
the LL misses?

Please let me know.

Thank you!

Re: [Valgrind-users] Valgrind-3.21.0.RC2 is available for testing

[Paul F. <pj...@wa...>]

On 23-04-23 13:58, Paul Floyd wrote:
> 

> 
> Still to come: Alpine/musl.

It configures.

It all builds.

It's not great.


== 619 tests, 102 stderr failures, 27 stdout failures, 1 stderrB 
failure, 2 stdoutB failures, 4 post failures ==

But that's good enough for me :-)

A+
Paul

Re: [Valgrind-users] Valgrind-3.21.0.RC2 is available for testing

[Paul F. <pj...@wa...>]

On 04/22/23 11:41 PM, Paul Floyd wrote:
>
> Nothing bad to report. I'll do one more test on Solaris 11.3 tomorrow.
>
Solaris 11.3 amd64

No python3, and I only have python3.4 (and no 'python3' metapackage 
either) on my machine that is long out of service contract. So I got 
some moans about that.

No aligned_alloc !

The memalign wrapper also wasn't correct for an alignment of zero.

I just fixed that. The changes should only affect non-Linux platforms. 
I'll try to do one more round of tests with these changes from git head.


Still to come: Alpine/musl.


A+

Paul

Re: [Valgrind-users] Valgrind-3.21.0.RC2 is available for testing

[Paul F. <pj...@wa...>]

On 22-04-23 23:41, Paul Floyd wrote:

> 
> OpenIndiana 22.10
> 
> Everything builds. All the gdbserver tests hang.

And I should have said the hangs aren't new.

A+
Paul

Re: [Valgrind-users] Valgrind-3.21.0.RC2 is available for testing

[Paul F. <pj...@wa...>]

Nothing bad to report. I'll do one more test on Solaris 11.3 tomorrow.

FreeBSD 13.1 amd64

No change

FreeBSD 13.2 amd64

I get a hang in drd/tests/pth_cancel_locked and a few more fails, most 
likely related to switching from clang 13 to 14.

FreedBSD 13.2 x86

As for amd64 but also a new hang in drd/tests/swapcontext

OpenIndiana 22.10

Everything builds. All the gdbserver tests hang.
== 832 tests, 82 stderr failures, 19 stdout failures, 15 stderrB 
failures, 17 stdoutB failures, 6 post failures ==


macOS 10.13

Everything builds but beyond that pretty grim.

== 706 tests, 322 stderr failures, 91 stdout failures, 0 stderrB 
failures, 0 stdoutB failures, 32 post failures ==


A+
Paul

[Valgrind-users] Valgrind-3.21.0.RC2 is available for testing

[Mark W. <ma...@kl...>]

An RC2 tarball for 3.21.0 is now available at
https://sourceware.org/pub/valgrind/valgrind-3.21.0.RC2.tar.bz2
(md5sum = f33407fdffbfa78f5014781cc92297cf)
(sha1sum = c520ee0c28d9e20d28aa25d05ce2525c39a69135)
https://sourceware.org/pub/valgrind/valgrind-3.21.0.RC2.tar.bz2.asc

Please give it a try in configurations that are important for you and
report any problems you have, either on this mailing list, or
(preferably) via our bug tracker at
https://bugs.kde.org/enter_bug.cgi?product=valgrind

Please check the NEWS entry below for new features that could use some
extra testing. Note that there has also been a dhat extension which
hasn't yet been added to NEWS.

There is now a a client request for DHAT to mark memory to be
histogrammed:
https://bugs.kde.org/464103
https://snapshots.sourceware.org/valgrind/trunk/latest/html/dh-manual.html#dh-access-counts

If nothing critical emerges, a final release will happen on
Friday 28 April.

* ==================== CORE CHANGES ===================

* When GDB is used to debug a program running under valgrind using
  the valgrind gdbserver, GDB will automatically load some
  python code provided in valgrind defining GDB front end commands
  corresponding to the valgrind monitor commands.
  These GDB front end commands accept the same format as
  the monitor commands directly sent to the Valgrind gdbserver.
  These GDB front end commands provide a better integration
  in the GDB command line interface, so as to use for example
  GDB auto-completion, command specific help, searching for
  a command or command help matching a regexp, ...
  For relevant monitor commands, GDB will evaluate arguments
  to make the use of monitor commands easier.
  For example, instead of having to print the address of a variable
  to pass it to a subsequent monitor command, the GDB front end
  command will evaluate the address argument. It is for example
  possible to do:
    (gdb) memcheck who_point_at &some_struct sizeof(some_struct)
  instead of:
    (gdb) p &some_struct
    $2 = (some_struct_type *) 0x1130a0 <some_struct>
    (gdb) p sizeof(some_struct)
    $3 = 40
    (gdb) monitor who_point_at 0x1130a0 40

* The vgdb utility now supports extended-remote protocol when
  invoked with --multi. In this mode the GDB run command is
  supported. Which means you don't need to run gdb and valgrind
  from different terminals. So for example to start you program
  in gdb and run it under valgrind you can do:
  $ gdb prog
  (gdb) set remote exec-file prog
  (gdb) set sysroot /
  (gdb) target extended-remote | vgdb --multi
  (gdb) start

* The behaviour of realloc with a size of zero can now
  be changed for tools that intercept malloc. Those
  tools are memcheck, helgrind, drd, massif and dhat.
  Realloc implementations generally do one of two things
     - free the memory like free() and return NULL
       (GNU libc and ptmalloc).
     - either free the memory and then allocate a
       minumum siized block or just return the
       original pointer. Return NULL if the
       allocation of the minimum sized block fails
       (jemalloc, musl, snmalloc, Solaris, macOS).
  When Valgrind is configured and built it will
  try to match the OS and libc behaviour. However
  if you are using a non-default library to replace
  malloc and family (e.g., musl on a glibc Linux or
  tcmalloc on FreeBSD) then you can use a command line
  option to change the behaviour of Valgrind:
    --realloc-zero-bytes-frees=yes|no [yes on Linux glibc, no otherwise]

* ================== PLATFORM CHANGES =================

* Make the address space limit on FreeBSD amd64 128Gbytes
  (the same as Linux and Solaris, it was 32Gbytes)

* ==================== TOOL CHANGES ===================

* Memcheck:
  - When doing a delta leak_search, it is now possible to only
    output the new loss records compared to the previous leak search.
    This is available in the memcheck monitor command 'leak_search'
    by specifying the "new" keyword or in your program by using
    the client request VALGRIND_DO_NEW_LEAK_CHECK.
    Whenever a "delta" leak search is done (i.e. when specifying
    "new" or "increased" or "changed" in the monitor command),
    the new loss records have a "new" marker.
  - Valgrind now contains python code that defines GDB memcheck
    front end monitor commands. See CORE CHANGES.
  - Performs checks for the use of realloc with a size of zero.
    This is non-portable and a source of errors. If memcheck
    detects such a usage it will generate an error
      realloc() with size 0
    followed by the usual callstacks.
    A switch has been added to allow this to be turned off:
      --show-realloc-size-zero=yes|no [yes]

* Helgrind:
  - The option ---history-backtrace-size=<number> allows to configure
    the number of entries to record in the stack traces of "old"
    accesses. Previously, this number was hardcoded to 8.
  - Valgrind now contains python code that defines GDB helgrind
    front end monitor commands. See CORE CHANGES.

* Cachegrind:
  - `--cache-sim=no` is now the default. The cache simulation is old and
    unlikely to match any real modern machine. This means only the `Ir`
    event are gathered by default, but that is by far the most useful
    event.
  - `cg_annotate`, `cg_diff`, and `cg_merge` have been rewritten in
    Python. As a result, they all have more flexible command line
    argument handling, e.g. supporting `--show-percs` and
    `--no-show-percs` forms as well as the existing `--show-percs=yes`
    and `--show-percs=no`.
  - `cg_annotate` has some functional changes.
    - It's much faster, e.g. 3-4x on common cases.
    - It now supports diffing (with `--diff`, `--mod-filename`, and
      `--mod-funcname`) and merging (by passing multiple data files).
    - It now provides more information at the file and function level.
      There are now "File:function" and "Function:file" sections. These
      are very useful for programs that use inlining a lot.
    - Support for user-annotated files and the `-I`/`--include` option
      has been removed, because it was of little use and blocked other
      improvements.
    - The `--auto` option is renamed `--annotate`, though the old
      `--auto=yes`/`--auto=no` forms are still supported.
  - `cg_diff` and `cg_merge` are now deprecated, because `cg_annotate`
    now does a better job of diffing and merging.
  - The Cachegrind output file format has changed very slightly, but in
    ways nobody is likely to notice.

* Callgrind:
  - Valgrind now contains python code that defines GDB callgrind
    front end monitor commands. See CORE CHANGES.

* Massif:
  - Valgrind now contains python code that defines GDB massif
    front end monitor commands. See CORE CHANGES.

Re: [Valgrind-users] unhandled instruction bytes

[Mark W. <ma...@kl...>]

On Wed, Apr 19, 2023 at 11:46:34AM +0200, folkert wrote:
> > > The 2 calls it does are:
> > > 
> > > print_char:
> > >      movb (%esi), %al
> > >      movb %al, buffer
> > >      movl $4, %eax
> > >      movl $1, %ebx
> > >      movl $buffer, %ecx
> > >      movl $1, %edx
> > >      int  $0x80
> > >      ret
> > > 
> > > exit:
> > >      movl $1, %eax
> > >      movl $0, %ebx
> > >      int  $0x80
> > 
> > Valgrind can't run just any executable binary. It has quite a lot of hard
> > coded limitations that correspont (mostly) to what compilers and link
> > editors will produce. So if you use assembler and use opcodes not normally
> > generated by compilers then it won't work.
> ...
> > So int 0x80 results in a decode error.
> > 
> > Can you use syscall?
> 
> That solves the problem.

Glad that resolved it.I also didn't know int 0x80 worked on amd64 as
syscal (but not under valgrind).  Note that this also
https://bugs.kde.org/show_bug.cgi?id=342988

Cheers,

Mark

Re: [Valgrind-users] unhandled instruction bytes

[folkert <fo...@va...>]

> > The 2 calls it does are:
> > 
> > print_char:
> >      movb (%esi), %al
> >      movb %al, buffer
> >      movl $4, %eax
> >      movl $1, %ebx
> >      movl $buffer, %ecx
> >      movl $1, %edx
> >      int  $0x80
> >      ret
> > 
> > exit:
> >      movl $1, %eax
> >      movl $0, %ebx
> >      int  $0x80
> 
> Valgrind can't run just any executable binary. It has quite a lot of hard
> coded limitations that correspont (mostly) to what compilers and link
> editors will produce. So if you use assembler and use opcodes not normally
> generated by compilers then it won't work.
...
> So int 0x80 results in a decode error.
> 
> Can you use syscall?

That solves the problem.
Thanks!

Re: [Valgrind-users] unhandled instruction bytes

[Floyd, P. <pj...@wa...>]

On 18/04/2023 17:46, folkert wrote:
> The 2 calls it does are:
>
> print_char:
>      movb (%esi), %al
>      movb %al, buffer
>      movl $4, %eax
>      movl $1, %ebx
>      movl $buffer, %ecx
>      movl $1, %edx
>      int  $0x80
>      ret
>
> exit:
>      movl $1, %eax
>      movl $0, %ebx
>      int  $0x80

Valgrind can't run just any executable binary. It has quite a lot of 
hard coded limitations that correspont (mostly) to what compilers and 
link editors will produce. So if you use assembler and use opcodes not 
normally generated by compilers then it won't work.

The code that handles this is

    case 0xCD: /* INT imm8 */
       d64 = getUChar(delta); delta++;

       /* Handle int $0xD2 (Solaris fasttrap syscalls). */
       if (d64 == 0xD2) {
          jmp_lit(dres, Ijk_Sys_int210, guest_RIP_bbstart + delta);
          vassert(dres->whatNext == Dis_StopHere);
          DIP("int $0xD2\n");
          return delta;
       }
       goto decode_failure;

So int 0x80 results in a decode error.

Can you use syscall?


A+

Paul

Re: [Valgrind-users] unhandled instruction bytes

[Eliot M. <mo...@cs...>]

On 4/18/2023 10:51 AM, folkert wrote:
> Hi,
> 
> I wrote a compiler for brainfuck to x86.
> The result is quite fast but I was curious if I could tune it even more.
> So I ran it in callgrind but this resulted in:
> 
> folkert@snsv ~/Projects/bf-compiler (master)$ valgrind --tool=callgrind ./test
> ==77043== Callgrind, a call-graph generating cache profiler
> ==77043== Copyright (C) 2002-2017, and GNU GPL'd, by Josef Weidendorfer et al.
> ==77043== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
> ==77043== Command: ./test
> ==77043==
> ==77043== For interactive control, run 'callgrind_control -h'.
> vex amd64->IR: unhandled instruction bytes: 0xCD 0x80 0xC3 0x67 0x80 0x3E 0x0 0x74 0x5 0x83
> vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
> vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
> ==77043== valgrind: Unrecognised instruction at address 0x40274e.
> ==77043==    at 0x40274E: ??? (in /home/folkert/Projects/bf-compiler/test)
> ==77043==    by 0x4020EE: ??? (in /home/folkert/Projects/bf-compiler/test)
> ==77043== Your program just tried to execute an instruction that Valgrind
> ==77043== did not recognise.  There are two possible reasons for this.
> ==77043== 1. Your program has a bug and erroneously jumped to a non-code
> ==77043==    location.  If you are running Memcheck and you just saw a
> ==77043==    warning about a bad jump, it's probably your program's fault.
> ==77043== 2. The instruction is legitimate but Valgrind doesn't handle it,
> ==77043==    i.e. it's Valgrind's fault.  If you think this is the case or
> ==77043==    you are not sure, please let us know and we'll try to fix it.
> ==77043== Either way, Valgrind will now raise a SIGILL signal which will
> ==77043== probably kill your program.
> ==77043==
> ==77043== Process terminating with default action of signal 4 (SIGILL)
> ==77043==  Illegal opcode at address 0x40274E
> ==77043==    at 0x40274E: ??? (in /home/folkert/Projects/bf-compiler/test)
> ==77043==    by 0x4020EE: ??? (in /home/folkert/Projects/bf-compiler/test)
> ==77043==
> ==77043== Events    : Ir
> ==77043== Collected : 28836
> ==77043==
> ==77043== I   refs:      28,836
> Illegal instruction (core dumped)
> 
> If you're curious what is going wrong here, the source assembly and the
> x86 binary can be retrieved from
> https://vanheusden.com/permshare/callgrind-error.tar.xz
> 
> Oh and if you would like to assemble the assembly yourself:
> 
> as -g mandelbrot.s
> ld -g a.out -o test
> 
> ./test then results in the mandelbrot-fractal.

Using an online disassembler, I found that the initial bytes decode to
int 0x80, which (under Linux) is a system call.  Maybe you're making a
system call that valgrind does not recognize?  One would need to know
register contents to go further with that.

Btw, naming a program "test" is not necessarily a wonderful idea if
the current directory happens to be on your path, since "test" is a
program often used by scripts.

Cheers - Eliot Moss

Re: [Valgrind-users] unhandled instruction bytes

[folkert <fo...@va...>]

> > I wrote a compiler for brainfuck to x86.
> > The result is quite fast but I was curious if I could tune it even more.
> > So I ran it in callgrind but this resulted in:
...
> > ==77043== Process terminating with default action of signal 4 (SIGILL)
> > ==77043==  Illegal opcode at address 0x40274E
> > ==77043==    at 0x40274E: ??? (in /home/folkert/Projects/bf-compiler/test)
> > ==77043==    by 0x4020EE: ??? (in /home/folkert/Projects/bf-compiler/test)
...
> > If you're curious what is going wrong here, the source assembly and the
> > x86 binary can be retrieved from
> > https://vanheusden.com/permshare/callgrind-error.tar.xz
...

> Using an online disassembler, I found that the initial bytes decode to
> int 0x80, which (under Linux) is a system call.  Maybe you're making a
> system call that valgrind does not recognize?  One would need to know
> register contents to go further with that.

The 2 calls it does are:

print_char:
    movb (%esi), %al
    movb %al, buffer
    movl $4, %eax
    movl $1, %ebx
    movl $buffer, %ecx
    movl $1, %edx
    int  $0x80
    ret

exit:
    movl $1, %eax
    movl $0, %ebx
    int  $0x80

When the program is ran directly from the command, it runs fine. So
that's not the problem.

[Valgrind-users] unhandled instruction bytes

[folkert <fo...@va...>]

Hi,

I wrote a compiler for brainfuck to x86.
The result is quite fast but I was curious if I could tune it even more.
So I ran it in callgrind but this resulted in:

folkert@snsv ~/Projects/bf-compiler (master)$ valgrind --tool=callgrind ./test
==77043== Callgrind, a call-graph generating cache profiler
==77043== Copyright (C) 2002-2017, and GNU GPL'd, by Josef Weidendorfer et al.
==77043== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==77043== Command: ./test
==77043== 
==77043== For interactive control, run 'callgrind_control -h'.
vex amd64->IR: unhandled instruction bytes: 0xCD 0x80 0xC3 0x67 0x80 0x3E 0x0 0x74 0x5 0x83
vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
==77043== valgrind: Unrecognised instruction at address 0x40274e.
==77043==    at 0x40274E: ??? (in /home/folkert/Projects/bf-compiler/test)
==77043==    by 0x4020EE: ??? (in /home/folkert/Projects/bf-compiler/test)
==77043== Your program just tried to execute an instruction that Valgrind
==77043== did not recognise.  There are two possible reasons for this.
==77043== 1. Your program has a bug and erroneously jumped to a non-code
==77043==    location.  If you are running Memcheck and you just saw a
==77043==    warning about a bad jump, it's probably your program's fault.
==77043== 2. The instruction is legitimate but Valgrind doesn't handle it,
==77043==    i.e. it's Valgrind's fault.  If you think this is the case or
==77043==    you are not sure, please let us know and we'll try to fix it.
==77043== Either way, Valgrind will now raise a SIGILL signal which will
==77043== probably kill your program.
==77043== 
==77043== Process terminating with default action of signal 4 (SIGILL)
==77043==  Illegal opcode at address 0x40274E
==77043==    at 0x40274E: ??? (in /home/folkert/Projects/bf-compiler/test)
==77043==    by 0x4020EE: ??? (in /home/folkert/Projects/bf-compiler/test)
==77043== 
==77043== Events    : Ir
==77043== Collected : 28836
==77043== 
==77043== I   refs:      28,836
Illegal instruction (core dumped)

If you're curious what is going wrong here, the source assembly and the
x86 binary can be retrieved from
https://vanheusden.com/permshare/callgrind-error.tar.xz

Oh and if you would like to assemble the assembly yourself:

as -g mandelbrot.s
ld -g a.out -o test

./test then results in the mandelbrot-fractal.


Regards,

Folkert van Heusden

Re: [Valgrind-users] RFC: proposal to remove user annotation from `cg_annotate`

[Nicholas N. <n.n...@gm...>]

I am planning to also remove the `-I`/`--include` option from cg_annotate,
for much the same reasons that I removed user annotated files: it's an
option that made sense in the very early days of cg_annotate, but is of
little or no use today, and it's getting in the way of some other changes I
want to make.

Nick

On Tue, 4 Apr 2023 at 15:52, Nicholas Nethercote <n.n...@gm...>
wrote:

> There were no objections, and I have now removed user annotations from
> `cg_annotate`.
>
> Nick
>
> On Wed, 29 Mar 2023 at 09:03, Nicholas Nethercote <n.n...@gm...>
> wrote:
>
>> Hi,
>>
>> I recently rewrote `cg_annotate`, `cg_diff`, and `cg_merge` in Python.
>> The old versions were written in Perl, Perl, and C, respectively. The new
>> versions are much nicer and easier to modify, and I have various ideas for
>> improving `cg_annotate`. This email is about one of those ideas.
>>
>> A typical way to invoke `cg_annotate` is like this:
>>
>> > cg_annotate cachegrind.out.12345
>>
>> This implies `--auto=yes`, which requests line-by-line "auto-annotation"
>> of source files. I.e. `cg_annotate` will automatically annotate all files
>> in the profile that meet the significance threshold.
>>
>> It's also possible to do something like this:
>>
>> > cg_annotate --auto=no cachegrind.out.12345 a.c b.c
>>
>> Which instead requests "user annotation" of the files `a.c` and `b.c`.
>>
>> My thesis is that auto-annotation suffices in practice for all reasonable
>> use cases, and that user annotation is unnecessary and can be removed.
>>
>> When I first wrote `cg_annotate` in 2002, only user annotation was
>> implemented. Shortly after, I added the `--auto={yes,no}` option. Since
>> then I've never used user annotation, and I suspect nobody else has either.
>> User annotation is ok when dealing with tiny programs, but as soon as you
>> are profiling a program with more than a handful of source files it becomes
>> impractical.
>>
>> The only possible use cases I can think of for user annotation are as
>> follows.
>>
>>    - If you want to see a particular file(s) annotated but you don't
>>    want to see any others, then you can use user annotation in combination
>>    with `--auto=no`. But it's trivial to search through the output for the
>>    particular file, so this doesn't seem important.
>>    - If the path to a file is somehow really messed up in the debug
>>    info, it might be possible that auto-annotation would fail to find it, but
>>    user annotation could find it, possibly in combination with `-I`. But this
>>    seems unlikely. Some basic testing shows that gcc, clang and rustc all
>>    default to using full paths in debug info. gcc supports
>>    `-fdebug-prefix-map` but that seems to mostly be used for changing full
>>    paths to relative paths, which will still work fine.
>>
>> Removing user annotation would (a) simplify the code and docs, and (b)
>> enable the possibility of moving the merge functionality from `cg_merge`
>> into `cg_annotate`, by allowing the user to specify multiple cachegrind.out
>> files as input.
>>
>> So: is anybody using user annotation? Does anybody see any problems with
>> this proposal?
>>
>> Thanks.
>>
>> Nick
>>
>

Re: [Valgrind-users] [Valgrind-developers] Valgrind-3.21.0.RC1 is available for testing

[Nicholas N. <n.n...@gm...>]

Hi,

My plans for the release:

   - I have one more significant improvement to `cg_annotate` to come,
   which will add merge and diff capability to it, in a way that is better
   than the merge/diff capability provided by `cg_merge` and `cg_diff`.
   - I need to update the Cachegrind docs and the NEWS file for all the
   changes I've made.

I know these will be happening late in the release cycle, but because it's
all Python code it should require less testing. The likelihood of
platform-specific differences in behaviour is much lower than in most other
code within Valgrind.

Nick

On Sat, 15 Apr 2023 at 12:07, Mark Wielaard <ma...@kl...> wrote:

> An RC1 tarball for 3.21.0 is now available at
> https://sourceware.org/pub/valgrind/valgrind-3.21.0.RC1.tar.bz2
> (md5sum = a3c7eeff47262cecdf5f1d68b38710b7)
> (sha1sum = 46fc5898415001e045abc1b4e2909a41144ed9c4)
> https://sourceware.org/pub/valgrind/valgrind-3.21.0.RC1.tar.bz2.asc
>
> Please give it a try in configurations that are important for you and
> report any problems you have, either on this mailing list, or
> (preferably) via our bug tracker at
> https://bugs.kde.org/enter_bug.cgi?product=valgrind
>
> There are still some patches being reviewed and a RC2 will appear end
> of next week.  If nothing critical emerges after that, a final release
> will happen on Friday 28 April.
>
>
>
> _______________________________________________
> Valgrind-developers mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-developers
>

[Valgrind-users] Valgrind-3.21.0.RC1 is available for testing

[Mark W. <ma...@kl...>]

An RC1 tarball for 3.21.0 is now available at
https://sourceware.org/pub/valgrind/valgrind-3.21.0.RC1.tar.bz2
(md5sum = a3c7eeff47262cecdf5f1d68b38710b7)
(sha1sum = 46fc5898415001e045abc1b4e2909a41144ed9c4)
https://sourceware.org/pub/valgrind/valgrind-3.21.0.RC1.tar.bz2.asc

Please give it a try in configurations that are important for you and
report any problems you have, either on this mailing list, or
(preferably) via our bug tracker at
https://bugs.kde.org/enter_bug.cgi?product=valgrind

There are still some patches being reviewed and a RC2 will appear end
of next week.  If nothing critical emerges after that, a final release
will happen on Friday 28 April.

[Valgrind-users] A bit more (pre-release) automation

[Mark W. <ma...@kl...>]

Hi,

Working towards a new release (3.21.0 currently planned for April 28)
there is a bit more automation to show pre-releases and documentation:

https://snapshots.sourceware.org/valgrind/trunk/

Every 15 minutes a buildbot will check for new commits and create a
dist, html manual pages and documentation downloads from latest git
trunk (currently the git master branch, after the release we'll switch
to using the main branch).

Be careful, these aren't official releases, it is as if getting a
random git checkout, but hopefully it is useful to see what is coming
and have the latest (draft) documentation for new features for the
next release. If you try these out please explicitly mention which
snapshot you used in any bug reports.

Cheers,

Mark

Re: [Valgrind-users] RFC: changing Cachegrind default to `--cache-sim=no`

[David F. <fa...@kd...>]

On mardi 4 avril 2023 12:11:18 CEST Nicholas Nethercote wrote:
> On Tue, 4 Apr 2023 at 19:24, David Faure <fa...@kd...> wrote:
> > But then, with no cache simulation and no call stacks, what's left in
> > `cachegrind --cache-sim=no`?
> 
> From the email that started this thread:
> 
> If you run with `--cache-sim=no` then the cache simulation is disabled and
> > you just get one event: Ir. (This is "instruction cache reads", which is
> > equivalent to "instructions executed".)

Ah, right, sorry.

So to summarize the big picture:
cachegrind -> instructions count, without call stacks, useful for overall numbers or with cg_annotate
callgrind -> instructions count, with call stacks, best viewed in kcachegrind


I wish those two could do cycles and not just instructions, but I guess this requires a good cache simulator again, back to square one ;)
(perf does cycles, but doesn't give exact number of method calls, that's one benefit of cachegrind/callgrind)

-- 
David Faure, fa...@kd..., http://www.davidfaure.fr
Working on KDE Frameworks 5

Re: [Valgrind-users] RFC: changing Cachegrind default to `--cache-sim=no`

[Nicholas N. <n.n...@gm...>]

On Tue, 4 Apr 2023 at 19:24, David Faure <fa...@kd...> wrote:

>
> But then, with no cache simulation and no call stacks, what's left in
> `cachegrind --cache-sim=no`?
>

>From the email that started this thread:

If you run with `--cache-sim=no` then the cache simulation is disabled and
> you just get one event: Ir. (This is "instruction cache reads", which is
> equivalent to "instructions executed".)
>

Re: [Valgrind-users] RFC: changing Cachegrind default to `--cache-sim=no`

[David F. <fa...@kd...>]

On lundi 3 avril 2023 23:46:46 CEST Nicholas Nethercote wrote:
> On Mon, 3 Apr 2023 at 21:36, David Faure <fa...@kd...> wrote:
> > But then, what's the difference between `cachegrind --cache-sim=no`
> > and `callgrind`?
> > 
> > https://accu.org/journals/overload/20/111/floyd_1886/ says
> > "The main differences are that Callgrind has more information about the
> > callstack whilst cachegrind gives more information about cache hit rates."
> > 
> > Wouldn't one want callstacks? (if this means stack traces).
> > I know I must be missing something, thanks for enlightening me.
> 
> Callgrind is a forked and extended version of Cachegrind. It also simulates
> a cache, with a slightly different simulation to Cachegrind's. The fact
> that both tools exist is due to historical reasons; if starting from
> scratch today you wouldn't deliberately split them.

Thanks for the information. This is indeed confusing - like anything that is 
"due to historical reasons" ;-)

> Call stacks are often useful (I regularly use Callgrind as well as
> Cachegrind) but they aren't always necessary. Without them, Cachegrind runs
> faster than Callgrind and produces smaller data files. Cachegrind also
> supports diffing and merging different files, while Callgrind does not.

OK. I thought call stacks were mandatory for any tool to be useful
(they certainly are for KCachegrind (*)), but I now found the documentation on 
cg_annotate.

But then, with no cache simulation and no call stacks, what's left in 
`cachegrind --cache-sim=no`?


(*) This naming adds to the confusion: kcachegrind requires callgrind, it 
can't work with cachegrind... I know, historical reasons :-)

-- 
David Faure, fa...@kd..., http://www.davidfaure.fr
Working on KDE Frameworks 5

Re: [Valgrind-users] RFC: proposal to remove user annotation from `cg_annotate`

[Nicholas N. <n.n...@gm...>]

There were no objections, and I have now removed user annotations from
`cg_annotate`.

Nick

On Wed, 29 Mar 2023 at 09:03, Nicholas Nethercote <n.n...@gm...>
wrote:

> Hi,
>
> I recently rewrote `cg_annotate`, `cg_diff`, and `cg_merge` in Python. The
> old versions were written in Perl, Perl, and C, respectively. The new
> versions are much nicer and easier to modify, and I have various ideas for
> improving `cg_annotate`. This email is about one of those ideas.
>
> A typical way to invoke `cg_annotate` is like this:
>
> > cg_annotate cachegrind.out.12345
>
> This implies `--auto=yes`, which requests line-by-line "auto-annotation"
> of source files. I.e. `cg_annotate` will automatically annotate all files
> in the profile that meet the significance threshold.
>
> It's also possible to do something like this:
>
> > cg_annotate --auto=no cachegrind.out.12345 a.c b.c
>
> Which instead requests "user annotation" of the files `a.c` and `b.c`.
>
> My thesis is that auto-annotation suffices in practice for all reasonable
> use cases, and that user annotation is unnecessary and can be removed.
>
> When I first wrote `cg_annotate` in 2002, only user annotation was
> implemented. Shortly after, I added the `--auto={yes,no}` option. Since
> then I've never used user annotation, and I suspect nobody else has either.
> User annotation is ok when dealing with tiny programs, but as soon as you
> are profiling a program with more than a handful of source files it becomes
> impractical.
>
> The only possible use cases I can think of for user annotation are as
> follows.
>
>    - If you want to see a particular file(s) annotated but you don't want
>    to see any others, then you can use user annotation in combination with
>    `--auto=no`. But it's trivial to search through the output for the
>    particular file, so this doesn't seem important.
>    - If the path to a file is somehow really messed up in the debug info,
>    it might be possible that auto-annotation would fail to find it, but user
>    annotation could find it, possibly in combination with `-I`. But this seems
>    unlikely. Some basic testing shows that gcc, clang and rustc all default to
>    using full paths in debug info. gcc supports `-fdebug-prefix-map` but that
>    seems to mostly be used for changing full paths to relative paths, which
>    will still work fine.
>
> Removing user annotation would (a) simplify the code and docs, and (b)
> enable the possibility of moving the merge functionality from `cg_merge`
> into `cg_annotate`, by allowing the user to specify multiple cachegrind.out
> files as input.
>
> So: is anybody using user annotation? Does anybody see any problems with
> this proposal?
>
> Thanks.
>
> Nick
>

Re: [Valgrind-users] RFC: changing Cachegrind default to `--cache-sim=no`

[Nicholas N. <n.n...@gm...>]

On Mon, 3 Apr 2023 at 21:36, David Faure <fa...@kd...> wrote:

>
> But then, what's the difference between `cachegrind --cache-sim=no`
> and `callgrind`?
>
> https://accu.org/journals/overload/20/111/floyd_1886/ says
> "The main differences are that Callgrind has more information about the
> callstack whilst cachegrind gives more information about cache hit rates."
>
> Wouldn't one want callstacks? (if this means stack traces).
> I know I must be missing something, thanks for enlightening me.
>

Callgrind is a forked and extended version of Cachegrind. It also simulates
a cache, with a slightly different simulation to Cachegrind's. The fact
that both tools exist is due to historical reasons; if starting from
scratch today you wouldn't deliberately split them.

Call stacks are often useful (I regularly use Callgrind as well as
Cachegrind) but they aren't always necessary. Without them, Cachegrind runs
faster than Callgrind and produces smaller data files. Cachegrind also
supports diffing and merging different files, while Callgrind does not.

Nick

Re: [Valgrind-users] RFC: changing Cachegrind default to `--cache-sim=no`

[David F. <fa...@kd...>]

[removing valgrind-developers, since I guess I can't post there]

On lundi 3 avril 2023 11:29:25 CEST Nicholas Nethercote wrote:
> I have been using `--cache-sim=no` almost exclusively for a long time. The
> cache simulation done by Valgrind is an approximation of the memory
> hierarchy of a 2002 AMD Athlon processor. Its accuracy for a modern memory
> hierarchy with three levels of cache, prefetching, non-LRU replacement, and
> who-knows-what-else is likely to be low. If you want to accurately know
> about cache behaviour you'd be much better off using hardware counters via
> `perf` or some other profiler.
> 
> But `--cache-sim=no` is still very useful because instruction execution
> counts are still very useful.
> 
> Therefore, I propose changing the default to `--cache-sim=no`. Does anyone
> have any objections to this?

I agree that simulating a cache from 2002 isn't very useful.

But then, what's the difference between `cachegrind --cache-sim=no`
and `callgrind`?

https://accu.org/journals/overload/20/111/floyd_1886/ says
"The main differences are that Callgrind has more information about the 
callstack whilst cachegrind gives more information about cache hit rates."

Wouldn't one want callstacks? (if this means stack traces).
I know I must be missing something, thanks for enlightening me.

-- 
David Faure, fa...@kd..., http://www.davidfaure.fr
Working on KDE Frameworks 5