#278 incorrect function name(s) in the oprofile report

[george]

Profiling user application (INTEL IMB benchmark) on Ubuntu 14.10
using operf generates incorrect function names that are not invoked during program execution,
while the output reports that this function is consuming 15% of the CPU cycles.

This has been verified by two methods: by comparing that "operf" output with the "perf" output
which reported the right function and by checking if the function is invoked in gdb (it was not).

Virtual address randomization has been disabled

cat /proc/sys/kernel/randomize_va_space

0

The library (libpami.so) has been built using GNU compilers. It is a mixed C and C++ code.

===========================================================================================
PROFILING IMB with operf tool
===========================================================================================
root@c656f6n01:/opt/at7.1/bin# ./operf -e CYCLES:1000000:0:0:1 -p 44212
operf: Press Ctl-c or 'kill -SIGINT 44458' to stop profiling
operf: Profiler started
^C

root@c656f6n01:/opt/at7.1/bin# ./opreport -l
Profiling done.
CPU: ppc64 POWER8, speed 3690 MHz (estimated)
Counted CYCLES events (Cycles) with a unit mask of 0x00 (No unit mask) count 1000000
samples % image name symbol name
4865 20.0305 libhal64_ibm.so hal_ibl_read_callback
3724 15.3327 libpami.so LapiImpl::Context::CheckUseShmHint(unsigned long, int)
2327 9.5809 libpami.so _pami_core_uint32_lxor
1966 8.0945 libmpi.so MPIDU_Sched_progress
1578 6.4970 libmlx5-rdmav2.so __mlx5_post_send

=======================================================================================
CHECHING LapiImpl::Context::CheckUseShmHint in gdb
========================================================================================
Breakpoint 1 at 0x10000054fff8: file /project/sprelvin/build/rvins001a/src/ppe/lapi/CheckParam.cpp, line 247.
(gdb) c
Continuing.

application terminated successfully without hitting LapiImpl::Context::CheckUseShmHint

=================================================================================================
PROFILING IMB benchmark using "perf" tool
================================================================================================

To display the perf.data header info, please use --header/--header-only options.

Samples: 29K of event 'cycles:u'

Event count (approx.): 27011079756

Overhead Command Shared Object Sym

........ ............... ......................... ..................................................................................................

17.04%  IMB-MPI1.mpich2  libhal64_ibm.so            [.] hal_ibl_read_callback
15.84%  IMB-MPI1.mpich2  libpami.so                 [.] _Z20_lapi_shm_dispatcherPN8LapiImpl7ContextE
14.70%  IMB-MPI1.mpich2  libpami.so                 [.] PAMI_Context_advance
10.38%  IMB-MPI1.mpich2  libpami.so                 [.] _Z16_lapi_dispatcherILb0EEvPN8LapiImpl7ContextE
 7.02%  IMB-MPI1.mpich2  libmpi.so                  [.] MPIDU_Sched_progress
 5.89%  IMB-MPI1.mpich2  libmlx5-rdmav2.so          [.] __mlx5_post_send

========================================================================================

The correct C++ function name is lapi_shm_dispatcher. The following two functions,
PAMI_Context_advance and _lapi_dispatcher are also reported incorrectly by the operf tool.

[William Cohen]

If you are using the Ubuntu release of oprofile, you should also file an Ubuntu bug (https://help.ubuntu.com/community/ReportingBugs) so it get fixed in Ubuntu.

What version of oprofile is being used?
Is Ubuntu using preload on shared libraries? I recall that causing issues on Fedora.

If oprofile-0.9 is being used would it be possible to locally build oprofile-1.0 on the machine and check to see if the problem still exists with the newer version?

[george]

oprofile 1.0.0 compiled on Oct 14 2014 13:05:44 is already installed.

The problem seems to be caused by the incorrect mapping of the virtual addresses to function names.

ldd /usr/bin/operf

    linux-vdso64.so.1 =>  (0x00003fffa16f0000)
    libpfm.so.4 => /usr/lib/powerpc64le-linux-gnu/libpfm.so.4 (0x00003fffa15e0000)
    libstdc++.so.6 => /usr/lib/powerpc64le-linux-gnu/libstdc++.so.6 (0x00003fffa1470000)
    libgcc_s.so.1 => /lib/powerpc64le-linux-gnu/libgcc_s.so.1 (0x00003fffa1440000)
    libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00003fffa1280000)
    libm.so.6 => /lib/powerpc64le-linux-gnu/libm.so.6 (0x00003fffa11a0000)
    /lib64/ld64.so.2 (0x00000000445e0000)