Profiling user application (INTEL IMB benchmark) on Ubuntu 14.10
using operf generates incorrect function names that are not invoked during program execution,
while the output reports that this function is consuming 15% of the CPU cycles.
This has been verified by two methods: by comparing that "operf" output with the "perf" output
which reported the right function and by checking if the function is invoked in gdb (it was not).
Virtual address randomization has been disabled
0
The library (libpami.so) has been built using GNU compilers. It is a mixed C and C++ code.
===========================================================================================
PROFILING IMB with operf tool
===========================================================================================
root@c656f6n01:/opt/at7.1/bin# ./operf -e CYCLES:1000000:0:0:1 -p 44212
operf: Press Ctl-c or 'kill -SIGINT 44458' to stop profiling
operf: Profiler started
^C
root@c656f6n01:/opt/at7.1/bin# ./opreport -l
Profiling done.
CPU: ppc64 POWER8, speed 3690 MHz (estimated)
Counted CYCLES events (Cycles) with a unit mask of 0x00 (No unit mask) count 1000000
samples % image name symbol name
4865 20.0305 libhal64_ibm.so hal_ibl_read_callback
3724 15.3327 libpami.so LapiImpl::Context::CheckUseShmHint(unsigned long, int)
2327 9.5809 libpami.so _pami_core_uint32_lxor
1966 8.0945 libmpi.so MPIDU_Sched_progress
1578 6.4970 libmlx5-rdmav2.so __mlx5_post_send
=======================================================================================
CHECHING LapiImpl::Context::CheckUseShmHint in gdb
========================================================================================
Breakpoint 1 at 0x10000054fff8: file /project/sprelvin/build/rvins001a/src/ppe/lapi/CheckParam.cpp, line 247.
(gdb) c
Continuing.
application terminated successfully without hitting LapiImpl::Context::CheckUseShmHint
=================================================================================================
PROFILING IMB benchmark using "perf" tool
================================================================================================
17.04% IMB-MPI1.mpich2 libhal64_ibm.so [.] hal_ibl_read_callback 15.84% IMB-MPI1.mpich2 libpami.so [.] _Z20_lapi_shm_dispatcherPN8LapiImpl7ContextE 14.70% IMB-MPI1.mpich2 libpami.so [.] PAMI_Context_advance 10.38% IMB-MPI1.mpich2 libpami.so [.] _Z16_lapi_dispatcherILb0EEvPN8LapiImpl7ContextE 7.02% IMB-MPI1.mpich2 libmpi.so [.] MPIDU_Sched_progress 5.89% IMB-MPI1.mpich2 libmlx5-rdmav2.so [.] __mlx5_post_send
========================================================================================
The correct C++ function name is lapi_shm_dispatcher. The following two functions,
PAMI_Context_advance and _lapi_dispatcher are also reported incorrectly by the operf tool.
If you are using the Ubuntu release of oprofile, you should also file an Ubuntu bug (https://help.ubuntu.com/community/ReportingBugs) so it get fixed in Ubuntu.
What version of oprofile is being used?
Is Ubuntu using preload on shared libraries? I recall that causing issues on Fedora.
If oprofile-0.9 is being used would it be possible to locally build oprofile-1.0 on the machine and check to see if the problem still exists with the newer version?
oprofile 1.0.0 compiled on Oct 14 2014 13:05:44 is already installed.
The problem seems to be caused by the incorrect mapping of the virtual addresses to function names.
ldd /usr/bin/operf