#123 opreport/annotate segfault at SEC_DEBUGGING check

closed-invalid
nobody
None
7
2004-09-11
2004-08-15
Dan Hollis
No

i'm unable to get any symbols from opreport.

redhat 7.3 (glibc 2.2.5)
kernel 2.6.8.1
gcc 2.96-113
oprofile 0.8

i get the nice usage breakdown by library and
application, but i'm unable to get any symbol breakdown
inside any of them.

[root@washuu goemon]# opcontrol
--vmlinux=/usr/src/linux-2.6.8.1/vmlinux --separate=kernel

[root@washuu goemon]# opcontrol --start
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/oprofiled.log
Daemon started.
Profiler running.

[root@washuu goemon]# opreport -f | more
CPU: AMD64 processors, speed 1402.67 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt
state) with a unit mask of 0x00 (No unit mask) count 1000
00
CPU_CLK_UNHALT...|
samples| %|
------------------
8703906 95.9390
/usr/local/games/enemy-territory-102/ettv.x86
CPU_CLK_UNHALT...|
samples| %|
------------------
4447002 51.0920
/usr/local/games/enemy-territory-102/ettv.x86
2507957 28.8142 /lib/libc-2.2.5.so
1588779 18.2536 /usr/src/linux-2.6.8.1/vmlinux
47289 0.5433
/home/tvmaster/.etwolf/etpro/tvgame.mp.i386.so
36089 0.4146
/home/wolf/.etwolf/etpro/tvgame.mp.i386.so
26167 0.3006 /home/tvmaster/.etwolf/pb/pbsv.so
15589 0.1791 /nvidia
13029 0.1497
/home/tvmaster/.etwolf/etpro/qagame.mp.i386.so
... blabla ..

[root@washuu goemon]# opreport -l
/usr/local/games/enemy-territory-102/ettv.x86 | more
warning: /ip_conntrack could not be found.
warning: /ip_tables could not be found.
warning: /ipt_MASQUERADE could not be found.
warning: /iptable_filter could not be found.
warning: /iptable_nat could not be found.
warning: /nvidia could not be found.
warning: /ohci1394 could not be found.
warning: /oprofile could not be found.
warning: /psmouse could not be found.
warning: /sk98lin could not be found.
warning: /snd_trident could not be found.
warning: /sunrpc could not be found.
warning: /uhci_hcd could not be found.
warning: /usbcore could not be found.
warning: /usbhid could not be found.

that's all the output i get!

[root@washuu goemon]# opreport -l /lib/libc-2.2.5.so | more
CPU: AMD64 processors, speed 1402.67 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt
state) with a unit mask of 0x00 (No unit mask) count 1000
00
samples % image name app name
symbol name
2710986 98.8347 libc-2.2.5.so ettv.x86
(no symbols)
8151 0.2972 libc-2.2.5.so cc1
(no symbols)
2788 0.1016 libc-2.2.5.so opreport
(no symbols)
2546 0.0928 libc-2.2.5.so cpp0
(no symbols)
[...etc etc...]

and that's it. no symbols no nothing!

i can guarantee 100% that ettv.x86 has symbols though.
it's compiled with -g, not stripped, and 'nm' output
confirms it has symbols:

[goemon@washuu src]$ nm
/usr/local/games/enemy-territory-102/ettv.x86 | more
[...]
08066e68 T ETTV_DEBUG
08c233ec B ettv_delay
08065754 T ettv_DemoNumber
080652e8 T ETTV_GetPlayerState
08065fa0 T ettv_init
080a0528 D ettv_lastmessagetime
08060990 T ettv_NextSlavePacket
08d0dd10 B ettv_paceslavepackets
08d0dd0c B ettv_password
080647a4 T ettv_ServerInfo
08c233e0 B ettv_shownet
080a052c D ettv_shownet_lasttime
[... etc etc ...]

i can also gdb this process with full symbolic info
etc. so the symbols are definitely there. it just seems
that oprofile 0.8 doesn't know how to read them?

or am I doing something wrong?

Discussion

  • Dan Hollis
    Dan Hollis
    2004-08-15

    Logged In: YES
    user_id=106082

    ok more info:

    looks like opreport is segfaulting:

    #0 op_bfd::op_bfd (this=0xbffff820, fname=@0x817df60,
    symbol_filter=@0x8175898, ok=@0xbffff7bf)
    at op_bfd.cpp:272
    272 if (sect->flags & SEC_DEBUGGING) {
    (gdb) bt
    #0 op_bfd::op_bfd (this=0xbffff820, fname=@0x817df60,
    symbol_filter=@0x8175898, ok=@0xbffff7bf)
    at op_bfd.cpp:272
    #1 0x080a0dba in populate_for_image (samples=@0xbffff910,
    ip=@0x817df60, symbol_filter=@0x8175898)
    at populate.cpp:56
    #2 0x08050362 in {anonymous}::opreport
    (non_options=@0xbffff980) at /usr/include/g++-3/stl_list.h:71
    #3 0x08065b80 in run_pp_tool (argc=3, argv=0xbffffa34,
    fct=0x804ffc0
    <{anonymous}::opreport(vector<basic_string<char,
    string_char_traits<char>, __default_alloc_template<true, 0>
    >, allocator<basic_string<char, string_char_traits<char>,
    __default_alloc_template<true, 0> > > > const &)>) at
    common_option.cpp:110
    #4 0x080506ba in main (argc=3, argv=0xbffffa34) at
    opreport.cpp:440
    #5 0x400b21c4 in __libc_start_main () from /lib/libc.so.6
    (gdb) print sect
    $1 = (asection *) 0x832175db
    (gdb) print sect->flags
    Cannot access memory at address 0x832175eb

    oprofile is linked against libbfd 2.15.90 ...

     
  • John Levon
    John Levon
    2004-08-24

    • priority: 5 --> 7
    • summary: no symbols --> opreport/annotate segfault at SEC_DEBUGGING check
     
  • Philippe Elie
    Philippe Elie
    2004-08-24

    Logged In: YES
    user_id=318973

    Hi, this puzzled me, the previous loop acceed the same data
    struct
    and doesn't segfault, it's possible the previous loop acceed
    incorrect
    memory address w/o segfault and as side-effect ensure a
    segfault in
    next loop, unlikely but possible.

    you can try two things :

    - first double check you don't have a bfd.h in your include
    path. We
    already get something related, a package was installing a
    bfd.h in
    /usr/local/include but didn't install the corresponding .so
    file so users
    compile oprofile against one version of bfd but dynamic
    link with a
    different version ...

    -second if first don't fix the problem try valgrind on
    opreport and post
    the output (or the relevant part if you understand valgrind
    output), if the
    output is too voluminous bzip it and send it at phil.el [at]
    wanadoo [point] fr

    regards,
    Phil

     
  • Logged In: NO

    I ran into something very similar on a SuSE 8.1 system,
    but once I built a debug version of binutils to look more
    closely at the problem, it vanished.

    SuSE 8.1
    glibc-2.2.5
    kernel 2.4.21-241-default
    gcc 3.2.2-5
    oprofile 0.8
    binutils 2.12.90.0.15.49

    opreport was seg faulting in op_bfd::op_bfd, though this time
    in the first loop, the one checking sect->flags &
    SECT_CODE. When I built a debug version of oprofile and
    examined ibfd, it looked wrong. Most of the fields in the bfd
    had default values. format was bfd_unknown, and the tdata
    union was all zeros. The ibfd->sections->next field pointed
    off into bad memory, which caused the segv.

    after playing around a bit with libbfd, I decided I needed a
    debug build to figure out what was going on, and downloaded
    and built binutils 2.15.91. I then rebuilt oprofile against my
    new libbfd, and the problem vanished. Now if I look at the
    bfd, it's properly identified the binary as bfd_object, and the
    other fields all seem to have correct values.

    I started searching for bfd.h files on my system, and found
    several, including some that were installed with the kernel
    source. The one in the kernel source tree was very old.
    Perhaps I was picking up the wrong header file in some
    portion of the oprofile build?

     
  • John Levon
    John Levon
    2004-09-11

    • status: open --> closed-invalid
     
  • John Levon
    John Levon
    2004-09-11

    Logged In: YES
    user_id=53034

    Original reporter is non-responsive. Very probably a
    mis-match between bfd.h and linked-against libbfd.

    Closing out.