#28 Segfault and core dump at end of run on 40Gb controller

[Jeff Lane]

Running Ubuntu 16.04 on a Power8 OpenPower server. I'm running iperf <ver> against a 56GbE NIC limited to 40Gb by infrastructure.</ver>

Any long runs result in Segfaults. Our typical test run is 4 cycles of 15 minutes of sustained throughput. However, it first segfaulted at 900 seconds. I then tryed 300 seconds, and it segfaulted again. 30 second runs seem to work:

ubuntu@fesenkov:~$ iperf -c 10.20.30.1 -P 4 -t 30

Client connecting to 10.20.30.1, TCP port 5001
TCP window size: 325 KByte (default)

---

[ 6] local 10.20.31.49 port 42130 connected with 10.20.30.1 port 5001
[ 3] local 10.20.31.49 port 42126 connected with 10.20.30.1 port 5001
[ 4] local 10.20.31.49 port 42124 connected with 10.20.30.1 port 5001
[ 5] local 10.20.31.49 port 42128 connected with 10.20.30.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-30.0 sec 33.8 GBytes 9.68 Gbits/sec
[ 3] 0.0-30.0 sec 34.0 GBytes 9.73 Gbits/sec
[ 4] 0.0-30.0 sec 31.2 GBytes 8.92 Gbits/sec
[ 5] 0.0-30.0 sec 31.2 GBytes 8.95 Gbits/sec
[SUM] 0.0-30.0 sec 130 GBytes 37.3 Gbits/sec

ubuntu@fesenkov:~$ iperf -c 10.20.30.1 -P 4 -t 300

Client connecting to 10.20.30.1, TCP port 5001
TCP window size: 325 KByte (default)

---

[ 6] local 10.20.31.49 port 42142 connected with 10.20.30.1 port 5001
[ 4] local 10.20.31.49 port 42136 connected with 10.20.30.1 port 5001
[ 3] local 10.20.31.49 port 42138 connected with 10.20.30.1 port 5001
[ 5] local 10.20.31.49 port 42140 connected with 10.20.30.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-300.0 sec 330 GBytes 9.45 Gbits/sec
[ 4] 0.0-300.0 sec 319 GBytes 9.14 Gbits/sec
[ 3] 0.0-300.0 sec 330 GBytes 9.45 Gbits/sec
[ 5] 0.0-300.0 sec 321 GBytes 9.19 Gbits/sec
Segmentation fault (core dumped)

I've attached a core dump from one of the failures (the rest didn't generate a dump and no stack trace appeared in syslog.

[Jeff Lane]

Errr, that's version 2.0.5, sorry. Forgot to fix that in the initial post.

[Jeff Lane]

I also built and tried 2.0.9 and got the same result:
ubuntu@fesenkov:~/iperf-2.0.9/src$ ./iperf --version
iperf version 2.0.9 (9 Sept 2016) pthreads
ubuntu@fesenkov:~/iperf-2.0.9/src$ ./iperf -c 10.20.30.1 -P 4 -t 250

---

Client connecting to 10.20.30.1, TCP port 5001
TCP window size: 325 KByte (default)

---

[ 6] local 10.20.31.49 port 42276 connected with 10.20.30.1 port 5001
[ 3] local 10.20.31.49 port 42272 connected with 10.20.30.1 port 5001
[ 5] local 10.20.31.49 port 42274 connected with 10.20.30.1 port 5001
[ 4] local 10.20.31.49 port 42270 connected with 10.20.30.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-250.0 sec 281 GBytes 9.66 Gbits/sec
[ 3] 0.0-250.0 sec 302 GBytes 10.4 Gbits/sec
[ 5] 0.0-250.0 sec 263 GBytes 9.02 Gbits/sec
[ 4] 0.0-250.0 sec 237 GBytes 8.14 Gbits/sec
Segmentation fault (core dumped)

Additionally, this is a POWER8NVL machine with 160 cores and 512Gb RAM, so it's not being resource starved here.

[Robert McMahon]

hmm, not sure how to read the crash file. Is it possible for you decode it with GDB and post the decoded stack trace?

Thanks,
Bob

[Jeff Lane]

I had to google this, debugging with gdb is not something I do very often at all...

(gdb) thread apply all bt full

Thread 3 (Thread 0x3fffb2d66910 (LWP 53527)):

0 0x00003fffb2a70110 in __pthread_cond_wait (cond=0x3aa765f8 <thread_snum_cond>, mutex=0x3aa76628 <thread_snum_cond+48>)</thread_snum_cond+48></thread_snum_cond>

at pthread_cond_wait.c:186
    r4 = 128
    r7 = 2
    r5 = 11
    r8 = 2
    arg3 = 11
    r0 = 221
    r3 = 512
    r6 = 0
    arg4 = 0
    __err = <optimized out>
    __ret = <optimized out>
    futex_val = 11
    buffer = {__routine = 0x3fffb2a6fd40 <__condvar_cleanup>, __arg = 0x3fffea7d48a0, __canceltype = 983960448, __prev = 0x0}
    cbuffer = {oldtype = 0, cond = 0x3aa765f8 <thread_sNum_cond>, mutex = 0x3aa76628 <thread_sNum_cond+48>, bc_seq = 0}
    err = <optimized out>
    pshared = 0
    pi_flag = 0
    val = <optimized out>
    seq = 5

1 0x000000003aa6108c in ?? ()

No symbol table info available.

2 0x000000003aa53eb4 in ?? ()

No symbol table info available.

3 0x00003fffb28b309c in generic_start_main (main=0x3aa53ce0, argc=<optimized out="">, argv=0x3fffea7d4e38, auxvec=0x3fffea7d4f20,</optimized>

init=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>, fini=<optimized out>) at ../csu/libc-start.c:291
    self = 0x3fffb2d66910
    result = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-1792641418412953196, 70367446397696, -1792641417485382668, 0 <repeats 13 times>, 
            70367449577240, 0, 70367449577248, 0, 983963648, 2449962745659916287, 0 <repeats 42 times>}, mask_was_saved = 0}}, 
      priv = {pad = {0x0, 0x0, 0x3fffb28d34d8 <__GI___cxa_atexit+56>, 0x3fffb2a57300}, data = {prev = 0x0, cleanup = 0x0, 
          canceltype = -1299368744}}}
    not_first_call = <optimized out>

4 0x00003fffb28b3298 in __libc_start_main (argc=<optimized out="">, argv=<optimized out="">, ev=<optimized out="">, auxvec=<optimized out="">,</optimized></optimized></optimized></optimized>

rtld_fini=<optimized out>, stinfo=<optimized out>, stack_on_entry=<optimized out>)
at ../sysdeps/unix/sysv/linux/powerpc/libc-start.c:116

No locals.

5 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 2 (Thread 0x3fffabfff180 (LWP 53531)):

0 0x000000003aa594a4 in ?? ()

No symbol table info available.

1 0x000000003aa55088 in ?? ()

No symbol table info available.

2 0x000000003aa556a0 in ?? ()

No symbol table info available.

3 0x000000003aa60e38 in ?? ()

No symbol table info available.

4 0x00003fffb2a68070 in start_thread (arg=0x3fffabfff180) at pthread_create.c:335

    pd = 0x3fffabfff180
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-1792641417364546540, 70367446662912, -1792641417486779512, 0, 0, 70367326502912, 
            8388608, 70367446647696, 70367334889696, 1099977858080, 1099977858512, 70367334888000, 983960448, 0, 0, 
            70367446647712, 70368383289344, 70368383289400, 4001536, 70367446466432, 70367334885296, 2450033535310888959, 
            0 <repeats 42 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, 
          canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00003fffb29b3a30 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

No locals.

Thread 1 (Thread 0x3fffb0f9f180 (LWP 53532)):

0 __strlen_power7 () at ../sysdeps/powerpc/powerpc64/power7/strlen.S:32

No locals.

1 0x00003fffb28eafc4 in _IO_vfprintf_internal (s=0x3fffb0f9e2b0, format=0x3aa63d40 "%4.2f %s", ap=0x3fffb0f9e498 "\300\340\245:")

at vfprintf.c:1632
    len = <optimized out>
    string_malloced = 0
    step0_jumps = {0, 1708, 1868, 1956, 2052, 2140, 2240, 2484, 2744, 2972, 3316, 3552, 6452, 6544, 6820, 6876, 6976, 6988, 1140, 
      5584, 3652, -556, 6344, 3892, 1000, 56, 6080, 6636, 6728, 2396}
    space = <optimized out>
    is_short = <optimized out>
    use_outdigits = <optimized out>
    step1_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 2972, 3316, 3552, 6452, 6544, 6820, 6876, 6976, 6988, 1140, 5584, 3652, -556, 6344, 
      3892, 1000, 56, 6080, 6636, 6728, 0}
    group = <optimized out>
    prec = -1
    step2_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3316, 3552, 6452, 6544, 6820, 6876, 6976, 6988, 1140, 5584, 3652, -556, 6344, 
      3892, 1000, 56, 6080, 6636, 6728, 0}
    string = <optimized out>
    left = 0
    is_long_double = <optimized out>
    width = <optimized out>
    step3a_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3456, 0, 0, 0, 6820, 6876, 6976, 6988, 1140, 0, 0, 0, 0, 3892, 0, 0, 0, 0, 0, 0}
    alt = <optimized out>
    showsign = <optimized out>
    is_long = <optimized out>
    is_char = <optimized out>
    pad = <optimized out>
    step3b_jumps = {0 <repeats 11 times>, 6452, 0, 0, 6820, 6876, 6976, 6988, 1140, 5584, 3652, -556, 6344, 3892, 1000, 56, 6080, 
      0, 0, 0}
    step4_jumps = {0 <repeats 14 times>, 6820, 6876, 6976, 6988, 1140, 5584, 3652, -556, 6344, 3892, 1000, 56, 6080, 0, 0, 0}
    is_negative = <optimized out>
    number = <optimized out>
    base = <optimized out>
    the_arg = {pa_wchar = 672094848 L'\x280f5a80', pa_int = 672094848, pa_long_int = 91226957170039424, 
      pa_long_long_int = 91226957170039424, pa_u_int = 672094848, pa_u_long_int = 91226957170039424, 
      pa_u_long_long_int = 91226957170039424, pa_double = 1.4657531154060746e-302, pa_long_double = <invalid float value>, 
      pa_string = 0x1441a6f280f5a80 <error: Cannot access memory at address 0x1441a6f280f5a80>, 
      pa_wstring = 0x1441a6f280f5a80 <error: Cannot access memory at address 0x1441a6f280f5a80>, pa_pointer = 0x1441a6f280f5a80, 
      pa_user = 0x1441a6f280f5a80}
    spec = <optimized out>
    _buffer = {__routine = 0x3fffb0f9e0c0, __arg = 0x3fffb0f9e438, __canceltype = 983973192, __prev = 0x0}
    _avail = <optimized out>
    thousands_sep = <optimized out>
    grouping = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>
    done = 5
    f = 0x3aa63d47 "s"
    lead_str_end = 0x3aa63d40 "%4.2f %s"
    end_of_spec = <optimized out>
    work_buffer = " \335\371\260\377?\000\000\000\000\000\000\000\000\000\000\220\336\371\260\377?\000\000\210H\000H\000\000\000\000\300\b\000\230\377?\000\000\062\000\000\000\000\000\000\000\200\335\371\260\377?\000\000\002\000\000\000\377?\000\000\354\340\231\262\377?\000\000\001", '\000' <repeats 15 times>, "\062\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\300\b\000\230\377?\000\000\062\000\000\000\000\000\000\000\006\000\000\000\000\000\000\000\271\071\225\f\000\300\022\000\300\027\245\262\377?\000\000\300\335\371\260\377?\000\000\210H\000H\000\000\000\000\354}\221\262\377?\000\000 \335\371\260\377?\000\000\244$\246:\000\000\000\000\300\b\000\230\377?\000\000\062\000\000\000\000\000\000\000"...
    workstart = <optimized out>
    workend = <optimized out>
    ap_save = 0x3fffb0f9e488 "\200Z\017(o\032D\001\300\340\002"
    nspecs_done = 1
    save_errno = 0
    readonly_format = 0
    __PRETTY_FUNCTION__ = "_IO_vfprintf_internal"

2 0x00003fffb29c8a34 in ___vsnprintf_chk (s=0x3aa765b8 <buffer> "0.00 GByte", maxlen=<optimized out="">, flags=<optimized out="">,</optimized></optimized></buffer>

slen=<optimized out>, format=0x3aa63d40 "%4.2f %s", args=0x3fffb0f9e488 "\200Z\017(o\032D\001\300\340\002") at vsnprintf_chk.c:63
    sf = {f = {_sbf = {_f = {_flags = -72515583, _IO_read_ptr = 0x3aa765b8 <buffer> "0.00 GByte", 
            _IO_read_end = 0x3aa765b8 <buffer> "0.00 GByte", _IO_read_base = 0x3aa765b8 <buffer> "0.00 GByte", 
            _IO_write_base = 0x3aa765b8 <buffer> "0.00 GByte", _IO_write_ptr = 0x3aa765bd <buffer+5> "GByte", 
            _IO_write_end = 0x3aa765d7 <buffer+31> "", _IO_buf_base = 0x3aa765b8 <buffer> "0.00 GByte", 
            _IO_buf_end = 0x3aa765d7 <buffer+31> "", _IO_save_base = 0x0, _IO_backup_base = 0x0, _IO_save_end = 0x0, 
            _markers = 0x0, _chain = 0x0, _fileno = 102, _flags2 = 4, _old_offset = 32, _cur_column = 0, 
            _vtable_offset = -107 '\225', _shortbuf = "\f", _lock = 0x0, _offset = 0, _codecvt = 0x3fffb0f9e330, 
            _wide_data = 0xffffffffffffffff, _freeres_list = 0x0, _freeres_buf = 0x0, __pad5 = 0, _mode = -1, 
            _unused2 = '\000' <repeats 12 times>, " \000\000\000\000\000\000"}, vtable = 0x3fffb2a4ead0 <_IO_strn_jumps>}, _s = {
          _allocate_buffer = 0x0, _free_buffer = 0x0}}, 
      overflow_buf = "0\343\371\260\377?", '\000' <repeats 26 times>, "X\343\371\260\377?\000\000\340\370\371\260\377?\000\000\370\342\371\260\377?\000\000\000\371\371\260\377?\000"}
    ret = <optimized out>

3 0x00003fffb29c8938 in ___snprintf_chk (s=<optimized out="">, maxlen=<optimized out="">, flags=<optimized out="">, slen=<optimized out="">,</optimized></optimized></optimized></optimized>

format=<optimized out>) at snprintf_chk.c:34
    arg = <optimized out>
    done = <optimized out>

4 0x000000003aa60448 in ?? ()

No symbol table info available.

5 0x000000003aa57c1c in ?? ()

No symbol table info available.

6 0x000000003aa5a100 in ?? ()

No symbol table info available.

7 0x000000003aa5a4e0 in ?? ()

No symbol table info available.

8 0x000000003aa5af4c in ?? ()

No symbol table info available.

9 0x000000003aa5b258 in ?? ()

No symbol table info available.

10 0x000000003aa60c48 in ?? ()

No symbol table info available.

11 0x00003fffb2a68070 in start_thread (arg=0x3fffb0f9f180) at pthread_create.c:335

    pd = 0x3fffb0f9f180
    now = <optimized out>
    unwind_buf = <error reading variable unwind_buf (Cannot access memory at address 0x124)>
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[Robert McMahon]

Hi Jeff,

Is it possible to try the latest code, i.e. use

git clone https://git.code.sf.net/p/iperf2/code iperf2-code

Then compile with symbols using from the iperf2-code directory:

./configure
make CFLAGS=-g CXXFLAGS=-g

Also, post the config.h file?

Thanks,
Bob

[Robert McMahon]

update to iperf 2.0.14