Menu
▾
▴
e1000-devel
|
From: Auer, J. <jen...@cg...> - 2016-08-26 10:51:43
|
Hi, we are experiencing packet reordering and thus bandwidth drops due to retransmissions on our servers. We are using RedHat 7.2 on HP Servers connected to a HP Aruba 1920 10GB switch. I can easily reproduce the issues by running iperf3 between the two servers: iperf3 -c 192.168.120.11 -p 12345 -b200m -P20 -t0 Connecting to host 192.168.120.11, port 12345 [ 4] local 192.168.120.10 port 59638 connected to 192.168.120.11 port 12345 [ 6] local 192.168.120.10 port 59640 connected to 192.168.120.11 port 12345 [ 8] local 192.168.120.10 port 59642 connected to 192.168.120.11 port 12345 [ 10] local 192.168.120.10 port 59644 connected to 192.168.120.11 port 12345 [ 12] local 192.168.120.10 port 59646 connected to 192.168.120.11 port 12345 [ 14] local 192.168.120.10 port 59648 connected to 192.168.120.11 port 12345 [ 16] local 192.168.120.10 port 59650 connected to 192.168.120.11 port 12345 [ 18] local 192.168.120.10 port 59652 connected to 192.168.120.11 port 12345 [ 20] local 192.168.120.10 port 59654 connected to 192.168.120.11 port 12345 [ 22] local 192.168.120.10 port 59656 connected to 192.168.120.11 port 12345 [ 24] local 192.168.120.10 port 59658 connected to 192.168.120.11 port 12345 [ 26] local 192.168.120.10 port 59660 connected to 192.168.120.11 port 12345 [ 28] local 192.168.120.10 port 59662 connected to 192.168.120.11 port 12345 [ 30] local 192.168.120.10 port 59664 connected to 192.168.120.11 port 12345 [ 32] local 192.168.120.10 port 59666 connected to 192.168.120.11 port 12345 [ 34] local 192.168.120.10 port 59668 connected to 192.168.120.11 port 12345 [ 36] local 192.168.120.10 port 59670 connected to 192.168.120.11 port 12345 [ 38] local 192.168.120.10 port 59672 connected to 192.168.120.11 port 12345 [ 40] local 192.168.120.10 port 59674 connected to 192.168.120.11 port 12345 [ 42] local 192.168.120.10 port 59676 connected to 192.168.120.11 port 12345 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 2 225 KBytes [ 6] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 1 202 KBytes [ 8] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 8 225 KBytes [ 10] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 2 132 KBytes [ 12] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 2 133 KBytes [ 14] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 2 132 KBytes [ 16] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 12 165 KBytes [ 18] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 2 151 KBytes [ 20] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 1 150 KBytes [ 22] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 4 225 KBytes [ 24] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 5 143 KBytes [ 26] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 3 132 KBytes [ 28] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 3 126 KBytes [ 30] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 1 127 KBytes [ 32] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 6 136 KBytes [ 34] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 2 139 KBytes [ 36] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 1 154 KBytes [ 38] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 1 136 KBytes [ 40] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 2 122 KBytes [ 42] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 4 139 KBytes [SUM] 0.00-1.00 sec 438 MBytes 3.67 Gbits/sec 64 - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 255 KBytes [ 6] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 243 KBytes [ 8] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 256 KBytes [ 10] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 191 KBytes [ 12] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 192 KBytes [ 14] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 192 KBytes [ 16] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 216 KBytes [ 18] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 208 KBytes [ 20] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 205 KBytes [ 22] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 259 KBytes [ 24] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 202 KBytes [ 26] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 198 KBytes [ 28] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 191 KBytes [ 30] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 192 KBytes [ 32] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 201 KBytes [ 34] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 198 KBytes [ 36] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 211 KBytes [ 38] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 198 KBytes [ 40] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 194 KBytes [ 42] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec 0 198 KBytes [SUM] 1.00-2.00 sec 480 MBytes 4.03 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 267 KBytes [ 6] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 250 KBytes [ 8] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 269 KBytes [ 10] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 201 KBytes [ 12] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 204 KBytes [ 14] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 2 202 KBytes [ 16] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 223 KBytes [ 18] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 216 KBytes [ 20] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 215 KBytes [ 22] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 270 KBytes [ 24] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 212 KBytes [ 26] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 208 KBytes [ 28] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 201 KBytes [ 30] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 202 KBytes [ 32] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 211 KBytes [ 34] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 208 KBytes [ 36] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 219 KBytes [ 38] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 209 KBytes [ 40] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 211 KBytes [ 42] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 211 KBytes [SUM] 2.00-3.00 sec 475 MBytes 3.99 Gbits/sec 2 - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 5 205 KBytes [ 6] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 260 KBytes [ 8] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 276 KBytes [ 10] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 221 KBytes [ 12] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 225 KBytes [ 14] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 221 KBytes [ 16] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 243 KBytes [ 18] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 230 KBytes [ 20] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 229 KBytes [ 22] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 281 KBytes [ 24] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 233 KBytes [ 26] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 226 KBytes [ 28] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 222 KBytes [ 30] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 218 KBytes [ 32] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 232 KBytes [ 34] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 229 KBytes [ 36] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 239 KBytes [ 38] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 229 KBytes [ 40] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 226 KBytes [ 42] 3.00-4.00 sec 23.9 MBytes 200 Mbits/sec 0 225 KBytes [SUM] 3.00-4.00 sec 478 MBytes 4.00 Gbits/sec 5 - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 1 212 KBytes [ 6] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 267 KBytes [ 8] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 283 KBytes [ 10] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 228 KBytes [ 12] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 232 KBytes [ 14] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 226 KBytes [ 16] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 249 KBytes [ 18] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 236 KBytes [ 20] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 235 KBytes [ 22] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 287 KBytes [ 24] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 239 KBytes [ 26] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 233 KBytes [ 28] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 230 KBytes [ 30] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 232 KBytes [ 32] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 238 KBytes [ 34] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 236 KBytes [ 36] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 247 KBytes [ 38] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 238 KBytes [ 40] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 233 KBytes [ 42] 4.00-5.00 sec 23.8 MBytes 199 Mbits/sec 0 232 KBytes [SUM] 4.00-5.00 sec 475 MBytes 3.99 Gbits/sec 1 ^C- - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 212 KBytes [ 6] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 267 KBytes [ 8] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 283 KBytes [ 10] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 228 KBytes [ 12] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 232 KBytes [ 14] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 226 KBytes [ 16] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 249 KBytes [ 18] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 236 KBytes [ 20] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 235 KBytes [ 22] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 287 KBytes [ 24] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 239 KBytes [ 26] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 233 KBytes [ 28] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 230 KBytes [ 30] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 232 KBytes [ 32] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 238 KBytes [ 34] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 236 KBytes [ 36] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 247 KBytes [ 38] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 238 KBytes [ 40] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 233 KBytes [ 42] 5.00-5.32 sec 9.50 MBytes 251 Mbits/sec 0 232 KBytes [SUM] 5.00-5.32 sec 190 MBytes 5.02 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 8 sender [ 4] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 6] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 6] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 8] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 8 sender [ 8] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 10] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 2 sender [ 10] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 12] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 2 sender [ 12] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 14] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 4 sender [ 14] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 16] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 12 sender [ 16] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 18] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 2 sender [ 18] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 20] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 20] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 22] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 4 sender [ 22] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 24] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 5 sender [ 24] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 26] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 3 sender [ 26] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 28] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 3 sender [ 28] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 30] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 30] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 32] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 6 sender [ 32] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 34] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 2 sender [ 34] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 36] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 36] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 38] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 38] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 40] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 2 sender [ 40] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 42] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 4 sender [ 42] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [SUM] 0.00-5.32 sec 2.48 GBytes 4.00 Gbits/sec 72 sender [SUM] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver iperf3: interrupt - the client has terminated In this test, I get some retransmissions every couple of seconds. We analyzed the retransmissions by capturing a sample with Wireshark and we can confirm that the retransmissions are fast retransmissions caused by reordering of the traffic at the receiving server only. I have found a description of problems caused by Flow Director and migrating processes from Fermilab (https://arxiv.org/pdf/1106.0443.pdf) together with a support case at RedHat (https://access.redhat.com/solutions/2403071). What is offical position concerning this issue? Is it considered as bug? The RedHat case recommends to either switch to Perfect Filtering or fix CPU assignment of processes. I have tried both solutions, but the first one still produces retransmissions even when pinning both iperf processes to a fixed CPU: taskset 0x1 iperf3 -s -p 12345 ----------------------------------------------------------- Server listening on 12345 ----------------------------------------------------------- Accepted connection from 192.168.120.10, port 59720 [ 5] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59722 [ 7] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59724 [ 9] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59726 [ 11] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59728 [ 13] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59730 [ 15] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59732 [ 17] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59734 [ 19] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59736 [ 21] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59738 [ 23] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59740 [ 25] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59742 [ 27] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59744 [ 29] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59746 [ 31] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59748 [ 33] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59750 [ 35] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59752 [ 37] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59754 [ 39] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59756 [ 41] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59758 [ 43] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59760 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 7] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 9] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 11] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 13] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 15] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 17] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 19] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 21] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 23] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 25] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 27] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 29] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 31] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 33] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 35] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 37] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 39] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 41] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [ 43] 0.00-1.00 sec 21.8 MBytes 182 Mbits/sec [SUM] 0.00-1.00 sec 435 MBytes 3.65 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 7] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 9] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 11] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 13] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 15] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 17] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 19] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 21] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 23] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 25] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 27] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 29] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 31] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 33] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 35] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 37] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 39] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 41] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [ 43] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec [SUM] 1.00-2.00 sec 478 MBytes 4.01 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 7] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 9] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 11] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 13] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 15] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 17] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 19] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 21] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 23] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 25] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 27] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 29] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 31] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 33] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 35] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 37] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 39] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 41] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [ 43] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec [SUM] 2.00-3.00 sec 478 MBytes 4.01 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 7] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 9] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 11] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 13] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 15] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 17] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 19] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 21] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 23] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 25] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 27] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 29] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 31] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 33] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 35] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 37] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 39] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 41] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [ 43] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec [SUM] 3.00-4.00 sec 475 MBytes 3.98 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 7] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 9] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 11] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 13] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 15] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 17] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 19] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 21] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 23] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 25] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 27] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 29] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 31] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 33] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 35] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 37] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 39] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 41] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 43] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [SUM] 4.00-5.00 sec 478 MBytes 4.01 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 7] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 9] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 11] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 13] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 15] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 17] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 19] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 21] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 23] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 25] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 27] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 29] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 31] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 33] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 35] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 37] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 39] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 41] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [ 43] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec [SUM] 4.00-5.00 sec 478 MBytes 4.01 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 7] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 7] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 9] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 9] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 11] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 11] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 13] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 13] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 15] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 15] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 17] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 17] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 19] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 19] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 21] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 21] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 23] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 23] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 25] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 25] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 27] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 27] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 29] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 29] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 31] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 31] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 33] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 33] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 35] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 35] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 37] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 37] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 39] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 39] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 41] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 41] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [ 43] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [ 43] 0.00-5.00 sec 125 MBytes 210 Mbits/sec receiver [SUM] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec sender [SUM] 0.00-5.00 sec 2.45 GBytes 4.20 Gbits/sec receiver iperf3: the client has terminated taskset 0x1 iperf3 -c 192.168.120.11 -p 12345 -b200m -P20 -t0 Connecting to host 192.168.120.11, port 12345 [ 4] local 192.168.120.10 port 59722 connected to 192.168.120.11 port 12345 [ 6] local 192.168.120.10 port 59724 connected to 192.168.120.11 port 12345 [ 8] local 192.168.120.10 port 59726 connected to 192.168.120.11 port 12345 [ 10] local 192.168.120.10 port 59728 connected to 192.168.120.11 port 12345 [ 12] local 192.168.120.10 port 59730 connected to 192.168.120.11 port 12345 [ 14] local 192.168.120.10 port 59732 connected to 192.168.120.11 port 12345 [ 16] local 192.168.120.10 port 59734 connected to 192.168.120.11 port 12345 [ 18] local 192.168.120.10 port 59736 connected to 192.168.120.11 port 12345 [ 20] local 192.168.120.10 port 59738 connected to 192.168.120.11 port 12345 [ 22] local 192.168.120.10 port 59740 connected to 192.168.120.11 port 12345 [ 24] local 192.168.120.10 port 59742 connected to 192.168.120.11 port 12345 [ 26] local 192.168.120.10 port 59744 connected to 192.168.120.11 port 12345 [ 28] local 192.168.120.10 port 59746 connected to 192.168.120.11 port 12345 [ 30] local 192.168.120.10 port 59748 connected to 192.168.120.11 port 12345 [ 32] local 192.168.120.10 port 59750 connected to 192.168.120.11 port 12345 [ 34] local 192.168.120.10 port 59752 connected to 192.168.120.11 port 12345 [ 36] local 192.168.120.10 port 59754 connected to 192.168.120.11 port 12345 [ 38] local 192.168.120.10 port 59756 connected to 192.168.120.11 port 12345 [ 40] local 192.168.120.10 port 59758 connected to 192.168.120.11 port 12345 [ 42] local 192.168.120.10 port 59760 connected to 192.168.120.11 port 12345 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 7 189 KBytes [ 6] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 7 122 KBytes [ 8] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 1 134 KBytes [ 10] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 0 158 KBytes [ 12] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 2 130 KBytes [ 14] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 1 168 KBytes [ 16] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 5 133 KBytes [ 18] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 0 181 KBytes [ 20] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 1 132 KBytes [ 22] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 5 129 KBytes [ 24] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 4 132 KBytes [ 26] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 4 133 KBytes [ 28] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 2 127 KBytes [ 30] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 0 173 KBytes [ 32] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 9 132 KBytes [ 34] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 3 130 KBytes [ 36] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 4 136 KBytes [ 38] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 1 130 KBytes [ 40] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 0 168 KBytes [ 42] 0.00-1.00 sec 21.9 MBytes 183 Mbits/sec 6 127 KBytes [SUM] 0.00-1.00 sec 438 MBytes 3.67 Gbits/sec 62 - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 189 KBytes [ 6] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 130 KBytes [ 8] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 134 KBytes [ 10] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 158 KBytes [ 12] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 134 KBytes [ 14] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 168 KBytes [ 16] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 3 148 KBytes [ 18] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 191 KBytes [ 20] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 141 KBytes [ 22] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 143 KBytes [ 24] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 134 KBytes [ 26] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 139 KBytes [ 28] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 129 KBytes [ 30] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 1 173 KBytes [ 32] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 133 KBytes [ 34] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 141 KBytes [ 36] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 137 KBytes [ 38] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 141 KBytes [ 40] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 168 KBytes [ 42] 1.00-2.00 sec 23.9 MBytes 200 Mbits/sec 0 133 KBytes [SUM] 1.00-2.00 sec 478 MBytes 4.01 Gbits/sec 4 - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 189 KBytes [ 6] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 130 KBytes [ 8] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 134 KBytes [ 10] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 158 KBytes [ 12] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 134 KBytes [ 14] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 168 KBytes [ 16] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 151 KBytes [ 18] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 211 KBytes [ 20] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 157 KBytes [ 22] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 157 KBytes [ 24] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 137 KBytes [ 26] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 147 KBytes [ 28] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 4 129 KBytes [ 30] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 173 KBytes [ 32] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 133 KBytes [ 34] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 143 KBytes [ 36] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 137 KBytes [ 38] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 144 KBytes [ 40] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 168 KBytes [ 42] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0 136 KBytes [SUM] 2.00-3.00 sec 478 MBytes 4.01 Gbits/sec 4 - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 189 KBytes [ 6] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 130 KBytes [ 8] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 134 KBytes [ 10] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 158 KBytes [ 12] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 134 KBytes [ 14] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 168 KBytes [ 16] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 1 156 KBytes [ 18] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 2 223 KBytes [ 20] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 163 KBytes [ 22] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 165 KBytes [ 24] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 139 KBytes [ 26] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 154 KBytes [ 28] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 130 KBytes [ 30] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 173 KBytes [ 32] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 134 KBytes [ 34] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 150 KBytes [ 36] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 137 KBytes [ 38] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 151 KBytes [ 40] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 168 KBytes [ 42] 3.00-4.00 sec 23.8 MBytes 199 Mbits/sec 0 146 KBytes [SUM] 3.00-4.00 sec 475 MBytes 3.98 Gbits/sec 3 - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 189 KBytes [ 6] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 130 KBytes [ 8] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 134 KBytes [ 10] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 1 158 KBytes [ 12] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 134 KBytes [ 14] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 168 KBytes [ 16] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 1 161 KBytes [ 18] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 223 KBytes [ 20] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 174 KBytes [ 22] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 175 KBytes [ 24] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 139 KBytes [ 26] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 154 KBytes [ 28] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 132 KBytes [ 30] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 173 KBytes [ 32] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 134 KBytes [ 34] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 150 KBytes [ 36] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 137 KBytes [ 38] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 151 KBytes [ 40] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 168 KBytes [ 42] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 146 KBytes [SUM] 4.00-5.00 sec 478 MBytes 4.01 Gbits/sec 2 ^C- - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 189 KBytes [ 6] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 130 KBytes [ 8] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 134 KBytes [ 10] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 158 KBytes [ 12] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 140 KBytes [ 14] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 168 KBytes [ 16] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 164 KBytes [ 18] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 223 KBytes [ 20] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 174 KBytes [ 22] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 175 KBytes [ 24] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 139 KBytes [ 26] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 154 KBytes [ 28] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 132 KBytes [ 30] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 173 KBytes [ 32] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 134 KBytes [ 34] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 150 KBytes [ 36] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 137 KBytes [ 38] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 151 KBytes [ 40] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 168 KBytes [ 42] 5.00-5.32 sec 9.38 MBytes 248 Mbits/sec 0 146 KBytes [SUM] 5.00-5.32 sec 188 MBytes 4.96 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 7 sender [ 4] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 6] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 7 sender [ 6] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 8] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 8] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 10] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 10] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 12] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 2 sender [ 12] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 14] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 14] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 16] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 10 sender [ 16] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 18] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 2 sender [ 18] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 20] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 20] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 22] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 5 sender [ 22] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 24] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 4 sender [ 24] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 26] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 4 sender [ 26] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 28] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 6 sender [ 28] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 30] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 30] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 32] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 9 sender [ 32] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 34] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 3 sender [ 34] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 36] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 4 sender [ 36] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 38] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 1 sender [ 38] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 40] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 0 sender [ 40] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [ 42] 0.00-5.32 sec 127 MBytes 200 Mbits/sec 6 sender [ 42] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver [SUM] 0.00-5.32 sec 2.47 GBytes 3.99 Gbits/sec 75 sender [SUM] 0.00-5.32 sec 0.00 Bytes 0.00 bits/sec receiver iperf3: interrupt - the client has terminated Switching to Perfect Forwarding mode fixed the problems and iperf runs without retransmissions. Are there other factors that can cause retransmissions, e.g. irq balancing at the receiver? Best wishes, Jens -- Jens Auer | CGI | Software-Engineer CGI (Germany) GmbH & Co. KG Rheinstraße 95 | 64295 Darmstadt | Germany T: +49 6151 36860 154 jen...@cg...<mailto:jen...@cg...> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>. CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail. |
|
From: Alexander D. <ale...@gm...> - 2016-08-26 14:26:21
|
On Fri, Aug 26, 2016 at 3:18 AM, Auer, Jens <jen...@cg...> wrote: > Hi, > > we are experiencing packet reordering and thus bandwidth drops due to retransmissions on our servers. We are using RedHat 7.2 on HP Servers connected to a HP Aruba 1920 10GB switch. I can easily reproduce the issues by running iperf3 between the two servers: <snipping out test results> > In this test, I get some retransmissions every couple of seconds. We analyzed the retransmissions by capturing a sample with Wireshark and we can confirm that the retransmissions are fast retransmissions caused by reordering of the traffic at the receiving server only. > > I have found a description of problems caused by Flow Director and migrating processes from Fermilab (https://arxiv.org/pdf/1106.0443.pdf) together with a support case at RedHat (https://access.redhat.com/solutions/2403071). What is offical position concerning this issue? Is it considered as bug? > > The RedHat case recommends to either switch to Perfect Filtering or fix CPU assignment of processes. I have tried both solutions, but the first one still produces retransmissions even when pinning both iperf processes to a fixed CPU: <snipping out some more test results> > > Switching to Perfect Forwarding mode fixed the problems and iperf runs without retransmissions. Are there other factors that can cause retransmissions, e.g. irq balancing at the receiver? > > Best wishes, > Jens Hi Jens, The main cause of the reordering is likely the ATR feature on the adapter. What ATR does is try to redirect all Rx traffic for a given flow back to the same Tx queue that the flow was received on. When combined with XPS which aligns the Tx queue selection with the CPU that the transmitting application is scheduled on this can lead to reordering. The reordering is due to the fact that the scheduler will move flows back and forth between CPUs. This in turn can lead to reordering if either the scheduler moves the application between CPUs or the IRQ balancing moves the Rx queue to a new CPU which leads to possible bouncing if acknowledgements are generated by the stack on a different CPU from the transmissions. On additional cause for possible issues is if you are running flows with a high connection rate. The filter table used for ATR has a limited amount of space available and if the table overflows it will cause the contents to be flushed. This also can lead to reordering related to ATR. Hope this helps to clarify some of what is going on. - Alex |
Thanks for helping keep SourceForge clean.
X