I have a machine that can drive 50Gbps on the NIC and 10Mpps with a 5Gbps limit per flow. When I try to stress test it with multicast I can't. I performed the following tests.
###
### 200 Bytes
###
root@ip-10-0-164-135:~# iperf -c 10.0.165.218 -u -e -b 5g -l 200 -t 20 -T 2
------------------------------------------------------------
Client connecting to 10.0.165.218, UDP port 5001 with pid 42869 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 200 byte datagrams, IPG target: 0.32 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 50362 connected with 10.0.165.218 port 5001 (sock=3) on 2023-03-15 23:44:08.583 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 2.30 GBytes 988 Mbits/sec 12352412/0 617621 pps
[ 1] Sent 12352413 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
WARN: Reporter thread may be too slow, await counter=1235, consider increasing NUM_REPORT_STRUCTS
################################
root@ip-10-0-164-135:~# iperf -c 224.1.2.1 -u -e -b 5g -l 200 -t 20 -T 2
------------------------------------------------------------
Client connecting to 224.1.2.1, UDP port 5001 with pid 42877 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 200 byte datagrams, IPG target: 0.32 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 47497 connected with 224.1.2.1 port 5001 (sock=3) on 2023-03-15 23:45:06.715 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 1.19 GBytes 509 Mbits/sec 6363107/0 318153 pps
[ 1] Sent 6363109 datagrams
###
### 250 Bytes
###
root@ip-10-0-164-135:~# iperf -c 10.0.165.218 -u -e -b 5g -l 250 -t 20 -T 2
------------------------------------------------------------
Client connecting to 10.0.165.218, UDP port 5001 with pid 42882 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 250 byte datagrams, IPG target: 0.40 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 38572 connected with 10.0.165.218 port 5001 (sock=3) on 2023-03-15 23:45:59.982 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 2.88 GBytes 1.24 Gbits/sec 12354408/0 617720 pps
[ 1] Sent 12354409 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
WARN: Reporter thread may be too slow, await counter=1235, consider increasing NUM_REPORT_STRUCTS
################################
root@ip-10-0-164-135:~# iperf -c 224.1.2.1 -u -e -b 5g -l 250 -t 20 -T 2
------------------------------------------------------------
Client connecting to 224.1.2.1, UDP port 5001 with pid 42886 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 250 byte datagrams, IPG target: 0.40 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 58790 connected with 224.1.2.1 port 5001 (sock=3) on 2023-03-15 23:46:51.854 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 1.44 GBytes 620 Mbits/sec 6204731/0 310237 pps
[ 1] Sent 6204733 datagrams
###
### 300 Bytes
###
root@ip-10-0-164-135:~# iperf -c 10.0.165.218 -u -e -b 5g -l 300 -t 20 -T 2
------------------------------------------------------------
Client connecting to 10.0.165.218, UDP port 5001 with pid 42893 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 300 byte datagrams, IPG target: 0.48 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 42124 connected with 10.0.165.218 port 5001 (sock=3) on 2023-03-15 23:48:27.434 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 3.45 GBytes 1.48 Gbits/sec 12351747/0 617587 pps
[ 1] Sent 12351748 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
WARN: Reporter thread may be too slow, await counter=1235, consider increasing NUM_REPORT_STRUCTS
################################
root@ip-10-0-164-135:~# iperf -c 224.1.2.1 -u -e -b 5g -l 300 -t 20 -T 2
------------------------------------------------------------
Client connecting to 224.1.2.1, UDP port 5001 with pid 42897 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 300 byte datagrams, IPG target: 0.48 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 39680 connected with 224.1.2.1 port 5001 (sock=3) on 2023-03-15 23:49:20.606 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 1.79 GBytes 768 Mbits/sec 6403652/0 320180 pps
[ 1] Sent 6403654 datagrams
###
### 500 Bytes
###
root@ip-10-0-164-135:~# iperf -c 10.0.165.218 -u -e -b 5g -l 500 -t 20 -T 2
------------------------------------------------------------
Client connecting to 10.0.165.218, UDP port 5001 with pid 42904 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 500 byte datagrams, IPG target: 0.80 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 44167 connected with 10.0.165.218 port 5001 (sock=3) on 2023-03-15 23:50:41.484 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 5.75 GBytes 2.47 Gbits/sec 12351860/0 617593 pps
[ 1] Sent 12351861 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
WARN: Reporter thread may be too slow, await counter=1235, consider increasing NUM_REPORT_STRUCTS
################################
root@ip-10-0-164-135:~# iperf -c 224.1.2.1 -u -e -b 5g -l 500 -t 20 -T 2
------------------------------------------------------------
Client connecting to 224.1.2.1, UDP port 5001 with pid 42908 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 500 byte datagrams, IPG target: 0.80 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 38793 connected with 224.1.2.1 port 5001 (sock=3) on 2023-03-15 23:51:27.444 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 2.94 GBytes 1.26 Gbits/sec 6319679/0 315982 pps
[ 1] Sent 6319681 datagrams
###
### 700 Bytes
###
root@ip-10-0-164-135:~# iperf -c 10.0.165.218 -u -e -b 5g -l 700 -t 20 -T 2
------------------------------------------------------------
Client connecting to 10.0.165.218, UDP port 5001 with pid 42912 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 700 byte datagrams, IPG target: 1.12 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 36802 connected with 10.0.165.218 port 5001 (sock=3) on 2023-03-15 23:52:17.429 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 8.06 GBytes 3.46 Gbits/sec 12357590/0 617879 pps
[ 1] Sent 12357591 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
WARN: Reporter thread may be too slow, await counter=1235, consider increasing NUM_REPORT_STRUCTS
################################
root@ip-10-0-164-135:~# iperf -c 224.1.2.1 -u -e -b 5g -l 700 -t 20 -T 2
------------------------------------------------------------
Client connecting to 224.1.2.1, UDP port 5001 with pid 42916 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 700 byte datagrams, IPG target: 1.12 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 40059 connected with 224.1.2.1 port 5001 (sock=3) on 2023-03-15 23:52:54.095 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 4.13 GBytes 1.77 Gbits/sec 6328553/0 316426 pps
[ 1] Sent 6328555 datagrams
###
### 1000 Bytes
###
root@ip-10-0-164-135:~# iperf -c 10.0.165.218 -u -e -b 5g -l 1000 -t 20 -T 2
------------------------------------------------------------
Client connecting to 10.0.165.218, UDP port 5001 with pid 42919 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 1000 byte datagrams, IPG target: 1.60 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 57456 connected with 10.0.165.218 port 5001 (sock=3) on 2023-03-15 23:53:41.781 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 11.2 GBytes 4.80 Gbits/sec 11989499/0 599475 pps
[ 1] Sent 11989500 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
################################
root@ip-10-0-164-135:~# iperf -c 224.1.2.1 -u -e -b 5g -l 1000 -t 20 -T 2
------------------------------------------------------------
Client connecting to 224.1.2.1, UDP port 5001 with pid 42923 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 1000 byte datagrams, IPG target: 1.60 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 48536 connected with 224.1.2.1 port 5001 (sock=3) on 2023-03-15 23:54:21.305 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 5.87 GBytes 2.52 Gbits/sec 6302025/0 315100 pps
[ 1] Sent 6302027 datagrams
###
### 1500 Bytes
###
root@ip-10-0-164-135:~# iperf -c 10.0.165.218 -u -e -b 5g -l 1500 -t 20 -T 2
------------------------------------------------------------
Client connecting to 10.0.165.218, UDP port 5001 with pid 42931 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 1500 byte datagrams, IPG target: 2.40 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 51157 connected with 10.0.165.218 port 5001 (sock=3) on 2023-03-15 23:55:08.733 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 11.3 GBytes 4.87 Gbits/sec 8108660/0 405431 pps
[ 1] Sent 8108662 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
################################
root@ip-10-0-164-135:~# iperf -c 224.1.2.1 -u -e -b 5g -l 1500 -t 20 -T 2
------------------------------------------------------------
Client connecting to 224.1.2.1, UDP port 5001 with pid 42935 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 1500 byte datagrams, IPG target: 2.40 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 58491 connected with 224.1.2.1 port 5001 (sock=3) on 2023-03-15 23:55:49.700 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 8.62 GBytes 3.70 Gbits/sec 6169914/0 308495 pps
[ 1] Sent 6169916 datagrams
###
### 8000 Bytes
###
root@ip-10-0-164-135:~# iperf -c 10.0.165.218 -u -e -b 5g -l 8000 -t 20 -T 2
------------------------------------------------------------
Client connecting to 10.0.165.218, UDP port 5001 with pid 42940 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 8000 byte datagrams, IPG target: 12.80 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 38545 connected with 10.0.165.218 port 5001 (sock=3) on 2023-03-15 23:56:52.792 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 11.6 GBytes 4.98 Gbits/sec 1554775/0 77739 pps
[ 1] Sent 1554777 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
################################
root@ip-10-0-164-135:~# iperf -c 224.1.2.1 -u -e -b 5g -l 8000 -t 20 -T 2
------------------------------------------------------------
Client connecting to 224.1.2.1, UDP port 5001 with pid 42944 (1 flows)
TOS set to 0x0 (Nagle on)
Sending 8000 byte datagrams, IPG target: 12.80 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 10.0.164.135%ens5 port 52781 connected with 224.1.2.1 port 5001 (sock=3) on 2023-03-15 23:57:30.862 (UTC)
[ ID] Interval Transfer Bandwidth Write/Err PPS
[ 1] 0.00-20.00 sec 11.6 GBytes 4.98 Gbits/sec 1554774/0 77739 pps
[ 1] Sent 1554776 datagrams
For 200, 250, 300, 500 and 700 Bytes Unicast I am CPU bound and understandably cannot drive the 5Gbps per flow, on 1000, 1500 and 8000bytes I can achieve the 5Gbps.
However with Multicast I cannot get close to the unicast numbers and CPU does not hit 100%. Can someone explain this behavior?
Log in to post a comment.
