UDP streams w/ >1 client IP crash without -B set
A means to measure network responsiveness and throughput
Brought to you by:
rjmcmahon
Menu
▾
▴
#270 UDP streams w/ >1 client IP crash without -B set
The issue occurs when the client has >1 ip address, either ipv6 or ipv4 of which both are reachable
Tested on both 2-1-4 and 2-1-9.
This was observed on both Linux and BSD.
Linux: Generates ~800 sockets and crashes
BSD: Fails on listener bind failed or connect UDP failed after attempting to bind() or connect() with EADDRINUSE
# iperf -us &
[1] 57139233
# ------------------------------------------------------------
Server listening on UDP port 5001
UDP buffer size: 41.1 KByte (default)
------------------------------------------------------------
# [ 1] local 192.168.0.31 port 5001 connected with 192.168.0.2 port 45547
listener bind failed: Address already in use
[root@linux ~]$ iperf -uc 192.168.0.68 -P 3 -b 50M
[ 2] local 192.168.0.2 port 51433 connected with 192.168.0.68 port 5001
------------------------------------------------------------
Client connecting to 192.168.0.68, UDP port 5001
Sending 1470 byte datagrams, IPG target: 224.30 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.2 port 54779 connected with 192.168.0.68 port 5001
[ 1] local 192.168.0.2 port 45547 connected with 192.168.0.68 port 5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-10.0004 sec 62.5 MBytes 52.4 Mbits/sec
[ 1] Sent 44587 datagrams
[ 2] 0.0000-10.0004 sec 62.5 MBytes 52.4 Mbits/sec
[ 2] Sent 44587 datagrams
[ 3] 0.0000-10.0004 sec 62.5 MBytes 52.4 Mbits/sec
[ 3] Sent 44587 datagrams
[SUM] 0.0000-10.0004 sec 188 MBytes 157 Mbits/sec
[SUM-3] Sent 133761 datagrams
[ 4] WARNING: did not receive ack of last datagram after 10 tries.
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
[ 5] WARNING: did not receive ack of last datagram after 10 tries.
Seen above the server accepts the connection on .31 though the client assumes it connected on .68.
From what I've looked at the connect() call in udp_accept() looks like the cause, as it chooses the best routable ip, which may not be the one the client is addressing, so all further messages end up at the listening socket rather than the 'connected' UDP socket. Attempted to remedy this by creating a new socket on receipt to the listening socket but further messages just went to the listening socket.
This can be worked around by specifying -B on the server side.
Can you compile from master? I did some fixes around UDP accept after 2.1.9. There were regressions introduced and fixed after 2.1.9
Compiled from master (both client and server), had the same output on Linux except now it segfaults when the client uses the same address that the server's connect() binds
192.168.0.2 and 192.168.0.102 are on the server
Client
master server w/ client using 192.168.0.2
2.1.9 server w/ client using 192.168.0.2
master server w/ client using 192.168.0.102
Last edit: Ryan Campbell 2023-09-06
I introduced a regression in UDP recently too. Need the latest per this commit. You'll need to git pull to get it.
No longer segfaulting on master w/ 192.168.0.2, but still seeing the same behaviour w/ 192.168.0.102
Too many open files seems like an os thing vs an iperf thing (though iperf may have stale open files - hopefully not.) I'm not great a debug here but the output of lsof might help.
hmm, looking closer, the multiple opens to port 55538 are a problem. Not sure why that's happening.
Can you run the server after compiling with debug?
Thanks
Bob
Output is very chatty
Last edit: Robert McMahon 2023-09-06
It created about ~1000 sockets this time so from some investigation I believe the following happens:
Looking at the how the active table is called in udp_accept it would return true if the host was there but the port wasn't.
Attached the thread-debug output because it filled the terminal.
Only got the following - Killed it from another terminal
yes, this is how it's supposed to work, only new sockets on new ip quintuples. But then this is a problem as the quintuples are the same yet new udp accepts occurred on the same quintuple.
Last edit: Robert McMahon 2023-09-06
This should stop the duplicates
udp accept checks the full quintuple per this
In active_table_update() it will only add an entry if the host isn't present, otherwise it increases the thread count.
ie if we received the following in this order to Iperf_push_host_port_conditional
So all subsequent connections would go through as when it increases thread count there is no tracking of which port it just created.
Last edit: Ryan Campbell 2023-09-06
Hmm, that's not what I'm reading. I don't think iperf_push_host should ever be called for UDP only iperf_push_host_port_conditional. The output with --configure --enable-thread-debug would show it though.
Last edit: Robert McMahon 2023-09-06
Yeah it doesn't look like it's calling iperf_push_host, only the _conditional. I cut out the rest of the output above and kept only the first few 'active table' entries here. There's only 1 'new entry' in the file
There is only 1 new entry because the summing is per source ip and not source ip/port, so that's right. What I don't understand is all the connect messages for the same src ip/src port. Those seem wrong.
can you post the full output. Probably easiest to read it as a log and attachment if that can be done.
Looks like sourceforge was down this morning.
I attached the output above as output.txt
https://sourceforge.net/p/iperf2/tickets/270/#c266/5acf/8073/3c07/7ab9/b17b/8446
I see the problem and need to think about the best solution. The listener tread is receiving packets for the server thread per using recvmsg.
A work around is to use port ranges on the server and incr-dstport on the client.
Last edit: Robert McMahon 2023-09-08
As a note I tried keeping the listener bound to *:5001 w/ the same socket and creating the server UDP sockets as needed but a large number of messages still made it to the listener.
Top udp4 socket shouldn't have been made + 36 432 messages on the listening socket is not great
I think I found the bug. Can you try the latest as of Sept 8?
[rjmcmahon@fedora iperf2-code]$ src/iperf -v
iperf version 2.1.n (8 September 2023) pthreads
Bob
Crashed again - attached output w/ enable-thread-debug
Can you try the latest and provide the output again?
Tried latest and uploaded the output .
It didn't create too many sockets, but it didn't destroy them or receive traffic.
Note: I did realize that this only happens when the server's two NIC's have ip addresses that are on the same subnet, which is not an ideal configuration
Another note, I had to cut the output at 20k lines as it wouldn't let me upload the whole 50 MB file. Cut from about 401K lines
I assume this is because connect() knows both routes will reach the destination. so picks whichever it believes is best
what happens when you compile without --enable-thread-debug? The new code should drop the duplicate flows but new flows should work.
Also, is this linux? Can you share the output of uname -r to get the kernel version? Most systems won't send the same flow to two different threads after a connect.