Sean Farner

Sorry for the delay, we ended up altering our test procedure to work around the issue as much as we could.(We started 64 servers to spread the load) I can also provide a little more insight on what is happening after digging in to the network side a little more. During our network exercise, we encountered this problem primarily on UDP downlink testing (-R) When we had 400-600 connections behind a NAT, whether it be a router or the EPC of a cellular network, there appears to be a problem with getting...

Sorry for the delay, we ended up altering our test procedure to work around the issue as much as we could. I can also provide a little more insight on what is happening after digging in to the network side a little more. During our network exercise, we encountered this problem primarily on UDP downlink testing (-R) When we had 400-600 connections behind a NAT, whether it be a router or the EPC of a cellular network, there appears to be a problem with getting the socket FIN packet through that tells...

I am familiar with valgrind. I'll take a look as soon as I get a chance.

We was able to recreate this with Alma 8.4 as well 4.18 kernel. Some additional information: We used a third box with Centos 7 behind a different NAT and were able to recreate the issue. Easy way to test, put the client behind a router and spawn 600 connections that repeat every 5 seconds. There will be a lack of reporting and you should see the memory usage of the process grow. We may have found a work around for our test by interleaving TCP uplink and UDP uplink tests. If we follow a TCP uplink...

Ubuntu 22.04 kernel 6.2.1-060201-generic

When we remove the -yC and replaced it with --reportstyle C it showed reports of 1 device across 17 uplink transfers, the other 39 devices were held back until Ctl-C We then removed the -yC all together, we got the first part of the server message: [ 24] local 172.16.2.200 port 5003 connected with 172.16.10.135 port 13177 (icwnd/mss/irtt=12/1288/45059) but the [ ID] Interval Transfer Bandwidth [157] 0.00-3.26 sec 440 KBytes 1.11 Mbits/sec [158] 0.00-2.75 sec 800 KBytes 2.38 Mbits/sec [159] 0.00-2.41...

We also tried breaking it up to 2 servers, one for downlink and one for uplink on another port: This partially fixes the problem. We are able to continue receiving uplink and downlink data so far (currently around the 115th iteration out of 500), but we only have seen the first output on the uplink test server. I assume that we wil probably see the rest after we hit Ctrl-C We are seeing another issue as well, 2 processes with the same iperf -s -w 1M -yC -e -p 5003 command are at 100% CPU usage and...

Additionally the memory used on the server continues to grow until it reaches this point, at which 3.8G have been used. For Reference System Specs: 12 core system with 30GB of ram. Also the Server output never shows the uplink output until Ctl-C Edit: Took a closer look with top and 14.1G Virtual memory in use and 101% of a single core when this state is reached.