#314 UDP accept race condition

[Robert McMahon]

There is a race between the listener thread and the server thread in the UDP accept.

Hi All,

A work around for this is to use -P 1 on the server. Can you try that? I believe UTF sets that by default. I also wonder if this is FC30 related though probably not.

There seems to be a race between the listener thread and the server thread on the recvfrom. I perf hangs a recvfrom in listener thread, connects the socket after a first packet arrives and then creates a new server thread to handle that traffic. The Listener thread then hangs another recvfrom. (This subsequent recvfrom won't happen if -P is 1) It's done this way so -P parallel threads will work for UDP.

I think the server thread startup is racing with the listener thread as both have a recvfrom. The Listener thread recvfom only has the port and optionally the -B IP addr. The server thread recvfrom has the full quintuple per the connect. My understanding is the OS will route packets properly per the connect. That's what I've seen but I need to verify this with the netdev kernel team.

The change was the multicast work. I tuned this path for very fast drops as that's the only way multicast can work well. I think that tuning caused the Listener to win the race for high pps initial loads.

[Robert McMahon]

Please use -P 1 on UDP for now. More on the issue is here. I need to experiment a bit.

I'm probably going to drop packets that are in the race vs try to hand them over between threads. This will look like packet loss per the network but it's really not. I'll probably add a separate counter for these packets. Comments?

This trick is basically reproducing the 'accept()` behaviour in UDP, where each ingress connection gets its own dedicated socket descriptor.

While this trick is nice, it's not without drawbacks — it's racy in two places. First, it's possible that the client will send more than one packet to the unconnected socket before the connected socket is created. The application code should work around it — if a packet received from the server socket belongs to an already existing connected flow, it shall be handed over to the right place. Then, during the creation of the connected socket, in the short window after bind() before connect() we might receive unexpected packets belonging to the unconnected socket! We don't want these packets here. It is necessary to filter the source IP/port when receiving early packets on the connected socket.

And from netdev

> Hi All,
>
> I have a question about the OS routing UDP packets to threads and connect
> vs unconnected sockets. Same src/dst IP and same dst port, different src
> port.
>
> If there are two UDP sockets listening on the same port, each serviced by
> its own thread and they both hang a recvfrom() or recv() (for the connected
> socket,) will the OS route packets only to the thread with a connected
> socket vs the thread with th unconnected socket? If not, what will happen?

Connected sockets receive packets matching 4-tuple, and unconnected sockets
receive packets that no connected socket matches.

  $ python3
  >>> from socket import *
  >>>
  >>> s1 = socket(AF_INET, SOCK_DGRAM)
  >>> s1.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
  >>> s1.bind(('0', 0))
  >>>
  >>> s2 = socket(AF_INET, SOCK_DGRAM)
  >>> s2.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
  >>> s2.bind(s1.getsockname())
  >>>
  >>> s1.connect(('10.0.0.53', 8000))
  >>> s1
  <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('10.0.0.53', 28947), raddr=('10.0.0.53', 8000)>
  >>> s2
  <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('0.0.0.0', 28947)>
  >>>
  >>> s3 = socket(AF_INET, SOCK_DGRAM)
  >>> s3.bind(('10.0.0.53', 8000))
  >>>
  >>> s4 = socket(AF_INET, SOCK_DGRAM)
  >>> s4.bind(('10.0.0.53', 8080))
  >>>
  >>> s3.sendto(b'hello', s1.getsockname())
  5
  >>> s4.sendto(b'world', s1.getsockname())
  5
  >>>
  >>> s1.recv(10)
  b'hello'
  >>> s2.recv(10)
  b'world'

Then, if you create a new unconnected socket, the old unconnected socket will
no longer receive packets until the new one is close()d.

  >>> s5 = socket(AF_INET, SOCK_DGRAM)
  >>> s5.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
  >>> s5.bind(s1.getsockname())
  >>>
  >>> s4.sendto(b'test', s1.getsockname())
  4
  >>> s2.recv(10)
  ^CTraceback (most recent call last):
    File "<stdin>", line 1, in <module>
  KeyboardInterrupt
  >>> s5.recv(10)
  b'test'

SO_REUSEADDR allows the newer socket to take over the port from the old
ones.

If you want to blance the loads across unconnected sockets, use SO_REUSPORT
instead.

[Robert McMahon]

From cloudflare post:

Established-over-unconnected technique
For some services we are using a technique which we call "established-over-unconnected". This comes from a realization that on Linux it's possible to create a connected socket over an unconnected one. Consider this code:

Does this look hacky? Well, it should. What we do here is:

— We start a UDP unconnected socket.

— We wait for a client to come in.

— As soon as we receive the first packet from the client, we immediately create a new fully connected socket, over the unconnected socket! It shares the same local port and local IP.

This is how it might look in ss:

see blog

Here you can see the two sockets managed in our python test server. Notice the established socket is sharing the unconnected socket port.

This trick is basically reproducing the 'accept()` behaviour in UDP, where each ingress connection gets its own dedicated socket descriptor.

While this trick is nice, it's not without drawbacks — it's racy in two places. First, it's possible that the client will send more than one packet to the unconnected socket before the connected socket is created. The application code should work around it — if a packet received from the server socket belongs to an already existing connected flow, it shall be handed over to the right place. Then, during the creation of the connected socket, in the short window after bind() before connect() we might receive unexpected packets belonging to the unconnected socket! We don't want these packets here. It is necessary to filter the source IP/port when receiving early packets on the connected socket.

Is this approach worth the extra complexity? It depends on the use case. For a relatively small number of long-lived flows, it might be ok. For a high number of short-lived flows (especially DNS or NTP) it's an overkill.

Keeping old flows stable during service restarts is particularly hard in UDP. The established-over-unconnected technique is just one of the simpler ways of handling it. We'll leave another technique, based on SO_REUSEPORT ebpf, for a future blog post.

[Robert McMahon]

One of the features merged in the 3.9 development cycle was TCP and UDP support for the SO_REUSEPORT socket option; that support was implemented in a series of patches by Tom Herbert. The new socket option allows multiple sockets on the same host to bind to the same port, and is intended to improve the performance of multithreaded network server applications running on top of multicore systems.

The basic concept of SO_REUSEPORT is simple enough. Multiple servers (processes or threads) can bind to the same port if they each set the option as follows:

int sfd = socket(domain, socktype, 0);

int optval = 1;
setsockopt(sfd, SOL_SOCKET, SO_REUSEPORT, &optval, sizeof(optval));

bind(sfd, (struct sockaddr *) &addr, addrlen);

So long as the first server sets this option before binding its socket, then any number of other servers can also bind to the same port if they also set the option beforehand. The requirement that the first server must specify this option prevents port hijacking—the possibility that a rogue application binds to a port already used by an existing server in order to capture (some of) its incoming connections or datagrams. To prevent unwanted processes from hijacking a port that has already been bound by a server using SO_REUSEPORT, all of the servers that later bind to that port must have an effective user ID that matches the effective user ID used to perform the first bind on the socket.

SO_REUSEPORT can be used with both TCP and UDP sockets. With TCP sockets, it allows multiple listening sockets—normally each in a different thread—to be bound to the same port. Each thread can then accept incoming connections on the port by calling accept(). This presents an alternative to the traditional approaches used by multithreaded servers that accept incoming connections on a single socket.

https://lwn.net/Articles/542629/