#66 udplso suicide

[Wes Eddy]

In LTP's udplso.c, when sending outgoing LTP segments over UDP, the sendto() return value is checked to determine success or failure.

If sending fails due to EINTR, then it's retried later.

If it fails for any other reason, the error handling logic triggers a shutdown of the whole task. Nothing else in ION attempts to respawn the LSO, so outgoing LTP segment flow comes to a grinding halt, even if there was only a "soft error" that doesn't indicate any permanent issue.

We experienced a sendto() return with errno of ENOBUF on-orbit, likely due to a temporary condition in a lower-level interface driver. This should be treated as a soft-error, with a "try again later" reaction, I believe. It doesn't indicate that the socket is broken in any way for future use, only that currently the OS was resource-limited.

There are likely other errno values that indicate soft errors rather than hard errors that should cause task death as well.

[Scott Burleigh]

The Linux 2.6.7 send(2) man page I'm looking at says that ENOBUFS means "The output queue for a network interface was full. This generally indicates that the interface has stopped sending." That doesn't sound like a soft error to me. (Nor do any of the other errno values listed on that man page, aside from EINTR.)

Obviously this interpretation is not going to be universally true across all operating systems, but I believe a large fraction of ION users are running on Linux platforms of one sort or another. It's an important perspective.

Implementing a "try again later" policy would clearly be doable, but getting such a policy right for the general, portable case is not easy: at all costs we want to avoid getting into any sort of tight infinite retry loop that would waste a lot of CPU. And in any case, anything we did would inevitably add complexity in core ION code.

That additional complexity would be entirely warranted if there were no alternative, but in this case I believe there is one. I think flight system executive code could pretty easily watch the ION status and diagnostic messages and, when "[i] udplso has ended" appears, consult the current status of the system and restart LTP if necessary.

This would be deployment-specific code that would have a much better understanding of flight system state than portable ION code ever will, enabling it to make much better decisions about how to respond to this sort of error (possibly including alerting mission operations to anomalous conditions in low-level interface drivers that might indicate trouble somewhere else).

As you say, nothing else in ION attempts to respawn the LSO, and I think that's actually appropriate: something outside of ION should be driving.