netconsole-devel

Re: [Netconsole-devel] Protocol notes

[<li...@ho...>]

> All packets are encapsulated in UDP datagrams having a port number of
> [21727] on the server side; [21726] on the client side.

> [IS THIS THE BEST WAY TO HANDLE PORT NUMBERS?  A FIXED PORT NUMBER
> SHOULD MAKE REATTACHMENT EASIEST, BUT DOES IT WORK WELL ON THE SERVER
> SIDE?]

Not too evil, but how does the client find out the server's IP address?
DHCP?  The presumed environment (e.g. Beowulf cluster) has a *lot* of clients,
so you don't want to have to configure them manually.

Would it make more sense for the client to broadcast to a statically
assigned multicast address?

Note that there appear to be no way for a server to elicit a response
from a client without sending it data (unless you want to reserve stream
0 for that purpose).  You could formalize this and require that the
client initiate communications, obviating the need to choose a client port.

Is there any way for a server to find out which streams a client supports?

Including the character set redundantly with each packet seems odd.
Think carefully if the number of such "header options" is going to grow.
There are other ways to achieve the same end if that is so
(a way for the client to answer queries from the server, perhaps).


One important thing to note is that the server can use the lock-step nature
to detect missing data.  The client will always transmit the lowest available
unacknowledged sequence number.  If the server sees a packet with a higher
sequence number than it is acknowledging, there has been a data gap.



How does a newly booted server achieve sync with the client?  It can't send
a packet including an ack sequence number, because it might land in the valid
sequence number space.  But the client has presumably backed off to quite
infrequent retrnamsissions, and the server probably doesn't want to
wait that long to establish communications.


Since the assumption here is that we can add almost arbitrary complexity
to the server to allow a lower-resource client, I can suggest at least
one header flag: "client has no timer".  If the client sets this bit, the
server must poll it as some suitable rate.  The client MUST transmit new
data in a stream when it arrives, and SHOULD make multiple transmission
attempts before its buffer space overruns, but it does not do timeout
retransmissions, relying on periodic server polls to tickle it into
retransmission as necessary.


Are there supposed to be previsions for the level-N boot loader to pass
the current sequence number to level N+1?  Should level-N wait for all
its data to be acknowledged before jumping to N+1 (and if so, how long),
or are there provisions for passing the unacknowledged data to level N+1?

Re: [Netconsole-devel] Protocol notes

[<li...@ho...>]

> I was considering it, but it adds a lot of complexity for what seems to
> be very little gain.  In the end I think the solution I came up with is
> cleaner.  Having a header end pointer so we can add new header fields in
> the future might be a good idea, though.

How about IP-style options?

> I was thinking about having a flags field in the header to denote that
> sequence numbers are lost -- one bit to indicate "my own sequence numbers
> have been reset" (START OF STREAM - typically issued when starting a new
> boot loader level) and one bit to indicate "I don't know where you are in
> your stream."

Okay, so we have a SYN bit and (the inverse of) an ACK bit.
Oh, above you've suggested a data pointer.

The UDP header gives us two port numbers and a checksum.

We already have two 32-bit byte checksums.

Remind me again why we don't just use TCP?

    0                   1                   2                   3   
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Source Port          |       Destination Port        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Sequence Number                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Acknowledgment Number                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Data |           |U|A|P|R|S|F|                               |
   | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
   |       |           |G|K|H|T|N|N|                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Checksum            |         Urgent Pointer        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

> I don't know if this is a feasible option, mostly because I worry about
> it requiring too much state at the server (which could cause restart
> problems.)

It does require that the server ping on startup.  It can back off dead
clients pretty severely after a while.  They're supposed to send packets
themselves on startup.

Re: [Netconsole-devel] Protocol notes

[H. P. A. <hp...@tr...>]

li...@ho... wrote:
> 
> > I was considering it, but it adds a lot of complexity for what seems to
> > be very little gain.  In the end I think the solution I came up with is
> > cleaner.  Having a header end pointer so we can add new header fields in
> > the future might be a good idea, though.
> 
> How about IP-style options?
> 

Painful to parse.  I'd like to avoid it.  Fixed-field offsets; even if
truncated by a data pointer, is a helluva lot easier to deal with.

> > I was thinking about having a flags field in the header to denote that
> > sequence numbers are lost -- one bit to indicate "my own sequence numbers
> > have been reset" (START OF STREAM - typically issued when starting a new
> > boot loader level) and one bit to indicate "I don't know where you are in
> > your stream."
> 
> Okay, so we have a SYN bit and (the inverse of) an ACK bit.
> Oh, above you've suggested a data pointer.
> 
> The UDP header gives us two port numbers and a checksum.
> 
> We already have two 32-bit byte checksums.
> 
> Remind me again why we don't just use TCP?

Once again, it will not handle interrupted connections (server reboots,
buffer overruns) correctly -- in fact, it would have disastrous effects. 
Saying you can have a small TCP implementation is not good enough -- the
OS on the server will implement TCP, not the small TCP derivative capable
of handling interrupted connections.  This is reason for not using TCP. 
That something looks a lot like TCP doesn't mean TCP is the right answer.

Once you cannot use TCP, you don't need to put in the things you don't
want into it.

>     0                   1                   2                   3
>     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |          Source Port          |       Destination Port        |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                        Sequence Number                        |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                    Acknowledgment Number                      |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |  Data |           |U|A|P|R|S|F|                               |
>    | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
>    |       |           |G|K|H|T|N|N|                               |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |           Checksum            |         Urgent Pointer        |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 
> > I don't know if this is a feasible option, mostly because I worry about
> > it requiring too much state at the server (which could cause restart
> > problems.)
> 
> It does require that the server ping on startup.  It can back off dead
> clients pretty severely after a while.  They're supposed to send packets
> themselves on startup.

I don't think we want to let the protocol back off quite as far as TCP
would.  Proper behaviour over high-latency/low-bandwidth links is
explicitly *not* a requirement.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] Protocol notes

[H. P. A. <hp...@tr...>]

li...@ho... wrote:
> 
> > All packets are encapsulated in UDP datagrams having a port number of
> > [21727] on the server side; [21726] on the client side.
> 
> > [IS THIS THE BEST WAY TO HANDLE PORT NUMBERS?  A FIXED PORT NUMBER
> > SHOULD MAKE REATTACHMENT EASIEST, BUT DOES IT WORK WELL ON THE SERVER
> > SIDE?]
> 
> Not too evil, but how does the client find out the server's IP address?
> DHCP?  The presumed environment (e.g. Beowulf cluster) has a *lot* of clients,
> so you don't want to have to configure them manually.
> 

DHCP.  It's the only sensible way.  You have to configure things like IP
address anyway.

> Would it make more sense for the client to broadcast to a statically
> assigned multicast address?
> 
> Note that there appear to be no way for a server to elicit a response
> from a client without sending it data (unless you want to reserve stream
> 0 for that purpose).  You could formalize this and require that the
> client initiate communications, obviating the need to choose a client port.

I was considering that; I'm wondering if that causes a problem if the
server is restarted, however.
 
> Is there any way for a server to find out which streams a client supports?
> 
> Including the character set redundantly with each packet seems odd.
> Think carefully if the number of such "header options" is going to grow.
> There are other ways to achieve the same end if that is so
> (a way for the client to answer queries from the server, perhaps).

I was considering it, but it adds a lot of complexity for what seems to
be very little gain.  In the end I think the solution I came up with is
cleaner.  Having a header end pointer so we can add new header fields in
the future might be a good idea, though.

> One important thing to note is that the server can use the lock-step nature
> to detect missing data.  The client will always transmit the lowest available
> unacknowledged sequence number.  If the server sees a packet with a higher
> sequence number than it is acknowledging, there has been a data gap.

Yes, that is true.
 
> How does a newly booted server achieve sync with the client?  It can't send
> a packet including an ack sequence number, because it might land in the valid
> sequence number space.  But the client has presumably backed off to quite
> infrequent retrnamsissions, and the server probably doesn't want to
> wait that long to establish communications.

I was thinking about having a flags field in the header to denote that
sequence numbers are lost -- one bit to indicate "my own sequence numbers
have been reset" (START OF STREAM - typically issued when starting a new
boot loader level) and one bit to indicate "I don't know where you are in
your stream."

> Since the assumption here is that we can add almost arbitrary complexity
> to the server to allow a lower-resource client, I can suggest at least
> one header flag: "client has no timer".  If the client sets this bit, the
> server must poll it as some suitable rate.  The client MUST transmit new
> data in a stream when it arrives, and SHOULD make multiple transmission
> attempts before its buffer space overruns, but it does not do timeout
> retransmissions, relying on periodic server polls to tickle it into
> retransmission as necessary.

I don't know if this is a feasible option, mostly because I worry about
it requiring too much state at the server (which could cause restart
problems.)

> Are there supposed to be previsions for the level-N boot loader to pass
> the current sequence number to level N+1?  Should level-N wait for all
> its data to be acknowledged before jumping to N+1 (and if so, how long),
> or are there provisions for passing the unacknowledged data to level N+1?

In normal operation, I think that you normally would have level N wait
until all data acked.  I don't think duration is something the protocol
needs to specify, although perhaps a minimum time would be good.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt