netconsole-devel — Developer mailing list

Re: [Netconsole-devel] status?

[H. P. A. <hp...@tr...>]

Jim Garlick wrote:
> 
> Thanks,
> 
> I need to go read your archives, but I will toss out a few points that come
> to mind:
> 
> Sorry if my point#1 was confusing - I just want to be able to use this stuff
> over networks that are not necessarily IP, such as CAN (control area network
> used on the Meiko CS/2 for consoles), RS485, whatever.  Maybe I can make them
> look like UDP, but that sounds ugly.
> 

It's easy enough to replace UDP with any other datagram protocol, I'd
think.  It would require appropriate modifications, of course.

> 
> On the architecture, it seems to me like the netcon server approach creates
> a single point of failure where there doesn't need to be one.  The way
> I did the elan consoles was I made a range of elancon tty "callout devices"
> which can be virtually connected to the elancon tty console device on any
> node via an ioctl.  The client user-space software then just becomes minicom
> with support for some new tty ioctls (because the connected callout device
> looks just like /dev/ttyS0), and it is fully peer-to-peer.
> 

This sounds like an awful lot of complexity.especially for the client. 
The client really does need to be lean and mean.  To be utterly frank, a
solution of the complexity level you seem to advocate sounds like you
might as well go right ahead and just use ssh.

As far as supporting multiple servers, there is a lot you could do but it
would all come at considerable cost in client complexity.  Multicasting,
for example, requires that the client understands IGMP.  Unicasting to
multiple servers would require waiting for ACKs from all of them.

> 
> For logging, one could preserve this peer-to-peer architecture and implement
> a separate read-only connect for the logging daemon so it could concurrently
> capture console output while console sessions come and go.  Support for
> redundant logging servers could even be added.
> 

Let's think about what 

	-hpa


-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] status?

[Jim G. <ga...@ll...>]

Thanks,

I need to go read your archives, but I will toss out a few points that come 
to mind:

Sorry if my point#1 was confusing - I just want to be able to use this stuff
over networks that are not necessarily IP, such as CAN (control area network
used on the Meiko CS/2 for consoles), RS485, whatever.  Maybe I can make them 
look like UDP, but that sounds ugly.

On the architecture, it seems to me like the netcon server approach creates
a single point of failure where there doesn't need to be one.  The way
I did the elan consoles was I made a range of elancon tty "callout devices"
which can be virtually connected to the elancon tty console device on any 
node via an ioctl.  The client user-space software then just becomes minicom
with support for some new tty ioctls (because the connected callout device
looks just like /dev/ttyS0), and it is fully peer-to-peer.

For logging, one could preserve this peer-to-peer architecture and implement
a separate read-only connect for the logging daemon so it could concurrently
capture console output while console sessions come and go.  Support for
redundant logging servers could even be added.

Jim

On Wed, 21 Feb 2001, H. Peter Anvin wrote:

> Jim Garlick wrote:
> > 
> > Sounds great.  So I take it a rough design has already been proposed?
> > I'd like to get my $0.02 in on the requirements:
> > 
> > 1. well defined, simple interface for the communication layer so ethernets,
> >    CAN (I maintain the Meiko CS/2 linux port), and cluster interconnects can
> >    plug in.
> > 
> > 2. in my world of clusters, the network is assumed to be physically
> >    secure and root on any node is equiv to root on any other node.  Ideally,
> >    console mangement would be peer-to-peer and root-only within that space.
> >    Outside of that space, access should be unavailable or sufficiently
> >    protected.
> > 
> > 3. ability to "broadcast" to some set of consoles, i.e. you open a display
> >    full of console windows talking to various nodes, and you have a separate
> >    window where everything typed is sent to all those nodes' consoles
> > 
> > 4. ability to force a panic remotely, esp when node is half way hung, like
> >    interrupts work but processes don't run
> > 
> > 5. console "stealing" protocol, i.e. joe has a console open and goes home,
> >    sue should be able to tell joe has the console open and usurp it, causing
> >    joe's session to report sue as the culprit.
> > 
> > 6. I already mentioned the problem of preserving connect state across a reboot.
> > 
> > I have 2-5 covered with my elan implementation, and I'd really like to get
> > #1 defined so I can take advantage of anything this group comes up with.
> > #6 is maybe a linuxbios issue - allocating bits of the system NVRAM.
> > 
> > I also have a minicom-like client program that manages the sessions (including
> > "stealing" and broadcasts) and a separate console logging/session management
> > system for serial consoles (lots of serial consoles connected to a digiboard).
> > I had planned to put these all together in one package so you get the same
> > interface no matter how the consoles are connected.
> > 
> > Do you think some collaboration is possible here?
> > 
> 
> We have already covered most of this.  I have discussed #2 with crypto
> people, and it pretty much boils down to the equivalent of spoofing DHCP;
> in other words, until someone comes up with a security standard for DHCP,
> it's pretty pointless to do anything in that vein.
> 
> I can't tell what the heck you're talking about in #1.  The idea is to
> define a UDP protocol, so if you can use IP over it it should work.
> 
> #4 pretty much boils down to supporting SysRq/Stop-A/BREAK remotely. 
> That's definitely planned.
> 
> We're planning a client/server/UI architecture, from a previous message:
> 
> --------------------
> 
> I'd like to see an architecture like this:
> 
> 
> +-------------+
> | Netcon user |
> |    agent    |
> +-------------+
>        |
>  TCP protocol
>        |
> +-------------+
> |    Netcon   |
> |    server   |
> +-------------+
>        |
> Netcon protocol
>        |
> +-------------+
> |    Netcon   |
> |    client   |
> +-------------+
> 
> 
> The purpose of the netcon server is to provide session management and
> (optionally) logging support.  The netcon user agent is what an admin
> uses to plug into a session.  By isolating the UI from the session
> management, we allow connecting and disconnecting to sessions.
> 
> Note that this is a useful capability in itself, and we may want to
> consider (later, please!) the netcon server to talk to a serial port or
> PTY session as well as to a netcon client.
> 
> Although the current assumption is that Netcon Protocol is a UDP
> protocol, it will provide a logical bidirectional stream capability.
> 
> I really want to define a specific terminal emulation for this protocol. 
> I definitely believe we do need more than a basic terminal, and having a
> specific terminal emulation is key.  The only logical such terminal is
> some ANSI/DEC VT subset.  Character set is yet another issue; I don't
> think I have the guts to mandate UTF-8 over this stream.
> 
> -----------------
> 
> Your points #3 and #5 are defined by the Netcon server protocol. 
> Basically what you're saying is that more than one user agent should be
> able to connect to the same client session; this is indeed important.
> 
> We're addressing #6 at the protocol level; this is in fact *the* main
> reason for going with a UDP protocol rather than using TCP.
> 
> It would be great if someone, such as yourself, would start coding up the
> UI <-> server connection system.  In the meantime, we're hashing out the
> protocol details.
> 
> One of the biggest open questions right now is whether or not Netcon
> Protocol should be a stream-based protocol in both directions (simulating
> a serial console) or a virtual screen protocol (simulating a CRT.)  Both
> have advantages and disadvantages.
> 
> 	-hpa
> 
> -- 
> <hp...@tr...> at work, <hp...@zy...> in private!
> "Unix gives you enough rope to shoot yourself in the foot."
> http://www.zytor.com/~hpa/puzzle.txt
> 
> _______________________________________________
> Netconsole-devel mailing list
> Net...@li...
> http://lists.sourceforge.net/lists/listinfo/netconsole-devel
> 

[Netconsole-devel] Since the archive seems to be broken...

[H. P. A. <hp...@tr...>]

... I have put the contents of my netcon mail folder available at:

	http://www.zytor.com/~hpa/netcon/

I'll update it every now and then.  If you're a new subscriber, please
look at it.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] status?

[H. P. A. <hp...@tr...>]

Jim Garlick wrote:
> 
> Sounds great.  So I take it a rough design has already been proposed?
> I'd like to get my $0.02 in on the requirements:
> 
> 1. well defined, simple interface for the communication layer so ethernets,
>    CAN (I maintain the Meiko CS/2 linux port), and cluster interconnects can
>    plug in.
> 
> 2. in my world of clusters, the network is assumed to be physically
>    secure and root on any node is equiv to root on any other node.  Ideally,
>    console mangement would be peer-to-peer and root-only within that space.
>    Outside of that space, access should be unavailable or sufficiently
>    protected.
> 
> 3. ability to "broadcast" to some set of consoles, i.e. you open a display
>    full of console windows talking to various nodes, and you have a separate
>    window where everything typed is sent to all those nodes' consoles
> 
> 4. ability to force a panic remotely, esp when node is half way hung, like
>    interrupts work but processes don't run
> 
> 5. console "stealing" protocol, i.e. joe has a console open and goes home,
>    sue should be able to tell joe has the console open and usurp it, causing
>    joe's session to report sue as the culprit.
> 
> 6. I already mentioned the problem of preserving connect state across a reboot.
> 
> I have 2-5 covered with my elan implementation, and I'd really like to get
> #1 defined so I can take advantage of anything this group comes up with.
> #6 is maybe a linuxbios issue - allocating bits of the system NVRAM.
> 
> I also have a minicom-like client program that manages the sessions (including
> "stealing" and broadcasts) and a separate console logging/session management
> system for serial consoles (lots of serial consoles connected to a digiboard).
> I had planned to put these all together in one package so you get the same
> interface no matter how the consoles are connected.
> 
> Do you think some collaboration is possible here?
> 

We have already covered most of this.  I have discussed #2 with crypto
people, and it pretty much boils down to the equivalent of spoofing DHCP;
in other words, until someone comes up with a security standard for DHCP,
it's pretty pointless to do anything in that vein.

I can't tell what the heck you're talking about in #1.  The idea is to
define a UDP protocol, so if you can use IP over it it should work.

#4 pretty much boils down to supporting SysRq/Stop-A/BREAK remotely. 
That's definitely planned.

We're planning a client/server/UI architecture, from a previous message:

--------------------

I'd like to see an architecture like this:


+-------------+
| Netcon user |
|    agent    |
+-------------+
       |
 TCP protocol
       |
+-------------+
|    Netcon   |
|    server   |
+-------------+
       |
Netcon protocol
       |
+-------------+
|    Netcon   |
|    client   |
+-------------+


The purpose of the netcon server is to provide session management and
(optionally) logging support.  The netcon user agent is what an admin
uses to plug into a session.  By isolating the UI from the session
management, we allow connecting and disconnecting to sessions.

Note that this is a useful capability in itself, and we may want to
consider (later, please!) the netcon server to talk to a serial port or
PTY session as well as to a netcon client.

Although the current assumption is that Netcon Protocol is a UDP
protocol, it will provide a logical bidirectional stream capability.

I really want to define a specific terminal emulation for this protocol. 
I definitely believe we do need more than a basic terminal, and having a
specific terminal emulation is key.  The only logical such terminal is
some ANSI/DEC VT subset.  Character set is yet another issue; I don't
think I have the guts to mandate UTF-8 over this stream.

-----------------

Your points #3 and #5 are defined by the Netcon server protocol. 
Basically what you're saying is that more than one user agent should be
able to connect to the same client session; this is indeed important.

We're addressing #6 at the protocol level; this is in fact *the* main
reason for going with a UDP protocol rather than using TCP.

It would be great if someone, such as yourself, would start coding up the
UI <-> server connection system.  In the meantime, we're hashing out the
protocol details.

One of the biggest open questions right now is whether or not Netcon
Protocol should be a stream-based protocol in both directions (simulating
a serial console) or a virtual screen protocol (simulating a CRT.)  Both
have advantages and disadvantages.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] status?

[Jim G. <ga...@ll...>]

Sounds great.  So I take it a rough design has already been proposed?
I'd like to get my $0.02 in on the requirements:

1. well defined, simple interface for the communication layer so ethernets, 
   CAN (I maintain the Meiko CS/2 linux port), and cluster interconnects can 
   plug in.

2. in my world of clusters, the network is assumed to be physically
   secure and root on any node is equiv to root on any other node.  Ideally,
   console mangement would be peer-to-peer and root-only within that space.
   Outside of that space, access should be unavailable or sufficiently 
   protected.

3. ability to "broadcast" to some set of consoles, i.e. you open a display 
   full of console windows talking to various nodes, and you have a separate 
   window where everything typed is sent to all those nodes' consoles

4. ability to force a panic remotely, esp when node is half way hung, like
   interrupts work but processes don't run
 
5. console "stealing" protocol, i.e. joe has a console open and goes home,
   sue should be able to tell joe has the console open and usurp it, causing
   joe's session to report sue as the culprit.

6. I already mentioned the problem of preserving connect state across a reboot.

I have 2-5 covered with my elan implementation, and I'd really like to get
#1 defined so I can take advantage of anything this group comes up with.
#6 is maybe a linuxbios issue - allocating bits of the system NVRAM.

I also have a minicom-like client program that manages the sessions (including
"stealing" and broadcasts) and a separate console logging/session management 
system for serial consoles (lots of serial consoles connected to a digiboard).
I had planned to put these all together in one package so you get the same 
interface no matter how the consoles are connected.

Do you think some collaboration is possible here?

Thanks,

Jim

On Wed, 21 Feb 2001, H. Peter Anvin wrote:

> Jim Garlick wrote:
> > 
> > Hi,
> > 
> > Anyone doing any work on this project?
> > 
> > A colleague pointed the sourceforge project out to me.  I have some code
> > already developed that enables me to do linux consoles over the Quadrics
> > Elan cluster interconnect, and I had planned to extend this to include
> > ethernet if I could figure out how to do it in a non-hardware-specific way.
> > 
> > On the elan, I can run a getty, capture printk's, even shutdown to single
> > user and come back up again.  What I can't currently do is store the
> > identity of the node that currently is connected to the console across a
> > reboot, or of course, to talk to the BIOS (unless the BIOS is LinuxBIOS!)
> > 
> > Ultimately I would like to have this work over ethernet on an API CS20 node,
> > which has ethernet wake-on-lan for power management and can do LinuxBIOS.
> > In theory I could then dispense with serial consoles in a large cluster of
> > CS20's.
> > 
> 
> Getting LinuxBIOS to support Netcon would be a big win.  That shouldn't
> even be that hard.  To make an unmodified BIOS support netcon probably
> would involve a hacked network card boot PROM that intercepts INT 10h;
> since a lot of network cards have EEPROM these days it might actually be
> doable (perhaps using NILO or somesuch.)
> 
> Anyway, Jim, to give you a primer: we're talking about using UDP,
> although a raw Ethernet option might be doable (the biggest problem with
> that is that the packet limit is much smaller.)  You do, of course, need
> an Ethernet card specific driver either way.
> 
> I am planning to add support for Netcon in the PXELINUX boot loader,
> since it has the PXE stack available.
> 
> 	-hpa
> 
> 
> P.S. I have complained about the archive for this list not working.  I
> hope it can get fixed.
> 
> -- 
> <hp...@tr...> at work, <hp...@zy...> in private!
> "Unix gives you enough rope to shoot yourself in the foot."
> http://www.zytor.com/~hpa/puzzle.txt
> 

Re: [Netconsole-devel] status?

[H. P. A. <hp...@tr...>]

Jim Garlick wrote:
> 
> Hi,
> 
> Anyone doing any work on this project?
> 
> A colleague pointed the sourceforge project out to me.  I have some code
> already developed that enables me to do linux consoles over the Quadrics
> Elan cluster interconnect, and I had planned to extend this to include
> ethernet if I could figure out how to do it in a non-hardware-specific way.
> 
> On the elan, I can run a getty, capture printk's, even shutdown to single
> user and come back up again.  What I can't currently do is store the
> identity of the node that currently is connected to the console across a
> reboot, or of course, to talk to the BIOS (unless the BIOS is LinuxBIOS!)
> 
> Ultimately I would like to have this work over ethernet on an API CS20 node,
> which has ethernet wake-on-lan for power management and can do LinuxBIOS.
> In theory I could then dispense with serial consoles in a large cluster of
> CS20's.
> 

Getting LinuxBIOS to support Netcon would be a big win.  That shouldn't
even be that hard.  To make an unmodified BIOS support netcon probably
would involve a hacked network card boot PROM that intercepts INT 10h;
since a lot of network cards have EEPROM these days it might actually be
doable (perhaps using NILO or somesuch.)

Anyway, Jim, to give you a primer: we're talking about using UDP,
although a raw Ethernet option might be doable (the biggest problem with
that is that the packet limit is much smaller.)  You do, of course, need
an Ethernet card specific driver either way.

I am planning to add support for Netcon in the PXELINUX boot loader,
since it has the PXE stack available.

	-hpa


P.S. I have complained about the archive for this list not working.  I
hope it can get fixed.

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

[Netconsole-devel] status?

[Jim G. <ga...@ll...>]

Hi,

Anyone doing any work on this project?

A colleague pointed the sourceforge project out to me.  I have some code
already developed that enables me to do linux consoles over the Quadrics 
Elan cluster interconnect, and I had planned to extend this to include 
ethernet if I could figure out how to do it in a non-hardware-specific way.

On the elan, I can run a getty, capture printk's, even shutdown to single
user and come back up again.  What I can't currently do is store the
identity of the node that currently is connected to the console across a 
reboot, or of course, to talk to the BIOS (unless the BIOS is LinuxBIOS!)

Ultimately I would like to have this work over ethernet on an API CS20 node,
which has ethernet wake-on-lan for power management and can do LinuxBIOS.
In theory I could then dispense with serial consoles in a large cluster of 
CS20's.

Jim

Re: [Netconsole-devel] Protocol notes

[H. P. A. <hp...@tr...>]

li...@ho... wrote:
> 
> > I was considering it, but it adds a lot of complexity for what seems to
> > be very little gain.  In the end I think the solution I came up with is
> > cleaner.  Having a header end pointer so we can add new header fields in
> > the future might be a good idea, though.
> 
> How about IP-style options?
> 

Painful to parse.  I'd like to avoid it.  Fixed-field offsets; even if
truncated by a data pointer, is a helluva lot easier to deal with.

> > I was thinking about having a flags field in the header to denote that
> > sequence numbers are lost -- one bit to indicate "my own sequence numbers
> > have been reset" (START OF STREAM - typically issued when starting a new
> > boot loader level) and one bit to indicate "I don't know where you are in
> > your stream."
> 
> Okay, so we have a SYN bit and (the inverse of) an ACK bit.
> Oh, above you've suggested a data pointer.
> 
> The UDP header gives us two port numbers and a checksum.
> 
> We already have two 32-bit byte checksums.
> 
> Remind me again why we don't just use TCP?

Once again, it will not handle interrupted connections (server reboots,
buffer overruns) correctly -- in fact, it would have disastrous effects. 
Saying you can have a small TCP implementation is not good enough -- the
OS on the server will implement TCP, not the small TCP derivative capable
of handling interrupted connections.  This is reason for not using TCP. 
That something looks a lot like TCP doesn't mean TCP is the right answer.

Once you cannot use TCP, you don't need to put in the things you don't
want into it.

>     0                   1                   2                   3
>     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |          Source Port          |       Destination Port        |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                        Sequence Number                        |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                    Acknowledgment Number                      |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |  Data |           |U|A|P|R|S|F|                               |
>    | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
>    |       |           |G|K|H|T|N|N|                               |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |           Checksum            |         Urgent Pointer        |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 
> > I don't know if this is a feasible option, mostly because I worry about
> > it requiring too much state at the server (which could cause restart
> > problems.)
> 
> It does require that the server ping on startup.  It can back off dead
> clients pretty severely after a while.  They're supposed to send packets
> themselves on startup.

I don't think we want to let the protocol back off quite as far as TCP
would.  Proper behaviour over high-latency/low-bandwidth links is
explicitly *not* a requirement.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] Protocol notes

[<li...@ho...>]

> I was considering it, but it adds a lot of complexity for what seems to
> be very little gain.  In the end I think the solution I came up with is
> cleaner.  Having a header end pointer so we can add new header fields in
> the future might be a good idea, though.

How about IP-style options?

> I was thinking about having a flags field in the header to denote that
> sequence numbers are lost -- one bit to indicate "my own sequence numbers
> have been reset" (START OF STREAM - typically issued when starting a new
> boot loader level) and one bit to indicate "I don't know where you are in
> your stream."

Okay, so we have a SYN bit and (the inverse of) an ACK bit.
Oh, above you've suggested a data pointer.

The UDP header gives us two port numbers and a checksum.

We already have two 32-bit byte checksums.

Remind me again why we don't just use TCP?

    0                   1                   2                   3   
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Source Port          |       Destination Port        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Sequence Number                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Acknowledgment Number                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Data |           |U|A|P|R|S|F|                               |
   | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
   |       |           |G|K|H|T|N|N|                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Checksum            |         Urgent Pointer        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

> I don't know if this is a feasible option, mostly because I worry about
> it requiring too much state at the server (which could cause restart
> problems.)

It does require that the server ping on startup.  It can back off dead
clients pretty severely after a while.  They're supposed to send packets
themselves on startup.

Re: [Netconsole-devel] Should netcon be a virtual screen-based protocol?

[H. P. A. <hp...@tr...>]

li...@ho... wrote:
> 
> > Originally I said "of course not", but in the car today I started to
> > wonder about it.  A (text) virtual screen would fit nicely in a single
> > UDP datagram, and would make it very easy to map native console
> > operations -- how weird they may be -- onto the virtual screen.  The
> > downside is that it makes server-side logging and scrollback harder.
> 
> Well, 80x25 characters (never mind attributes) is larger than 1500 bytes.
> Do we want to require support for fragmentation?
> 

Fragmentation is trivial.  Reassembly isn't, but that is the server's
job.

> > Thoughts on this subject?
> 
> It also makes defining "catching everything" for logging a bit difficult.
> 
> It would theoretically permit using BIOS setup utilities across the
> protocol, but is the probability of the support being added non-negligible
> enough to worry about it?  What other benefits are there?

For one thing, it makes the issue of retransmissions trivial... there is
only one buffer and it's the current screen buffer.  Similarly, if a
server is rebooted and has to reattach, the screen is still intact as it
will be included in full by the next transmission from the client.  A lot
of the issues of terminal emulation and the potential need for SIGWINCH,
etc, go away.

> It *would* be kind of nice for a Linux boot loader to convert the text-mode
> BIOS screen to console logs for dumping, similar to the way that the
> various stages manage to coordinate the screen cursor and not overwrite
> each other...

Yes, one could do that that way.


-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] Protocol notes

[H. P. A. <hp...@tr...>]

li...@ho... wrote:
> 
> > All packets are encapsulated in UDP datagrams having a port number of
> > [21727] on the server side; [21726] on the client side.
> 
> > [IS THIS THE BEST WAY TO HANDLE PORT NUMBERS?  A FIXED PORT NUMBER
> > SHOULD MAKE REATTACHMENT EASIEST, BUT DOES IT WORK WELL ON THE SERVER
> > SIDE?]
> 
> Not too evil, but how does the client find out the server's IP address?
> DHCP?  The presumed environment (e.g. Beowulf cluster) has a *lot* of clients,
> so you don't want to have to configure them manually.
> 

DHCP.  It's the only sensible way.  You have to configure things like IP
address anyway.

> Would it make more sense for the client to broadcast to a statically
> assigned multicast address?
> 
> Note that there appear to be no way for a server to elicit a response
> from a client without sending it data (unless you want to reserve stream
> 0 for that purpose).  You could formalize this and require that the
> client initiate communications, obviating the need to choose a client port.

I was considering that; I'm wondering if that causes a problem if the
server is restarted, however.
 
> Is there any way for a server to find out which streams a client supports?
> 
> Including the character set redundantly with each packet seems odd.
> Think carefully if the number of such "header options" is going to grow.
> There are other ways to achieve the same end if that is so
> (a way for the client to answer queries from the server, perhaps).

I was considering it, but it adds a lot of complexity for what seems to
be very little gain.  In the end I think the solution I came up with is
cleaner.  Having a header end pointer so we can add new header fields in
the future might be a good idea, though.

> One important thing to note is that the server can use the lock-step nature
> to detect missing data.  The client will always transmit the lowest available
> unacknowledged sequence number.  If the server sees a packet with a higher
> sequence number than it is acknowledging, there has been a data gap.

Yes, that is true.
 
> How does a newly booted server achieve sync with the client?  It can't send
> a packet including an ack sequence number, because it might land in the valid
> sequence number space.  But the client has presumably backed off to quite
> infrequent retrnamsissions, and the server probably doesn't want to
> wait that long to establish communications.

I was thinking about having a flags field in the header to denote that
sequence numbers are lost -- one bit to indicate "my own sequence numbers
have been reset" (START OF STREAM - typically issued when starting a new
boot loader level) and one bit to indicate "I don't know where you are in
your stream."

> Since the assumption here is that we can add almost arbitrary complexity
> to the server to allow a lower-resource client, I can suggest at least
> one header flag: "client has no timer".  If the client sets this bit, the
> server must poll it as some suitable rate.  The client MUST transmit new
> data in a stream when it arrives, and SHOULD make multiple transmission
> attempts before its buffer space overruns, but it does not do timeout
> retransmissions, relying on periodic server polls to tickle it into
> retransmission as necessary.

I don't know if this is a feasible option, mostly because I worry about
it requiring too much state at the server (which could cause restart
problems.)

> Are there supposed to be previsions for the level-N boot loader to pass
> the current sequence number to level N+1?  Should level-N wait for all
> its data to be acknowledged before jumping to N+1 (and if so, how long),
> or are there provisions for passing the unacknowledged data to level N+1?

In normal operation, I think that you normally would have level N wait
until all data acked.  I don't think duration is something the protocol
needs to specify, although perhaps a minimum time would be good.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] Should netcon be a virtual screen-based protocol?

[<li...@ho...>]

> Originally I said "of course not", but in the car today I started to
> wonder about it.  A (text) virtual screen would fit nicely in a single
> UDP datagram, and would make it very easy to map native console
> operations -- how weird they may be -- onto the virtual screen.  The
> downside is that it makes server-side logging and scrollback harder.

Well, 80x25 characters (never mind attributes) is larger than 1500 bytes.
Do we want to require support for fragmentation?

> Thoughts on this subject?

It also makes defining "catching everything" for logging a bit difficult.

It would theoretically permit using BIOS setup utilities across the
protocol, but is the probability of the support being added non-negligible
enough to worry about it?  What other benefits are there?


It *would* be kind of nice for a Linux boot loader to convert the text-mode
BIOS screen to console logs for dumping, similar to the way that the
various stages manage to coordinate the screen cursor and not overwrite
each other...

[Netconsole-devel] Should netcon be a virtual screen-based protocol?

[H. P. A. <hp...@tr...>]

Originally I said "of course not", but in the car today I started to
wonder about it.  A (text) virtual screen would fit nicely in a single
UDP datagram, and would make it very easy to map native console
operations -- how weird they may be -- onto the virtual screen.  The
downside is that it makes server-side logging and scrollback harder.

Thoughts on this subject?

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] Protocol notes

[<li...@ho...>]

> All packets are encapsulated in UDP datagrams having a port number of
> [21727] on the server side; [21726] on the client side.

> [IS THIS THE BEST WAY TO HANDLE PORT NUMBERS?  A FIXED PORT NUMBER
> SHOULD MAKE REATTACHMENT EASIEST, BUT DOES IT WORK WELL ON THE SERVER
> SIDE?]

Not too evil, but how does the client find out the server's IP address?
DHCP?  The presumed environment (e.g. Beowulf cluster) has a *lot* of clients,
so you don't want to have to configure them manually.

Would it make more sense for the client to broadcast to a statically
assigned multicast address?

Note that there appear to be no way for a server to elicit a response
from a client without sending it data (unless you want to reserve stream
0 for that purpose).  You could formalize this and require that the
client initiate communications, obviating the need to choose a client port.

Is there any way for a server to find out which streams a client supports?

Including the character set redundantly with each packet seems odd.
Think carefully if the number of such "header options" is going to grow.
There are other ways to achieve the same end if that is so
(a way for the client to answer queries from the server, perhaps).


One important thing to note is that the server can use the lock-step nature
to detect missing data.  The client will always transmit the lowest available
unacknowledged sequence number.  If the server sees a packet with a higher
sequence number than it is acknowledging, there has been a data gap.



How does a newly booted server achieve sync with the client?  It can't send
a packet including an ack sequence number, because it might land in the valid
sequence number space.  But the client has presumably backed off to quite
infrequent retrnamsissions, and the server probably doesn't want to
wait that long to establish communications.


Since the assumption here is that we can add almost arbitrary complexity
to the server to allow a lower-resource client, I can suggest at least
one header flag: "client has no timer".  If the client sets this bit, the
server must poll it as some suitable rate.  The client MUST transmit new
data in a stream when it arrives, and SHOULD make multiple transmission
attempts before its buffer space overruns, but it does not do timeout
retransmissions, relying on periodic server polls to tickle it into
retransmission as necessary.


Are there supposed to be previsions for the level-N boot loader to pass
the current sequence number to level N+1?  Should level-N wait for all
its data to be acknowledged before jumping to N+1 (and if so, how long),
or are there provisions for passing the unacknowledged data to level N+1?

[Netconsole-devel] Protocol notes

[H. P. A. <hp...@tr...>]

Hi everyone,

I've done my first *VERY CRUDE* draft of the protocol.  Note that it is
intended as a lock-step protocol, although acking is done at a byte
position level, which means that if new data has arrived between an
original transmission and the retransmission, the retransmission may
include the new data as well.

Again, this is a very crude and early draft.  Please help me pick it
apart and get it fleshed out.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

[Netconsole-devel] Tool to generate docs including protocol pics

[H. P. A. <hp...@tr...>]

Does anyone know of a decent tool to generate printed documentation (i.e
PDF) including such things as protocol stack images?  Or should I just go
with old RFC-style tradition and just use text and my trusty ole Emacs?

	-hpa

[Netconsole-devel] Re: LILO and serial speeds over 9600

[H. P. A. <hp...@tr...>]

Rogier Wolff wrote:
> 
> Keep the old ring buffer. Keep a pointer to where there has been an
> ack.
> 
> (Hmm. May not hold for bootloader stuff. )
> 

No; I was planning to use a byte position based ACK system, though.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] Getting this show on the road...

[H. P. A. <hp...@tr...>]

li...@ho... wrote:
> 
> Excellent point!  The logging server has to maintain logs and associated
> sequence numbers on stable storage, so it can resume the conversation at
> te right place.  This means that we can't use in-kernel TCP, so...
> 

You can't even require that.  You have to assume that the server
physically was destroyed; you still need to be able to resume
connectivity; obviously data loss.

> >> We are working with minimal boot loaders and the like.  Do you think
> >> they need significant termcap fanciness?  (Are you trying to support
> >> BIOS config screens and the like?)
> 
> > Yes, or "make menuconfig" if you need to rebuild the server kernel, or
> > what have you.
> 
> Um, once you have enough envronment running that you can *run* make
> menuconfig, you have enough that you can just use the regular kernel,
> ssh and the like.

Assuming it's safe for you to enable it.
 
> But I understand the disconnect now.  I was thinking /var/log/kern.log
> and you were thinking /dev/tty1.
> 
> You want to export the keyboard and screen across the network for
> everything.

Yes, way, way early before the kernel even boots.
 
> One point that raises is that you want to be able to connect even if the
> headless box has spewed so much that its buffers have overrun and the server
> has lost sync.  Does the headless box have to keep track a virtual
> screen that can be downloaded at will?  Or is it okay to say that it might
> get corrupted and require a ^L to fix?

I think so.
 
> When I said "crashing problems", I meant "things that print a message to
> the screen and lock the machine."
> 
> In *this* case, a big issue will be sharing access to the ethernet with
> the OS driver.  One way to do it, that I have used for serial ports is
> to maintain an interrupt-safe buffer of data to send, which normally
> the low-level code queues and the OS device drivers sends out.
> 
> But if the buffer overruns, the low-level code leaves interrupts off
> and runs a special polling driver to push the buffer out synchronously,
> on the asumption that the OS is hosed.
> 
> I don't know if this is amenable to ethernets.  How *are* we going to
> do this?  Require a dedicated NIC for the console?  It's actually not
> unreasonable given the price of NICs and hub ports these days.

Most NICs these days require a lot of machinery to work at all.  I'm sure
you can isolate it, but it almost certainly requires a lot of work.  I
think there is a completely different project out there that deals with
Linux crash dumping (postmortem debugging.)  It's a fundamentally
different problem, and personally I'd like to consider that out of scope.

One thing worth considering is whether or not there should be a separate
log stream in addition to the two-way data stream, though.  A log stream
would at least avoid the "I can't log into the server console because of
all the spewage" problem.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] Getting this show on the road...

[<li...@ho...>]

> Actually, the biggest reason *not* to use TCP is what happens when the
> server dies, and then is restarted.  It is utterly imperative that it has
> no effect on the clients, but you should still resume the connection
> (even though you have probably lost data) which is unachievable using TCP
> on stock Unix systems.

Excellent point!  The logging server has to maintain logs and associated
sequence numbers on stable storage, so it can resume the conversation at
te right place.  This means that we can't use in-kernel TCP, so...

>> We are working with minimal boot loaders and the like.  Do you think
>> they need significant termcap fanciness?  (Are you trying to support
>> BIOS config screens and the like?)

> Yes, or "make menuconfig" if you need to rebuild the server kernel, or
> what have you.

Um, once you have enough envronment running that you can *run* make
menuconfig, you have enough that you can just use the regular kernel,
ssh and the like.


But I understand the disconnect now.  I was thinking /var/log/kern.log
and you were thinking /dev/tty1.

You want to export the keyboard and screen across the network for
everything.


One point that raises is that you want to be able to connect even if the
headless box has spewed so much that its buffers have overrun and the server
has lost sync.  Does the headless box have to keep track a virtual
screen that can be downloaded at will?  Or is it okay to say that it might
get corrupted and require a ^L to fix?

> That is *one* application.  The much more important application is to
> *manage* a headless machine in a machine room somewhere.  The network is
> a really bad choice for the crash dump application: there are way too
> many things that can cause that to die.

When I said "crashing problems", I meant "things that print a message to
the screen and lock the machine."


In *this* case, a big issue will be sharing access to the ethernet with
the OS driver.  One way to do it, that I have used for serial ports is
to maintain an interrupt-safe buffer of data to send, which normally
the low-level code queues and the OS device drivers sends out.

But if the buffer overruns, the low-level code leaves interrupts off
and runs a special polling driver to push the buffer out synchronously,
on the asumption that the OS is hosed.

I don't know if this is amenable to ethernets.  How *are* we going to
do this?  Require a dedicated NIC for the console?  It's actually not
unreasonable given the price of NICs and hub ports these days.

Re: [Netconsole-devel] Getting this show on the road...

[H. P. A. <hp...@tr...>]

li...@ho... wrote:
> 
> > Okay, let's look at the basic design of this puppy.  I don't
> > so much mean protocol details and the be or not be of security
> > (which I'd rather wait with until we have a prototype) but more
> > the general architecture.
> 
> Security can be postponed a bit, but a bit of protocol design would
> be relevant to the architecture.
> 
> The most important piece of architecture is how it fits in to the
> "client", as you have called it, the machine containing the netcon stub.
> The protocol is a big enough part of that to be an "architecture" issue.
> 
> Your UA/server/client (although I'd like a different word than "client",
> maybe debugger-style "target"?) design makes sense, particularly if the
> netcon protocol will perform poorly over a long-haul link.
> 
> > Although the current assumption is that Netcon Protocol is a UDP
> > protocol, it will provide a logical bidirectional stream capability.
> 
> Once you're supporting a bidirectional stream, think hard about
> a stripped-down TCP.  The only thing you might be able to do better
> with a hand-rolled one is pass state between boot loader stages.
> 

Actually, the biggest reason *not* to use TCP is what happens when the
server dies, and then is restarted.  It is utterly imperative that it has
no effect on the clients, but you should still resume the connection
(even though you have probably lost data) which is unachievable using TCP
on stock Unix systems.

Some multiport serial cards have been known to glitch the TxD line during
initialization.  The system at the other end tends to interpret that as a
BREAK.  I have seen what happens if the multiport serial card happens to
be hooked up to a rackful of Suns.  We most emphatically do NOT want the
equivalent problem.

> > I really want to define a specific terminal emulation for this protocol.
> > I definitely believe we do need more than a basic terminal, and having a
> > specific terminal emulation is key.  The only logical such terminal is
> > some ANSI/DEC VT subset.  Character set is yet another issue; I don't
> > think I have the guts to mandate UTF-8 over this stream.
> 
> Ack!  Pfft!  Where did *this* idea come from?
> 
> To put it more moderately, please explain why that is other than a Bad Idea.
> 
> You said you want the server to do logging.  ANSI cursor control seems
> like it would make that difficult, at least.
> 

Hardly so.  I wrote that code myself a long time ago -- it's utterly
trivial.

> We are working with minimal boot loaders and the like.  Do you think
> they need significant termcap fanciness?  (Are you trying to support
> BIOS config screens and the like?)
> 

Yes, or "make menuconfig" if you need to rebuild the server kernel, or
what have you.

> 
> I was thinking of something of the sophistication of Sun's serial console.
> The idea is to be able to debug booting and crashing problems in a
> headless machine in a machine room somewhere.
> 
> If not, could we have a little discussion of intended applications?
> 

That is *one* application.  The much more important application is to
*manage* a headless machine in a machine room somewhere.  The network is
a really bad choice for the crash dump application: there are way too
many things that can cause that to die.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] Getting this show on the road...

[<li...@ho...>]

> Okay, let's look at the basic design of this puppy.  I don't
> so much mean protocol details and the be or not be of security
> (which I'd rather wait with until we have a prototype) but more
> the general architecture.

Security can be postponed a bit, but a bit of protocol design would
be relevant to the architecture.

The most important piece of architecture is how it fits in to the
"client", as you have called it, the machine containing the netcon stub.
The protocol is a big enough part of that to be an "architecture" issue.


Your UA/server/client (although I'd like a different word than "client",
maybe debugger-style "target"?) design makes sense, particularly if the
netcon protocol will perform poorly over a long-haul link.

> Although the current assumption is that Netcon Protocol is a UDP
> protocol, it will provide a logical bidirectional stream capability.

Once you're supporting a bidirectional stream, think hard about
a stripped-down TCP.  The only thing you might be able to do better
with a hand-rolled one is pass state between boot loader stages.

> I really want to define a specific terminal emulation for this protocol. 
> I definitely believe we do need more than a basic terminal, and having a
> specific terminal emulation is key.  The only logical such terminal is
> some ANSI/DEC VT subset.  Character set is yet another issue; I don't
> think I have the guts to mandate UTF-8 over this stream.

Ack!  Pfft!  Where did *this* idea come from?

To put it more moderately, please explain why that is other than a Bad Idea.

You said you want the server to do logging.  ANSI cursor control seems
like it would make that difficult, at least.

We are working with minimal boot loaders and the like.  Do you think
they need significant termcap fanciness?  (Are you trying to support
BIOS config screens and the like?)


I was thinking of something of the sophistication of Sun's serial console.
The idea is to be able to debug booting and crashing problems in a
headless machine in a machine room somewhere.

If not, could we have a little discussion of intended applications?

[Netconsole-devel] Getting this show on the road...

[H. P. A. <hp...@tr...>]

Okay, let's look at the basic design of this puppy.  I don't
so much mean protocol details and the be or not be of security
(which I'd rather wait with until we have a prototype) but more
the general architecture.

I'd like to see an architecture like this:


+-------------+
| Netcon user |
|    agent    |
+-------------+
       |
 TCP protocol
       |
+-------------+
|    Netcon   |
|    server   |
+-------------+
       |
Netcon protocol
       |
+-------------+
|    Netcon   |
|    client   |
+-------------+


The purpose of the netcon server is to provide session management and
(optionally) logging support.  The netcon user agent is what an admin
uses to plug into a session.  By isolating the UI from the session
management, we allow connecting and disconnecting to sessions.

Note that this is a useful capability in itself, and we may want to
consider (later, please!) the netcon server to talk to a serial port or
PTY session as well as to a netcon client.

Although the current assumption is that Netcon Protocol is a UDP
protocol, it will provide a logical bidirectional stream capability.

I really want to define a specific terminal emulation for this protocol. 
I definitely believe we do need more than a basic terminal, and having a
specific terminal emulation is key.  The only logical such terminal is
some ANSI/DEC VT subset.  Character set is yet another issue; I don't
think I have the guts to mandate UTF-8 over this stream.

Thoughts?

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt