netconsole-devel

Re: [Netconsole-devel] Getting this show on the road...

[<li...@ho...>]

> Okay, let's look at the basic design of this puppy.  I don't
> so much mean protocol details and the be or not be of security
> (which I'd rather wait with until we have a prototype) but more
> the general architecture.

Security can be postponed a bit, but a bit of protocol design would
be relevant to the architecture.

The most important piece of architecture is how it fits in to the
"client", as you have called it, the machine containing the netcon stub.
The protocol is a big enough part of that to be an "architecture" issue.


Your UA/server/client (although I'd like a different word than "client",
maybe debugger-style "target"?) design makes sense, particularly if the
netcon protocol will perform poorly over a long-haul link.

> Although the current assumption is that Netcon Protocol is a UDP
> protocol, it will provide a logical bidirectional stream capability.

Once you're supporting a bidirectional stream, think hard about
a stripped-down TCP.  The only thing you might be able to do better
with a hand-rolled one is pass state between boot loader stages.

> I really want to define a specific terminal emulation for this protocol. 
> I definitely believe we do need more than a basic terminal, and having a
> specific terminal emulation is key.  The only logical such terminal is
> some ANSI/DEC VT subset.  Character set is yet another issue; I don't
> think I have the guts to mandate UTF-8 over this stream.

Ack!  Pfft!  Where did *this* idea come from?

To put it more moderately, please explain why that is other than a Bad Idea.

You said you want the server to do logging.  ANSI cursor control seems
like it would make that difficult, at least.

We are working with minimal boot loaders and the like.  Do you think
they need significant termcap fanciness?  (Are you trying to support
BIOS config screens and the like?)


I was thinking of something of the sophistication of Sun's serial console.
The idea is to be able to debug booting and crashing problems in a
headless machine in a machine room somewhere.

If not, could we have a little discussion of intended applications?

Re: [Netconsole-devel] Getting this show on the road...

[<li...@ho...>]

> Actually, the biggest reason *not* to use TCP is what happens when the
> server dies, and then is restarted.  It is utterly imperative that it has
> no effect on the clients, but you should still resume the connection
> (even though you have probably lost data) which is unachievable using TCP
> on stock Unix systems.

Excellent point!  The logging server has to maintain logs and associated
sequence numbers on stable storage, so it can resume the conversation at
te right place.  This means that we can't use in-kernel TCP, so...

>> We are working with minimal boot loaders and the like.  Do you think
>> they need significant termcap fanciness?  (Are you trying to support
>> BIOS config screens and the like?)

> Yes, or "make menuconfig" if you need to rebuild the server kernel, or
> what have you.

Um, once you have enough envronment running that you can *run* make
menuconfig, you have enough that you can just use the regular kernel,
ssh and the like.


But I understand the disconnect now.  I was thinking /var/log/kern.log
and you were thinking /dev/tty1.

You want to export the keyboard and screen across the network for
everything.


One point that raises is that you want to be able to connect even if the
headless box has spewed so much that its buffers have overrun and the server
has lost sync.  Does the headless box have to keep track a virtual
screen that can be downloaded at will?  Or is it okay to say that it might
get corrupted and require a ^L to fix?

> That is *one* application.  The much more important application is to
> *manage* a headless machine in a machine room somewhere.  The network is
> a really bad choice for the crash dump application: there are way too
> many things that can cause that to die.

When I said "crashing problems", I meant "things that print a message to
the screen and lock the machine."


In *this* case, a big issue will be sharing access to the ethernet with
the OS driver.  One way to do it, that I have used for serial ports is
to maintain an interrupt-safe buffer of data to send, which normally
the low-level code queues and the OS device drivers sends out.

But if the buffer overruns, the low-level code leaves interrupts off
and runs a special polling driver to push the buffer out synchronously,
on the asumption that the OS is hosed.

I don't know if this is amenable to ethernets.  How *are* we going to
do this?  Require a dedicated NIC for the console?  It's actually not
unreasonable given the price of NICs and hub ports these days.

Re: [Netconsole-devel] Getting this show on the road...

[H. P. A. <hp...@tr...>]

li...@ho... wrote:
> 
> Excellent point!  The logging server has to maintain logs and associated
> sequence numbers on stable storage, so it can resume the conversation at
> te right place.  This means that we can't use in-kernel TCP, so...
> 

You can't even require that.  You have to assume that the server
physically was destroyed; you still need to be able to resume
connectivity; obviously data loss.

> >> We are working with minimal boot loaders and the like.  Do you think
> >> they need significant termcap fanciness?  (Are you trying to support
> >> BIOS config screens and the like?)
> 
> > Yes, or "make menuconfig" if you need to rebuild the server kernel, or
> > what have you.
> 
> Um, once you have enough envronment running that you can *run* make
> menuconfig, you have enough that you can just use the regular kernel,
> ssh and the like.

Assuming it's safe for you to enable it.
 
> But I understand the disconnect now.  I was thinking /var/log/kern.log
> and you were thinking /dev/tty1.
> 
> You want to export the keyboard and screen across the network for
> everything.

Yes, way, way early before the kernel even boots.
 
> One point that raises is that you want to be able to connect even if the
> headless box has spewed so much that its buffers have overrun and the server
> has lost sync.  Does the headless box have to keep track a virtual
> screen that can be downloaded at will?  Or is it okay to say that it might
> get corrupted and require a ^L to fix?

I think so.
 
> When I said "crashing problems", I meant "things that print a message to
> the screen and lock the machine."
> 
> In *this* case, a big issue will be sharing access to the ethernet with
> the OS driver.  One way to do it, that I have used for serial ports is
> to maintain an interrupt-safe buffer of data to send, which normally
> the low-level code queues and the OS device drivers sends out.
> 
> But if the buffer overruns, the low-level code leaves interrupts off
> and runs a special polling driver to push the buffer out synchronously,
> on the asumption that the OS is hosed.
> 
> I don't know if this is amenable to ethernets.  How *are* we going to
> do this?  Require a dedicated NIC for the console?  It's actually not
> unreasonable given the price of NICs and hub ports these days.

Most NICs these days require a lot of machinery to work at all.  I'm sure
you can isolate it, but it almost certainly requires a lot of work.  I
think there is a completely different project out there that deals with
Linux crash dumping (postmortem debugging.)  It's a fundamentally
different problem, and personally I'd like to consider that out of scope.

One thing worth considering is whether or not there should be a separate
log stream in addition to the two-way data stream, though.  A log stream
would at least avoid the "I can't log into the server console because of
all the spewage" problem.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: [Netconsole-devel] Getting this show on the road...

[H. P. A. <hp...@tr...>]

li...@ho... wrote:
> 
> > Okay, let's look at the basic design of this puppy.  I don't
> > so much mean protocol details and the be or not be of security
> > (which I'd rather wait with until we have a prototype) but more
> > the general architecture.
> 
> Security can be postponed a bit, but a bit of protocol design would
> be relevant to the architecture.
> 
> The most important piece of architecture is how it fits in to the
> "client", as you have called it, the machine containing the netcon stub.
> The protocol is a big enough part of that to be an "architecture" issue.
> 
> Your UA/server/client (although I'd like a different word than "client",
> maybe debugger-style "target"?) design makes sense, particularly if the
> netcon protocol will perform poorly over a long-haul link.
> 
> > Although the current assumption is that Netcon Protocol is a UDP
> > protocol, it will provide a logical bidirectional stream capability.
> 
> Once you're supporting a bidirectional stream, think hard about
> a stripped-down TCP.  The only thing you might be able to do better
> with a hand-rolled one is pass state between boot loader stages.
> 

Actually, the biggest reason *not* to use TCP is what happens when the
server dies, and then is restarted.  It is utterly imperative that it has
no effect on the clients, but you should still resume the connection
(even though you have probably lost data) which is unachievable using TCP
on stock Unix systems.

Some multiport serial cards have been known to glitch the TxD line during
initialization.  The system at the other end tends to interpret that as a
BREAK.  I have seen what happens if the multiport serial card happens to
be hooked up to a rackful of Suns.  We most emphatically do NOT want the
equivalent problem.

> > I really want to define a specific terminal emulation for this protocol.
> > I definitely believe we do need more than a basic terminal, and having a
> > specific terminal emulation is key.  The only logical such terminal is
> > some ANSI/DEC VT subset.  Character set is yet another issue; I don't
> > think I have the guts to mandate UTF-8 over this stream.
> 
> Ack!  Pfft!  Where did *this* idea come from?
> 
> To put it more moderately, please explain why that is other than a Bad Idea.
> 
> You said you want the server to do logging.  ANSI cursor control seems
> like it would make that difficult, at least.
> 

Hardly so.  I wrote that code myself a long time ago -- it's utterly
trivial.

> We are working with minimal boot loaders and the like.  Do you think
> they need significant termcap fanciness?  (Are you trying to support
> BIOS config screens and the like?)
> 

Yes, or "make menuconfig" if you need to rebuild the server kernel, or
what have you.

> 
> I was thinking of something of the sophistication of Sun's serial console.
> The idea is to be able to debug booting and crashing problems in a
> headless machine in a machine room somewhere.
> 
> If not, could we have a little discussion of intended applications?
> 

That is *one* application.  The much more important application is to
*manage* a headless machine in a machine room somewhere.  The network is
a really bad choice for the crash dump application: there are way too
many things that can cause that to die.

	-hpa

-- 
<hp...@tr...> at work, <hp...@zy...> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt