etherboot-developers — Discussion of developer issues

Re: [Etherboot-developers] http/dns

[<ke...@et...>]

>I believe, this should already work. The ip transmit code checks whether 
>the packet should be sent to a host on the local network, and in that 
>case sends it directly to that machine; if the the IP address does not 
>match the local network address, all packets are sent to the default 
>gateway. So, in a sense we do have a routing table -- it's just hardcoded.

Well it's just a case of a two entry routing table, local network plus
default gateway, which works for most sites. I'm not sure why the BOOTP
RFC allowed the gateway option to be a list of gateway addresses with no
means of specifying the associated netmasks.

>> But note that many sites use a HTTP proxy so you'd have to get the HTTP
>> code to understand proxies as well. The URL should also be kept short
>> since it has to fit in 128 bytes, or was it 64?
>
>It doesn't necessarily have to be this short. Things just get a little 
>less efficient, because the request will be split across multiple TCP 
>packets. In reality, this is not much of a problem.

But the initial URL is in the DHCP offer. But of course you are
proposing to hardcode it.

>A lot of organizations have a pretty strict policy against running any 
>kind of unauthorized server. That's the whole reason why I thought 
>implementing HTTP could be useful. Fortunately, most of these places 
>already have a DHCP server in place. So, they are almost set for Etherboot.

I would think such a place would also be upset if someone replaced an
existing OS with Debian. :-)

>The only part that would missing is the URL that the DHCP server should 
>sent. Users that find themselves in this position, can either try to 
>persuade their IT department to advertise the right filename, or they'll 
>have to resort to hardcoding the name into the image. I don't really 
>like the latter, but realize that it is sometimes quite useful.

There is still the problem that such strict sites will almost certainly
have a HTTP proxy so you will have to handle proxies also.

You don't really have to use Debian to persuade me that's it's a good
idea to implement HTTP and DNS, those are useful additions in
themselves, provided it doesn't cost users who don't want to compile it
in, but your scenario highlights various issues to think of for many
releases to come.

Re: [Etherboot-developers] http/dns

[Markus G. <ma...@gu...>]

Ken Yap wrote:
> Oh and don't forget HTTP support should probably include support for
> 3xx redirects, which are very commonly used. Support for metatag
> redirects is probably too hard.

The code already has support for HTTP redirects. These are reasonably 
common, and I thought users would be really confused if Etherboot did 
not attempt to resolve them.


Markus

Re: [Etherboot-developers] http/dns

[Markus G. <ma...@gu...>]

Ken Yap wrote:
> But they have to go through one menu to choose the NIC so why not do it
> all at once?

Do they? Can't we build a floppy image that includes all of our drivers? 
I thought this was supported with 5.2, but I haven't tested myself.

> I think gatewaying works for TFTP, which uses the first gateway
> specified in option-whatever as per RFC. There's no option to select a
> second or third gateway because there's no routing table. The other
> protocols should be written to do the same. This is of course providing
> the address handed out by DHCP is allowed by firewalls to connect to the
> Internet directly.

I believe, this should already work. The ip transmit code checks whether 
the packet should be sent to a host on the local network, and in that 
case sends it directly to that machine; if the the IP address does not 
match the local network address, all packets are sent to the default 
gateway. So, in a sense we do have a routing table -- it's just hardcoded.

> But note that many sites use a HTTP proxy so you'd have to get the HTTP
> code to understand proxies as well. The URL should also be kept short
> since it has to fit in 128 bytes, or was it 64?

It doesn't necessarily have to be this short. Things just get a little 
less efficient, because the request will be split across multiple TCP 
packets. In reality, this is not much of a problem.

> Well I have no objections if you write a resolver client that can be
> ifdefed. You'd have to query the DNS servers in option-whatever.

Having a resolver client would certainly be useful, if it could be made 
to fit in a small footprint.

> All this Etherboot development to support Debian loading? Now I know why
> dselect was so clunky. :-) Surely it's simpler to ask the user to run a
> HTTP proxy (probably about 50 lines of Perl) on an internal machine.

A lot of organizations have a pretty strict policy against running any 
kind of unauthorized server. That's the whole reason why I thought 
implementing HTTP could be useful. Fortunately, most of these places 
already have a DHCP server in place. So, they are almost set for Etherboot.

The only part that would missing is the URL that the DHCP server should 
sent. Users that find themselves in this position, can either try to 
persuade their IT department to advertise the right filename, or they'll 
have to resort to hardcoding the name into the image. I don't really 
like the latter, but realize that it is sometimes quite useful.



Markus

Re: [Etherboot-developers] http/dns (was: List of NICs by popularity)

[<ke...@et...>]

>Should we have support for dns resolution?
>- Yes we should, if not only for the debian case! They could have e.g.
>  etherboot-install.debian.org revolving A record to all of their http
>  servers or so!

Oh and don't forget HTTP support should probably include support for
3xx redirects, which are very commonly used. Support for metatag
redirects is probably too hard.

Re: [Etherboot-developers] http/dns (was: List of NICs by popularity)

[<ke...@et...>]

>> Except that I don't think the Debian project would appreciate one server
>> getting all the hits. So Debian should probably start a debian-o-matic
>> site and ask the user for the region in addition to asking for the NIC.
>
>They could build a menu.nbi that does this for them. It would be
>reasonably small and thus be comparative to the user hitting
>www.debian.org with their browser before they find out what their
>mirror is called exactly.

But they have to go through one menu to choose the NIC so why not do it
all at once?

>Does etherboot support loading over network boundaries at all, e.g.
>select the specified "routers" entry from the dhcp response if the
>requested ip is obviously out of range? That is one of the things I
>thought about on my long train ride home this evening...
>
>Do 5.2 and 5.0.11 support (if the proto is compiled in) URLs to be
>specified, e.g.
>tftp:// [<serverip> [:<port>]] / <filename>
>nfs:// [<serverip>] / <filename>
>?

I think gatewaying works for TFTP, which uses the first gateway
specified in option-whatever as per RFC. There's no option to select a
second or third gateway because there's no routing table. The other
protocols should be written to do the same. This is of course providing
the address handed out by DHCP is allowed by firewalls to connect to the
Internet directly.

But note that many sites use a HTTP proxy so you'd have to get the HTTP
code to understand proxies as well. The URL should also be kept short
since it has to fit in 128 bytes, or was it 64?

>How far did multicast support get?

I think Eric is disappointed that nobody fixed up the other drivers to
do multicast. Hungry for work?

>Should we have support for dns resolution?
>- Yes we should, if not only for the debian case! They could have e.g.
>  etherboot-install.debian.org revolving A record to all of their http
>  servers or so!

Well I have no objections if you write a resolver client that can be
ifdefed. You'd have to query the DNS servers in option-whatever.

All this Etherboot development to support Debian loading? Now I know why
dselect was so clunky. :-) Surely it's simpler to ask the user to run a
HTTP proxy (probably about 50 lines of Perl) on an internal machine. It
can serve on a port > 1024 so you don't have to be root and do the DNS
and HTTP proxying, including understanding proxy.pac as some sites have.
It could also contain the smarts to choose the right server.  Then you'd
only have to give Etherboot a url like:
http://10.10.10.10:8000/debian.nb and it would get forwarded to the
appropriate Debian server.

Re: [Etherboot-developers] http/dns (was: List of NICs by popularity)

[Anselm M. H. <an...@ho...>]

Hello Ken,

>>Ah, yet another incentive for getting HTTP support working on 5.2. If
>>Debian's URL was hardcoded in the Etherboot image (I know, this is 
>>normally frowned upon), the kernel could be loaded directly from 
>>Debian's server. All the user needed would be a DHCP server and the 
>>Etherboot floppy.

> Except that I don't think the Debian project would appreciate one server
> getting all the hits. So Debian should probably start a debian-o-matic
> site and ask the user for the region in addition to asking for the NIC.

They could build a menu.nbi that does this for them. It would be
reasonably small and thus be comparative to the user hitting
www.debian.org with their browser before they find out what their
mirror is called exactly.

Does etherboot support loading over network boundaries at all, e.g.
select the specified "routers" entry from the dhcp response if the
requested ip is obviously out of range? That is one of the things I
thought about on my long train ride home this evening...

Do 5.2 and 5.0.11 support (if the proto is compiled in) URLs to be
specified, e.g.
tftp:// [<serverip> [:<port>]] / <filename>
nfs:// [<serverip>] / <filename>
?

How far did multicast support get?

Should we have support for dns resolution?
- Yes we should, if not only for the debian case! They could have e.g.
  etherboot-install.debian.org revolving A record to all of their http
  servers or so!

Best regards,
 Anselm Martin Hoffmeister
 Stockholm Projekt Computer-Service
 <an...@ho...>

-- 
Disclaimer - These opiini^H^H damn! ^H^H ^Q ^[ .. :w :q :wq :wq! ^d X^?
exit X Q ^C ^c ^? :quitbye  CtrlAltDel  ~~q :~q logout save/quit :!QUIT
^[zz ^[ZZZZZZ ^H  man vi ^@ ^L ^[c ^# ^E ^X ^I ^T ? help helpquit ^D ^d
man help ^C exit ?Quit ?q CtrlShftDel "Hey, what does this button d..."

Re: [Etherboot-developers] List of NICs by popularity

[<ke...@et...>]

>Ah, yet another incentive for getting HTTP support working on 5.2. If 
>Debian's URL was hardcoded in the Etherboot image (I know, this is 
>normally frowned upon), the kernel could be loaded directly from 
>Debian's server. All the user needed would be a DHCP server and the 
>Etherboot floppy.

Except that I don't think the Debian project would appreciate one server
getting all the hits. So Debian should probably start a debian-o-matic
site and ask the user for the region in addition to asking for the NIC.

Re: [Etherboot-developers] List of NICs by popularity

[Markus G. <ma...@gu...>]

Anselm Martin Hoffmeister wrote:
> Could you support debian be installed from an etherboot NBI?
> Meaning... you setup a tftp/dhcp server on one debian server box and
> then - putting one rom-o-matic floppy into a client - can install from
> a netbooted kernel and a ftp.*.debian.org server? I think it's possible
> with some manual tweeking right now, but why not support right out of
> the box?

Ah, yet another incentive for getting HTTP support working on 5.2. If 
Debian's URL was hardcoded in the Etherboot image (I know, this is 
normally frowned upon), the kernel could be loaded directly from 
Debian's server. All the user needed would be a DHCP server and the 
Etherboot floppy.

I have a preliminary patch for 5.2, but there are still some bugs in the 
TCP flow control, and the code currently suffers from corruption of the 
downloaded image file. I hope, I'll have a working patch soon.


Markus

Re: [Etherboot-developers] List of NICs by popularity

[Anselm M. H. <an...@ho...>]

Hello Alastair,

> I'm a debian developer working on the next-generation Debian Installer.

While that is talked about...

> For people still using floppies + net to install from, we need to make
> all the NIC driver modules for Linux available on floppies - this will
> take several floppies, and wish to prioritise them according to how
> popular each of the NICs are.

Could you support debian be installed from an etherboot NBI?
Meaning... you setup a tftp/dhcp server on one debian server box and
then - putting one rom-o-matic floppy into a client - can install from
a netbooted kernel and a ftp.*.debian.org server? I think it's possible
with some manual tweeking right now, but why not support right out of
the box?

Best regards,
 Anselm Martin Hoffmeister
 Stockholm Projekt Computer-Service
 <an...@ho...>

[Etherboot-developers] Etherboot and the RTL8139D

[Timothy L. <tl...@ro...>]

Hi

There have been reports that the current rtl8139 etherboot driver does
not work for the "D" revision of the RTL8139 (i.e. RTL8139D).

If you have a model of this card that does not work (I would also like
to hear if it does work), please send me some information about the
card.  I am looking to obtain a card, so I need the card manufacturer,
card model, where you bought it.

If someone has an extra one of these cards that I could have that would
be great (I am willing to trade for another card).  When I get some
time, I will try to take a look at the Linux driver and update the
Etherboot driver, but it would be a lot easier if I had the card.

Regards

Tim Legge
Etherboot Developer

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.502 / Virus Database: 300 - Release Date: 7/18/2003
 

[Etherboot-developers] PCNET32 driver has been ported to Etherboot

[Timothy L. <tl...@ro...>]

Hi All

I just checked in pcnet32.c to the cvs.  As you know, the old lance
driver did not support relocation in 5.1/5.2.  This initial version of
the driver supports both 5.0 and 5.2.  

I have successfully tested it on two different pcnet based NICS and I
have successfully tested the 5.0 version with the VMWare pcnet based
virtual nic.

Now that a driver supports relocation for pcnet based nics, we just need
to figure out how to make 5.2 work with vmware.

Regards

Tim

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.502 / Virus Database: 300 - Release Date: 7/18/2003
 

[Etherboot-developers] webcam up! http://www.thinguin.org/

[Marty C. <ma...@et...>]

The LinuxWorld Expo webcam is up!

Come see us at:

     http://www.thinguin.org/

There are lots of people at the show, and we're having a nice time 
talking about network booting.

Marty

-- 
     Try: http://rom-o-matic.net/ to make Etherboot images instantly.

    Name: Marty Connor
US Mail: Entity Cyber, Inc.; P.O. Box 391827;
          Cambridge, MA 02139; USA
   Voice: (617) 491-6935; Fax: (617) 491-7046
   Email: md...@et...
     Web: http://www.etherboot.org/

Re: TCP and HTTP support (Was: [Etherboot-developers] 3c900-tpo hang after DHCPOFFER)

[<ja...@Mc...>]

Markus,

You never fail to amaze me.

I can't wait to buy you a beer (or 3) this week, while in San Fran.

Jim McQuillan
ja...@Lt...



On Sun, 3 Aug 2003, Markus Gutschke wrote:

> Ken Yap wrote:
> >>Dig all you want, but I still like my HTTP idea :)
> > 
> > So do I but I think it should be done by writing a server side TFTP to
> > HTTP proxy. I might knock up a Perl version sometime. That might scare
> > some of you into doing it first with a more readable language. :-)
> 
> I always thought that HTTP couldn't be that difficult to do, if it was 
> strictly limited to the small subset that Etherboot needs. 15 hours 
> later, I have the proof ;-)
> 
> The attached patch is against 5.0.11, as that was easiest to test with; 
> I might try to port it next week at LinuxExpo, if I can find a more 
> up-to-date test environment.
> 
> Now, whether this is really a good idea, and whether it should go into 
> Etherboot, is a a completely different question. I leave that up to Ken.
> 
> 
> 
> Markus
> 

-- 

Re: [Etherboot-developers] ANNOUNCE: limited tagged image support for grub (0.93)

[<ke...@et...>]

>Definitely yes in a perfect world, but reality is biting: think of PXE. 
>But in the etherboot case, we can at least try to get there.

Well the problem of booting images without having to prepare them is an
old one. Frankly I don't see image preparation as a hassle, but then
people are lazy and also people prefer GRUB to LILO because you don't
have to remember to run LILO after changing the kernel, for the same
reason of laziness. PXE loads a first stage which then makes calls to
the core to load the kernel. It's one way of doing it, but I don't like
the specification and implementation.

I toyed with the idea of loading header only files which would not be
jumped to but be used to guide the loading of the rest of the image.
Maybe we could define an indirect segment type which would not contain
data to be placed in memory but a URL to where the data can be fetched.
Like a symlink in fact.

Another idea is maybe Etherboot could acquire the ability to load
dynamic modules, that would have powerful (and dangerous) uses. But it
might make the compressed code too big to fit in 32kB of ROM which I
think is a reasonable goal for the NICs on the market.

Anyway if you have any ideas, feel free to post.

Re: TCP and HTTP support (Was: [Etherboot-developers] 3c900-tpo hang after DHCPOFFER)

[Markus G. <ma...@gu...>]

Eric W. Biederman wrote:
> Ok.  The important thing is that the stack does not have any nasty
> interaction effects when congestion occurs.  Which I know some of the
> proposed TCP algorithms have.  The worse case I can imagine is everyone
> assuming they have an infinite TCP window and just blasting packets.
> That only works for one client, and one server, on an unoccupied network.

The following is a discussion of the short-cuts that I took and why I do 
not expect them to be cause for any serious problems; if somebody can 
point out why those assumptions are wrong, I'll try to implement fixes.

Technically, my code is broken, because it does not check window sizes 
for sending data. It should not be overly complicated to add that code, 
but realistically I'd never expect it to trigger.

In the case of HTTP (and iSCSI is probably similar), we only ever send a 
very small packet at the beginning of the connection and afterwards read 
a large response. If the sending window is originally closed, we 
incorrectly keep retransmitting data, whereas we really should wait 
until the window opens before attempting to do so. But if we do manage 
to trigger this problem, the server has far bigger problems (i.e. 
accepting new TCP connections, but not having any available network 
buffers). And the packets that we send do not actually cause any 
problems other than using up some network bandwidth.

The other bug is that we only send a single packet and then wait for the 
ACK. While not completely wrong, waiting for the ACK causes a small 
performance penalty. This is a deliberate choice, because the code gets 
a lot easier if the only packet which could ever need retransmission is 
the very last one we sent. As we typically never send more than one 
packet worth of data during the entire connection (the HTTP GET request 
is quite small), implementing a sliding window on the sending side is 
quite some overkill.

As for dealing with congested networks, I believe this is mostly done on 
the server side, although I will check my documentation and see if there 
is anything we should do (other than sending an early ACK if we see out 
of order packets, but the code already handles that case). The server 
just keeps sending data at us as fast as it thinks OK, and we keep 
acknowledging it as we get it, and adjusting the window size as needed; 
I believe, there is not much more that we can do about congestion control.

The last protocol violation that I am aware of, is the missing protocol 
state for receiving the final ACK package in the session. This is not 
typically a problem; but if our FIN/ACK got dropped, this will keep us 
from retransmitting it and thus the server could stay in FIN_WAIT_1 
until the connection times out. I do not expect this bug to show unless 
the network is very congested (and even then, its not a big problem), 
but if anybody ever notices, it can be fixed with a little effort.

Another simplification of the code is that it only ever allows for one 
open connection, but none of my use cases are affected by that. It also 
does not implement listening for incoming connections, but those are 
unlikely to ever be needed in a boot Prom. And finally, it currently 
does not allow for sending anything other than one initial block of 
data; I think, if needed, this restrictions could be removed with a one 
or two line change, though.

In short, this TCP stack follows the same design philosophy as the rest 
of Etherboot. It does not attempt to be a full general purpose 
implementation, but rather something very compact which is tailored 
exactly to the use patterns that we need. If you think that any of the 
above limitations are not acceptable, make a case for it, and I'll see 
if I can address it.

> The hard case is the initial block of data.  Which is assumed to have
> enough information to identify the file formant and to start parsing
> the file.  I believe several of the OS loaders will choke if this is
> less than 512 bytes.  After that we are pretty safe, variable size TFTP
> blocks need to be handled already.  They all are all >= 512 bytes but
> there is nothing like a power of two requirement.  We should be able
> to handle 513 byte packets without a hitch.

Again, I'd be very surprised, if any real-life deployment caused 
problems here. Given how all modern network stacks (other than my really 
simple-minded one ;-) implement Nagle and/or have applications that try 
to fill buffers before sending, the first packet received should almost 
certainly contain more than 512 bytes of payload. The only exceptions 
that I could see would be unusually small MTUs (unlikely in a LAN) or 
unusually large HTTP headers in front of the payload.

It would be easy enough to reblock to 512 chunks, but it is going to 
take up 512 of space in the BSS or on the stack. I'd rather wait for bug 
reports from the field, before implementing an expensive work-around for 
a bug that never triggers. I could be persuaded otherwise, if somebody 
feels very strongly about this.



Markus

Re: [Etherboot-developers] references for booting a PC

[<ke...@et...>]

>However, there is still much I do not understand about booting a PC. For
>example, I still don't quite understand why it's necessary to relocate to the
>top of the memory, just below the 4Gb limit.

The main reason is it gives Etherboot more room to work in so we can use
the larger drivers that recent NICs require, and also put in the
features users ask for but I keep resisting :-). Low memory is already
crowded.

>My question is: are there any books, online references or manufacturer
>specifications I could read in order to have a better understanding of bootin
>g a PC?

Etherboot is not a first stage booter. Your PC is already booted when
Etherboot gains control via the industry standard hooks provided by the
BIOS. Etherboot is a kernel loader, kernel being loosely defined as the
initial program.  It's in the same family of software as LILO, GRUB,
SYSLINUX, etc, only it boots from a network connection, not from
rotating media.

To learn how to initialise a PC's hardware you should look for documents
that explain how a BIOS works.  Initialising hardware is usually chipset
specific and done by the BIOS. Perhaps you might get something out of
reading LinuxBIOS.

[Etherboot-developers] references for booting a PC

[<in...@ca...>]

I would like to have a better understanding of the booting process of a PC. I
have read the developper manual of etherboot and learned quite a bit, notably
from the section "The execution environment of Etherboot". I have also read the
source code and learned even more, notably from floppyload.S and start16.S.
However, there is still much I do not understand about booting a PC. For
example, I still don't quite understand why it's necessary to relocate to the
top of the memory, just below the 4Gb limit.

My question is: are there any books, online references or manufacturer
specifications I could read in order to have a better understanding of booting a PC?

My suggestion is: other people might find the answer to the above question
useful, so perhaps a reference section could be added to the developper manual.

Re: [Etherboot-developers] Re: TCP and HTTP support

[<ke...@et...>]

>> if you patch against 5.3.0 (5.1.10 is close enough) and remove all the
>> large buffers from the stack
>
>I had been wondering how to do that best. I don't really need much 
>space. The only large structures are the ones that hold the current HTTP 
>header/state (about 100 bytes), any URL found in redirects (80 bytes), 
>and the buffer used for assembling outgoing packets (1500 bytes). The 
>latter is the problem. I know that none of my packets are large, so I 
>guess I could just shrink this buffer (576 would make a lot of sense), 
>and impose a maximum segment size for outgoing TCP packets. Would that 
>be acceptable?
>
>Alternatively, I could allocate the buffer in the BSS.
>
>What is the prefered solution?

Put large objects in the BSS, there's no ROM space penalty for that and
I'm less concerned about footprint now that Etherboot runs relocated.  I
don't mind a full size packet buffer. The size should be configurable as
I suspect somebody called Eric may want to use Etherboot across media
with larger frame sizes someday. :-)

What I think will be interesting is learning how well it plays with
server TCP stacks, whether breaking off a boot will leave dangling
connections which have to timeout, and that sort of thing. And of
course, if 50 clients all boot at once, the web server will think it's
under attack. Anyway Jim's mob will do this testing for us. :-)

Re: [Etherboot-developers] stumbling block for 5.2

[<ke...@et...>]

>Just a small comment on what I was thinking when I coded this.
>1) Unless we reserve all of the memory dos is using it is non-trivial
>   to return to, especially after etherboot has had a failed image load.
>   So the quit function was just a best effort kind of thing.

It may be necessary to use .exe format to ensure that all the memory
that Etherboot uses is known to DOS and also to support images > 64kB.
But this would mean obtaining information from the load map or branching
off earlier than .img. I'm not in a hurry to do anything about this.
.com is just a carrier. Maybe there's no public demand for fixing the Q
problem.  Maybe somebody will want it badly enough to fix it himself.
I'll leave it as is.

The 5.2 tarball has been uploaded and is ready to release. The CVS area
is in place. I'm just wondering when will be the first 5.2.1 patch. :-)

Re: [Etherboot-developers] ANNOUNCE: limited tagged image support for grub (0.93)

[Hans-Peter J. <hp...@ur...>]

Hi Eric,

thank you very much for your comments. It's always a pleasure for me.

On Sunday 03 August 2003 19:52, Eric W. Biederman wrote:
> me wrote:
> >
> > Yep, I've looked at it. In general, if I need to load some
> > secondary menuing system, it does't matter, if this is 10L or 166K.
> > But when I'm forced to use a pxe loader, etherboot >= 5.1 isn't an
> > option anymore, since despite of the pxe target, it doesn't provide
> > any surplus value anymore, no, better than that, it creates a nice
> > useless boot loop. Options: use 5.0 with menus enabled, or go the
> > grub way.
>
> I have no problem at all using 5.1 with pxe.  All it takes is a
> single if statement in the dhcpv3 configuration file to distinguish
> between etherboot and pxe.

Well, you're right, off cause. It's more the comfort of menus in my 
usual office like environments, I highly depend on: I'm surrounded with 
5 systems here, while none has a harddisk running! In fact, I don't 
like harddisks at all in their single form, doing my hd math like: 3+1 
hot spare most of the time ;-).
When it comes to linux installations, repartition or rescue harddisks, I 
usually not even touch any removable media. But I can imagine that 
these are old hats for you.

> > The nice thing in grub from developers pov is the approach to
> > provide some limited posix like f* functions to operate on files
> > (one at a time), which seems to work fine with tftp, too.
>
> I am amazed you are looking at the grub code and not going blind...
> I guess if you are not digging into it too deeply the code looks o.k.

Well, I kept my eyesight and have some experience with fragile 
(commercial) software systems, where even grub compares positively 8|. 
As long as I get it done in a timely fashion, I simply don't look.

That project seems to be missing a benevolent dictatorship, other 
projects get their notable, but ordered advances from, like what is 
happening in etherboots lucky case, especially when its leader is 
supported such greatly from some magic genius called Eric :) 

BTW: the real fun comes in with the gfxboot package, which is an ps like 
interpreter, implemented in assembler (140K source), with hooks for
3 different boot loaders: syslinux, lilo and grub. crazyness approaching
infinity!

> > This is one thing, I didn't like in this scenario: the repetition
> > of device drivers in the different loaders. Much cooler would be to
> > install and use an interrupt vector for basic nic device
> > operations, such that chained boot loaders are able to use one set
> > of device drivers. That would spare me the now necessary port of
> > the current etherboot drivers to grub to not loose the track. I
> > hope, you don't have any objections.
>
> Chained drivers are so wrong.   The reason they are wrong is very
> simple, software on a ROM is never updated.  As long as the software
> on the ROM is good enough to do it's one thing that is all that is
> required of it.

Definitely yes in a perfect world, but reality is biting: think of PXE. 
But in the etherboot case, we can at least try to get there.

> As for porting please notice that the 5.2 drivers are much easier to
> host.

Thanks to Axel Dittrich, I've also an example backport of e1000 drivers, 
which I ported over to grub without any major hassles. I expect to get 
it also done timely.

> Now if you want something really fun that saves all of the work
> of replicating drivers.  I can hook you up with a kernel with kexec
> support and you can use all of it's drivers and boot anything you
> want.

That sounds like a crazy idea, but doesn't that impose opening a bulk of 
other cans of worms: libc, complexity, etc.. One real nice thing of 
etherboot is its concise implementation. In the linux bios case, I can 
see all its beauties, but do you expect to equip nic flash roms with a 
linux (micro?) kernel, just to boot linux? Well, the e1000 ships with a 
256K flash. BTW: Does somebody has flashed that parts from linux? It's
one of the tasks, I currently do within a nbi dos session...

> Eric

Kind regards,
Pete

Re: TCP and HTTP support (Was: [Etherboot-developers] 3c900-tpo hang after DHCPOFFER)

[<ebi...@ln...>]

Markus Gutschke <ma...@gu...> writes:

> This would be different, if somebody really went to the trouble of
> implementing more complicated TCP based protocols (such as the iSCSI
> stuff that you mentioned). If there is a lot of data being sent in
> both directions, then the TCP implementation probably needs to be more
> sophisticated than what I do right now.

Even iSCSI would be mostly a read case. Just a whole bunch
of read commands probably 64K reads at a time or 512 bytes if we are
plain lazy.  We don't have write support in any of our disk drivers
and I don't see that making any kind of sense, in a bootloader anyway. 
 
> As for the more advanced tuning algorithms that modern TCP stacks use
> (e.g. slow start, restart after a stopped connection, ECN, ...), I'll
> check with my documentation and see if any of those are applicable,
> but I don't expect that much of it is useable.

Ok.  The important thing is that the stack does not have any nasty
interaction effects when congestion occurs.  Which I know some of the
proposed TCP algorithms have.  The worse case I can imagine is everyone
assuming they have an infinite TCP window and just blasting packets.
That only works for one client, and one server, on an unoccupied network.
 
> >>>3) Are alignment considerations that need to be taken care of.
> >>
> >>??? Can you elaborate what you were thinking of?
> > Compilers on non-x86 assume there certain kinds of data are aligned
> > to certain boundaries.  4 bytes for a 32bit int for example.  Network
> > packets have a tendency to misalign structures.
> > For running the code on the Itanium for example it is very important
> > that we don't have that kind of issue.
> 
> I see. I don't believe any of my code violates any alignment
> requirements (other than the HTTP handler calling getdec() in a
> slightly broken way; we really should have something like getndec()),
> but I am not sure about the OS loaders. The TCP code just hands the
> received data off as it gets it. Can all of our loaders deal with
> arbitrarily fragmented blocks or do I have to reassemble blocks before
> I can hand them off? If I have to do this, what block size do I have
> to use? Is any power of two OK, or does it have to be 512 bytes?

The hard case is the initial block of data.  Which is assumed to have
enough information to identify the file formant and to start parsing
the file.  I believe several of the OS loaders will choke if this is
less than 512 bytes.  After that we are pretty safe, variable size TFTP
blocks need to be handled already.  They all are all >= 512 bytes but
there is nothing like a power of two requirement.  We should be able
to handle 513 byte packets without a hitch.

Eric

Re: TCP and HTTP support (Was: [Etherboot-developers] 3c900-tpo hang after DHCPOFFER)

[Markus G. <ma...@gu...>]

>>>2) How well does it the tcp stack work in a congested environment.
>>>   - Does it play badly with other simultaneous tcp connections?
>>>   - Does the implemented window scaling work well.
>>
>>I don't have much in the form of test environments. Hopefully, somebody else can
>>
>>take a stab at testing the code in more real-life situations.
> 
> 
> Some of that can be done by simple inspection and theoretical calculation.
> Testing helps two of course.  But I know I put in all of the exponential
> back off code into etherboot based on pure inspection.

Apart from the fact that I forgot all my documentation at work and had 
to base the code on what I remembered and what I could find online, I 
think the algorithm used mostly makes sense.

The window scaling algorithms that you usually find in TCP stacks assume 
that the kernel receives packets in interrupt service handlers and 
delivers them once the user space application is ready to consume data. 
If the application cannot consume fast enough, then flow control must 
kick in.

None of this is really true for Etherboot. Receiving data is the bottle 
neck, as we can only poll the driver and not all cards support 
sufficiently large buffers; but once the data is received, it can be 
consumed immediately. Also, we don't really know, if we just dropped 
packets on the floor, because we did not poll fast enough.

That's why my code slowly grows the window size, but as soon as it 
discovers problems, shrinks it back to half. It could try to do smarter 
things based on timeouts and RTT estimates, but I am not sure this would 
really help. On the other hand, in my small test environment, I have 
never seen it try to shrink the window; but in more congested networks 
or on slower clients, I wouldn't be surprised if it happened occasionally.

In fact, I do try to correctly compute smoothed estimates of the 
round-trip-time, but realistically that is just overkill. The client 
never sends more than the initial SYN packet, one data packet and 
occasional ACKs. This is definitly not enough to estimate RTT very 
precisely.

This would be different, if somebody really went to the trouble of 
implementing more complicated TCP based protocols (such as the iSCSI 
stuff that you mentioned). If there is a lot of data being sent in both 
directions, then the TCP implementation probably needs to be more 
sophisticated than what I do right now.

As for the more advanced tuning algorithms that modern TCP stacks use 
(e.g. slow start, restart after a stopped connection, ECN, ...), I'll 
check with my documentation and see if any of those are applicable, but 
I don't expect that much of it is useable.

>>>3) Are alignment considerations that need to be taken care of.
>>
>>??? Can you elaborate what you were thinking of?
> 
> 
> Compilers on non-x86 assume there certain kinds of data are aligned
> to certain boundaries.  4 bytes for a 32bit int for example.  Network
> packets have a tendency to misalign structures.
> 
> For running the code on the Itanium for example it is very important
> that we don't have that kind of issue.

I see. I don't believe any of my code violates any alignment 
requirements (other than the HTTP handler calling getdec() in a slightly 
broken way; we really should have something like getndec()), but I am 
not sure about the OS loaders. The TCP code just hands the received data 
off as it gets it. Can all of our loaders deal with arbitrarily 
fragmented blocks or do I have to reassemble blocks before I can hand 
them off? If I have to do this, what block size do I have to use? Is any 
power of two OK, or does it have to be 512 bytes?


Markus

Re: [Etherboot-developers] stumbling block for 5.2

[<ebi...@ln...>]

ke...@et... (Ken Yap) writes:

> >The reason zeroing BSS wipes out the return path is simple: BSS contains
> >the 32 bit stack with the return address on it! Now to figure out what
> >to do about this.
> 
> Ok, I've fixed this. I put the protected mode stack in its own segment
> between _ebss and _end, so that it isn't zeroed at the beginning of
> main.
> 
> But that's not all that was needed. At the start of comprefix.S I
> changed %ss to somewhere out of Etherboot's way since the real mode
> stack with the return address was being clobbered by Etherboot too.
> 
> So now it returns to the DOS prompt ok, but DOS doesn't quite work right
> after that, I don't know if this is because of the old FreeDOS I'm using
> or Etherboot is messing up FreeDOS another way, possibly via the GDT. If
> somebody could try .com images with another DOS it would be appreciated.
> The latest sources are in CVS.
> 
> And now the ROM version exits to the BIOS ok too after the stack segment
> fix.

Ouch.  Was this failing before?  I am pretty certain that case was working.

>  Marty and anybody else with a moment to spare, could you please
> check out the latest changes and test the ROM version (really burn it on
> an EPROM or EEPROM, not the floppy simulation) and check that the Quit
> command returns control to the BIOS to boot the next device in the BIOS
> list? It works for this ISA NIC of mine but I'd like confirmation with
> PCI NICs.
> 
> I will release 5.1.10 (release candidate 2) in a day or two.

Just a small comment on what I was thinking when I coded this.
1) Unless we reserve all of the memory dos is using it is non-trivial
   to return to, especially after etherboot has had a failed image load.
   So the quit function was just a best effort kind of thing.

I did burn it in a ROM to very that case worked.  But I really didn't verify
anything else.

Eric

[Etherboot-developers] A few random thoughts....

[<ebi...@ln...>]

- Talking to some BIOS developers it became clear that if they had
  a good alternative to the current situation they would change.

  So there is a very practical case to implementing PXE support.

- On the other side of things I have a solution for very high powered
  bootloaders.  The linux kexec system call.  And I am debating with
  myself with how I should bring this out as a usable piece of code.
  Especially since I have places where I can use it that are still
  in ROM firmware.

  For a disk based bootloader I can do some very awesome things.  Like
  a CD or even a floppy that will enable a network boot off of any driver
  in the Linux kernel.

  So I have two questions...
  1) Would it make any sense to port etherboot to the Linux kernel
     because of it's tiny udp/ip stack.  For my space constrained
     situations.

  2) How do I keep from competing two strongly with etherboot if I
     unveil the all singing all dancing bootloader base on the Linux
     kernel?

Eric

Re: TCP and HTTP support (Was: [Etherboot-developers] 3c900-tpo hang after DHCPOFFER)

[<ebi...@ln...>]

Markus Gutschke <ma...@gu...> writes:

> Eric W. Biederman wrote:
> > A couple of interesting things to find out are:
> > 1) How much does it increase the size of a rom?
> 
> A quick inspection of the object files suggests that selecting
> DOWNLOAD_PROTO_HTTP increases the image size by about 2kB.

Then it looks like it is probably useable.  The network drivers
are increasing faster than that.
 
> > 2) How well does it the tcp stack work in a congested environment.
> >    - Does it play badly with other simultaneous tcp connections?
> >    - Does the implemented window scaling work well.
> 
> I don't have much in the form of test environments. Hopefully, somebody else can
> 
> take a stab at testing the code in more real-life situations.

Some of that can be done by simple inspection and theoretical calculation.
Testing helps two of course.  But I know I put in all of the exponential
back off code into etherboot based on pure inspection.

> > 3) Are alignment considerations that need to be taken care of.
> 
> ??? Can you elaborate what you were thinking of?

Compilers on non-x86 assume there certain kinds of data are aligned
to certain boundaries.  4 bytes for a 32bit int for example.  Network
packets have a tendency to misalign structures.

For running the code on the Itanium for example it is very important
that we don't have that kind of issue.

Eric