etherboot-developers — Discussion of developer issues

Re: [Etherboot-developers] About WINCE_IMAGE

[<ke...@et...>]

>Hi,
>  In Config it says
>    "-D WINCE_IMAGE
>        Add the ability to boot WINCE.... now only sis630 OK!  "
>  If I want to add WINCE_IMAGE support in other mainboards, what should I>
>do?

I suspect it means that the person who added the code only tested it
with the SiS630. I see no reason why an image format should be specific
to a particular NIC, all the Etherboot drivers offer the same API to the
core code. So give it a try and let us know if it works with other NICs.
If it does I'll remove that comment.

[Etherboot-developers] About WINCE_IMAGE

[elife e. <rim...@ho...>]

Hi,
  In Config it says 
    "-D WINCE_IMAGE 
        Add the ability to boot WINCE.... now only sis630 OK!  "
  If I want to add WINCE_IMAGE support in other mainboards, what should I 
do?

Best Regards!

_________________________________________________________________
免费下载 MSN Explorer:   http://explorer.msn.com/lccn/  

[Etherboot-developers] Etherboot 5.3.1 (development) released

[<ke...@et...>]

I have released Etherboot-5.3.1 at http://www.etherboot.org/

Changes from 5.3.0:

+ Timothy Legge rewrite proto_tftm.c, got multicast working with atftp
and enabled multicast for the Tulip.

Notes on setting up for multicast are at the Etherboot Wiki, linked to
from the documentation page.  Please test the multicast and let us know
how it works or not. Feedback on the new e1000 driver is also awaited,
so that I can make it the 5.2.x driver if all is well.

Marty, another one for rom-o-matic.

MD5 sums:

95c6c91a02bdce5b16625be75706abbb  etherboot-5.3.1.tar.bz2

SHA1 sums:

bcb7cf01af542fd47ce281a7b2e25638c3ce3081  etherboot-5.3.1.tar.bz2

RE: [Etherboot-developers] Re: TCP and HTTP support

[Timothy L. <tl...@ro...>]

> I meant just steal the idea. Clients who don't have the whole file go
> asking around for someone who has some or all of the missing pieces.
It
> was the bitmap of blocks in both slam and bittorrent that triggered my
> pattern matching brain cells.

That was part of the reason I thought of tftm sending out the packets
they received.  A client that comes on line part way through the
multicast file receives the last half of the file from the server and
must wait till another client asks for the initial blocks (or it becomes
the master client).  

What if some of the clients started sending out the initial blocks?  If
sent to the same multicast address it could allow the other clients to
fill in their missing blocks without requiring the server to resend
them.  

Of course, in a setting of 10s, 100s of clients this could result in a
lot of transmits for no good reason.  I suppose tftm could be extended
to send requests to the multicast group if it is not the master client.
Other clients in the group could respond to this transmission by sending
requested blocks of the file.

Tim

Re: [Etherboot-developers] Re: TCP and HTTP support

[<ke...@et...>]

>> Or you could do something bittorrent-like.
>
>Hmm.   That is an interesting idea.
>
>The size of BitTorrent causes my some concern.  As does the fact that
>it is written in python.  Not because of python itself but more because
>people who write good low level code tend to prefer lower level languages.
>
>But I think it is worth looking at as I survey the possibilities.

I meant just steal the idea. Clients who don't have the whole file go
asking around for someone who has some or all of the missing pieces. It
was the bitmap of blocks in both slam and bittorrent that triggered my
pattern matching brain cells.

Re: [Etherboot-developers] Re: TCP and HTTP support

[<ebi...@ln...>]

ke...@et... (Ken Yap) writes:

> >> implementation of multicast).  That alternative is to build a tree
> >> of machines where the root only transmits the image to some of them.
> >
> >As I was working on tftm it crossed my mind that EB might be able to
> >transmit a received packet to another multicast address as it is
> >received.  However since the NICs are not filtering the addresses, it
> >would probably just add clutter to the network.
> 
> Or you could do something bittorrent-like.

Hmm.   That is an interesting idea.

The size of BitTorrent causes my some concern.  As does the fact that
it is written in python.  Not because of python itself but more because
people who write good low level code tend to prefer lower level languages.

But I think it is worth looking at as I survey the possibilities.

Eric

Re: [Etherboot-developers] Re: TCP and HTTP support

[<ebi...@ln...>]

"Timothy Legge" <tl...@ro...> writes:

> > implementation of multicast).  That alternative is to build a tree
> > of machines where the root only transmits the image to some of them.
> 
> As I was working on tftm it crossed my mind that EB might be able to
> transmit a received packet to another multicast address as it is
> received.  However since the NICs are not filtering the addresses, it
> would probably just add clutter to the network.

Right.  With multicast at the IP level all you need on the large scale
are multicast routers.  Nothing else should need to manually distribute
multicast packets except the transmitter.

And no machine actually owns multicast packets so that does not help much
either.  If you are on a good switch it will snoop the IGMP traffic and only
those ports that need it will get the multicast traffic.

> > novel use of http support.  As a machine comes up it can temporarily
> > be an http cache for a particular boot file.
> 
> But could it finish transmitting it before the loaded boot file takes
> over?

What beoboot (my inspiration) does is to have the boot client hang
around for a while and work as a sever.  After that point is up
and all connections are closed it will actually boot up.  But it uses
a very home grown protocol.   Which is fine, but I would love
to find a way to do that with a standard protocol.

Eric

Re: [Etherboot-developers] Re: TCP and HTTP support

[<ke...@et...>]

>> implementation of multicast).  That alternative is to build a tree
>> of machines where the root only transmits the image to some of them.
>
>As I was working on tftm it crossed my mind that EB might be able to
>transmit a received packet to another multicast address as it is
>received.  However since the NICs are not filtering the addresses, it
>would probably just add clutter to the network.

Or you could do something bittorrent-like.

RE: [Etherboot-developers] Re: TCP and HTTP support

[Timothy L. <tl...@ro...>]

> implementation of multicast).  That alternative is to build a tree
> of machines where the root only transmits the image to some of them.

As I was working on tftm it crossed my mind that EB might be able to
transmit a received packet to another multicast address as it is
received.  However since the NICs are not filtering the addresses, it
would probably just add clutter to the network.
 
> novel use of http support.  As a machine comes up it can temporarily
> be an http cache for a particular boot file.

But could it finish transmitting it before the loaded boot file takes
over?

Re: [Etherboot-developers] Re: TCP and HTTP support

[<ebi...@ln...>]

ke...@et... (Ken Yap) writes:

> >> if you patch against 5.3.0 (5.1.10 is close enough) and remove all the
> >> large buffers from the stack
> >
> >I had been wondering how to do that best. I don't really need much 
> >space. The only large structures are the ones that hold the current HTTP 
> >header/state (about 100 bytes), any URL found in redirects (80 bytes), 
> >and the buffer used for assembling outgoing packets (1500 bytes). The 
> >latter is the problem. I know that none of my packets are large, so I 
> >guess I could just shrink this buffer (576 would make a lot of sense), 
> >and impose a maximum segment size for outgoing TCP packets. Would that 
> >be acceptable?
> >
> >Alternatively, I could allocate the buffer in the BSS.
> >
> >What is the prefered solution?
> 
> Put large objects in the BSS, there's no ROM space penalty for that and
> I'm less concerned about footprint now that Etherboot runs relocated.  I
> don't mind a full size packet buffer. The size should be configurable as
> I suspect somebody called Eric may want to use Etherboot across media
> with larger frame sizes someday. :-)

I just might.
 
> What I think will be interesting is learning how well it plays with
> server TCP stacks, whether breaking off a boot will leave dangling
> connections which have to timeout, and that sort of thing. And of
> course, if 50 clients all boot at once, the web server will think it's
> under attack. Anyway Jim's mob will do this testing for us. :-)

50 clients all at once...
A bitty box of a cluster :)

I have been brainstorming now that we have multicast support working
pretty well in etherboot. 

There is an alternative to multicast that in some circumstances
may be prefered.  (Say when your switch does not provide a good 
implementation of multicast).  That alternative is to build a tree
of machines where the root only transmits the image to some of them.

Does any one know a way for a machine to tell an http server it
wants to be a proxy/caching server for it?  If so I have a very
novel use of http support.  As a machine comes up it can temporarily
be an http cache for a particular boot file.

If nothing else there are some nics (like the tg3) where streaming data
works much better than a tftp ping pong because of chipset/driver
bugs.  Although tftp has always been latency limited.

Eric

[Etherboot-developers] Multicast TFTP General Release

[Timothy L. <tl...@ro...>]

Hi Ken

I found a few minutes this morning and cleaned up a few things.  I have ifdefed the debugging information but left TFTM_DEBUG defined so that it shows for now.  

Please go ahead and release the next 5.3 version at your leisure.

The tulip driver has been Multicast enabled.  There is one small update to etherboot.h.  As per the Wiki, Multicast and tftm needs to be enabled in the Config file.

I have reliably tested that it works with three clients.  Hopefully multicast tftp will be of use to someone.

Regards

Tim Legge

Re: Re: [Etherboot-developers] Multicast TFTP

[Timothy L. <tl...@ro...>]

> Great work. When you think it's ready for release, we can cut a 5.3.1
> tarball, get it onto rom-o-matic and solicit wider testing.

Thanks, I won't get anything done this weekend, so probably Monday or Tuesday night.

Tim

Re: [Etherboot-developers] Multicast TFTP

[<ke...@et...>]

>It looks like Multicast TFTP is pretty much ready.  I have resolved the last 
>couple of issues I had.  It looks like some of the UDP checksum errors I had 
>are related to a "stupid" Intel NIC.  (I had sworn off Intel NIC's once, may 
>be I should have stuck with that).
>...
>I will continue to do some cleanup and maybe even comments  (It took me the l
>ongest time to figure out that >> 3 means devide by 8) but the structure of t
>he code will probably stay as is.  I had fleeting thought of looking at the a
>tftp client to implement a cleaner implementation, but this seems to meet my 
>needs (of course I have no real need for EB multicast tftp) and atftp is prob
>ably overkill.

Great work. When you think it's ready for release, we can cut a 5.3.1
tarball, get it onto rom-o-matic and solicit wider testing.

[Etherboot-developers] Multicast TFTP

[Timothy L. <tl...@ro...>]

Hi

It looks like Multicast TFTP is pretty much ready.  I have resolved the last couple of issues I had.  It looks like some of the UDP checksum errors I had are related to a "stupid" Intel NIC.  (I had sworn off Intel NIC's once, may be I should have stuck with that).

I was using PXE to load EB to multicast load tom's floppy linux via the same NIC.  I switched the EB to a different rom and it worked fine with the tlan.

Even the memory corruption seems to have gone away.  I am not sure why, so it may rear its head in a larger test base.  ( I have successfully used multicast EB to boot 3 machines)

I will continue to do some cleanup and maybe even comments  (It took me the longest time to figure out that >> 3 means devide by 8) but the structure of the code will probably stay as is.  I had fleeting thought of looking at the atftp client to implement a cleaner implementation, but this seems to meet my needs (of course I have no real need for EB multicast tftp) and atftp is probably overkill.

Regards

Tim

Re: [Etherboot-developers] DHCP option Bootfile name and extension path

[<ke...@et...>]

>> According to man dhcp-options, option 52 is automatically inserted by
>> dhcpd when overflow happens and cannot be configured by the user.
>
>If overflow happens, there is no space for these options in etherboot and the
>loaded images, and my solutions won't work anyway. So, not much point in
>implementing it anywhere yet without an API change.

Presumably what happens is DHCPD sees that the number of options are too
many for the packet, and that the servername and filename fields doesn't
use up all of the 64 and 128 bytes allocated to them respectively and so
decides to recover a few bytes by turning on option 52 and making the
servername and filename fields options instead.

The problem is that it would only recover a small number of bytes, and
that it would be troublesome to test; one would have to put lots of
options for this overflow to be triggered.

So I'm happy to leave it option 52 alone.

In DHCP6, all fields are options so there is no wastage of space due to
fixed length fields. But that's another story for the future.

Re: [Etherboot-developers] DHCP option Bootfile name and extension path

[Vasil V. <vas...@sy...>]

On Wed, 3 Sep 2003, Ken Yap wrote:

> >> I have one question, what happens when option overload (52) is present?
> >> I suppose that the definition of KERNEL_BUF then fails badly.
> >
> >Option overload (52) is not currently supported by Etherboot. It can be
> >trivially supported and I have implemented it. KERNEL_BUF is not a problem
> >even then within Etherboot. It becomes a problem when the bootp structure is
> >passed to loaded image, which then may modify it and return to Etherboot to
> >retry and possibly load a different image. A few images, e.g. the nbi menus,
> >expect the KERNEL_BUF where it is at the moment, and implementing BOOTP/DHCP
> >option overload (52) will mean breaking them.
> >
> >If option overload (52) is required, then just copy the options to the end of
> >the buffer would be a solution compatible with the current API. So, I've
> >corrected the implementation and the result is the patch attached for
> >etherboot 5.0.11 (as of now CVS HEAD of etherboot 5.0).
> >
> >Any comments?
>
> I don't want to add new features for a 5.0 series in maintenance mode,
> i.e. only bug fixes and demonstrably safe changes are accepted. So I'll
> accept the first set of changes but would rather leave the situation
> w.r.t.  option 52 alone. I was just double checking with you that you
> didn't need option 52. But if you want to implement this feature for
> testing in 5.3 for backporting to 5.2 or incorporation in 5.4, that's ok
> with me.

I am not fast. I thought option 52 is required somehow.

> According to man dhcp-options, option 52 is automatically inserted by
> dhcpd when overflow happens and cannot be configured by the user.

If overflow happens, there is no space for these options in etherboot and the
loaded images, and my solutions won't work anyway. So, not much point in
implementing it anywhere yet without an API change.

--
Vasil

RE: [Etherboot-developers] proto_tftm.c -- Etherboot Multicast TFTP

[Timothy L. <tl...@ro...>]

> Is this printed by Etherboot or a hardware error? I notice that the

The tomsrtbt produces the error, I will post the exact message after I
review how I call allot.

> protocol handler allocates a buffer on the heap the size of the file.
> Are you doing multiple allocations or something, like what Eric
> suggests?  Maybe the master handover isn't correct? Put some debugging
> statements there.

I will take a look.

> Another thing you can do is put some code to make a checksum of the
> whole buffer before handing it off to osloader since it's effectively
> one big block. Then you can compare this with the file checksum or
> between machines.

In this case, I don't think it is an issue with the file image but I
could be wrong.

Tim

RE: [Etherboot-developers] proto_tftm.c -- Etherboot Multicast TFTP

[Timothy L. <tl...@ro...>]

> Will each machine work individually?
> 
> Do the nodes work if they never become the master?  I don't even know
> if that is possible.

Yes, they work independently and two work together.  (I should try to
see if throwing the 486 into the mix caused issues).  It is possible for
a node to work if it never becomes master.  It simply need to receive
all the packets.  It should still send a disconnect or an error message
to tell the server that it is done.

> Are you possibly calling allot multiple times and never calling
forget.
> 

I thought I had coded to prevent that, but since the two machines that
are not master get it, you may be correct.

> What it really sounds like if this happens on startup is that the
memory
> map the kernel gets passed is corrupted.  What does the e820 map the
> kernel prints out a boot time look like?

I will take a look.

> I might be able to find some time later..  I am so swamped I can
barely
> find time to reply to some of my email. :)  That plus a set of test
> machines
> take some work to commandeer.

Indeed, I am trying to think of a way to commandeer more, but I don't
have a method at the moment.

Thanks for the hints

Tim

Re: [Etherboot-developers] proto_tftm.c -- Etherboot Multicast TFTP

[<ke...@et...>]

>Now that the corruption is out of the way, I have successfully boot two
>machines via multicast with atftp.  However, if I throw a third one into
>the mix, all three get the file and start to boot tomsrtbt.  However,
>two of the three get Out of memory errors and System Halt.

Is this printed by Etherboot or a hardware error? I notice that the
protocol handler allocates a buffer on the heap the size of the file.
Are you doing multiple allocations or something, like what Eric
suggests?  Maybe the master handover isn't correct? Put some debugging
statements there.

Another thing you can do is put some code to make a checksum of the
whole buffer before handing it off to osloader since it's effectively
one big block. Then you can compare this with the file checksum or
between machines.

Re: [Etherboot-developers] proto_tftm.c -- Etherboot Multicast TFTP

[<ebi...@ln...>]

"Timothy Legge" <tl...@ro...> writes:

> Now that the corruption is out of the way, I have successfully boot two
> machines via multicast with atftp.

Cool.

> However, if I throw a third one into
> the mix, all three get the file and start to boot tomsrtbt.  However,
> two of the three get Out of memory errors and System Halt.

Will each machine work individually?

Do the nodes work if they never become the master?  I don't even know
if that is possible.

Are you possibly calling allot multiple times and never calling forget.

What it really sounds like if this happens on startup is that the memory
map the kernel gets passed is corrupted.  What does the e820 map the
kernel prints out a boot time look like? 

> All three machines are quite different, 486-25 (3c515), P200 (eepro100),
> P4 2Ghz (eepro100).  Since all three are of different speeds, I have to
> force them to multicast by hitting the pause button and holding them
> till the others catch up.

Maybe but I can't imagine how the pause key would mess things up...

> Could this be a symptom of that?  If not, what should I look at?  If you
> have a number of similarly configured clients, the code could use some
> more realistic testing...

I might be able to find some time later..  I am so swamped I can barely
find time to reply to some of my email. :)  That plus a set of test machines
take some work to commandeer.

Eric

RE: [Etherboot-developers] proto_tftm.c -- Etherboot Multicast TFTP

[Timothy L. <tl...@ro...>]

> > I removed the -1 and my problem went away.  I did not figure it out
till
> > I used all your checks for bad packets.
> 
> Cool.  I am glad my sanity checks helped.
> 

Now that the corruption is out of the way, I have successfully boot two
machines via multicast with atftp.  However, if I throw a third one into
the mix, all three get the file and start to boot tomsrtbt.  However,
two of the three get Out of memory errors and System Halt.

All three machines are quite different, 486-25 (3c515), P200 (eepro100),
P4 2Ghz (eepro100).  Since all three are of different speeds, I have to
force them to multicast by hitting the pause button and holding them
till the others catch up.

Could this be a symptom of that?  If not, what should I look at?  If you
have a number of similarly configured clients, the code could use some
more realistic testing...

Tim

Re: [Etherboot-developers] proto_tftm.c -- Etherboot Multicast TFTP

[<ebi...@ln...>]

"Timothy Legge" <tl...@ro...> writes:

> > It should just be a simple walk of the bitmap.   The slam
> > protocol batches all of it into a single request which is certainly
> > more than you need.
> 
> I found the source of my corruption.  Slam calculates the size of the
> second last packet (???) instead of the last packet.  Does it not use
> the last packet?

I believe slam uses zero based number for it's packets, where
tftp uses one based number.  Which leads naturally to an off
by one difference between the pieces of code.  It has been a long
time since I looked at the code in detail.

> 	/* Compute the expected data length */
> 	if (packet != state.total_packets -1) {
> 		data_len = state.block_size;
> 	} else {
> 		data_len = state.total_bytes % state.block_size;
> 	}
> 
> I removed the -1 and my problem went away.  I did not figure it out till
> I used all your checks for bad packets.

Cool.  I am glad my sanity checks helped.

Eric

RE: [Etherboot-developers] proto_tftm.c -- Etherboot Multicast TFTP

[Timothy L. <tl...@ro...>]

> It should just be a simple walk of the bitmap.   The slam
> protocol batches all of it into a single request which is certainly
> more than you need.

I found the source of my corruption.  Slam calculates the size of the
second last packet (???) instead of the last packet.  Does it not use
the last packet?

	/* Compute the expected data length */
	if (packet != state.total_packets -1) {
		data_len = state.block_size;
	} else {
		data_len = state.total_bytes % state.block_size;
	}

I removed the -1 and my problem went away.  I did not figure it out till
I used all your checks for bad packets.

Tim

Re: [Etherboot-developers] DHCP option Bootfile name and extension path

[<ke...@et...>]

>> I have one question, what happens when option overload (52) is present?
>> I suppose that the definition of KERNEL_BUF then fails badly.
>
>Option overload (52) is not currently supported by Etherboot. It can be
>trivially supported and I have implemented it. KERNEL_BUF is not a problem
>even then within Etherboot. It becomes a problem when the bootp structure is
>passed to loaded image, which then may modify it and return to Etherboot to
>retry and possibly load a different image. A few images, e.g. the nbi menus,
>expect the KERNEL_BUF where it is at the moment, and implementing BOOTP/DHCP
>option overload (52) will mean breaking them.
>
>If option overload (52) is required, then just copy the options to the end of
>the buffer would be a solution compatible with the current API. So, I've
>corrected the implementation and the result is the patch attached for
>etherboot 5.0.11 (as of now CVS HEAD of etherboot 5.0).
>
>Any comments?

I don't want to add new features for a 5.0 series in maintenance mode,
i.e. only bug fixes and demonstrably safe changes are accepted. So I'll
accept the first set of changes but would rather leave the situation
w.r.t.  option 52 alone. I was just double checking with you that you
didn't need option 52. But if you want to implement this feature for
testing in 5.3 for backporting to 5.2 or incorporation in 5.4, that's ok
with me.

According to man dhcp-options, option 52 is automatically inserted by
dhcpd when overflow happens and cannot be configured by the user.

Re: [Etherboot-developers] DHCP option Bootfile name and extension path

[Vasil V. <vas...@sy...>]

On Tue, 2 Sep 2003, Ken Yap wrote:

> >I have the following patch for making a solution I needed. It also fixes a
> >small bug. I can commit this to CVS if there are no objections.
>
> I have one question, what happens when option overload (52) is present?
> I suppose that the definition of KERNEL_BUF then fails badly.

Option overload (52) is not currently supported by Etherboot. It can be
trivially supported and I have implemented it. KERNEL_BUF is not a problem
even then within Etherboot. It becomes a problem when the bootp structure is
passed to loaded image, which then may modify it and return to Etherboot to
retry and possibly load a different image. A few images, e.g. the nbi menus,
expect the KERNEL_BUF where it is at the moment, and implementing BOOTP/DHCP
option overload (52) will mean breaking them.

If option overload (52) is required, then just copy the options to the end of
the buffer would be a solution compatible with the current API. So, I've
corrected the implementation and the result is the patch attached for
etherboot 5.0.11 (as of now CVS HEAD of etherboot 5.0).

Any comments?

Vasil