User-mode Linux kernel port

Good day, Matthew,

On Mon, 28 Oct 2002, Matthew Bloch wrote:

> I'm just off tomorrow to go fix our UML host machine which, after I started a 
> small build going on one of the UMLs, stopped responding to any TCP 
> connections.  Pings came back fine, and TCP connections to the UMLs *and* to 
> the host machine were answered, but no data came back from them.  So the host 
> kernel seems to have stiffed itself in some way; the IP stack seems okay but 
> whatever process is meant to be connected to incoming connections isn't 
> responding, or data isn't coming through.

	I'm running into the same problem.
	Zaphod (see 
http://www.stearns.org/slartibartfast/uml-coop.current.html for more info) 
is running into the same problem; the tcp connection finishes, but no 
userspace app can be forked (I'm guessing on that part).  I have a vague 
suspicion it _might_ be related to a problem we had on thhe same host 
where running ps/w/top (on an already open ssh connection) never returns; 
Jeff was kind enough to narrow that down to a problem with the host kernel 
task lock.

	Here's a tcpdump of the connection attempt:

64.91.163.166.33473 > 66.59.109.137.1500: SWE 1287174553:1287174553(0) win 
2048 <mss 512,sackOK,timestamp 14605734 0,nop,wscale 0> (DF)

66.59.109.137.1500 > 64.91.163.166.33473: SE 822818040:822818040(0) ack 
1287174554 win 5792 <mss 1460,sackOK,timestamp 87494946 
14605734,nop,wscale 0> (DF)

64.91.163.166.33473 > 66.59.109.137.1500: . ack 1 win 2048 
<nop,nop,timestamp 14605773 87494946> (DF)

64.91.163.166.33473 > 66.59.109.137.1500: P 1:414(413) ack 1 win 2048 
<nop,nop,timestamp 14605773 87494946> (DF) [tos 0x2,ECT] 

64.91.163.166.33473 > 66.59.109.137.1500: P 1:414(413) ack 1 win 2048 
<nop,nop,timestamp 14605890 87494946> (DF)

66.59.109.137.1500 > 64.91.163.166.33473: SE 822818040:822818040(0) ack 
1287174554 win 5792 <mss 1460,sackOK,timestamp 87495286 
14605890,nop,wscale 0> (DF)

64.91.163.166.33473 > 66.59.109.137.1500: . ack 1 win 2048 
<nop,nop,timestamp 14606088 87495286,nop,nop,sack sack 1 {0:1} > (DF)

64.91.163.166.33473 > 66.59.109.137.1500: P 1:414(413) ack 1 win 2048 
<nop,nop,timestamp 14606124 87495286> (DF)

64.91.163.166.33473 > 66.59.109.137.1500: P 1:414(413) ack 1 win 2048 
<nop,nop,timestamp 14606592 87495286> (DF)

66.59.109.137.1500 > 64.91.163.166.33473: SE 822818040:822818040(0) ack 
1287174554 win 5792 <mss 1460,sackOK,timestamp 87495886 
14606592,nop,wscale 0> (DF)

64.91.163.166.33473 > 66.59.109.137.1500: . ack 1 win 2048 
<nop,nop,timestamp 14606679 87495886,nop,nop,sack sack 1 {0:1} > (DF)

64.91.163.166.33473 > 66.59.109.137.1500: P 1:414(413) ack 1 win 2048 
<nop,nop,timestamp 14607528 87495886> (DF)

66.59.109.137.1500 > 64.91.163.166.33473: SE 822818040:822818040(0) ack 
1287174554 win 5792 <mss 1460,sackOK,timestamp 87497086 
14607528,nop,wscale 0> (DF)

64.91.163.166.33473 > 66.59.109.137.1500: . ack 1 win 2048 
<nop,nop,timestamp 14607852 87497086,nop,nop,sack sack 1 {0:1} > (DF)

64.91.163.166.33473 > 66.59.109.137.1500: P 1:414(413) ack 1 win 2048 
<nop,nop,timestamp 14609400 87497086> (DF)

66.59.109.137.1500 > 64.91.163.166.33473: SE 822818040:822818040(0) ack 
1287174554 win 5792 <mss 1460,sackOK,timestamp 87499506 
14609400,nop,wscale 0> (DF)

64.91.163.166.33473 > 66.59.109.137.1500: . ack 1 win 2048 
<nop,nop,timestamp 14610221 87499506,nop,nop,sack sack 1 {0:1} > (DF)

64.91.163.166.33473 > 66.59.109.137.1500: P 1:414(413) ack 1 win 2048 
<nop,nop,timestamp 14613144 87499506> (DF)

66.59.109.137.1500 > 64.91.163.166.33473: SE 822818040:822818040(0) ack 
1287174554 win 5792 <mss 1460,sackOK,timestamp 87504326 
14613144,nop,wscale 0> (DF)

64.91.163.166.33473 > 66.59.109.137.1500: . ack 1 win 2048 
<nop,nop,timestamp 14614934 87504326,nop,nop,sack sack 1 {0:1} > (DF)

64.91.163.166.33473 > 66.59.109.137.1500: P 1:414(413) ack 1 win 2048 
<nop,nop,timestamp 14620632 87504326> (DF)

64.91.163.166.33473 > 66.59.109.137.1500: P 1:414(413) ack 1 win 2048 
<nop,nop,timestamp 14632632 87504326> (DF)

66.59.109.137.1500 > 64.91.163.166.33473: R 822818041:822818041(0) win 0 
(DF)


	(To everyone, not just Matthew)  Is there any chance that we're 
exposing some race in the host kernel with repeated process name changes 
or some other facet of uml?  </mode straw-grasping=off>

> I'm can't be sure exactly what's going on until we can attach a monitor & 
> keyboard to the server, but we were using a 2.4.19 host kernel, and whatever 
> the latest 2.4.19 UML patch for 2.4.19 was at the start of the month.
> 
> We've got about 6-7hrs tomorrow to sort out our machine; does anyone have 
> opinions on the most stable combination of UML host and client kernels to use 
> in a production environment (apart from "not at all, bozo" :-) )?  Or indeed 
> what might have gone wrong?  I'm interested in the most likely ways that a 
> UML kernel could stiff its host.

	No promises, but David Coulson was seeing daily lockups with 
2.4.20-pre9 which went away with 2.4.20-pre10.  I just updated Zaphod from 
the 2.4.20-pre7-ac2 to 2.4.20-pre11.  I have my fingers crossed.
	Hell, I have my _toes_ crossed.
	Cheers,
	- Bill

---------------------------------------------------------------------------
	If it happens once, it's a bug.  If it happens twice, it's 
a feature.  If it happens more than twice, it's a design philosophy.
(Courtesy of Slashdot)
--------------------------------------------------------------------------
William Stearns (wstearns@...).  Mason, Buildkernel, named2hosts, 
and ipfwadm2ipchains are at:                        http://www.stearns.org
--------------------------------------------------------------------------

William Stearns wrote:
> 	I'm running into the same problem.
> 	Zaphod (see 
> http://www.stearns.org/slartibartfast/uml-coop.current.html for more info) 
> is running into the same problem; the tcp connection finishes, but no 
> userspace app can be forked (I'm guessing on that part).  I have a vague 
> suspicion it _might_ be related to a problem we had on thhe same host 
> where running ps/w/top (on an already open ssh connection) never returns; 
> Jeff was kind enough to narrow that down to a problem with the host kernel 
> task lock.

This is something I've battled with for well over a month with 2.4.19 
and 2.4.20-pre kernels. They seem to get into a state where they can't 
fork processes or do very much, but kernel-level stuff like TCP 
connections and pings will work. As you said, Jeff tracked this down to 
a mm_sem locking issue, but I don't know if/where it has been fixed.

> 	(To everyone, not just Matthew)  Is there any chance that we're 
> exposing some race in the host kernel with repeated process name changes 
> or some other facet of uml?  </mode straw-grasping=off>

Process name changing?

> 	No promises, but David Coulson was seeing daily lockups with 
> 2.4.20-pre9 which went away with 2.4.20-pre10. 

Indeed. For the record, I'm running 2.4.20-pre10, along with Rik Van 
Riel's fairsched patch.

> I just updated Zaphod from 
> the 2.4.20-pre7-ac2 to 2.4.20-pre11.  I have my fingers crossed.
> 	Hell, I have my _toes_ crossed.

You don't want to know what I have crossed ;-)

David

-- 
David Coulson                                  http://davidcoulson.net/
d@...                       http://journal.davidcoulson.net/

Good day, David,

On Mon, 28 Oct 2002, David Coulson wrote:

> William Stearns wrote:
> > 	I'm running into the same problem.
> > 	Zaphod (see 
> > http://www.stearns.org/slartibartfast/uml-coop.current.html for more info) 
> > is running into the same problem; the tcp connection finishes, but no 
> > userspace app can be forked (I'm guessing on that part).  I have a vague 
> > suspicion it _might_ be related to a problem we had on thhe same host 
> > where running ps/w/top (on an already open ssh connection) never returns; 
> > Jeff was kind enough to narrow that down to a problem with the host kernel 
> > task lock.
> 
> This is something I've battled with for well over a month with 2.4.19 
> and 2.4.20-pre kernels. They seem to get into a state where they can't 
> fork processes or do very much, but kernel-level stuff like TCP 
> connections and pings will work. As you said, Jeff tracked this down to 
> a mm_sem locking issue, but I don't know if/where it has been fixed.

	I'm not aware of any progress on it, but that certainly doesn't 
equate to "still broken".

> > 	(To everyone, not just Matthew)  Is there any chance that we're 
> > exposing some race in the host kernel with repeated process name changes 
> > or some other facet of uml?  </mode straw-grasping=off>
> 
> Process name changing?

	Where the uml kernel changes "httpd" to "./uml-kernel-name 
(umid) [httpd]"

> > 	No promises, but David Coulson was seeing daily lockups with 
> > 2.4.20-pre9 which went away with 2.4.20-pre10. 
> 
> Indeed. For the record, I'm running 2.4.20-pre10, along with Rik Van 
> Riel's fairsched patch.

	Correct, sorry about that.

> > I just updated Zaphod from 
> > the 2.4.20-pre7-ac2 to 2.4.20-pre11.  I have my fingers crossed.
> > 	Hell, I have my _toes_ crossed.
> 
> You don't want to know what I have crossed ;-)

	And on that point we are in _complete_ agreement.  ;-)
	Cheers,
	- Bill

---------------------------------------------------------------------------
	Q: Do you view Linux and the open-source movement as a threat to
Microsoft?

	A: Yeah. It's good competition. It will force us to be
innovative. It will force us to justify the prices and value that we
deliver. And that's only healthy. The only thing we have a problem with
is when the government funds open-source work. Government funding should
be for work that is available to everybody. Open source is not available
to commercial companies. The way the license is written, if you use any
open-source software, you have to make the rest of your software open
source. If the government wants to put something in the public domain,
it should. Linux is not in the public domain. Linux is a cancer that
attaches itself in an intellectual property sense to everything it
touches. That's the way that the license works. 

	-- Steve Ballmer, http://www.suntimes.com/output/tech/cst-fin-micro01.html
--------------------------------------------------------------------------
William Stearns (wstearns@...).  Mason, Buildkernel, named2hosts, 
and ipfwadm2ipchains are at:                        http://www.stearns.org
--------------------------------------------------------------------------

Hi Matthew,

> We've got about 6-7hrs tomorrow to sort out our machine; does anyone have 
> opinions on the most stable combination of UML host and client kernels to use 
> in a production environment (apart from "not at all, bozo" :-) )?  Or indeed 
> what might have gone wrong?  I'm interested in the most likely ways that a 
> UML kernel could stiff its host.

As you will have noticed by previous posts to this list by myself and 
Bill, this is quite a common issue and something which we've both 
battled with over the last couple of months. AFAIK, this is a host 
kernel issue and from my own experiences, it is completly independent of 
the UML kernel which is running on the host.

You might want to look at running the latest 2.4.20-preX kernel on it, 
or back pedeling a little and looking at 2.4.18 or something and seeing 
if they are any more successful - Generally, the closer I've got to 
2.4.19, the more unstable my host has been.

As always, I'd be interested to hear if you have any progress with this 
and whether or not upgrading the kernel actually changes anything.

David

-- 
David Coulson                                  http://davidcoulson.net/
d@...                       http://journal.davidcoulson.net/

William Stearns wrote:
> 	Where the uml kernel changes "httpd" to "./uml-kernel-name 
> (umid) [httpd]"

Surely it changes it from <name of uml binary>? I can't see how that 
would be an issue, although any guess is as good as any right now.

> 	And on that point we are in _complete_ agreement.  ;-)

Thank God for that :-)

David

-- 
David Coulson                                  http://davidcoulson.net/
d@...                       http://journal.davidcoulson.net/

On Mon, Oct 28, 2002 at 05:57:46PM +0000, Matthew Bloch wrote:
> We've got about 6-7hrs tomorrow to sort out our machine; does anyone have 
> opinions on the most stable combination of UML host and client kernels to use 
> in a production environment (apart from "not at all, bozo" :-) )?

FWIW, I'm now using the latest Red Hat 8.0 errata kernel
(2.4.18-17.8.0).  It seems to be working fine, although my best uptime
is only 9 days.  Before that, I was running one of the 7.3 kernels
(2.4.18-3 or 2.4.18-5, not sure which) and had several months uptime.

The Red Hat 8.0 kernel has a higher HZ, so it should have better
latency for the UML systems.

Steve
-- 
steve@...           | Southern Illinois Linux Users Group
(618)398-7360             | See web site for meeting details.
Steven Pritchard          | http://www.silug.org/

On Monday 28 October 2002 19:19, David Coulson wrote:
> Hi Matthew,
>
> > We've got about 6-7hrs tomorrow to sort out our machine; does anyone have
> > opinions on the most stable combination of UML host and client kernels to
> > use in a production environment (apart from "not at all, bozo" :-) )?  Or
> > indeed what might have gone wrong?  I'm interested in the most likely
> > ways that a UML kernel could stiff its host.
>
> As you will have noticed by previous posts to this list by myself and
> Bill, this is quite a common issue and something which we've both
> battled with over the last couple of months. AFAIK, this is a host
> kernel issue and from my own experiences, it is completly independent of
> the UML kernel which is running on the host.
>
> You might want to look at running the latest 2.4.20-preX kernel on it,
> or back pedeling a little and looking at 2.4.18 or something and seeing
> if they are any more successful - Generally, the closer I've got to
> 2.4.19, the more unstable my host has been.

Thankyou, this is interesting stuff from both you and Bill.  My next question 
before I set off tomorrow is: can you find a surefire way of triggering the 
lockup behaviour?  All I tried to do was build qmail but maybe something a 
bit more heavyweight inside the UML should break it more reliably?  I apt-get 
installed a bunch of debian packages on two UMLs at once, and there was no 
problem.  I'm going to investigate the host kernel a bit when we get a 
monitor attached, but I'm not sure how much we can find out from it.

If the host kernel is at fault I'd be interested to try putting on Redhat's 
"legendary" 2.4.9 kernel from a while back which is meant to be very stable 
indeed, but I believe may be useless because we need some RAID stuff only 
introduced in 2.4.19.  I'll check with my partner because he sorted out the 
host kernel.

-- 
Matthew Bloch         Bytemark Computer Consulting Limited
                                http://www.bytemark.co.uk/
                                  tel. +44 (0) 8707 455026

On Mon, Oct 28, 2002 at 09:01:54PM +0000, Matthew Bloch wrote:
> If the host kernel is at fault I'd be interested to try putting on Redhat's 
> "legendary" 2.4.9 kernel from a while back which is meant to be very stable 
> indeed, but I believe may be useless because we need some RAID stuff only 
> introduced in 2.4.19.

Red Hat has moved on to their 2.4.18-based kernel for everything
except *maybe* the AS product with their latest errata kernels.  It
seems to be every bit as stable as their 2.4.9-based kernels, perhaps
even more so, and the performance is *much* better with O(1), rmap,
etc.

Oh, and I'm using software RAID on quite a few systems with Red Hat
2.4.18 kernels...  I'd be really surprised if whatever you are looking
for in 2.4.19 isn't already in their 2.4.18 kernels if it is something
good.  :-)

(Not trying to be Red Hat fanboy here, but their kernels are one area
where they definitely seem to do a *really* good job.)

Steve
-- 
steve@...           | Southern Illinois Linux Users Group
(618)398-7360             | See web site for meeting details.
Steven Pritchard          | http://www.silug.org/

On Mon, 28 Oct 2002, William Stearns wrote:
> Good day, Matthew,
>
> On Mon, 28 Oct 2002, Matthew Bloch wrote:
>
> > I'm just off tomorrow to go fix our UML host machine which, after I started a
> > small build going on one of the UMLs, stopped responding to any TCP
> > connections.  Pings came back fine, and TCP connections to the UMLs *and* to
> > the host machine were answered, but no data came back from them.  So the host
> > kernel seems to have stiffed itself in some way; the IP stack seems okay but
> > whatever process is meant to be connected to incoming connections isn't
> > responding, or data isn't coming through.
>
> 	I'm running into the same problem.
> 	Zaphod (see
> http://www.stearns.org/slartibartfast/uml-coop.current.html for more info)
> is running into the same problem; the tcp connection finishes, but no
> userspace app can be forked (I'm guessing on that part).  I have a vague
> suspicion it _might_ be related to a problem we had on thhe same host
> where running ps/w/top (on an already open ssh connection) never returns;
> Jeff was kind enough to narrow that down to a problem with the host kernel
> task lock.
>
> 	(To everyone, not just Matthew)  Is there any chance that we're
> exposing some race in the host kernel with repeated process name changes
> or some other facet of uml?  </mode straw-grasping=off>
>
> 	No promises, but David Coulson was seeing daily lockups with
> 2.4.20-pre9 which went away with 2.4.20-pre10.  I just updated Zaphod from
> the 2.4.20-pre7-ac2 to 2.4.20-pre11.  I have my fingers crossed.

Just curious, what are you folks running for a host kernel?  I've got two
4-way Xeon boxes with roughly 20 UML instances each running a Redhat
2.4.18 kernel on the host.  One of them is running the last 2.4.18 UML
kernel, and the other the last released 2.4.19 UML kernel, and neither has
seen any of the chaos that you are both reporting.  Are you getting alot
of network throughput on these boxes?

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lonni J Friedman				netllama@...
Linux Step-by-step & TyGeMo		     http://netllama.ipfox.com

On Monday 28 October 2002 22:08, Net Llama! wrote:
[snip]
> Just curious, what are you folks running for a host kernel?  I've got two
> 4-way Xeon boxes with roughly 20 UML instances each running a Redhat
> 2.4.18 kernel on the host.  One of them is running the last 2.4.18 UML
> kernel, and the other the last released 2.4.19 UML kernel, and neither has
> seen any of the chaos that you are both reporting.  Are you getting alot
> of network throughput on these boxes?

None at all.  But from what you say, and Steven's recommendation of the Redhat 
2.4.18 kernel, that sounds like the kernel we want to be on.  Our problem 
with the kernel, thought I doubt it's precisely familiar to anyone who hasn't 
been through this specific pain, is that our motherboar has an Intel ICH6 IDE 
controller on, and a Highpoint 373 (I think).  And I believe that only 
2.4.19-ac1 had the right combination of drivers and RAID support which allows 
both controllers to work as a RAID1 array using DMA.  I think 2.4.18 wouldn't 
use DMA on the Intel controller and earlier versions had other problems... 
argh, all a bit of a muddle.

Anyhow I'll download this host kernel:

ftp://ftp.mirror.ac.uk/sites/ftp.redhat.com/pub/redhat/linux/8.0/en/os/i386/SRPMS/kernel-2.4.18-14.src.rpm

and see whether it looks promising for tomorrow.  I think I should in fact 
veer *away* from trying to find a UML patch which is "better behaved" because 
that's just trying to mask the problem.

-- 
Matthew Bloch         Bytemark Computer Consulting Limited
                                http://www.bytemark.co.uk/
                                  tel. +44 (0) 8707 455026

On Monday 28 October 2002 22:08, Net Llama! wrote:
> Just curious, what are you folks running for a host kernel?  

Sorry, in answer to your question I think it's 2.4.19-ac1 or 2.

-- 
Matthew Bloch         Bytemark Computer Consulting Limited
                                http://www.bytemark.co.uk/
                                  tel. +44 (0) 8707 455026

Matthew Bloch wrote:
> Thankyou, this is interesting stuff from both you and Bill.  My next question 
> before I set off tomorrow is: can you find a surefire way of triggering the 
> lockup behaviour? 

I've never found one. Most of my lock-ups occured in the early hours 
(GMT) when it wasn't being used much. I could build kernels, Apache, et 
al on the thing all day and it'd be great, but for some reason, it would 
just occasionally die.

> All I tried to do was build qmail but maybe something a 
> bit more heavyweight inside the UML should break it more reliably?  I apt-get 
> installed a bunch of debian packages on two UMLs at once, and there was no 
> problem.  I'm going to investigate the host kernel a bit when we get a 
> monitor attached, but I'm not sure how much we can find out from it.

I never got any useful information following lock-ups. Jeff and Bill 
managed to get some stacks and things which they sent to lkml, although 
AFAIK, they never had a response. You might want to check out the 
user-mode-linux-devel and lkml archives.

> If the host kernel is at fault I'd be interested to try putting on Redhat's 
> "legendary" 2.4.9 kernel from a while back which is meant to be very stable 
> indeed, but I believe may be useless because we need some RAID stuff only 
> introduced in 2.4.19.  I'll check with my partner because he sorted out the 
> host kernel.

I'm unaware of any new RAID stuff in 2.4.19, but are there no patches 
for it you could apply to RH's kernel tree?

David

-- 
David Coulson                                  http://davidcoulson.net/
d@...                       http://journal.davidcoulson.net/

Net Llama! wrote:
> Just curious, what are you folks running for a host kernel? 

2.4.19-pre-something.

> I've got two
> 4-way Xeon boxes with roughly 20 UML instances each running a Redhat
> 2.4.18 kernel on the host.  One of them is running the last 2.4.18 UML
> kernel, and the other the last released 2.4.19 UML kernel, and neither has
> seen any of the chaos that you are both reporting.  Are you getting alot
> of network throughput on these boxes?

Depends - Network throughput does not seem to be factor when looking at 
the lock-ups.

David

-- 
David Coulson                                  http://davidcoulson.net/
d@...                       http://journal.davidcoulson.net/

Hi!

Steven Pritchard wrote:
> On Mon, Oct 28, 2002 at 05:57:46PM +0000, Matthew Bloch wrote:
> 
>>We've got about 6-7hrs tomorrow to sort out our machine; does anyone have 
>>opinions on the most stable combination of UML host and client kernels to use 
>>in a production environment (apart from "not at all, bozo" :-) )?
> 
> 
> FWIW, I'm now using the latest Red Hat 8.0 errata kernel
> (2.4.18-17.8.0).  It seems to be working fine, although my best uptime
> is only 9 days.  Before that, I was running one of the 7.3 kernels


Just to clarify, have you installed it 9 days ago and it never crashed 
or did it crash several times?



> (2.4.18-3 or 2.4.18-5, not sure which) and had several months uptime.
> 


While running umls?

Sorry, I see ambigous terms everywhere this late at night :)

After "upgrading" from stock 2.4.18 to stock 2.4.19 linux kernel one 
host that runs 3 umls (almost no load at all) panic'ed on me with that 
behaviour: you can ping but not much more... The machine had 9 days of 
uptime... hmmmmm

I'm thinking about 2.4.18 because stability is my main concern. Right 
now i'm rebooting every week, just because :)

Thanks,
Nuno Silva





> The Red Hat 8.0 kernel has a higher HZ, so it should have better
> latency for the UML systems.
> 
> Steve

On Tue, Oct 29, 2002 at 06:00:08AM +0000, Nuno Silva wrote:
> Just to clarify, have you installed it 9 days ago and it never crashed 
> or did it crash several times?

I just installed it 9 days ago (well, 10 now).  It hasn't crashed.

> >(2.4.18-3 or 2.4.18-5, not sure which) and had several months uptime.
> 
> While running umls?

Right.  It was really getting hammered when I wasn't using tmpfs too.
The newer kernel is getting it easy...  The box hardly has any load
now even though it is running 14 (I think) UMLs at the moment.

Steve
-- 
steve@...           | Southern Illinois Linux Users Group
(618)398-7360             | See web site for meeting details.
Steven Pritchard          | http://www.silug.org/