Re: [Openipmi-developer] IPMI driver performance problem

[Matt D. <ma...@do...>]

On Mon, Mar 17, 2008 at 03:11:27PM +1100, Nathan Scott wrote:
> Hi all,
> 
> We're seeing an IPMI related performance problem on our production
> servers, which I hope someone can help me with.  These are Dell
> boxes so I've CC'd Matt in case the answer is known already (sorry
> for the intrusion if not, Matt).

yes, this is quite known.

Short story is, the KCS interface is horrible, but unfortunately has
become the typical implementation.  Longer story is, you can either
poll for each character to arrive, or you can get an interrupt for
each character.  Except that there's no interrupt line present in most
systems, including all Dell servers with KCS to date.  So, the driver
must poll (which is that the kipmi0 thread is all about).

You can disable the polling thread, but then the driver operates at
the speed of the timer interrupt, so 1ms per character in RHEL4.
System startup, starting OMSA, in this mode, takes an extra 3-5
minutes.  Firmware updates take ~15 minutes.  With the polling thread,
these times are "a few seconds" and 1.5 minutes respectively.

I tried to get a hardware interrupt line added to the Dell 10G
PowerEdge servers.  However, this greatly confused some "other
operating systems" to the point of making them completely unusable, so
it had to be disabled.

The polling thread runs at the lowest possible priority, so it
shouldn't get scheduled that often.  However, when it does get
scheduled, because it's not pre-emptable, it will consume CPU cycles
polling until a command completes, at which point it'll get scheduled
out again.

If this is seriously impacting your system performance (more than just
appearing in top/ps/etc.), one option is to disable OMSA (if you
aren't using it), which is the userspace systems management software
that occasionally polls the state of all the sensors, which is why you
see the CPU spikes.

Thanks,
Matt