Re: [kvm-devel] pinning, tsc and apic

[Marcelo T. <mto...@re...>]

On Mon, May 12, 2008 at 02:19:24PM -0500, Ryan Harper wrote:
> I've been digging into some of the instability we see when running
> larger numbers of guests at the same time.  The test I'm currently using
> involves launching 64 1vcpu guests on an 8-way AMD box.  With the latest
> kvm-userspace git and kvm.git + Gerd's kvmclock fixes, I can launch all
> 64 of these 1 second apart, and only a handful (1 to 3)  end up not
> making it up.  In dmesg on the host, I get a couple messages:
> 
> [321365.362534] vcpu not ready for apic_round_robin
> 
> and 
> 
> [321503.023788] Unsupported delivery mode 7
> 
> Now, the interesting bit for me was when I used numactl to pin the guest
> to a processor, all of the guests come up with no issues at all.  As I
> looked into it, it means that we're not running any of the vcpu
> migration code which on svm is comprised of tsc_offset recalibration and
> apic migration, and on vmx, a little more per-vcpu work

Hi Ryan,

There are two places that attempt to use delivery mode 7: kexec crash
and io_apic_64.c::check_timer().

The later will happen if the guest fails to receive PIT IRQ's for 10
ticks. If you're using HZ=1000 thats 10ms. See timer_irq_works().

The in-kernel pit emulation has logic which avoids injecting more than
one IRQ during 10ms.

Note that the guest 10ms delay is TSC based and uses only the lower
32-bits of the value. It is quite likely that the TSC adjustment results
in them increasing more rapidly then they should.

So can you try setting KVM_MAX_PIT_INTR_INTERVAL to a lower value? HZ/10
or something.

You can confirm this theory by booting the guests with "apic=debug".