Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

[Avi K. <av...@qu...>]

David S. Ahern wrote:
> I added the traces and captured data over another apparent lockup of the guest.
> This seems to be representative of the sequence (pid/vcpu removed).
>
> (+4776)  VMEXIT         [ exitcode = 0x00000000, rip = 0x00000000 c016127c ]
> (+   0)  PAGE_FAULT     [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ]
> (+3632)  VMENTRY
> (+4552)  VMEXIT         [ exitcode = 0x00000000, rip = 0x00000000 c016104a ]
> (+   0)  PAGE_FAULT     [ errorcode = 0x0000000b, virt = 0x00000000 fffb61c8 ]
> (+   54928)  VMENTRY
>   

Can you oprofile the host to see where the 54K cycles are spent?

> (+4568)  VMEXIT         [ exitcode = 0x00000000, rip = 0x00000000 c01610e7 ]
> (+   0)  PAGE_FAULT     [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ]
> (+   0)  PTE_WRITE      [ gpa = 0x00000000 00009db4 gpte = 0x00000000 41c5d363 ]
> (+8432)  VMENTRY
> (+3936)  VMEXIT         [ exitcode = 0x00000000, rip = 0x00000000 c01610ee ]
> (+   0)  PAGE_FAULT     [ errorcode = 0x00000003, virt = 0x00000000 c0009db0 ]
> (+   0)  PTE_WRITE      [ gpa = 0x00000000 00009db0 gpte = 0x00000000 00000000 ]
> (+   13832)  VMENTRY
>
>
> (+5768)  VMEXIT         [ exitcode = 0x00000000, rip = 0x00000000 c016127c ]
> (+   0)  PAGE_FAULT     [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ]
> (+3712)  VMENTRY
> (+4576)  VMEXIT         [ exitcode = 0x00000000, rip = 0x00000000 c016104a ]
> (+   0)  PAGE_FAULT     [ errorcode = 0x0000000b, virt = 0x00000000 fffb61d0 ]
> (+   0)  PTE_WRITE      [ gpa = 0x00000000 3d5981d0 gpte = 0x00000000 3d55d047 ]
>   

This indeed has the accessed bit clear.

> (+   65216)  VMENTRY
> (+4232)  VMEXIT         [ exitcode = 0x00000000, rip = 0x00000000 c01610e7 ]
> (+   0)  PAGE_FAULT     [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ]
> (+   0)  PTE_WRITE      [ gpa = 0x00000000 00009db4 gpte = 0x00000000 3d598363 ]
>   

This has the accessed bit set and the user bit clear, and the pte 
pointing at the previous pte_write gpa.  Looks like a kmap_atomic().

> (+8640)  VMENTRY
> (+3936)  VMEXIT         [ exitcode = 0x00000000, rip = 0x00000000 c01610ee ]
> (+   0)  PAGE_FAULT     [ errorcode = 0x00000003, virt = 0x00000000 c0009db0 ]
> (+   0)  PTE_WRITE      [ gpa = 0x00000000 00009db0 gpte = 0x00000000 00000000 ]
> (+   14160)  VMENTRY
>
> I can forward a more complete time snippet if you'd like. vcpu0 + corresponding
> vcpu1 files have 85000 total lines and compressed the files total ~500k.
>
> I did not see the FLOODED trace come out during this sample though I did bump
> the count from 3 to 4 as you suggested.
>
>
>   

Bumping the count was supposed to remove the flooding...

> Correlating rip addresses to the 2.4 kernel:
>
> c0160d00-c0161290 = page_referenced
>
> It looks like the event is kscand running through the pages. I suspected this
> some time ago, and tried tweaking the kscand_work_percent sysctl variable. It
> appeared to lower the peak of the spikes, but maybe I imagined it. I believe
> lowering that value makes kscand wake up more often but do less work (page
> scanning) each time it is awakened.
>
>   

What does 'top' in the guest show (perhaps sorted by total cpu time 
rather than instantaneous usage)?

What host kernel are you running?  How many host cpus?

-- 
error compiling committee.c: too many arguments to function