[kvm-devel] RHEL5 smp guests on RHE5.1 hosts hang with kvm-52

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

(Changed the subject to correspond with email.)

I am having the same problem on the 64-bit host running RHEL5.1 as well, it just takes more reboots. Same symptoms as I mentioned for the 32-bit host. kernel side stack traces for each qemu thread for one of the lockups is attached; the file contains traces for each thread at 3 sample times in case it helps get some insight.

david

david ahern wrote:
> With kvm-52 my 32-bit host running RHEL5.1 can start an RHEL 5 SMP guest only once. Second and subsequent attempts hang. Removing kvm and kvm_intel modules have no affect; I need to reboot the host to get an SMP guest to start. My similarly configured 64-bit host does not seem to have this problem.
> 
> 
> Second attempts to start the RHEL5 SMP guest hang at:
> Starting udev: _
> 
> 
> Looking at top on the host shows qemu in a loop:
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                
>  3909 root      18   0 1625m  67m 9476 R  400  2.1   2:52.32 qemu-system-x86                                         
> 
> 
> In this case the qemu threads are:
> 
>   PID   LWP TTY          TIME CMD
>  3909  3909 pts/0    00:01:12 qemu-system-x86
>  3909  3911 pts/0    00:01:05 qemu-system-x86
>  3909  3912 pts/0    00:01:05 qemu-system-x86
>  3909  3913 pts/0    00:01:07 qemu-system-x86
>  3909  3917 pts/0    00:00:00 qemu-system-x86
> 
> 
> 
> and their kernel side backtraces are:
> 
>  process trace for qemu-system-x86(3909)
>         f5967d88 00000082 f8c125e4 bbdec465 000001c6 f5230da4 00000001 f7acf000 
>         f7d7d000 bbded629 000001c6 000011c4 00000000 f7acf110 c30126e0 00000001 
>         f4d8a000 f5967d90 f5967d80 f5230da0 f5967000 f8c11120 f5230da0 f4d8a000 
>  Call Trace:
>   [<f8c125e4>] vmx_vcpu_put+0xef/0xf6 [kvm_intel]
>   [<f8c11120>] handle_external_interrupt+0x0/0xc [kvm_intel]
>   [<c042169f>] __cond_resched+0x16/0x34
>   [<c0604218>] cond_resched+0x2a/0x31
>   [<f8b96d7f>] kvm_arch_vcpu_ioctl_run+0x28d/0x333 [kvm]
>   [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
>   [<f8b943d4>] kvm_vcpu_ioctl+0xbb/0x366 [kvm]
>   [<c042169f>] __cond_resched+0x16/0x34
>   [<c0604218>] cond_resched+0x2a/0x31
>   [<c0480305>] core_sys_select+0x1ef/0x2ca
>   [<c041ea84>] __wake_up_common+0x2f/0x53
>   [<c0604141>] schedule+0x90d/0x9ba
>   [<c0405953>] reschedule_interrupt+0x1f/0x24
>   [<c042e759>] __dequeue_signal+0x151/0x15c
>   [<c042fa99>] dequeue_signal+0x2d/0x9c
>   [<c043062c>] sys_rt_sigtimedwait+0xc5/0x2c2
>   [<c042cc0e>] getnstimeofday+0x30/0xb6
>   [<c04386d6>] ktime_get_ts+0x16/0x44
>   [<c04388b6>] ktime_get+0x12/0x34
>   [<c04352a6>] common_timer_get+0xee/0x129
>   [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
>   [<c047f1e8>] do_ioctl+0x1c/0x5d
>   [<c047f473>] vfs_ioctl+0x24a/0x25c
>   [<c047f4cd>] sys_ioctl+0x48/0x5f
>   [<c0404eff>] syscall_call+0x7/0xb
>   =======================
>  process trace for qemu-system-x86(3911)
>         c301a6e0 00000100 000001c7 f749baa0 00000001 c301a6e0 f749baa0 00000001 
>         f51fed44 f51fed44 f51fed6c 00000001 00000001 00000046 f579ce20 f57ee000 
>         00000001 c04059bf f579ce20 8005003b 00006c00 f8c113e5 f579ce20 f579ce20 
>  Call Trace:
>   [<c04059bf>] apic_timer_interrupt+0x1f/0x24
>   [<f8c113e5>] vmcs_writel+0x1b/0x2c [kvm_intel]
>   [<f8b96cf4>] kvm_arch_vcpu_ioctl_run+0x202/0x333 [kvm]
>   [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
>   [<f8b943d4>] kvm_vcpu_ioctl+0xbb/0x366 [kvm]
>   [<c041fa31>] enqueue_task+0x29/0x39
>   [<c041fa5d>] __activate_task+0x1c/0x29
>   [<c04202a7>] try_to_wake_up+0x371/0x37b
>   [<c0604141>] schedule+0x90d/0x9ba
>   [<c041ea84>] __wake_up_common+0x2f/0x53
>   [<c041f871>] __wake_up+0x2a/0x3d
>   [<c0438eb7>] wake_futex+0x3a/0x44
>   [<c0439187>] futex_wake+0xa9/0xb3
>   [<c0439d66>] do_futex+0x20d/0xb15
>   [<f8b94696>] kvm_ack_smp_call+0x17/0x27 [kvm]
>   [<c042e759>] __dequeue_signal+0x151/0x15c
>   [<c042fa99>] dequeue_signal+0x2d/0x9c
>   [<f8b93ea9>] kvm_vm_ioctl+0x0/0x277 [kvm]
>   [<f8b9410d>] kvm_vm_ioctl+0x264/0x277 [kvm]
>   [<c04202b1>] default_wake_function+0x0/0xc
>   [<c040599b>] call_function_interrupt+0x1f/0x24
>   [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
>   [<c047f1e8>] do_ioctl+0x1c/0x5d
>   [<c047f473>] vfs_ioctl+0x24a/0x25c
>   [<c047f4cd>] sys_ioctl+0x48/0x5f
>   [<c0404eff>] syscall_call+0x7/0xb
>   =======================
>  process trace for qemu-system-x86(3912)
>         f560fd88 00000082 f8c125e4 193272c5 000001c8 f52b6074 00000004 f7f09000 
>         f7f09000 19328fc9 000001c8 00001d04 00000002 f52b6070 55eefb90 00000000 
>         f52b6070 f5693000 f52b6070 8005003b 00006c00 f8c113e5 f52b6070 f52b6070 
>  Call Trace:
>   [<f8c125e4>] vmx_vcpu_put+0xef/0xf6 [kvm_intel]
>   [<f8c113e5>] vmcs_writel+0x1b/0x2c [kvm_intel]
>   [<f8b96cf4>] kvm_arch_vcpu_ioctl_run+0x202/0x333 [kvm]
>   [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
>   [<f8b943d4>] kvm_vcpu_ioctl+0xbb/0x366 [kvm]
>   [<c0604141>] schedule+0x90d/0x9ba
>   [<c041ea84>] __wake_up_common+0x2f/0x53
>   [<c0461e10>] find_extend_vma+0x12/0x49
>   [<c0438d53>] get_futex_key+0x40/0xd0
>   [<c0439187>] futex_wake+0xa9/0xb3
>   [<c0439d66>] do_futex+0x20d/0xb15
>   [<f888f9b0>] ext3_ordered_writepage+0x0/0x162 [ext3]
>   [<c042e759>] __dequeue_signal+0x151/0x15c
>   [<c042fa99>] dequeue_signal+0x2d/0x9c
>   [<f8b93ea9>] kvm_vm_ioctl+0x0/0x277 [kvm]
>   [<f8b9410d>] kvm_vm_ioctl+0x264/0x277 [kvm]
>   [<c04202b1>] default_wake_function+0x0/0xc
>   [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
>   [<c047f1e8>] do_ioctl+0x1c/0x5d
>   [<c047f473>] vfs_ioctl+0x24a/0x25c
>   [<c047f4cd>] sys_ioctl+0x48/0x5f
>   [<c0404eff>] syscall_call+0x7/0xb
>   =======================
>  process trace for qemu-system-x86(3913)
>         c302a6e0 00000100 000001c8 f7488550 00000003 c302a6e0 f7488550 00000003 
>         f4d92d44 f4d92d44 f4d92d6c 00000001 00000001 00000046 f52b6de0 f5aaa000 
>         00000001 c04059bf f52b6de0 8005003b 00006c00 f8c113e5 f52b6de0 f52b6de0 
>  Call Trace:
>   [<c04059bf>] apic_timer_interrupt+0x1f/0x24
>   [<f8c113e5>] vmcs_writel+0x1b/0x2c [kvm_intel]
>   [<f8b96cf4>] kvm_arch_vcpu_ioctl_run+0x202/0x333 [kvm]
>   [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
>   [<f8b943d4>] kvm_vcpu_ioctl+0xbb/0x366 [kvm]
>   [<c0604141>] schedule+0x90d/0x9ba
>   [<c041ea84>] __wake_up_common+0x2f/0x53
>   [<c0461e10>] find_extend_vma+0x12/0x49
>   [<c0438d53>] get_futex_key+0x40/0xd0
>   [<c0439187>] futex_wake+0xa9/0xb3
>   [<c0439d66>] do_futex+0x20d/0xb15
>   [<c040599b>] call_function_interrupt+0x1f/0x24
>   [<c042e759>] __dequeue_signal+0x151/0x15c
>   [<f8b93ea9>] kvm_vm_ioctl+0x0/0x277 [kvm]
>   [<f8b9410d>] kvm_vm_ioctl+0x264/0x277 [kvm]
>   [<c04202b1>] default_wake_function+0x0/0xc
>   [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
>   [<c047f1e8>] do_ioctl+0x1c/0x5d
>   [<c047f473>] vfs_ioctl+0x24a/0x25c
>   [<c047f4cd>] sys_ioctl+0x48/0x5f
>   [<c0404eff>] syscall_call+0x7/0xb
>   =======================
> 
> david
> 
> 
> Avi Kivity wrote:
>> david ahern wrote:
>>> The patch worked for me -- rhel4 smp guests boot fine on stock RHEL5
>>> hosts, both 32-bit and 64-bit.
>>>
>>>   
>> Excellent.  I had a premonition so it is already committed.
>>
>> Do note that smp_call_function_mask() emulation is pretty bad in terms
>> of performance on large multicores.  On a dual code it's basically
>> equivalent to mainline, I guess it's okay for four-way, but above
>> four-way you will need either mainline or a better
>> smp_call_function_mask() (which is nontrivial but doable).
>>
>>> david
>>>
>>>
>>> Avi Kivity wrote:
>>>  
>>>> david ahern wrote:
>>>>    
>>>>> In RHEL 5.1 <linux/notifier.h> defines:
>>>>>
>>>>> #define CPU_TASKS_FROZEN    0x0010
>>>>>
>>>>> #define CPU_ONLINE_FROZEN   (CPU_ONLINE | CPU_TASKS_FROZEN)
>>>>> #define CPU_DEAD_FROZEN     (CPU_DEAD | CPU_TASKS_FROZEN)
>>>>>
>>>>> which means in kvm-51/kernel/external-module-compat.h the '#ifndef
>>>>> CPU_TASKS_FROZEN' needs to have a case. For my purposes, I just moved
>>>>> up the endif around what was defined.
>>>>>         
>>>> I committed a change which renders this unnecessary.  Will be part of
>>>> kvm-52.
>>>>
>>>>    
>>>>> With that change, kvm-51 compiles. I am still seeing 32-bit SMP guests
>>>>> hang on boot for both 32-bit and 64-bit hosts (again running RHEL5.1).
>>>>>         
>>>> I still don't.  Can you test the attached patch?
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> -------------------------------------------------------------------------
>>>>
>>>> This SF.net email is sponsored by: Splunk Inc.
>>>> Still grepping through log files to find problems?  Stop.
>>>> Now Search log events and configuration files using AJAX and a browser.
>>>> Download your FREE copy of Splunk now >> http://get.splunk.com/
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> kvm-devel mailing list
>>>> kvm...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>>>>     
>>
> 