From: Ho, M. <min...@in...> - 2010-09-02 17:23:07
|
Hello, I am having trouble running profiling tools on the 2.6.33.7 kernel with the rt29 patch. I am running on an Intel Core 2 machine. I run oprofile and received the following kernel oops when starting the oprofile daemon. Here is the log info: oprofile: using NMI interrupt. BUG: sleeping function called from invalid context at kernel/rtmutex.c:707 pcnt: 2 0 in_atomic(): 1, irqs_disabled(): 1, pid: 4020, name: oprofiled Pid: 4020, comm: oprofiled Not tainted 2.6.33.7-rt29 #3 Call Trace: [<c104e8c4>] ? rt_spin_lock_fastlock+0x26/0x58 [<c108ff56>] ? _slab_irq_disable+0x22/0x42 [<c109108a>] ? __kmalloc+0x7d/0xf0 [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] [<e0070e8d>] ? nmi_cpu_setup+0x87/0xcc [oprofile] [<e0070e06>] ? nmi_cpu_setup+0x0/0xcc [oprofile] [<c102fa7b>] ? on_each_cpu+0x25/0x50 [<e0070dd9>] ? nmi_setup+0x11d/0x14a [oprofile] [<e006f133>] ? oprofile_setup+0x2d/0x86 [oprofile] [<e006fede>] ? event_buffer_open+0x42/0x60 [oprofile] [<c109363d>] ? __dentry_open+0x1a4/0x29a [<c109b09b>] ? generic_permission+0xc/0x7e [<c10937c1>] ? nameidata_to_filp+0x27/0x38 [<e006fe9c>] ? event_buffer_open+0x0/0x60 [oprofile] [<c109d214>] ? do_filp_open+0x439/0x843 [<c10808f6>] ? __do_fault+0x2bf/0x2ef [<c1090091>] ? slab_irq_enable+0x45/0x79 [<c10933a2>] ? do_sys_open+0x4c/0xe4 [<c109347e>] ? sys_open+0x1e/0x23 [<c1002750>] ? sysenter_do_call+0x12/0x26 When I run perf record I get a slightly different kernel oops. Here is the relevant log info: BUG: sleeping function called from invalid context at arch/x86/mm/highmem_32.c:9 pcnt: 14010003 0 in_atomic(): 1, irqs_disabled(): 1, pid: 3640, name: perf Pid: 3640, comm: perf Not tainted 2.6.33.7-rt29 #3 Call Trace: [<c101d0c3>] ? kmap+0x41/0x52 [<c100b66e>] ? perf_callchain+0x20d/0x2ac [<c106abd9>] ? perf_prepare_sample+0x1d1/0x23b [<c106ce0b>] ? __perf_event_overflow+0x20a/0x25e [<c106d4bc>] ? perf_event_overflow+0xf/0x12 [<c100c906>] ? intel_pmu_handle_irq+0x1fd/0x25d [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae [<c1386478>] ? perf_event_nmi_handler+0x34/0x3f [<c13877dd>] ? notifier_call_chain+0x2a/0x47 [<c138781e>] ? __atomic_notifier_call_chain+0x24/0x33 [<c1387836>] ? atomic_notifier_call_chain+0x9/0xc [<c1387869>] ? notify_die+0x30/0x34 [<c1385c75>] ? do_nmi+0x73/0x25f [<c13858a9>] ? nmi_stack_correct+0x28/0x2d [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae [<c100afda>] ? hw_perf_enable+0xf/0x10 [<c1069b42>] ? __perf_event_sched_in+0x10c/0x119 [<c1069b7a>] ? perf_event_task_sched_in+0x2b/0x31 [<c1024088>] ? finish_task_switch+0x2b/0x72 [<c1383d18>] ? __schedule+0x55f/0x5c3 [<c1024c79>] ? try_to_wake_up+0x2b5/0x2bf [<c1384f5b>] ? _raw_spin_lock+0xd/0x23 [<c1383e0e>] ? preempt_schedule+0x40/0x5a [<c13848f1>] ? rt_mutex_slowunlock+0x37/0x44 [<c109a317>] ? pipe_release+0x7c/0x84 [<c109598a>] ? __fput+0xe7/0x17b [<c10932b5>] ? filp_close+0x4e/0x54 [<c109331d>] ? sys_close+0x62/0x9b [<c1002750>] ? sysenter_do_call+0x12/0x26 Both perf and oprofile appear to collect data successfully although when I ran perf on a long-running program it generated the same kernel oops multiple times and eventually the machine hung. I would appreciate any guidance that you can give. Best Regards, Minnie Ho |
From: Maynard J. <may...@us...> - 2010-09-07 15:53:42
|
Ho, Minnie wrote: > Hello, This message below is an exact copy of a message sent to the linux-rt-users mailing list by Jerry Sydir on Aug 18. What's up with copying his note and forwarding it to oprofile-list as if it's your own? And the email thread on linux-rt-users provided a solution to the problem. Why are you posting this to oprofile-list and wasting our time? -Maynard > > I am having trouble running profiling tools on the 2.6.33.7 kernel with the rt29 > patch. I am running on an Intel Core 2 machine. I run oprofile and received the following kernel oops when > starting the oprofile daemon. Here is the log info: > > oprofile: using NMI interrupt. > BUG: sleeping function called from invalid context at kernel/rtmutex.c:707 > pcnt: 2 0 in_atomic(): 1, irqs_disabled(): 1, pid: 4020, name: oprofiled > Pid: 4020, comm: oprofiled Not tainted 2.6.33.7-rt29 #3 > Call Trace: > [<c104e8c4>] ? rt_spin_lock_fastlock+0x26/0x58 > [<c108ff56>] ? _slab_irq_disable+0x22/0x42 > [<c109108a>] ? __kmalloc+0x7d/0xf0 > [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] > [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] > [<e0070e8d>] ? nmi_cpu_setup+0x87/0xcc [oprofile] > [<e0070e06>] ? nmi_cpu_setup+0x0/0xcc [oprofile] > [<c102fa7b>] ? on_each_cpu+0x25/0x50 > [<e0070dd9>] ? nmi_setup+0x11d/0x14a [oprofile] > [<e006f133>] ? oprofile_setup+0x2d/0x86 [oprofile] > [<e006fede>] ? event_buffer_open+0x42/0x60 [oprofile] > [<c109363d>] ? __dentry_open+0x1a4/0x29a > [<c109b09b>] ? generic_permission+0xc/0x7e > [<c10937c1>] ? nameidata_to_filp+0x27/0x38 > [<e006fe9c>] ? event_buffer_open+0x0/0x60 [oprofile] > [<c109d214>] ? do_filp_open+0x439/0x843 > [<c10808f6>] ? __do_fault+0x2bf/0x2ef > [<c1090091>] ? slab_irq_enable+0x45/0x79 > [<c10933a2>] ? do_sys_open+0x4c/0xe4 > [<c109347e>] ? sys_open+0x1e/0x23 > [<c1002750>] ? sysenter_do_call+0x12/0x26 > > When I run perf record I get a slightly different kernel oops. Here is the relevant log info: > > BUG: sleeping function called from invalid context at arch/x86/mm/highmem_32.c:9 > pcnt: 14010003 0 in_atomic(): 1, irqs_disabled(): 1, pid: 3640, name: perf > Pid: 3640, comm: perf Not tainted 2.6.33.7-rt29 #3 > Call Trace: > [<c101d0c3>] ? kmap+0x41/0x52 > [<c100b66e>] ? perf_callchain+0x20d/0x2ac > [<c106abd9>] ? perf_prepare_sample+0x1d1/0x23b > [<c106ce0b>] ? __perf_event_overflow+0x20a/0x25e > [<c106d4bc>] ? perf_event_overflow+0xf/0x12 > [<c100c906>] ? intel_pmu_handle_irq+0x1fd/0x25d > [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae > [<c1386478>] ? perf_event_nmi_handler+0x34/0x3f > [<c13877dd>] ? notifier_call_chain+0x2a/0x47 > [<c138781e>] ? __atomic_notifier_call_chain+0x24/0x33 > [<c1387836>] ? atomic_notifier_call_chain+0x9/0xc > [<c1387869>] ? notify_die+0x30/0x34 > [<c1385c75>] ? do_nmi+0x73/0x25f > [<c13858a9>] ? nmi_stack_correct+0x28/0x2d > [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae > [<c100afda>] ? hw_perf_enable+0xf/0x10 > [<c1069b42>] ? __perf_event_sched_in+0x10c/0x119 > [<c1069b7a>] ? perf_event_task_sched_in+0x2b/0x31 > [<c1024088>] ? finish_task_switch+0x2b/0x72 > [<c1383d18>] ? __schedule+0x55f/0x5c3 > [<c1024c79>] ? try_to_wake_up+0x2b5/0x2bf > [<c1384f5b>] ? _raw_spin_lock+0xd/0x23 > [<c1383e0e>] ? preempt_schedule+0x40/0x5a > [<c13848f1>] ? rt_mutex_slowunlock+0x37/0x44 > [<c109a317>] ? pipe_release+0x7c/0x84 > [<c109598a>] ? __fput+0xe7/0x17b > [<c10932b5>] ? filp_close+0x4e/0x54 > [<c109331d>] ? sys_close+0x62/0x9b > [<c1002750>] ? sysenter_do_call+0x12/0x26 > > Both perf and oprofile appear to collect data successfully although when I ran > perf on a long-running program it generated the same kernel oops multiple times > and eventually the machine hung. > > I would appreciate any guidance that you can give. > > Best Regards, > Minnie Ho > > > > > ------------------------------------------------------------------------------ > This SF.net Dev2Dev email is sponsored by: > > Show off your parallel programming skills. > Enter the Intel(R) Threading Challenge 2010. > http://p.sf.net/sfu/intel-thread-sfd > > > > _______________________________________________ > oprofile-list mailing list > opr...@li... > https://lists.sourceforge.net/lists/listinfo/oprofile-list |
From: Ho, M. <min...@in...> - 2010-09-07 22:24:01
|
Hi Maynard and all, I am Jerry Sydir's colleague. We were running into issues installing oprofile and perf. The solution provided on the e-mail thread in response to Jerry's original posting on linux-rt-users works for perf, but not for oprofile. (With the perf solution posted on linux-rt-users, we still get the same kernel error when installing oprofile.) So I posted the error to the oprofile list hoping that folks on this list might have some tips on helping us with oprofile. (We prefer using oprofile to perf). My apologies for not having explained this in my previous posting. Minnie -----Original Message----- From: Maynard Johnson [mailto:may...@us...] Sent: Tuesday, September 07, 2010 8:54 AM To: Ho, Minnie Cc: opr...@li... Subject: Re: Kernel oops while running oprofile and perf on 2.6.33.7-rt29 kernel Ho, Minnie wrote: > Hello, This message below is an exact copy of a message sent to the linux-rt-users mailing list by Jerry Sydir on Aug 18. What's up with copying his note and forwarding it to oprofile-list as if it's your own? And the email thread on linux-rt-users provided a solution to the problem. Why are you posting this to oprofile-list and wasting our time? -Maynard > > I am having trouble running profiling tools on the 2.6.33.7 kernel with the rt29 > patch. I am running on an Intel Core 2 machine. I run oprofile and received the following kernel oops when > starting the oprofile daemon. Here is the log info: > > oprofile: using NMI interrupt. > BUG: sleeping function called from invalid context at kernel/rtmutex.c:707 > pcnt: 2 0 in_atomic(): 1, irqs_disabled(): 1, pid: 4020, name: oprofiled > Pid: 4020, comm: oprofiled Not tainted 2.6.33.7-rt29 #3 > Call Trace: > [<c104e8c4>] ? rt_spin_lock_fastlock+0x26/0x58 > [<c108ff56>] ? _slab_irq_disable+0x22/0x42 > [<c109108a>] ? __kmalloc+0x7d/0xf0 > [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] > [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] > [<e0070e8d>] ? nmi_cpu_setup+0x87/0xcc [oprofile] > [<e0070e06>] ? nmi_cpu_setup+0x0/0xcc [oprofile] > [<c102fa7b>] ? on_each_cpu+0x25/0x50 > [<e0070dd9>] ? nmi_setup+0x11d/0x14a [oprofile] > [<e006f133>] ? oprofile_setup+0x2d/0x86 [oprofile] > [<e006fede>] ? event_buffer_open+0x42/0x60 [oprofile] > [<c109363d>] ? __dentry_open+0x1a4/0x29a > [<c109b09b>] ? generic_permission+0xc/0x7e > [<c10937c1>] ? nameidata_to_filp+0x27/0x38 > [<e006fe9c>] ? event_buffer_open+0x0/0x60 [oprofile] > [<c109d214>] ? do_filp_open+0x439/0x843 > [<c10808f6>] ? __do_fault+0x2bf/0x2ef > [<c1090091>] ? slab_irq_enable+0x45/0x79 > [<c10933a2>] ? do_sys_open+0x4c/0xe4 > [<c109347e>] ? sys_open+0x1e/0x23 > [<c1002750>] ? sysenter_do_call+0x12/0x26 > > When I run perf record I get a slightly different kernel oops. Here is the relevant log info: > > BUG: sleeping function called from invalid context at arch/x86/mm/highmem_32.c:9 > pcnt: 14010003 0 in_atomic(): 1, irqs_disabled(): 1, pid: 3640, name: perf > Pid: 3640, comm: perf Not tainted 2.6.33.7-rt29 #3 > Call Trace: > [<c101d0c3>] ? kmap+0x41/0x52 > [<c100b66e>] ? perf_callchain+0x20d/0x2ac > [<c106abd9>] ? perf_prepare_sample+0x1d1/0x23b > [<c106ce0b>] ? __perf_event_overflow+0x20a/0x25e > [<c106d4bc>] ? perf_event_overflow+0xf/0x12 > [<c100c906>] ? intel_pmu_handle_irq+0x1fd/0x25d > [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae > [<c1386478>] ? perf_event_nmi_handler+0x34/0x3f > [<c13877dd>] ? notifier_call_chain+0x2a/0x47 > [<c138781e>] ? __atomic_notifier_call_chain+0x24/0x33 > [<c1387836>] ? atomic_notifier_call_chain+0x9/0xc > [<c1387869>] ? notify_die+0x30/0x34 > [<c1385c75>] ? do_nmi+0x73/0x25f > [<c13858a9>] ? nmi_stack_correct+0x28/0x2d > [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae > [<c100afda>] ? hw_perf_enable+0xf/0x10 > [<c1069b42>] ? __perf_event_sched_in+0x10c/0x119 > [<c1069b7a>] ? perf_event_task_sched_in+0x2b/0x31 > [<c1024088>] ? finish_task_switch+0x2b/0x72 > [<c1383d18>] ? __schedule+0x55f/0x5c3 > [<c1024c79>] ? try_to_wake_up+0x2b5/0x2bf > [<c1384f5b>] ? _raw_spin_lock+0xd/0x23 > [<c1383e0e>] ? preempt_schedule+0x40/0x5a > [<c13848f1>] ? rt_mutex_slowunlock+0x37/0x44 > [<c109a317>] ? pipe_release+0x7c/0x84 > [<c109598a>] ? __fput+0xe7/0x17b > [<c10932b5>] ? filp_close+0x4e/0x54 > [<c109331d>] ? sys_close+0x62/0x9b > [<c1002750>] ? sysenter_do_call+0x12/0x26 > > Both perf and oprofile appear to collect data successfully although when I ran > perf on a long-running program it generated the same kernel oops multiple times > and eventually the machine hung. > > I would appreciate any guidance that you can give. > > Best Regards, > Minnie Ho > > > > > ------------------------------------------------------------------------------ > This SF.net Dev2Dev email is sponsored by: > > Show off your parallel programming skills. > Enter the Intel(R) Threading Challenge 2010. > http://p.sf.net/sfu/intel-thread-sfd > > > > _______________________________________________ > oprofile-list mailing list > opr...@li... > https://lists.sourceforge.net/lists/listinfo/oprofile-list |
From: Maynard J. <may...@us...> - 2010-09-08 15:26:40
|
On 09/07/2010 5:23 PM, Ho, Minnie wrote: > Hi Maynard and all, > > I am Jerry Sydir's colleague. We were running into issues installing oprofile and perf. The solution provided on the e-mail thread in response to Jerry's original posting on linux-rt-users works for perf, but not for oprofile. (With the perf solution posted on linux-rt-users, we still get the same kernel error when installing oprofile.) > > So I posted the error to the oprofile list hoping that folks on this list might have some tips on helping us with oprofile. (We prefer using oprofile to perf). > > My apologies for not having explained this in my previous posting. Thanks for clarifying. Here's my two cents worth. The kmalloc call in ppro_setup_ctrs was changed to a kzalloc call in the linux base in 2.6.33 (I believe). But in either case, GFP_ATOMIC is used, which is supposed to guarantee that the call doesn't sleep. Yet, the failure you're seeing is "sleeping function called from invalid context". You should double-check that the kmalloc call in arch/x86/op_model_ppro.c:ppro_setup_ctrs does in fact use GFP_KERNEL. Maybe *Robert Richter* (on cc) might have a better insight . . . -Maynard > > Minnie > > -----Original Message----- > From: Maynard Johnson [mailto:may...@us...] > Sent: Tuesday, September 07, 2010 8:54 AM > To: Ho, Minnie > Cc: opr...@li... > Subject: Re: Kernel oops while running oprofile and perf on 2.6.33.7-rt29 kernel > > Ho, Minnie wrote: >> Hello, > > This message below is an exact copy of a message sent to the linux-rt-users mailing list by Jerry Sydir on Aug 18. What's up with copying his note and forwarding it to oprofile-list as if it's your own? And the email thread on linux-rt-users provided a solution to the problem. Why are you posting this to oprofile-list and wasting our time? > > -Maynard >> >> I am having trouble running profiling tools on the 2.6.33.7 kernel with the rt29 >> patch. I am running on an Intel Core 2 machine. I run oprofile and received the following kernel oops when >> starting the oprofile daemon. Here is the log info: >> >> oprofile: using NMI interrupt. >> BUG: sleeping function called from invalid context at kernel/rtmutex.c:707 >> pcnt: 2 0 in_atomic(): 1, irqs_disabled(): 1, pid: 4020, name: oprofiled >> Pid: 4020, comm: oprofiled Not tainted 2.6.33.7-rt29 #3 >> Call Trace: >> [<c104e8c4>] ? rt_spin_lock_fastlock+0x26/0x58 >> [<c108ff56>] ? _slab_irq_disable+0x22/0x42 >> [<c109108a>] ? __kmalloc+0x7d/0xf0 >> [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] >> [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] >> [<e0070e8d>] ? nmi_cpu_setup+0x87/0xcc [oprofile] >> [<e0070e06>] ? nmi_cpu_setup+0x0/0xcc [oprofile] >> [<c102fa7b>] ? on_each_cpu+0x25/0x50 >> [<e0070dd9>] ? nmi_setup+0x11d/0x14a [oprofile] >> [<e006f133>] ? oprofile_setup+0x2d/0x86 [oprofile] >> [<e006fede>] ? event_buffer_open+0x42/0x60 [oprofile] >> [<c109363d>] ? __dentry_open+0x1a4/0x29a >> [<c109b09b>] ? generic_permission+0xc/0x7e >> [<c10937c1>] ? nameidata_to_filp+0x27/0x38 >> [<e006fe9c>] ? event_buffer_open+0x0/0x60 [oprofile] >> [<c109d214>] ? do_filp_open+0x439/0x843 >> [<c10808f6>] ? __do_fault+0x2bf/0x2ef >> [<c1090091>] ? slab_irq_enable+0x45/0x79 >> [<c10933a2>] ? do_sys_open+0x4c/0xe4 >> [<c109347e>] ? sys_open+0x1e/0x23 >> [<c1002750>] ? sysenter_do_call+0x12/0x26 >> >> When I run perf record I get a slightly different kernel oops. Here is the relevant log info: >> >> BUG: sleeping function called from invalid context at arch/x86/mm/highmem_32.c:9 >> pcnt: 14010003 0 in_atomic(): 1, irqs_disabled(): 1, pid: 3640, name: perf >> Pid: 3640, comm: perf Not tainted 2.6.33.7-rt29 #3 >> Call Trace: >> [<c101d0c3>] ? kmap+0x41/0x52 >> [<c100b66e>] ? perf_callchain+0x20d/0x2ac >> [<c106abd9>] ? perf_prepare_sample+0x1d1/0x23b >> [<c106ce0b>] ? __perf_event_overflow+0x20a/0x25e >> [<c106d4bc>] ? perf_event_overflow+0xf/0x12 >> [<c100c906>] ? intel_pmu_handle_irq+0x1fd/0x25d >> [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae >> [<c1386478>] ? perf_event_nmi_handler+0x34/0x3f >> [<c13877dd>] ? notifier_call_chain+0x2a/0x47 >> [<c138781e>] ? __atomic_notifier_call_chain+0x24/0x33 >> [<c1387836>] ? atomic_notifier_call_chain+0x9/0xc >> [<c1387869>] ? notify_die+0x30/0x34 >> [<c1385c75>] ? do_nmi+0x73/0x25f >> [<c13858a9>] ? nmi_stack_correct+0x28/0x2d >> [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae >> [<c100afda>] ? hw_perf_enable+0xf/0x10 >> [<c1069b42>] ? __perf_event_sched_in+0x10c/0x119 >> [<c1069b7a>] ? perf_event_task_sched_in+0x2b/0x31 >> [<c1024088>] ? finish_task_switch+0x2b/0x72 >> [<c1383d18>] ? __schedule+0x55f/0x5c3 >> [<c1024c79>] ? try_to_wake_up+0x2b5/0x2bf >> [<c1384f5b>] ? _raw_spin_lock+0xd/0x23 >> [<c1383e0e>] ? preempt_schedule+0x40/0x5a >> [<c13848f1>] ? rt_mutex_slowunlock+0x37/0x44 >> [<c109a317>] ? pipe_release+0x7c/0x84 >> [<c109598a>] ? __fput+0xe7/0x17b >> [<c10932b5>] ? filp_close+0x4e/0x54 >> [<c109331d>] ? sys_close+0x62/0x9b >> [<c1002750>] ? sysenter_do_call+0x12/0x26 >> >> Both perf and oprofile appear to collect data successfully although when I ran >> perf on a long-running program it generated the same kernel oops multiple times >> and eventually the machine hung. >> >> I would appreciate any guidance that you can give. >> >> Best Regards, >> Minnie Ho >> >> >> >> >> ------------------------------------------------------------------------------ >> This SF.net Dev2Dev email is sponsored by: >> >> Show off your parallel programming skills. >> Enter the Intel(R) Threading Challenge 2010. >> http://p.sf.net/sfu/intel-thread-sfd >> >> >> >> _______________________________________________ >> oprofile-list mailing list >> opr...@li... >> https://lists.sourceforge.net/lists/listinfo/oprofile-list > |
From: Ho, M. <min...@in...> - 2010-09-09 13:48:18
|
Hi Maynard, Thanks very much for your response! We checked the code, and the ppro_setup_ctrs calls kzalloc with the GFP_ATOMIC flag. If we understand the kmalloc documentation correctly, this call should not be sleeping as Maynard indicated. So we are still not quite sure why we get this kernel error. Might you have any suggestions on how to proceed next? Minnie -----Original Message----- From: Maynard Johnson [mailto:may...@us...] Sent: Wednesday, September 08, 2010 8:26 AM To: Ho, Minnie Cc: opr...@li...; Sydir, Jerry; Robert Richter Subject: Re: Kernel oops while running oprofile and perf on 2.6.33.7-rt29 kernel On 09/07/2010 5:23 PM, Ho, Minnie wrote: > Hi Maynard and all, > > I am Jerry Sydir's colleague. We were running into issues installing oprofile and perf. The solution provided on the e-mail thread in response to Jerry's original posting on linux-rt-users works for perf, but not for oprofile. (With the perf solution posted on linux-rt-users, we still get the same kernel error when installing oprofile.) > > So I posted the error to the oprofile list hoping that folks on this list might have some tips on helping us with oprofile. (We prefer using oprofile to perf). > > My apologies for not having explained this in my previous posting. Thanks for clarifying. Here's my two cents worth. The kmalloc call in ppro_setup_ctrs was changed to a kzalloc call in the linux base in 2.6.33 (I believe). But in either case, GFP_ATOMIC is used, which is supposed to guarantee that the call doesn't sleep. Yet, the failure you're seeing is "sleeping function called from invalid context". You should double-check that the kmalloc call in arch/x86/op_model_ppro.c:ppro_setup_ctrs does in fact use GFP_KERNEL. Maybe *Robert Richter* (on cc) might have a better insight . . . -Maynard > > Minnie > > -----Original Message----- > From: Maynard Johnson [mailto:may...@us...] > Sent: Tuesday, September 07, 2010 8:54 AM > To: Ho, Minnie > Cc: opr...@li... > Subject: Re: Kernel oops while running oprofile and perf on 2.6.33.7-rt29 kernel > > Ho, Minnie wrote: >> Hello, > > This message below is an exact copy of a message sent to the linux-rt-users mailing list by Jerry Sydir on Aug 18. What's up with copying his note and forwarding it to oprofile-list as if it's your own? And the email thread on linux-rt-users provided a solution to the problem. Why are you posting this to oprofile-list and wasting our time? > > -Maynard >> >> I am having trouble running profiling tools on the 2.6.33.7 kernel with the rt29 >> patch. I am running on an Intel Core 2 machine. I run oprofile and received the following kernel oops when >> starting the oprofile daemon. Here is the log info: >> >> oprofile: using NMI interrupt. >> BUG: sleeping function called from invalid context at kernel/rtmutex.c:707 >> pcnt: 2 0 in_atomic(): 1, irqs_disabled(): 1, pid: 4020, name: oprofiled >> Pid: 4020, comm: oprofiled Not tainted 2.6.33.7-rt29 #3 >> Call Trace: >> [<c104e8c4>] ? rt_spin_lock_fastlock+0x26/0x58 >> [<c108ff56>] ? _slab_irq_disable+0x22/0x42 >> [<c109108a>] ? __kmalloc+0x7d/0xf0 >> [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] >> [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] >> [<e0070e8d>] ? nmi_cpu_setup+0x87/0xcc [oprofile] >> [<e0070e06>] ? nmi_cpu_setup+0x0/0xcc [oprofile] >> [<c102fa7b>] ? on_each_cpu+0x25/0x50 >> [<e0070dd9>] ? nmi_setup+0x11d/0x14a [oprofile] >> [<e006f133>] ? oprofile_setup+0x2d/0x86 [oprofile] >> [<e006fede>] ? event_buffer_open+0x42/0x60 [oprofile] >> [<c109363d>] ? __dentry_open+0x1a4/0x29a >> [<c109b09b>] ? generic_permission+0xc/0x7e >> [<c10937c1>] ? nameidata_to_filp+0x27/0x38 >> [<e006fe9c>] ? event_buffer_open+0x0/0x60 [oprofile] >> [<c109d214>] ? do_filp_open+0x439/0x843 >> [<c10808f6>] ? __do_fault+0x2bf/0x2ef >> [<c1090091>] ? slab_irq_enable+0x45/0x79 >> [<c10933a2>] ? do_sys_open+0x4c/0xe4 >> [<c109347e>] ? sys_open+0x1e/0x23 >> [<c1002750>] ? sysenter_do_call+0x12/0x26 >> >> When I run perf record I get a slightly different kernel oops. Here is the relevant log info: >> >> BUG: sleeping function called from invalid context at arch/x86/mm/highmem_32.c:9 >> pcnt: 14010003 0 in_atomic(): 1, irqs_disabled(): 1, pid: 3640, name: perf >> Pid: 3640, comm: perf Not tainted 2.6.33.7-rt29 #3 >> Call Trace: >> [<c101d0c3>] ? kmap+0x41/0x52 >> [<c100b66e>] ? perf_callchain+0x20d/0x2ac >> [<c106abd9>] ? perf_prepare_sample+0x1d1/0x23b >> [<c106ce0b>] ? __perf_event_overflow+0x20a/0x25e >> [<c106d4bc>] ? perf_event_overflow+0xf/0x12 >> [<c100c906>] ? intel_pmu_handle_irq+0x1fd/0x25d >> [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae >> [<c1386478>] ? perf_event_nmi_handler+0x34/0x3f >> [<c13877dd>] ? notifier_call_chain+0x2a/0x47 >> [<c138781e>] ? __atomic_notifier_call_chain+0x24/0x33 >> [<c1387836>] ? atomic_notifier_call_chain+0x9/0xc >> [<c1387869>] ? notify_die+0x30/0x34 >> [<c1385c75>] ? do_nmi+0x73/0x25f >> [<c13858a9>] ? nmi_stack_correct+0x28/0x2d >> [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae >> [<c100afda>] ? hw_perf_enable+0xf/0x10 >> [<c1069b42>] ? __perf_event_sched_in+0x10c/0x119 >> [<c1069b7a>] ? perf_event_task_sched_in+0x2b/0x31 >> [<c1024088>] ? finish_task_switch+0x2b/0x72 >> [<c1383d18>] ? __schedule+0x55f/0x5c3 >> [<c1024c79>] ? try_to_wake_up+0x2b5/0x2bf >> [<c1384f5b>] ? _raw_spin_lock+0xd/0x23 >> [<c1383e0e>] ? preempt_schedule+0x40/0x5a >> [<c13848f1>] ? rt_mutex_slowunlock+0x37/0x44 >> [<c109a317>] ? pipe_release+0x7c/0x84 >> [<c109598a>] ? __fput+0xe7/0x17b >> [<c10932b5>] ? filp_close+0x4e/0x54 >> [<c109331d>] ? sys_close+0x62/0x9b >> [<c1002750>] ? sysenter_do_call+0x12/0x26 >> >> Both perf and oprofile appear to collect data successfully although when I ran >> perf on a long-running program it generated the same kernel oops multiple times >> and eventually the machine hung. >> >> I would appreciate any guidance that you can give. >> >> Best Regards, >> Minnie Ho >> >> >> >> >> ------------------------------------------------------------------------------ >> This SF.net Dev2Dev email is sponsored by: >> >> Show off your parallel programming skills. >> Enter the Intel(R) Threading Challenge 2010. >> http://p.sf.net/sfu/intel-thread-sfd >> >> >> >> _______________________________________________ >> oprofile-list mailing list >> opr...@li... >> https://lists.sourceforge.net/lists/listinfo/oprofile-list > |
From: Maynard J. <may...@us...> - 2010-09-09 14:58:22
|
On 09/09/2010 8:47 AM, Ho, Minnie wrote: > Hi Maynard, > > Thanks very much for your response! > > We checked the code, and the ppro_setup_ctrs calls kzalloc with the GFP_ATOMIC flag. If we understand the kmalloc documentation correctly, this call should not be sleeping as Maynard indicated. So we are still not quite sure why we get this kernel error. > > Might you have any suggestions on how to proceed next? I spoke to one of my colleagues who does some RT kernel work, and he recommended reporting this to the linux-rt-users mailing list, especially in light of this only happening on an RT kernel. -Maynard > > Minnie > > -----Original Message----- > From: Maynard Johnson [mailto:may...@us...] > Sent: Wednesday, September 08, 2010 8:26 AM > To: Ho, Minnie > Cc: opr...@li...; Sydir, Jerry; Robert Richter > Subject: Re: Kernel oops while running oprofile and perf on 2.6.33.7-rt29 kernel > > On 09/07/2010 5:23 PM, Ho, Minnie wrote: >> Hi Maynard and all, >> >> I am Jerry Sydir's colleague. We were running into issues installing oprofile and perf. The solution provided on the e-mail thread in response to Jerry's original posting on linux-rt-users works for perf, but not for oprofile. (With the perf solution posted on linux-rt-users, we still get the same kernel error when installing oprofile.) >> >> So I posted the error to the oprofile list hoping that folks on this list might have some tips on helping us with oprofile. (We prefer using oprofile to perf). >> >> My apologies for not having explained this in my previous posting. > Thanks for clarifying. Here's my two cents worth. The kmalloc call in > ppro_setup_ctrs was changed to a kzalloc call in the linux base in 2.6.33 (I > believe). But in either case, GFP_ATOMIC is used, which is supposed to > guarantee that the call doesn't sleep. Yet, the failure you're seeing is > "sleeping function called from invalid context". You should double-check that > the kmalloc call in arch/x86/op_model_ppro.c:ppro_setup_ctrs does in fact use > GFP_KERNEL. > > Maybe *Robert Richter* (on cc) might have a better insight . . . > > -Maynard >> >> Minnie >> >> -----Original Message----- >> From: Maynard Johnson [mailto:may...@us...] >> Sent: Tuesday, September 07, 2010 8:54 AM >> To: Ho, Minnie >> Cc: opr...@li... >> Subject: Re: Kernel oops while running oprofile and perf on 2.6.33.7-rt29 kernel >> >> Ho, Minnie wrote: >>> Hello, >> >> This message below is an exact copy of a message sent to the linux-rt-users mailing list by Jerry Sydir on Aug 18. What's up with copying his note and forwarding it to oprofile-list as if it's your own? And the email thread on linux-rt-users provided a solution to the problem. Why are you posting this to oprofile-list and wasting our time? >> >> -Maynard >>> >>> I am having trouble running profiling tools on the 2.6.33.7 kernel with the rt29 >>> patch. I am running on an Intel Core 2 machine. I run oprofile and received the following kernel oops when >>> starting the oprofile daemon. Here is the log info: >>> >>> oprofile: using NMI interrupt. >>> BUG: sleeping function called from invalid context at kernel/rtmutex.c:707 >>> pcnt: 2 0 in_atomic(): 1, irqs_disabled(): 1, pid: 4020, name: oprofiled >>> Pid: 4020, comm: oprofiled Not tainted 2.6.33.7-rt29 #3 >>> Call Trace: >>> [<c104e8c4>] ? rt_spin_lock_fastlock+0x26/0x58 >>> [<c108ff56>] ? _slab_irq_disable+0x22/0x42 >>> [<c109108a>] ? __kmalloc+0x7d/0xf0 >>> [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] >>> [<e0071529>] ? ppro_setup_ctrs+0x29/0x1b8 [oprofile] >>> [<e0070e8d>] ? nmi_cpu_setup+0x87/0xcc [oprofile] >>> [<e0070e06>] ? nmi_cpu_setup+0x0/0xcc [oprofile] >>> [<c102fa7b>] ? on_each_cpu+0x25/0x50 >>> [<e0070dd9>] ? nmi_setup+0x11d/0x14a [oprofile] >>> [<e006f133>] ? oprofile_setup+0x2d/0x86 [oprofile] >>> [<e006fede>] ? event_buffer_open+0x42/0x60 [oprofile] >>> [<c109363d>] ? __dentry_open+0x1a4/0x29a >>> [<c109b09b>] ? generic_permission+0xc/0x7e >>> [<c10937c1>] ? nameidata_to_filp+0x27/0x38 >>> [<e006fe9c>] ? event_buffer_open+0x0/0x60 [oprofile] >>> [<c109d214>] ? do_filp_open+0x439/0x843 >>> [<c10808f6>] ? __do_fault+0x2bf/0x2ef >>> [<c1090091>] ? slab_irq_enable+0x45/0x79 >>> [<c10933a2>] ? do_sys_open+0x4c/0xe4 >>> [<c109347e>] ? sys_open+0x1e/0x23 >>> [<c1002750>] ? sysenter_do_call+0x12/0x26 >>> >>> When I run perf record I get a slightly different kernel oops. Here is the relevant log info: >>> >>> BUG: sleeping function called from invalid context at arch/x86/mm/highmem_32.c:9 >>> pcnt: 14010003 0 in_atomic(): 1, irqs_disabled(): 1, pid: 3640, name: perf >>> Pid: 3640, comm: perf Not tainted 2.6.33.7-rt29 #3 >>> Call Trace: >>> [<c101d0c3>] ? kmap+0x41/0x52 >>> [<c100b66e>] ? perf_callchain+0x20d/0x2ac >>> [<c106abd9>] ? perf_prepare_sample+0x1d1/0x23b >>> [<c106ce0b>] ? __perf_event_overflow+0x20a/0x25e >>> [<c106d4bc>] ? perf_event_overflow+0xf/0x12 >>> [<c100c906>] ? intel_pmu_handle_irq+0x1fd/0x25d >>> [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae >>> [<c1386478>] ? perf_event_nmi_handler+0x34/0x3f >>> [<c13877dd>] ? notifier_call_chain+0x2a/0x47 >>> [<c138781e>] ? __atomic_notifier_call_chain+0x24/0x33 >>> [<c1387836>] ? atomic_notifier_call_chain+0x9/0xc >>> [<c1387869>] ? notify_die+0x30/0x34 >>> [<c1385c75>] ? do_nmi+0x73/0x25f >>> [<c13858a9>] ? nmi_stack_correct+0x28/0x2d >>> [<c100b7a3>] ? intel_pmu_enable_all+0x3d/0xae >>> [<c100afda>] ? hw_perf_enable+0xf/0x10 >>> [<c1069b42>] ? __perf_event_sched_in+0x10c/0x119 >>> [<c1069b7a>] ? perf_event_task_sched_in+0x2b/0x31 >>> [<c1024088>] ? finish_task_switch+0x2b/0x72 >>> [<c1383d18>] ? __schedule+0x55f/0x5c3 >>> [<c1024c79>] ? try_to_wake_up+0x2b5/0x2bf >>> [<c1384f5b>] ? _raw_spin_lock+0xd/0x23 >>> [<c1383e0e>] ? preempt_schedule+0x40/0x5a >>> [<c13848f1>] ? rt_mutex_slowunlock+0x37/0x44 >>> [<c109a317>] ? pipe_release+0x7c/0x84 >>> [<c109598a>] ? __fput+0xe7/0x17b >>> [<c10932b5>] ? filp_close+0x4e/0x54 >>> [<c109331d>] ? sys_close+0x62/0x9b >>> [<c1002750>] ? sysenter_do_call+0x12/0x26 >>> >>> Both perf and oprofile appear to collect data successfully although when I ran >>> perf on a long-running program it generated the same kernel oops multiple times >>> and eventually the machine hung. >>> >>> I would appreciate any guidance that you can give. >>> >>> Best Regards, >>> Minnie Ho >>> >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> This SF.net Dev2Dev email is sponsored by: >>> >>> Show off your parallel programming skills. >>> Enter the Intel(R) Threading Challenge 2010. >>> http://p.sf.net/sfu/intel-thread-sfd >>> >>> >>> >>> _______________________________________________ >>> oprofile-list mailing list >>> opr...@li... >>> https://lists.sourceforge.net/lists/listinfo/oprofile-list >> > |