OProfile

This email lists some known regressions in 2.6.19-rc5 compared to 2.6.18
that are not yet fixed in Linus' tree.

If you find your name in the Cc header, you are either submitter of one
of the bugs, maintainer of an affectected subsystem or driver, a patch
of you caused a breakage or I'm considering you in any other way possibly
involved with one or more of these issues.

Due to the huge amount of recipients, please trim the Cc when answering.


Subject    : PCI MSI setting corrupted during resume
References : http://bugzilla.kernel.org/show_bug.cgi?id=7479
Submitter  : Stephen Hemminger <shemminger@...>
Status     : unknown


Subject    : x86_64 boot failure: irq 22: nobody cared (hda_intel MSI)
References : http://lkml.org/lkml/2006/11/8/98
Submitter  : Olivier Nicolas <olivn@...>
Status     : unknown


Subject    : SMP kernel can not generate ISA irq properly
References : http://lkml.org/lkml/2006/10/22/15
             http://lkml.org/lkml/2006/11/10/142
Submitter  : Komuro <komurojun-mbn@...>
Handled-By : Thomas Gleixner <tglx@...>
Status     : Thomas is investigating


Subject    : x86_64: Fix partial page check to ensure unusable memory
                     is not being marked usable
References : http://lkml.org/lkml/2006/11/9/239
Submitter  : Aaron Durbin <adurbin@...>
Caused-By  : Mel Gorman <mel@...>
             commit 5cb248abf5ab65ab543b2d5fc16c738b28031fc0
Patch      : http://lkml.org/lkml/2006/11/9/239
Status     : patch available


Subject    : x86_64: Bad page state in process 'swapper'
References : http://lkml.org/lkml/2006/11/10/135
             http://lkml.org/lkml/2006/11/10/208
Submitter  : Andre Noll <maan@...>
Handled-By : Andi Kleen <ak@...>
Status     : Andi is investigating


Subject    : x86_64: oprofile doesn't work
References : http://lkml.org/lkml/2006/10/27/3
Submitter  : Prakash Punnoor <prakash@...>
Status     : unknown


Subject    : weird battery charge level reported
             ACPI Error method parse / execution failed
References : http://bugzilla.kernel.org/show_bug.cgi?id=7466
Submitter  : Olivier Mondoloni <olivier.mondoloni@...>
Status     : unknown


Subject    : ThinkPad R50p: boot fail with (lapic && on_battery)
References : http://lkml.org/lkml/2006/10/31/333
Submitter  : Ernst Herzberg <earny@...>
Handled-By : Len Brown <len.brown@...>
Status     : problem is being debugged


Subject    : BUG: scheduling while atomic: events/0/0x00000001/4
             after resume
References : http://lkml.org/lkml/2006/11/2/209
Submitter  : Paolo Ornati <ornati@...>
Status     : unknown


Subject    : sata-via doesn't detect anymore disks attached to VIA vt6421
References : http://bugzilla.kernel.org/show_bug.cgi?id=7255
Submitter  : Thierry Vignaud <tvignaud@...>
Status     : unknown


Subject    : libata must be initialized earlier
References : http://ozlabs.org/pipermail/linuxppc-dev/2006-November/027945.html
Submitter  : Paul Mackerras <paulus@...>
Handled-By : Brian King <brking@...>
Patch      : http://marc.theaimsgroup.com/?l=linux-ide&m=116169938407596&w=2
Status     : patch available


Subject    : unable to rip cd
References : http://lkml.org/lkml/2006/10/13/100
             http://lkml.org/lkml/2006/11/8/42
Submitter  : Alex Romosan <romosan@...>
Handled-By : Jens Axboe <jens.axboe@...>
Status     : Jens is investigating

This email lists some known regressions in 2.6.19-rc5 compared to 2.6.18
that are not yet fixed in Linus' tree.

If you find your name in the Cc header, you are either submitter of one
of the bugs, maintainer of an affectected subsystem or driver, a patch
of you caused a breakage or I'm considering you in any other way possibly
involved with one or more of these issues.

Due to the huge amount of recipients, please trim the Cc when answering.


Subject    : PCI MSI setting corrupted during resume
References : http://bugzilla.kernel.org/show_bug.cgi?id=7479
Submitter  : Stephen Hemminger <shemminger@...>
Status     : unknown


Subject    : SMP kernel can not generate ISA irq properly
References : http://lkml.org/lkml/2006/10/22/15
             http://lkml.org/lkml/2006/11/10/142
Submitter  : Komuro <komurojun-mbn@...>
Handled-By : "Eric W. Biederman" <ebiederm@...>
             Ingo Molnar <mingo@...>
Status     : problem is being debugged


Subject    : ThinkPad R50p: boot fail with (lapic && on_battery)
References : http://lkml.org/lkml/2006/10/31/333
Submitter  : Ernst Herzberg <earny@...>
Handled-By : Len Brown <len.brown@...>
Status     : problem is being debugged


Subject    : x86_64: Bad page state in process 'swapper'
References : http://lkml.org/lkml/2006/11/10/135
             http://lkml.org/lkml/2006/11/10/208
Submitter  : Andre Noll <maan@...>
Handled-By : Andi Kleen <ak@...>
Status     : Andi is investigating


Subject    : x86_64: oprofile doesn't work
References : http://lkml.org/lkml/2006/10/27/3
Submitter  : Prakash Punnoor <prakash@...>
Status     : unknown


Subject    : unable to rip cd
References : http://lkml.org/lkml/2006/10/13/100
             http://lkml.org/lkml/2006/11/8/42
Submitter  : Alex Romosan <romosan@...>
Handled-By : Jens Axboe <jens.axboe@...>
Status     : Jens is investigating


Subject    : can't disable OHCI wakeup via sysfs
References : http://lkml.org/lkml/2006/11/11/33
Submitter  : Andrey Borzenkov <arvidjaar@...>
Handled-By : Alan Stern <stern@...>
Patch      : http://lkml.org/lkml/2006/11/13/261
Status     : patch available

> On a working kernel on an Opteron, we have normally 4 directories 
> in /dev/oprofile :
> 
> # ls -ld /dev/oprofile/?
> drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/0
> drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/1
> drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/2
> drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/3
> 
> With linux-2.6.19-rc5, the first one (0) is missing and we get 1,2,3

That's because 0 was never available. It is used by the NMI watchdog.
The new kernel doesn't give it to oprofile anymore.

> Maybe the 'bug' is in oprofile tools, that currently expect to find '0'

Yes, it's likely a user space issue.

-Andi

On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:

> Subject    : x86_64: oprofile doesn't work
> References : http://lkml.org/lkml/2006/10/27/3
> Submitter  : Prakash Punnoor <prakash@...>
> Status     : unknown
>

I confirm a got this one too.
On a working kernel on an Opteron, we have normally 4 directories 
in /dev/oprofile :

# ls -ld /dev/oprofile/?
drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/0
drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/1
drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/2
drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/3

With linux-2.6.19-rc5, the first one (0) is missing and we get 1,2,3

Maybe the 'bug' is in oprofile tools, that currently expect to find '0'

Eric

Andi Kleen wrote:
>>On a working kernel on an Opteron, we have normally 4 directories 
>>in /dev/oprofile :
>>
>># ls -ld /dev/oprofile/?
>>drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/0
>>drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/1
>>drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/2
>>drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/3
>>
>>With linux-2.6.19-rc5, the first one (0) is missing and we get 1,2,3
> 
> 
> That's because 0 was never available. It is used by the NMI watchdog.
> The new kernel doesn't give it to oprofile anymore.
> 
> 
>>Maybe the 'bug' is in oprofile tools, that currently expect to find '0'
> 
> 
> Yes, it's likely a user space issue.
> 
> -Andi

OProfile has a simplistic view of the performance monitoring hardware. The 
routines in libop/op_alloc_counter.c determine what set of performance registers 
is available from the processor in use. There is no check to see what registers 
are actually available in the /dev/oprofile directory.

opcontrol executes ophelp to determine which specific counters to count which 
events. The function map_event_to_counter() in libop/op_alloc_counter.c does the 
actual selection. It seems what is needed is for map_event_to_counter() to check 
to see which counters are available and mark the others as unavailable.

-Will

> OProfile has a simplistic view of the performance monitoring hardware. The 
> routines in libop/op_alloc_counter.c determine what set of performance registers 
> is available from the processor in use. There is no check to see what registers 
> are actually available in the /dev/oprofile directory.
> 
> opcontrol executes ophelp to determine which specific counters to count which 
> events. The function map_event_to_counter() in libop/op_alloc_counter.c does the 
> actual selection. It seems what is needed is for map_event_to_counter() to check 
> to see which counters are available and mark the others as unavailable

Thanks for the explanation. Can you please fix it and release a new version? 
Documentation/Changes could be adapted then.

-Andi

On Wed, 15 Nov 2006 17:48:05 +0100
Andi Kleen <ak@...> wrote:

> 
> > OProfile has a simplistic view of the performance monitoring hardware. The 
> > routines in libop/op_alloc_counter.c determine what set of performance registers 
> > is available from the processor in use. There is no check to see what registers 
> > are actually available in the /dev/oprofile directory.
> > 
> > opcontrol executes ophelp to determine which specific counters to count which 
> > events. The function map_event_to_counter() in libop/op_alloc_counter.c does the 
> > actual selection. It seems what is needed is for map_event_to_counter() to check 
> > to see which counters are available and mark the others as unavailable
> 
> Thanks for the explanation. Can you please fix it and release a new version? 
> Documentation/Changes could be adapted then.
> 

Meanwhile we should restore the NMI counter to fix this bug.

On Wednesday 15 November 2006 19:39, Andrew Morton wrote:
> On Wed, 15 Nov 2006 17:48:05 +0100
> Andi Kleen <ak@...> wrote:
> 
> > 
> > > OProfile has a simplistic view of the performance monitoring hardware. The 
> > > routines in libop/op_alloc_counter.c determine what set of performance registers 
> > > is available from the processor in use. There is no check to see what registers 
> > > are actually available in the /dev/oprofile directory.
> > > 
> > > opcontrol executes ophelp to determine which specific counters to count which 
> > > events. The function map_event_to_counter() in libop/op_alloc_counter.c does the 
> > > actual selection. It seems what is needed is for map_event_to_counter() to check 
> > > to see which counters are available and mark the others as unavailable
> > 
> > Thanks for the explanation. Can you please fix it and release a new version? 
> > Documentation/Changes could be adapted then.
> > 
> 
> Meanwhile we should restore the NMI counter to fix this bug.

No, it was always oprofile who was buggy here, silently taking 
the nmi watchdog away.

-Andi

On Wed, 15 Nov 2006, Andi Kleen wrote:
> > 
> > Meanwhile we should restore the NMI counter to fix this bug.
> 
> No, it was always oprofile who was buggy here, silently taking 
> the nmi watchdog away.

Andi, your "blame game" doesn't matter.

The fact is, it used to work, and the kernel changed interfaces, so now it 
doesn't. 

In other words, a kernel interface to user land changed. THAT IS ALWAYS A 
BUG. We don't change UI.

Yes, "oprofile" should be fixed to not depend on that, but the kernel 
shouldn't change the interfaces, and we should add back the zero entry.

		Linus

> The fact is, it used to work, and the kernel changed interfaces, so now it 
> doesn't. 

No, it didn't work. oprofile may have done something, but it 
just silently killed the NMI watchdog in the process.
That was never acceptable.

Now we do proper accounting of NMI sources and also proper allocation
of performance counters.

 
> Yes, "oprofile" should be fixed to not depend on that, but the kernel 
> shouldn't change the interfaces, and we should add back the zero entry.

That would break the nmi watchdog again.

Anyways, there is a sysctl to disable the nmi watchdog if someone
is desperate.

But I think it is clearly oprofile who did wrong here and needs
to be fixed.

-Andi

On Wed, 15 Nov 2006 20:23:53 +0100
Andi Kleen <ak@...> wrote:

> 
> > The fact is, it used to work, and the kernel changed interfaces, so now it 
> > doesn't. 
> 
> No, it didn't work. oprofile may have done something, but it 
> just silently killed the NMI watchdog in the process.
> That was never acceptable.

But people could get profiles out.  I know, I've seen them!

> Now we do proper accounting of NMI sources and also proper allocation
> of performance counters.
> 
>  
> > Yes, "oprofile" should be fixed to not depend on that, but the kernel 
> > shouldn't change the interfaces, and we should add back the zero entry.
> 
> That would break the nmi watchdog again.
> 
> Anyways, there is a sysctl to disable the nmi watchdog if someone
> is desperate.
> 
> But I think it is clearly oprofile who did wrong here and needs
> to be fixed.
> 

Is it correct to say that oprofile-on-2.6.18 works, and that
oprofile-on-2.6.19-rc5 does not?

Or is there some sort of workaround for this, or does 2.6.19-rc5 only fail
in some particular scenarios?

If it's really true that oprofile is simply busted then that's a serious
problem and we should find some way of unbusting it.  If that means just
adding a dummy "0" entry which always returns zero or something like that,
then fine.

But we can't just go and bust it.

Andrew Morton <akpm@...> writes:

> Is it correct to say that oprofile-on-2.6.18 works, and that
> oprofile-on-2.6.19-rc5 does not?
>
> Or is there some sort of workaround for this, or does 2.6.19-rc5 only fail
> in some particular scenarios?
>
> If it's really true that oprofile is simply busted then that's a serious
> problem and we should find some way of unbusting it.  If that means just
> adding a dummy "0" entry which always returns zero or something like that,
> then fine.
>
> But we can't just go and bust it.

The simple question.  If we turn off the NMI watchdog on 2.6.19-rc5 
does oprofile work?  I believe that is what Andi said.

The description I read was a resource conflict. The resources oprofile
just expects it can used are already in use so we tell it no and
the user space oprofile doesn't cope.

Now I don't know the interface allows us to rename the interfaces
from 1 2 3 to 0 1 2.  If we can then that looks like something we can
fix.  Otherwise from the description I tend to agree with Andi.

The user space application assumed it own hardware that it did not.

Hmm.  I bet if nothing else we could move the NMI watchdog from 0 to 3
and make things work that way...


Eric

On Wed, 15 Nov 2006 14:18:24 -0700
ebiederm@... (Eric W. Biederman) wrote:

> Andrew Morton <akpm@...> writes:
> 
> > Is it correct to say that oprofile-on-2.6.18 works, and that
> > oprofile-on-2.6.19-rc5 does not?
> >
> > Or is there some sort of workaround for this, or does 2.6.19-rc5 only fail
> > in some particular scenarios?
> >
> > If it's really true that oprofile is simply busted then that's a serious
> > problem and we should find some way of unbusting it.  If that means just
> > adding a dummy "0" entry which always returns zero or something like that,
> > then fine.
> >
> > But we can't just go and bust it.
> 
> The simple question.  If we turn off the NMI watchdog on 2.6.19-rc5 
> does oprofile work?  I believe that is what Andi said.
> 
> The description I read was a resource conflict. The resources oprofile
> just expects it can used are already in use so we tell it no and
> the user space oprofile doesn't cope.

That would have been a bug in earlier kernels.

> Now I don't know the interface allows us to rename the interfaces
> from 1 2 3 to 0 1 2.  If we can then that looks like something we can
> fix.  Otherwise from the description I tend to agree with Andi.
> 
> The user space application assumed it own hardware that it did not.
> 
> Hmm.  I bet if nothing else we could move the NMI watchdog from 0 to 3
> and make things work that way...

Surely the appropriate behaviour is to allow oprofile to steal the NMI and
to then put the NMI back to doing the watchdog thing after oprofile has
finished with it.

If that's not a feasible thing to do for 2.6.19 then some short-term
hack which makes oprofile work again is needed.

On Wed, Nov 15, 2006 at 12:21:18PM -0800, Andrew Morton wrote:
> Andi Kleen <ak@...> wrote:
> 
> > 
> > > The fact is, it used to work, and the kernel changed interfaces, so now it 
> > > doesn't. 
> > 
> > No, it didn't work. oprofile may have done something, but it 
> > just silently killed the NMI watchdog in the process.
> > That was never acceptable.
> 
> But people could get profiles out.  I know, I've seen them!

Just the nmi watchdog was gone then.

> 
> > Now we do proper accounting of NMI sources and also proper allocation
> > of performance counters.
> > 
> >  
> > > Yes, "oprofile" should be fixed to not depend on that, but the kernel 
> > > shouldn't change the interfaces, and we should add back the zero entry.
> > 
> > That would break the nmi watchdog again.
> > 
> > Anyways, there is a sysctl to disable the nmi watchdog if someone
> > is desperate.
> > 
> > But I think it is clearly oprofile who did wrong here and needs
> > to be fixed.
> > 
> 
> Is it correct to say that oprofile-on-2.6.18 works, and that
> oprofile-on-2.6.19-rc5 does not?
> 
> Or is there some sort of workaround for this, or does 2.6.19-rc5 only fail

echo 0 > /proc/sys/kernel/nmi_watchdog before the oprofile module is loaded.
With builtin oprofile probably nmi_watchdog=0

> in some particular scenarios?

On x86-64 and on newer i386 machines (based on DMI year)


> 
> If it's really true that oprofile is simply busted then that's a serious
> problem and we should find some way of unbusting it.  If that means just
> adding a dummy "0" entry which always returns zero or something like that,
> then fine.

That could be probably done.

> But we can't just go and bust it.

It just did something unbelievable broken before. I would say it busted
itself.

-Andi

On Thu, 16 Nov 2006 04:21:09 +0100
Andi Kleen <ak@...> wrote:

> > 
> > If it's really true that oprofile is simply busted then that's a serious
> > problem and we should find some way of unbusting it.  If that means just
> > adding a dummy "0" entry which always returns zero or something like that,
> > then fine.
> 
> That could be probably done.

I'm told that this is exactly what it was doing before it got changed.

> > But we can't just go and bust it.
> 
> It just did something unbelievable broken before.

What did it do?

> I would say it busted
> itself.

It gave profiles, which was fairly handy.

On Thursday 16 November 2006 06:05, Andrew Morton wrote:
> On Thu, 16 Nov 2006 04:21:09 +0100
> Andi Kleen <ak@...> wrote:
> 
> > > 
> > > If it's really true that oprofile is simply busted then that's a serious
> > > problem and we should find some way of unbusting it.  If that means just
> > > adding a dummy "0" entry which always returns zero or something like that,
> > > then fine.
> > 
> > That could be probably done.
> 
> I'm told that this is exactly what it was doing before it got changed.

Hmm, ok perhaps that can be arranged again.

The trouble is that I want to use this performance counter for
other purposes too, so we would run into trouble again 
if oprofile keeps stealing it.

> > > But we can't just go and bust it.
> > 
> > It just did something unbelievable broken before.
> 
> What did it do?

Silently kill the nmi watchdog.

> 
> > I would say it busted
> > itself.
> 
> It gave profiles, which was fairly handy.

I'm sure it can be fixed there. Ok ok I keep sounding like a sysfs maintainer
now @)

-Andi

Andrew Morton writes:
 > Surely the appropriate behaviour is to allow oprofile to steal the NMI and
 > to then put the NMI back to doing the watchdog thing after oprofile has
 > finished with it.

Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
the in-kernel API allowing real performance counter drivers like
oprofile (and perfctr) to claim the HW from the NMI watchdog,
do their work, and then release it which resumed the watchdog.

Note that oprofile (and perfctr) didn't do anything behind the
NMI watchdog's back. They went via the API. Nothing dodgy going on.

Andi Kleen wrote:
> On Thursday 16 November 2006 06:05, Andrew Morton wrote:
> 
>>On Thu, 16 Nov 2006 04:21:09 +0100
>>Andi Kleen <ak@...> wrote:
>>
>>
>>>>If it's really true that oprofile is simply busted then that's a serious
>>>>problem and we should find some way of unbusting it.  If that means just
>>>>adding a dummy "0" entry which always returns zero or something like that,
>>>>then fine.
>>>
>>>That could be probably done.
>>
>>I'm told that this is exactly what it was doing before it got changed.
> 
> 
> Hmm, ok perhaps that can be arranged again.
> 
> The trouble is that I want to use this performance counter for
> other purposes too, so we would run into trouble again 
> if oprofile keeps stealing it.

What other purposes do you see the performance counters useful for? To collect 
information on process characteristics so they can be scheduled more efficiently?

Is this going to require sharing the nmi interrupt and knowing which perfcounter 
register triggered the interrupt to get the correct action?  Currently the 
oprofile interrupt handler assumes any performance monitoring counter it sees 
overflowing is something it should count.

-Will

> What other purposes do you see the performance counters useful for? 

Export one to user space as a cycle counter for benchmarking. RDTSC doesn't 
do this job anymore.

> To collect information on process characteristics so they can be scheduled more efficiently?

That might happen at some point in the future, but i would expect
us to wait for CPUs with more performance counters first.

> Is this going to require sharing the nmi interrupt and knowing which perfcounter 
> register triggered the interrupt to get the correct action?  Currently the 
> oprofile interrupt handler assumes any performance monitoring counter it sees 
> overflowing is something it should count.

Yes. That needs to be fixed.

-Andi

On Thu, 16 Nov 2006 11:55:46 +0100
Mikael Pettersson <mikpe@...> wrote:

> Andrew Morton writes:
>  > Surely the appropriate behaviour is to allow oprofile to steal the NMI and
>  > to then put the NMI back to doing the watchdog thing after oprofile has
>  > finished with it.
> 
> Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
> the in-kernel API allowing real performance counter drivers like
> oprofile (and perfctr) to claim the HW from the NMI watchdog,
> do their work, and then release it which resumed the watchdog.

OK.  But from Andi's comments it seems that the NMI watchdog was failing to
resume its operation.

> Note that oprofile (and perfctr) didn't do anything behind the
> NMI watchdog's back. They went via the API. Nothing dodgy going on.

Hello,

On Thu, Nov 16, 2006 at 10:34:56AM -0500, William Cohen wrote:
> 
> Is this going to require sharing the nmi interrupt and knowing which perfcounter 
> register triggered the interrupt to get the correct action?  Currently the 
> oprofile interrupt handler assumes any performance monitoring counter it sees 
> overflowing is something it should count.
> 
Yes, you need to share the NMI interrupt. In my next perfmon patch you will
see that this can be made to work. You just need to add one check in the 
NMI handler callback: is it for me or else try perfmon? Perfmon can auto-detect
if NMI is active and give up the right counter (there is an API to check
what is reserved). The interface propagates the list of available counters
to apps which then pass the information onto libpfm which tries to use
the remaining counters.

-- 
-Stephane

Andrew Morton writes:
 > On Thu, 16 Nov 2006 11:55:46 +0100
 > Mikael Pettersson <mikpe@...> wrote:
 > 
 > > Andrew Morton writes:
 > >  > Surely the appropriate behaviour is to allow oprofile to steal the NMI and
 > >  > to then put the NMI back to doing the watchdog thing after oprofile has
 > >  > finished with it.
 > > 
 > > Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
 > > the in-kernel API allowing real performance counter drivers like
 > > oprofile (and perfctr) to claim the HW from the NMI watchdog,
 > > do their work, and then release it which resumed the watchdog.
 > 
 > OK.  But from Andi's comments it seems that the NMI watchdog was failing to
 > resume its operation.

It certainly worked when I originally implemented it. If it didn't work
that way before 2.6.19-rc1 butchered it then that would have been a bug
that should have been fixed.

On Fri, 17 Nov 2006 10:59:07 +0100
Mikael Pettersson <mikpe@...> wrote:

> Andrew Morton writes:
>  > On Thu, 16 Nov 2006 11:55:46 +0100
>  > Mikael Pettersson <mikpe@...> wrote:
>  > 
>  > > Andrew Morton writes:
>  > >  > Surely the appropriate behaviour is to allow oprofile to steal the NMI and
>  > >  > to then put the NMI back to doing the watchdog thing after oprofile has
>  > >  > finished with it.
>  > > 
>  > > Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
>  > > the in-kernel API allowing real performance counter drivers like
>  > > oprofile (and perfctr) to claim the HW from the NMI watchdog,
>  > > do their work, and then release it which resumed the watchdog.
>  > 
>  > OK.  But from Andi's comments it seems that the NMI watchdog was failing to
>  > resume its operation.
> 
> It certainly worked when I originally implemented it. If it didn't work
> that way before 2.6.19-rc1 butchered it then that would have been a bug
> that should have been fixed.

Oh.  OK.

Meanwhile, 2.6.19-rc6 remains unfixed.

On Friday 17 November 2006 10:59, Mikael Pettersson wrote:

> It certainly worked when I originally implemented it. 

I don't think so. NMI watchdog never recovered no matter if oprofile
used the counter or not.

-Andi

Andrew Morton wrote:
> On Fri, 17 Nov 2006 10:59:07 +0100
> Mikael Pettersson <mikpe@...> wrote:
> 
>> Andrew Morton writes:
>>  > On Thu, 16 Nov 2006 11:55:46 +0100
>>  > Mikael Pettersson <mikpe@...> wrote:
>>  > 
>>  > > Andrew Morton writes:
>>  > >  > Surely the appropriate behaviour is to allow oprofile to steal the NMI and
>>  > >  > to then put the NMI back to doing the watchdog thing after oprofile has
>>  > >  > finished with it.
>>  > > 
>>  > > Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
>>  > > the in-kernel API allowing real performance counter drivers like
>>  > > oprofile (and perfctr) to claim the HW from the NMI watchdog,
>>  > > do their work, and then release it which resumed the watchdog.
>>  > 
>>  > OK.  But from Andi's comments it seems that the NMI watchdog was failing to
>>  > resume its operation.
>>
>> It certainly worked when I originally implemented it. If it didn't work
>> that way before 2.6.19-rc1 butchered it then that would have been a bug
>> that should have been fixed.
> 
> Oh.  OK.
> 
> Meanwhile, 2.6.19-rc6 remains unfixed.
> 
Has anyone verified that nmi watchdog works at all in 2.6.19-rc6? I 
haven't built a kernel since rc2, other things have been taking my time.

-- 
Bill Davidsen <davidsen@...>
   Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.

On Wednesday 22 November 2006 11:28, Eric Dumazet wrote:
> On Wednesday 15 November 2006 11:35, Eric Dumazet wrote:
> > On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:
> > > Subject    : x86_64: oprofile doesn't work
> > > References : http://lkml.org/lkml/2006/10/27/3
> > > Submitter  : Prakash Punnoor <prakash@...>
> > > Status     : unknown
> >
> 
> I hit the same problem on i386 architecture too, if CONFIG_ACPI is not set.

oprofile is still broken because it cannot deal with the lack of perfctr 0.
You can disable the nmi watchdog as a workaround.

-Andi

On Wednesday 15 November 2006 11:35, Eric Dumazet wrote:
> On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:
> > Subject    : x86_64: oprofile doesn't work
> > References : http://lkml.org/lkml/2006/10/27/3
> > Submitter  : Prakash Punnoor <prakash@...>
> > Status     : unknown
>

I hit the same problem on i386 architecture too, if CONFIG_ACPI is not set.

# opcontrol --setup --event=RESOURCE_STALLS:1000 --vmlinux=$VMFILE
# opcontrol --start
/usr/bin/opcontrol: line 911: /dev/oprofile/0/enabled: No such file or 
directory
/usr/bin/opcontrol: line 911: /dev/oprofile/0/event: No such file or directory
/usr/bin/opcontrol: line 911: /dev/oprofile/0/count: No such file or directory
/usr/bin/opcontrol: line 911: /dev/oprofile/0/kernel: No such file or 
directory
/usr/bin/opcontrol: line 911: /dev/oprofile/0/user: No such file or directory
/usr/bin/opcontrol: line 911: /dev/oprofile/0/unit_mask: No such file or 
directory
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/oprofiled.log
Daemon started.
Profiler running.

# ls -l /dev/oprofile/
total 0
drwxr-xr-x 1 root root 0 Nov 22 11:18 1
-rw-r--r-- 1 root root 0 Nov 22 11:18 backtrace_depth
-rw-r--r-- 1 root root 0 Nov 22 11:18 buffer
-rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_size
-rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_watershed
-rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_buffer_size
-rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_type
-rw-rw-rw- 1 root root 0 Nov 22 11:18 dump
-rw-r--r-- 1 root root 0 Nov 22 11:18 enable
-rw-r--r-- 1 root root 0 Nov 22 11:18 pointer_size
drwxr-xr-x 1 root root 0 Nov 22 11:18 stats
# dmesg | grep oprofile
oprofile: using NMI interrupt.
# opcontrol --version
opcontrol: oprofile 0.9.2 compiled on Nov 22 2006 11:24:09

Eric

Eric Dumazet wrote:
> On Wednesday 15 November 2006 11:35, Eric Dumazet wrote:
> 
>>On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:
>>
>>>Subject    : x86_64: oprofile doesn't work
>>>References : http://lkml.org/lkml/2006/10/27/3
>>>Submitter  : Prakash Punnoor <prakash@...>
>>>Status     : unknown
>>
> 
> I hit the same problem on i386 architecture too, if CONFIG_ACPI is not set.
> 
> # opcontrol --setup --event=RESOURCE_STALLS:1000 --vmlinux=$VMFILE
> # opcontrol --start
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/enabled: No such file or 
> directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/event: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/count: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/kernel: No such file or 
> directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/user: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/unit_mask: No such file or 
> directory
> Using 2.6+ OProfile kernel interface.
> Reading module info.
> Using log file /var/lib/oprofile/oprofiled.log
> Daemon started.
> Profiler running.
> 
> # ls -l /dev/oprofile/
> total 0
> drwxr-xr-x 1 root root 0 Nov 22 11:18 1
> -rw-r--r-- 1 root root 0 Nov 22 11:18 backtrace_depth
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_size
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_watershed
> -rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_buffer_size
> -rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_type
> -rw-rw-rw- 1 root root 0 Nov 22 11:18 dump
> -rw-r--r-- 1 root root 0 Nov 22 11:18 enable
> -rw-r--r-- 1 root root 0 Nov 22 11:18 pointer_size
> drwxr-xr-x 1 root root 0 Nov 22 11:18 stats
> # dmesg | grep oprofile
> oprofile: using NMI interrupt.
> # opcontrol --version
> opcontrol: oprofile 0.9.2 compiled on Nov 22 2006 11:24:09
> 
> Eric

Could you try the patch that I posted on the oprofile mailing list last week 
November 17 2005 for op_allocate.c and see if that resolves the problem you are 
having?

http://sourceforge.net/mailarchive/message.php?msg_id=37316102

-Will

Eric Dumazet wrote:
> On Wednesday 15 November 2006 11:35, Eric Dumazet wrote:
> 
>>On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:
>>
>>>Subject    : x86_64: oprofile doesn't work
>>>References : http://lkml.org/lkml/2006/10/27/3
>>>Submitter  : Prakash Punnoor <prakash@...>
>>>Status     : unknown
>>
> 
> I hit the same problem on i386 architecture too, if CONFIG_ACPI is not set.
> 
> # opcontrol --setup --event=RESOURCE_STALLS:1000 --vmlinux=$VMFILE
> # opcontrol --start
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/enabled: No such file or 
> directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/event: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/count: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/kernel: No such file or 
> directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/user: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/unit_mask: No such file or 
> directory
> Using 2.6+ OProfile kernel interface.
> Reading module info.
> Using log file /var/lib/oprofile/oprofiled.log
> Daemon started.
> Profiler running.
> 
> # ls -l /dev/oprofile/
> total 0
> drwxr-xr-x 1 root root 0 Nov 22 11:18 1
> -rw-r--r-- 1 root root 0 Nov 22 11:18 backtrace_depth
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_size
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_watershed
> -rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_buffer_size
> -rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_type
> -rw-rw-rw- 1 root root 0 Nov 22 11:18 dump
> -rw-r--r-- 1 root root 0 Nov 22 11:18 enable
> -rw-r--r-- 1 root root 0 Nov 22 11:18 pointer_size
> drwxr-xr-x 1 root root 0 Nov 22 11:18 stats
> # dmesg | grep oprofile
> oprofile: using NMI interrupt.
> # opcontrol --version
> opcontrol: oprofile 0.9.2 compiled on Nov 22 2006 11:24:09
> 
> Eric

You will also need another patch checked into the oprofile cvs last week mentioned:

http://sourceforge.net/mailarchive/message.php?msg_id=35422937

-Will

On Wednesday 22 November 2006 19:05, William Cohen wrote:

> You will also need another patch checked into the oprofile cvs last week
> mentioned:
>
> http://sourceforge.net/mailarchive/message.php?msg_id=35422937
>
> -Will

Thank you William.

I confirm that CVS oprofile version + patches you gave here works with 
linux-2.6.16-rc6 on i386, regardless of disabling nmi_watchdog  (adding or 
not nmi_watchdog=0 in boot params)

Eric

On Wed, 22 Nov 2006 11:36:14 +0100
Andi Kleen <ak@...> wrote:

> On Wednesday 22 November 2006 11:28, Eric Dumazet wrote:
> > On Wednesday 15 November 2006 11:35, Eric Dumazet wrote:
> > > On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:
> > > > Subject    : x86_64: oprofile doesn't work
> > > > References : http://lkml.org/lkml/2006/10/27/3
> > > > Submitter  : Prakash Punnoor <prakash@...>
> > > > Status     : unknown
> > >
> > 
> > I hit the same problem on i386 architecture too, if CONFIG_ACPI is not set.
> 
> oprofile is still broken because it cannot deal with the lack of perfctr 0.

The kernel is still broken because we changed the interface.

> You can disable the nmi watchdog as a workaround.

I don't understand why you think this is acceptable.

On 11/22/06, Andrew Morton <akpm@...> wrote:
> On Wed, 22 Nov 2006 11:36:14 +0100
> Andi Kleen <ak@...> wrote:
>
> > On Wednesday 22 November 2006 11:28, Eric Dumazet wrote:
> > > On Wednesday 15 November 2006 11:35, Eric Dumazet wrote:
> > > > On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:
> > > > > Subject    : x86_64: oprofile doesn't work
> > > > > References : http://lkml.org/lkml/2006/10/27/3
> > > > > Submitter  : Prakash Punnoor <prakash@...>
> > > > > Status     : unknown
> > > >
> > >
> > > I hit the same problem on i386 architecture too, if CONFIG_ACPI is not set.
> >
> > oprofile is still broken because it cannot deal with the lack of perfctr 0.
>
> The kernel is still broken because we changed the interface.

I just got bit by this on 2.6.20-latest (well, of two days ago anyway)
while trying to debug another transient 'kacpid sucks all available
cpu time'. But that's okay, I'm sure it will happen again in a week or
two.

In the meantime, who won this pis^H^H^H discussion?

Mikael Pettersson wrote:
> Andrew Morton writes:
> > Surely the appropriate behaviour is to allow oprofile to steal the NMI and
> > to then put the NMI back to doing the watchdog thing after oprofile has
> > finished with it.
>
> Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
> the in-kernel API allowing real performance counter drivers like
> oprofile (and perfctr) to claim the HW from the NMI watchdog,
> do their work, and then release it which resumed the watchdog.
>
> Note that oprofile (and perfctr) didn't do anything behind the
> NMI watchdog's back. They went via the API. Nothing dodgy going on.

Well, that seems clear.