After consulting with Milena, I made a small change to the original patch that fixed this hang problem. I've tested the new patch, and attached it to this posting.
-------- Original Message --------
Subject: [PATCH] Fix intermittent itrace hang on ppc64
Date: Wed, 25 Nov 2009 13:12:00 -0600
From: Maynard Johnson <maynardj@...>
To: perfinsp-list@..., Milena Milenkovic <mmilenko@...>
As I've mentioned to you before, a bug was reported by the IBM test team against the ITrace package that ships with SLES 11. When running our ITrace testsuite in a loop, they occasionally would encounter a system hang symptom. Because of the difficulty of reproducing the problem and the late stage of the SLES 11 release cycle in which the problem was found, we were unable to fix this problem for SLES 11. We were, however, able to provide users a workaround for the problem. I've debugged this problem now and have attached a patch to fix this. Please commit as soon as possible so I can push this fix into SLES.
Details of problem
On the ppc64 platform, we have a need to avoid tracing through reservations (lwarx or ldarx). For single step tracing, the crude mechanism we have to prevent this is, once we detect such a reservation instruction, we suspend tracing and set up the performance monitor to count some number of instructions completed. The number of instructions is chosen to give ample time for the reservation to have completed. After the performance monitor interrupt, we resume normal tracing. An interesting fact about the "hang" problem was that, if you waited long enough (say, a couple hours), the system would eventually break out of the hang. So I knew it was not a deadlock that was causing the problem. The problem involved our use of the performance monitor. If the performance monitor is enabled and counting at the time of calling ITrace_off, the performance monitor interrupt occurs at some point during the ITrace_off processing, resulting in running our perfmon interrupt handler, ITr
ace_irq(). The ITrace_irq function does some special processing when it sees that tracing is being turned off, but instead of continuing on to disable performance monitor interrupts, it does an early return. The result was that the PMAO bit of the perfmon control register MMCR0 stayed set to '1', causing new perfmon interrupts to be delivered -- so very little real work got done between interrupts. Eventually, though, we got through the ITrace_off and the userspace testcase called ITrace_remove, which unregisters the performance monitor and replaces our perfmon interrupt handler with a system default handler, which properly sets PMAO to '0'.