From: Maynard P. J. <may...@us...> - 2005-01-21 21:39:11
Attachments:
oprof-docpatch-for-ppc64.txt
|
John, When I first sent this doc patch for the new ppc64 support in OProfile, the feedback I got was understandably not real positive. I've pressed people here to the best of my ability to be able to release more information, but it's just going to happen real soon. I don't know what else I can do at this point. I really hope that you and the other committers can accept that this is only a starting point -- not the final version. As more information is made public, I'll be able to update the OProfile doc. I don't know when 0.8.2 will be released, but I would hope the ppc64 support to be included in that release could be documented, even if that documentation isn't much more than indicating to the user that the platform support is now present. If there are some specific suggestions that you or others have that might make this doc patch more palatable, I'm certainly willing to work on it some more. The original patch I posted with the October 15 note no longer applies to OProfile CVS. I will re-do the patch with current CVS once we can come to an agreement on what should be contained in it. Thanks. -Maynard ----- Original Message ----- From: "Maynard P. Johnson" <may...@us...> To: <opr...@li...> Sent: Friday, October 15, 2004 3:23 PM Subject: doc patch for PowerPC 64-bit processor support > Attached is the doc patch I've been promising that documents the > addition of the Power4, Power5, and PPC970 support to OProfile. > > Thanks. > -- > Maynard Johnson > ---------------------------------------------------------------------------- ---- > diff -paurNX ../diff_fileExclusionFilter oprofile/ChangeLog oprof2/ChangeLog > --- oprofile/ChangeLog 2004-10-15 13:40:35.000000000 -0500 > +++ oprof2/ChangeLog 2004-10-15 16:10:45.511935408 -0500 > @@ -1,3 +1,6 @@ > +2004-10-15 Maynard Johnson <may...@us...> > + * doc/oprofile.xml: Add PowerPC 64-bit processor support information > + > 2004-10-15 Will Cohen <wc...@re...> > > * events/ppc64/power4/events: Corrected group 4 counter assignments. > diff -paurNX ../diff_fileExclusionFilter oprofile/doc/oprofile.xml oprof2/doc/oprofile.xml > --- oprofile/doc/oprofile.xml 2004-08-20 08:14:49.000000000 -0500 > +++ oprof2/doc/oprofile.xml 2004-10-15 16:08:03.853016576 -0500 > @@ -84,6 +84,10 @@ OProfile is not a panacea. OProfile migh > with the line <constant>EXPORT_SYMBOL(do_fork);</constant> present in <filename>kernel/ksyms.c</filename>. > Such a kernel is present in the <ulink url="http://www.x86-64.org">x86-64.org</ulink> CVS repository. > 2.6 kernels are supported with the in-kernel OProfile driver. > + </para> > + <para> > + PPC64 processors (Power4/Power5/PPC970) require a recent (> 2.6.5) kernel with the line > + <constant>#define PV_970</constant> present in <filename>include/asm-ppc64/processor.h</filename>. > <!-- FIXME: do we require always gte 2.4.10 for nosmp ? --> > </para></listitem> > </varlistentry> > @@ -748,6 +752,10 @@ hardware counters as necessary. Note tha > allowed by the CPU; running <command>opcontrol --list-events</command> gives the details > of each event. The event specification is a colon-separated string > of the form <option><emphasis>name</emphasis>:<emphasis>count</emphasis>:<emphasis>unitm ask</emphasis>:<emphasis>kernel</emphasis>:<emphasis>user</emphasis></option > as described in this table: > +<note><para> > +For the PowerPC platforms, all events specified must be in the same group; i.e., the group number > +appended to the event name (e.g. <constant><<emphasis>some-event-name</emphasis>>_GRP9</constant>) must be the same. > +</para></note> > </para> > <informaltable frame="all"> > <tgroup cols='2'> > @@ -797,8 +805,8 @@ The table below lists the events selecte > <row><entry>Itanium</entry><entry>ia64/itanium</entry><entry>CPU_CYCLES:1000 00:0:1:1</entry></row> > <row><entry>Itanium 2</entry><entry>ia64/itanium2</entry><entry>CPU_CYCLES:100000:0:1:1</entry>< /row> > <row><entry>TIMER_INT</entry><entry>timer</entry><entry>None selectable</entry></row> > -<row><entry>IBM iseries</entry><entry>timer</entry><entry>None selectable</entry></row> > -<row><entry>IBM pseries</entry><entry>timer</entry><entry>None selectable</entry></row> > +<row><entry>IBM iseries</entry><entry>PowerPC 4/5/970</entry><entry>CYCLES:10000:0:1:1</entry></row> > +<row><entry>IBM pseries</entry><entry>PowerPC 4/5/970</entry><entry>CYCLES:10000:0:1:1</entry></row> > <row><entry>IBM s390</entry><entry>timer</entry><entry>None selectable</entry></row> > <row><entry>IBM s390x</entry><entry>timer</entry><entry>None selectable</entry></row> > </tbody> > @@ -849,10 +857,11 @@ events other than the default event chos > </para> > </note> > <para> > -The hardware performance counters are detailed in the Intel IA-32 Architecture Manual, Volume 3, available > +The Intel hardware performance counters are detailed in the Intel IA-32 Architecture Manual, Volume 3, available > from <ulink url="http://developer.intel.com/">http://developer.intel.com/</ulink>. The AMD Athlon/Duron > implementation is detailed in <ulink url="http://www.amd.com/products/cpg/athlon/techdocs/pdf/22007.pdf"> > -http://www.amd.com/products/cpg/athlon/techdocs/pdf/22007.pdf</ulink>. > +http://www.amd.com/products/cpg/athlon/techdocs/pdf/22007.pdf</ulink>. For PowerPC64 processors in > +IBM iSeries and pSeries systems, contact IBM for more information. > These processors are capable of delivering an interrupt when a counter overflows. > This is the basic mechanism on which OProfile is based. The delivery mode is <acronym>NMI</acronym>, > so blocking interrupts in the kernel does not prevent profiling. When the interrupt handler is called, > @@ -979,6 +988,17 @@ existing interrupt-based model to the PM > </para> > </sect2> > > +<sect2 id="ppc64"> > +<title>PowerPC64 support</title> > +<para> > +The performance monitoring unit (PMU) for the PowerPC 64-bit processors > +consists of between 6 and 8 counters (depending on the model), plus three > +special purpose registers used for programming the counters -- MMCR0, MMCR1, > +and MMCRA. Advanced features such as instruction matching and thresholding are > +not supported by this version of OProfile. > +</para> > +</sect2> > + > <sect2 id="misuse"> > <title>Dangerous counter settings</title> > <para> > |
From: Maynard P. J. <may...@us...> - 2005-01-24 13:57:34
|
Maynard P. Johnson wrote: > John, > When I first sent this doc patch for the new ppc64 support in OProfile, the > feedback I got was understandably not real positive. I've pressed people > here to the best of my ability to be able to release more information, but > it's just going to happen real soon. hmmm . . . sorry, I meant to say "it's just NOT going to happen real soon." I don't know what else I can do at > this point. > > I really hope that you and the other committers can accept that this is only > a starting point -- not the final version. As more information is made > public, I'll be able to update the OProfile doc. I don't know when 0.8.2 > will be released, but I would hope the ppc64 support to be included in that > release could be documented, even if that documentation isn't much more than > indicating to the user that the platform support is now present. If there > are some specific suggestions that you or others have that might make this > doc patch more palatable, I'm certainly willing to work on it some more. > > The original patch I posted with the October 15 note no longer applies to > OProfile CVS. I will re-do the patch with current CVS once we can come to > an agreement on what should be contained in it. > > Thanks. > > -Maynard -- Maynard Johnson |
From: Maynard P. J. <may...@us...> - 2005-01-26 15:03:31
|
Thanks, that's good news. I hadn't seen a confirmation email that the patch had been applied, but I may have missed it. And when I recently updated from OProfile CVS and didn't see my ChangeLog entry for my doc patch, I assumed it hadn't been applied yet. I see that the controversial part of my patch has been removed (i.e., telling users to "contact IBM for more information"). I'm fine with that decision. As I said, as documentation becomes publicly available, I'll update that section. ----- Original Message ----- From: "Philippe Elie" <ph...@wa...> To: "Maynard P. Johnson" <may...@us...> Cc: "John Levon" <le...@mo...>; <opr...@li...> Sent: Wednesday, January 26, 2005 12:52 AM Subject: Re: doc patch for PowerPC 64-bit processor support > On Mon, 24 Jan 2005 at 07:57 +0000, Maynard P. Johnson wrote: > > > Maynard P. Johnson wrote: > > >John, > > >When I first sent this doc patch for the new ppc64 support in OProfile, the > > >feedback I got was understandably not real positive. I've pressed people > > >here to the best of my ability to be able to release more information, but > > >it's just going to happen real soon. > > hmmm . . . sorry, I meant to say "it's just NOT going to happen real soon." > > I don't know what else I can do at > > >this point. > > I applied this patch > > ~~~~ > > > |
From: William C. <wc...@re...> - 2005-02-15 18:21:44
|
Maynard P. Johnson wrote: > Thanks, that's good news. I hadn't seen a confirmation email that the patch > had been applied, but I may have missed it. And when I recently updated > from OProfile CVS and didn't see my ChangeLog entry for my doc patch, I > assumed it hadn't been applied yet. I see that the controversial part of my > patch has been removed (i.e., telling users to "contact IBM for more > information"). I'm fine with that decision. As I said, as documentation > becomes publicly available, I'll update that section. I happen to see Maynard at a workshop on Performance monitoring hardware this past weekend and he mentioned that IBM has released more information about the performance monitoring hardware. The "IBM PowerPC 970FX RISC Microprocessor User's Manual" has a chapter describing the performance monitoring hardware. http://www-306.ibm.com/chips/techlib/techlib.nsf/techdocs/AE818B5D1DBB02EC87256DDE00007821 -Will |
From: John L. <le...@mo...> - 2005-02-15 18:36:04
|
On Tue, Feb 15, 2005 at 01:21:27PM -0500, William Cohen wrote: > I happen to see Maynard at a workshop on Performance monitoring hardware > this past weekend and he mentioned that IBM has released more Hi Will, any chance of a write-up on the workshop for us? john |
From: William C. <wc...@re...> - 2005-02-17 15:38:45
|
John Levon wrote: > On Tue, Feb 15, 2005 at 01:21:27PM -0500, William Cohen wrote: > > >>I happen to see Maynard at a workshop on Performance monitoring hardware >>this past weekend and he mentioned that IBM has released more > > > Hi Will, any chance of a write-up on the workshop for us? > > john Sunday February 13 I attended the Hardware Performance Monitor Design and Functionality workshop in San Francisco. http://www.c3.lanl.gov/~mlang/perf-count-ws.html The goal of the workshop was to discuss issues related to performance monitoring. A summary of paper should be coming out of the discussions. There should be postings of the presentations at (but it may take a little while be cause people HPCA11 runs through most of the week): http://lacsi.rice.edu/workshops/hpca11/ There appeared to be about 50 people at the workshop. Designers of performance monitoring hardware in the ia64, p6, p4, ibm power processors were there. People that work on software that use the performance monitoring hardware, e.g. OProfile, PAPI, and Perfmon. There were also a number of people there from the national laboratories which often have very computationally large computations. Didn't see many people from the Independent Software Vendors (ISVs) that should be interested in getting better insight into how to tune their applications. There were some common themes: 1) Monitoring memory system is key. Dynamic memory allocation and data layout have a huge impact on memery system performance. 2) Need to make it easier to use the appropriate events to find common performance. Most performance monitoring hardware has hundreds of possible events to measure and users are getting lost in the vast number of choices. 3) Often events are put in by the HW designer benefit to test something out. A side effect of this is that the events may not be something directly mappable to the user code (e.g. speculative execution of instructions). Perfmon hardware often low on the priorities and not usually thought of in the same manner as the typical user visible features in the processor, e.g. instruction set extensions. 4) Performance monitoring need to provide insight into behavior of system (hardware/software). The information needs to be mappable back to the program the developer is working on. For example counters counting cache misses is often not enough. Often the problem is due to the layout of data structures and the same code may be used to traverse a number of data structures. Some data structures fit in cache fine and others are layed out poorly in memory. More detailed comments about some of the talks are below. --------------- Rob Fowler from Rice University talked about experiences with performance monitoring tools. Their tools (http://www.hipersoft.rice.edu/) are being used by Los Alamos. Stated that "Existing tools didn't significantly help productivity" and "TLB issues not widely appreciated". Memory performance is often the limiting factor on large applications. Allocating and reallocation memory over time can lay out objects so that TLB and cache performance suffers, e.g. stride through memory touching every 2,000th object. Philip Mucci had a similar observation about an application running on a Opteron later in the day. Would like better information about cache conflicts. Setting a flag and recompiling the application may not be feasible because the build take a day or more. Given the size of the applications some of the instrumenation process needs to be automated. Preferred profiling to calipers. People are okay with 1% or 2% overhead with the instrumentation. Fowler found the number of performance counters available in the processor to be smaller than what was desired and had to resort to multiple runs to collect all the pieces of data. Complained that the constrains on the Pentium 4 hardware limited the number of events. Fowler would like information on OS issues like Locks contention. Rice is porting the HPCToolkit to use oprofile. Yet another application that could make use to a formal oprofile sample database library. He also mentioned the issue with OProfile requiring root and storing things in /var/lib/oprofile. Normal users don't have access to that. Having the ability for normal users to do the sampling would be a good thing. --------------- Martin Schultz from LLNL discussed next generation monitoring system. This is collaboration between LLNL, Cornell, and Georgia Tech. Want introspection and to automate the seach. They want to look at things at a higher level than just counting number of times events happen, e.g. memory access patterns, and how network devices are used. However, they are talking about using hardware such as FPGA to aggregate the data. I am not wild about throwing more hardware at the problem because this is going to exclude most regular systems. There should be more detailed specification on the performance monitoring system coming out. Additional information on it at: http://owl.csl.cornell.edu/ The work sounded interesting and wa trying to extend the performance monitoring hardware outside the CPU. ------------------ Bill Brantley from AMD during his talk about NUMA performance had an entire slide about virtualization. He felt that virtualization is going to be a big issue for future performance monitoring. I guess we have seen a little of that with HT Pentium 4. Bill Brantley was talking more along the line of Xen. Should looking into running performance monitoring tools (e.g. OProfile) on Xen. ------------------ Stephane Eranian from HP (developer of Perfmon) pitched new interface to try to unify differences between performance monitoring hardware in different architectures. Difference between different PMU interfaces impeding tool developmen. Interface based on the linux interface for perfmon on ia64. http://www.hpl.hp.com/techreports/2004/HPL-2004-200R1.html ---- Performance monitoring hardware designer talk about ia64, power pc, and pentium 4. A number of people not satisfied with the pentium 4 sharing of performance monitoring hardware. Also some people not satisified with the 32-bit or different sizes of performance monitoring counters (need to know size to set properly for overflow based sampling). Hardware designers pointed out some of the issues with hardware design of performance monitoring hardware. Usually low on the list of features needed and very little chip area allocated (sometimes bundle with debugging hardware). The instrumentation points are all over the chip and wires are expensive. Arbitrary selection of events for each perf counter expensive area wise (cross-bar switch), so solutions like IBM power5 selections of certain event sets or Intel Pentium 4 where certain counters can count only certain events end up being used. The events that the programmer may be interested in may not be available at the chip level or adding a probe for that might increase delay on a critical path. What the processor does and what the programmer thinks the processor does can be very different things due to the deep pipelines and speculative execution. Thus, some events that people are interested in are not available. Also things something get lost in translation between architect and circuit implementor. Thus, things like counting number of floating point instructions entering the processor pipeline rather than the number of floating point instructions retired (speculative ones vs. real). The hardware developers often put in undocumented events to verify the hardware is doing what they expect. Having to support the performance monitoring events for general users has a cost associated (writing documentation and testing). However, it seems like IBM is getting more willing to release information. Maynard Johnson of IBM mentioned hat IBM has finally released a manual with information about the performance monitoring hardware in the power 970 chip. Raw addresses in PEBS are not particularly useful in systems such as Linux because the linear addresses are not mapped back to code. Talking with Stephane Eranian the mechanism that OProfile uses to map samples back to files and offsets could probably be used to address that situation. -Will |
From: Philippe E. <ph...@wa...> - 2005-01-26 15:00:57
|
On Mon, 24 Jan 2005 at 07:57 +0000, Maynard P. Johnson wrote: >> I really hope that you and the other committers can accept that this is >> only a starting point -- not the final version. As more information >> is made public I commited this patch -- phe |