From: CC Y <yec...@gm...> - 2015-11-05 20:15:52
|
Hi, Looks like ocount can not collect the performance events of PM_L2_LD_MISS on my power8 machine. The information and command is: *cye@...$ ocount-1.2 -e PM_L2_LD_MISS ls* *perf_event_open failed with Invalid argument* *Caught runtime error while setting up counters* *Internal Error. Perf event setup failed.* *Error running ocount* Actually I found all of the events of PM_L2_XXXX does not work. Does this happens to anyone else? Here are some information might useful: *ocount-1.2 --version* *ocount-1.2: oprofile 1.2.0git compiled on Nov 5 2015 15:00:21* *uname -a* *Linux node-ibm-822 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:13 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux* Regards, Chencheng Ye -- Chencheng Ye Cluster and Grid Computing Lab (CGCL) Services Computing Technology and System Lab (SCTS) School of Computer Science and Technology (CS) Huazhong University of Science and Technology (HUST) Wuhan, 430074, China |
From: Carl E. L. <ce...@us...> - 2015-11-05 22:33:58
|
On Thu, 2015-11-05 at 20:15 +0000, CC Y wrote: > Hi, > > > Looks like ocount can not collect the performance events of > PM_L2_LD_MISS on my power8 machine. The information and command is: > cye@...$ ocount-1.2 -e PM_L2_LD_MISS ls > > perf_event_open failed with Invalid argument > > > Caught runtime error while setting up counters > > Internal Error. Perf event setup failed. > > Error running ocount > Chencheng Ye: I took a look at the issue you reported on Power 8. I am seeing the same issue on my lab machine with OProfile 1.1. ophelp lists the PM_L2_LD_MISS event a a legal event. I checked the oprofile config.log file for libpfm. The config file lists: | #define HAVE_LIBPFM3 0 | #define HAVE_LIBPFM 1 I was checking the libpfm library and noticed that the PM_L2_LD_MISS is supported in libpfm 4.6 but not in libpfm 4.5. The libpfm library is important because OProfile uses it to convert the event name to the raw code for the event. The raw code is then passed down to the linux perf interface to do the event counting. I don't remember the details of the libpfm setup for OProfile and what the HAVE_LIBPFM in the config.log file means. I will have to go dig some. Will might remember the specifics. Anyway, my first thought is that oprofile is not using the latest libpfm library??? Not sure at this point. I will have to look some more to see if that might be the issue. Carl Love > > Actually I found all of the events of PM_L2_XXXX does not work. > > Does this happens to anyone else? > > > Here are some information might useful: > > ocount-1.2 --version > > ocount-1.2: oprofile 1.2.0git compiled on Nov 5 2015 15:00:21 > > > uname -a > > Linux node-ibm-822 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 > 23:32:13 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux > > > Regards, > > Chencheng Ye > > > -- > > Chencheng Ye > Cluster and Grid Computing Lab (CGCL) > Services Computing Technology and System Lab (SCTS) > School of Computer Science and Technology (CS) > Huazhong University of Science and Technology (HUST) > Wuhan, 430074, China > ------------------------------------------------------------------------------ > _______________________________________________ > oprofile-list mailing list > opr...@li... > https://lists.sourceforge.net/lists/listinfo/oprofile-list |
From: Carl E. L. <ce...@us...> - 2015-11-06 17:50:19
|
Chencheng Ye: I looked into the issue some more. The libpfm is not an issue. The name to event number lookup is working fine. The PM_L2_LD_MISS event generates the event code 0x426084. You can see the event code ocount is using if you use the option --verbose. It will print something like: ocount --verbose -e PM_L2_LD_MISS ls Final event code is 426084 Number of events passed is 1 Exec args are: ls where the "Final event code" is printed in decimal. The event requires the cache selector bit to be set to be set to 1 in the processor performance counter setup register. On Power 8, this register only writable by the hypervisor. So the kernel must reject any event where the selector bit (bit 22) is not equal to 0. So until they come up with an API so the kernel can issue a request to the hypervisor to write the bit, any event with bit 22 set to a 1 will be rejected. This is an issue for OProfile and perf. OProfile (operf and ocount) are actually using perf under the cover to setup and read the hardware performance counters. The kernel code that does the rejection of the event is actually the perf kernel code for Power 8. If you are really interested, you can see the code in arch/powerpc/perf/power8-pmu.c. There is a comment at about line 77 of the file that describes the register layout. Then lines 327 to 337 is where the code is that actually checks that bit 22 is 1 and rejects the event if the bit is a 1. So, that said, we need to update the list of supported OProfile events to remove these until such time that the kernel is fixed. Carl Love On Thu, 2015-11-05 at 14:33 -0800, Carl E. Love wrote: > On Thu, 2015-11-05 at 20:15 +0000, CC Y wrote: > > Hi, > > > > > > Looks like ocount can not collect the performance events of > > PM_L2_LD_MISS on my power8 machine. The information and command is: > > cye@...$ ocount-1.2 -e PM_L2_LD_MISS ls > > > > perf_event_open failed with Invalid argument > > > > > > Caught runtime error while setting up counters > > > > Internal Error. Perf event setup failed. > > > > Error running ocount > > > > Chencheng Ye: > > I took a look at the issue you reported on Power 8. I am seeing the > same issue on my lab machine with OProfile 1.1. ophelp lists the > PM_L2_LD_MISS event a a legal event. > > I checked the oprofile config.log file for libpfm. The config file > lists: > > | #define HAVE_LIBPFM3 0 > | #define HAVE_LIBPFM 1 > > I was checking the libpfm library and noticed that the PM_L2_LD_MISS is > supported in libpfm 4.6 but not in libpfm 4.5. > > The libpfm library is important because OProfile uses it to convert the > event name to the raw code for the event. The raw code is then passed > down to the linux perf interface to do the event counting. > > I don't remember the details of the libpfm setup for OProfile and what > the HAVE_LIBPFM in the config.log file means. I will have to go dig > some. Will might remember the specifics. Anyway, my first thought is > that oprofile is not using the latest libpfm library??? Not sure at > this point. I will have to look some more to see if that might be the > issue. > > Carl Love > > > > > > > Actually I found all of the events of PM_L2_XXXX does not work. > > > > Does this happens to anyone else? > > > > > > Here are some information might useful: > > > > ocount-1.2 --version > > > > ocount-1.2: oprofile 1.2.0git compiled on Nov 5 2015 15:00:21 > > > > > > uname -a > > > > Linux node-ibm-822 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 > > 23:32:13 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux > > > > > > Regards, > > > > Chencheng Ye > > > > > > -- > > > > Chencheng Ye > > Cluster and Grid Computing Lab (CGCL) > > Services Computing Technology and System Lab (SCTS) > > School of Computer Science and Technology (CS) > > Huazhong University of Science and Technology (HUST) > > Wuhan, 430074, China > > ------------------------------------------------------------------------------ > > _______________________________________________ > > oprofile-list mailing list > > opr...@li... > > https://lists.sourceforge.net/lists/listinfo/oprofile-list > > > > ------------------------------------------------------------------------------ > _______________________________________________ > oprofile-list mailing list > opr...@li... > https://lists.sourceforge.net/lists/listinfo/oprofile-list > |
From: CC Y <yec...@gm...> - 2015-11-06 18:04:45
|
Dear Carl, Thanks for replying this post. Really appreciate! Your explanation is quit specific, now i totally understand the issue. Sincerely, On Fri, Nov 6, 2015 at 12:50 PM Carl E. Love <ce...@us...> wrote: > Chencheng Ye: > > I looked into the issue some more. The libpfm is not an issue. The > name to event number lookup is working fine. The PM_L2_LD_MISS event > generates the event code 0x426084. You can see the event code ocount is > using if you use the option --verbose. It will print something like: > > ocount --verbose -e PM_L2_LD_MISS ls > Final event code is 426084 > Number of events passed is 1 > Exec args are: ls > > where the "Final event code" is printed in decimal. > > The event requires the cache selector bit to be set to be set to 1 in > the processor performance counter setup register. On Power 8, this > register only writable by the hypervisor. So the kernel must reject any > event where the selector bit (bit 22) is not equal to 0. So until they > come up with an API so the kernel can issue a request to the hypervisor > to write the bit, any event with bit 22 set to a 1 will be rejected. > > This is an issue for OProfile and perf. OProfile (operf and ocount) are > actually using perf under the cover to setup and read the hardware > performance counters. The kernel code that does the rejection of the > event is actually the perf kernel code for Power 8. If you are really > interested, you can see the code in arch/powerpc/perf/power8-pmu.c. > > There is a comment at about line 77 of the file that describes the > register layout. Then lines 327 to 337 is where the code is that > actually checks that bit 22 is 1 and rejects the event if the bit is a > 1. > > So, that said, we need to update the list of supported OProfile events > to remove these until such time that the kernel is fixed. > > Carl Love > > On Thu, 2015-11-05 at 14:33 -0800, Carl E. Love wrote: > > On Thu, 2015-11-05 at 20:15 +0000, CC Y wrote: > > > Hi, > > > > > > > > > Looks like ocount can not collect the performance events of > > > PM_L2_LD_MISS on my power8 machine. The information and command is: > > > cye@...$ ocount-1.2 -e PM_L2_LD_MISS ls > > > > > > perf_event_open failed with Invalid argument > > > > > > > > > Caught runtime error while setting up counters > > > > > > Internal Error. Perf event setup failed. > > > > > > Error running ocount > > > > > > > Chencheng Ye: > > > > I took a look at the issue you reported on Power 8. I am seeing the > > same issue on my lab machine with OProfile 1.1. ophelp lists the > > PM_L2_LD_MISS event a a legal event. > > > > I checked the oprofile config.log file for libpfm. The config file > > lists: > > > > | #define HAVE_LIBPFM3 0 > > | #define HAVE_LIBPFM 1 > > > > I was checking the libpfm library and noticed that the PM_L2_LD_MISS is > > supported in libpfm 4.6 but not in libpfm 4.5. > > > > The libpfm library is important because OProfile uses it to convert the > > event name to the raw code for the event. The raw code is then passed > > down to the linux perf interface to do the event counting. > > > > I don't remember the details of the libpfm setup for OProfile and what > > the HAVE_LIBPFM in the config.log file means. I will have to go dig > > some. Will might remember the specifics. Anyway, my first thought is > > that oprofile is not using the latest libpfm library??? Not sure at > > this point. I will have to look some more to see if that might be the > > issue. > > > > Carl Love > > > > > > > > > > > > Actually I found all of the events of PM_L2_XXXX does not work. > > > > > > Does this happens to anyone else? > > > > > > > > > Here are some information might useful: > > > > > > ocount-1.2 --version > > > > > > ocount-1.2: oprofile 1.2.0git compiled on Nov 5 2015 15:00:21 > > > > > > > > > uname -a > > > > > > Linux node-ibm-822 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 > > > 23:32:13 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux > > > > > > > > > Regards, > > > > > > Chencheng Ye > > > > > > > > > -- > > > > > > Chencheng Ye > > > Cluster and Grid Computing Lab (CGCL) > > > Services Computing Technology and System Lab (SCTS) > > > School of Computer Science and Technology (CS) > > > Huazhong University of Science and Technology (HUST) > > > Wuhan, 430074, China > > > > ------------------------------------------------------------------------------ > > > _______________________________________________ > > > oprofile-list mailing list > > > opr...@li... > > > https://lists.sourceforge.net/lists/listinfo/oprofile-list > > > > > > > > > ------------------------------------------------------------------------------ > > _______________________________________________ > > oprofile-list mailing list > > opr...@li... > > https://lists.sourceforge.net/lists/listinfo/oprofile-list > > > > > -- Chencheng Ye Cluster and Grid Computing Lab (CGCL) Services Computing Technology and System Lab (SCTS) School of Computer Science and Technology (CS) Huazhong University of Science and Technology (HUST) Wuhan, 430074, China |
From: William C. <wc...@re...> - 2015-11-07 03:03:17
|
On 11/06/2015 12:50 PM, Carl E. Love wrote: > Chencheng Ye: > > I looked into the issue some more. The libpfm is not an issue. The > name to event number lookup is working fine. The PM_L2_LD_MISS event > generates the event code 0x426084. You can see the event code ocount is > using if you use the option --verbose. It will print something like: > > ocount --verbose -e PM_L2_LD_MISS ls > Final event code is 426084 > Number of events passed is 1 > Exec args are: ls > > where the "Final event code" is printed in decimal. > > The event requires the cache selector bit to be set to be set to 1 in > the processor performance counter setup register. On Power 8, this > register only writable by the hypervisor. So the kernel must reject any > event where the selector bit (bit 22) is not equal to 0. So until they > come up with an API so the kernel can issue a request to the hypervisor > to write the bit, any event with bit 22 set to a 1 will be rejected. > > This is an issue for OProfile and perf. OProfile (operf and ocount) are > actually using perf under the cover to setup and read the hardware > performance counters. The kernel code that does the rejection of the > event is actually the perf kernel code for Power 8. If you are really > interested, you can see the code in arch/powerpc/perf/power8-pmu.c. > > There is a comment at about line 77 of the file that describes the > register layout. Then lines 327 to 337 is where the code is that > actually checks that bit 22 is 1 and rejects the event if the bit is a > 1. > > So, that said, we need to update the list of supported OProfile events > to remove these until such time that the kernel is fixed. > > Carl Love Hi Carl, Given that virtualization has been supported a long time on power it is a bit surprising that some of the performance monitoring events are not selectable for the guest vm. Is there something that describes the reason for this limitation besides the comment in the kernel source code? If you have a patch to disable the problem power8 events, I will merge it into the upstream oprofile. -Will > > On Thu, 2015-11-05 at 14:33 -0800, Carl E. Love wrote: >> On Thu, 2015-11-05 at 20:15 +0000, CC Y wrote: >>> Hi, >>> >>> >>> Looks like ocount can not collect the performance events of >>> PM_L2_LD_MISS on my power8 machine. The information and command is: >>> cye@...$ ocount-1.2 -e PM_L2_LD_MISS ls >>> >>> perf_event_open failed with Invalid argument >>> >>> >>> Caught runtime error while setting up counters >>> >>> Internal Error. Perf event setup failed. >>> >>> Error running ocount >>> >> >> Chencheng Ye: >> >> I took a look at the issue you reported on Power 8. I am seeing the >> same issue on my lab machine with OProfile 1.1. ophelp lists the >> PM_L2_LD_MISS event a a legal event. >> >> I checked the oprofile config.log file for libpfm. The config file >> lists: >> >> | #define HAVE_LIBPFM3 0 >> | #define HAVE_LIBPFM 1 >> >> I was checking the libpfm library and noticed that the PM_L2_LD_MISS is >> supported in libpfm 4.6 but not in libpfm 4.5. >> >> The libpfm library is important because OProfile uses it to convert the >> event name to the raw code for the event. The raw code is then passed >> down to the linux perf interface to do the event counting. >> >> I don't remember the details of the libpfm setup for OProfile and what >> the HAVE_LIBPFM in the config.log file means. I will have to go dig >> some. Will might remember the specifics. Anyway, my first thought is >> that oprofile is not using the latest libpfm library??? Not sure at >> this point. I will have to look some more to see if that might be the >> issue. >> >> Carl Love >> >> >> >>> >>> Actually I found all of the events of PM_L2_XXXX does not work. >>> >>> Does this happens to anyone else? >>> >>> >>> Here are some information might useful: >>> >>> ocount-1.2 --version >>> >>> ocount-1.2: oprofile 1.2.0git compiled on Nov 5 2015 15:00:21 >>> >>> >>> uname -a >>> >>> Linux node-ibm-822 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 >>> 23:32:13 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux >>> >>> >>> Regards, >>> >>> Chencheng Ye >>> >>> >>> -- >>> >>> Chencheng Ye >>> Cluster and Grid Computing Lab (CGCL) >>> Services Computing Technology and System Lab (SCTS) >>> School of Computer Science and Technology (CS) >>> Huazhong University of Science and Technology (HUST) >>> Wuhan, 430074, China >>> ------------------------------------------------------------------------------ >>> _______________________________________________ >>> oprofile-list mailing list >>> opr...@li... >>> https://lists.sourceforge.net/lists/listinfo/oprofile-list >> >> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> oprofile-list mailing list >> opr...@li... >> https://lists.sourceforge.net/lists/listinfo/oprofile-list >> > > > > ------------------------------------------------------------------------------ > _______________________________________________ > oprofile-list mailing list > opr...@li... > https://lists.sourceforge.net/lists/listinfo/oprofile-list > |
From: Michael P. <mp...@re...> - 2015-11-16 18:20:54
|
Hi Carl, just a question... What does 'hypervisor' exactly mean here? Do you mean a PowerKVM host or a PowerVM one? Or does it apply to both? And finally, how does that work on a POWER8 machine that runs without virtualization? Are the events still not accessible there? Thanks, Michael On Fri, 2015-11-06 at 09:50 -0800, Carl E. Love wrote: > Chencheng Ye: > > I looked into the issue some more. The libpfm is not an issue. The > name to event number lookup is working fine. The PM_L2_LD_MISS event > generates the event code 0x426084. You can see the event code ocount is > using if you use the option --verbose. It will print something like: > > ocount --verbose -e PM_L2_LD_MISS ls > Final event code is 426084 > Number of events passed is 1 > Exec args are: ls > > where the "Final event code" is printed in decimal. > > The event requires the cache selector bit to be set to be set to 1 in > the processor performance counter setup register. On Power 8, this > register only writable by the hypervisor. So the kernel must reject any > event where the selector bit (bit 22) is not equal to 0. So until they > come up with an API so the kernel can issue a request to the hypervisor > to write the bit, any event with bit 22 set to a 1 will be rejected. > > This is an issue for OProfile and perf. OProfile (operf and ocount) are > actually using perf under the cover to setup and read the hardware > performance counters. The kernel code that does the rejection of the > event is actually the perf kernel code for Power 8. If you are really > interested, you can see the code in arch/powerpc/perf/power8-pmu.c. > > There is a comment at about line 77 of the file that describes the > register layout. Then lines 327 to 337 is where the code is that > actually checks that bit 22 is 1 and rejects the event if the bit is a > 1. > > So, that said, we need to update the list of supported OProfile events > to remove these until such time that the kernel is fixed. > > Carl Love > |