Thanks for your help again Maynard,

You were right -- i actually didn't install the headers, but rather the kernel-devel package. I can try again to see if the operf works now with the base kernel, but in the meantime, i pulled the latest from GIT and applied the patch. Everything seems to be in order:

./configure --with-kernel=/home/username/linux-3.2.32/usr 
make
sudo make install

Then i run:

operf -s --vmlinux=/home/username/linux-3.2.32/arch/x86/boot/compressed/vmlinux 

When i run it for 10 minutes or so, i get a "stats" folder created in:

oprofile_data/samples/current

The stats folder is completely empty. In the operf.log file, here's what it says:

-- OProfile/operf Statistics --
Nr. non-backtrace samples: 0
Nr. kernel samples: 0
Nr. user space samples: 0
Nr. samples lost due to sample address not in expected range for domain: 0
Nr. lost kernel samples: 0
Nr. samples lost due to sample file open failure: 0
Nr. samples lost due to no permanent mapping: 0
Nr. user context kernel samples lost due to no app info available: 0
Nr. user samples lost due to no app info available: 0
Nr. backtraces skipped due to no file mapping: 0
Nr. hypervisor samples dropped due to address out-of-range: 0
Nr. samples lost reported by perf_events kernel: 0

Prior to doing a manual compile on oprofile, i did a yum install oprofile and it installed a 0.9.7 version. When i ran the opcontrol and opreport commands, even if i only ran opcontrol for only a few minutes, i would get tons and tons of records. Perhaps i'm not configuring it correctly to record the right things? Thanks again for your help..

-Bow-Nan


On Fri, Apr 5, 2013 at 4:54 PM, Maynard Johnson <maynardj@us.ibm.com> wrote:
On 04/05/2013 02:47 PM, Bow-Nan Cheng wrote:
> Hello Maynard,
>
> Thank you again for all your help, to answer your questions below:
>
> 1. Kernel headers -- i have kernel headers installed for the kernel version i'm running:
> yum install kernel-headers
> uname -a yields 2.6.32-358.2.1.el6.x86_64 and /usr/src/kernels/2.6.32-358.2.1.el6.x86_64/ has the headers
>
> 2. strace -- i ran the command you suggested and the number after syscall (298) matches what's in the one in the kernel headers:
>
> From strace ===
> syscall_298(0x7fff3530a0f0, 0x5b77, 0, 0xffffffff, 0, 0x4800000000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 2)
> write(2, "Your kernel's Performance Events"..., 59Your kernel's Performance Events Subsystem does not support) = 59
>
> From greping for __NR_perf_event_open in the /usr/src/kernels/2.6.32-358.2.1.el6.x86_64/ folder ===
>
> arch/x86/include/asm/unistd_64.h:#define __NR_perf_event_open298
>
> 3. Pathname of asm/unistd in -save-temps:
>
> # 90 "../libperf_events/operf_utils.h"
> # 1 "/usr/include/asm/unistd.h" 1 3 4
>
>
> # 1 "/usr/include/asm/unistd_64.h" 1 3 4
> # 16 "/usr/include/asm/unistd_64.h" 3 4
>
> Does this mean there's a discrepancy? Thanks again for your help
Does 'rpm -qa | grep kernel-headers' show the kernel headers package is installed?  I don't know about CentOS, but for RHEL, /usr/src/kernels/`uname -r`/include is the typical location where header files for the kernel-devel package are installed.  The kernel-headers (which is for user space program development) are typically installed in /usr/include.  Nevertheless, the strace output looks like it's calling the correct syscall for perf_event_open, but the errno returned (2) is ENOENT, which implies (as operf's message says) that the kernel does not support this processor. But Vince Weaver's website that you pointed to in your earlier posting seems to refute that.  Can you install the 'perf' package (if not already installed) and try something simple like 'perf record ls'.  Does that work?  If it doesn't, what's the error?

Can you try again to use the 3.2 kernel that you initially wanted to use, but first, apply the oprofile patch I mentioned in my initial response.  Unfortunately, that patch does not apply cleanly to the official 0.9.8 release, so pull down the latest code
from our shared git repo (git clone git://oprofile.git.sourceforge.net/gitroot/oprofile/oprofile).  After applying that oprofile patch, do the following (from the root oprofile dir):
        ./autogen.sh
        ./configure --help
        Read the help text for the '--with-kernel' option (there's some new help information there, added by my patch)
        Follow the instructions from the help text to properly install your custom kernel's header files (I recommend
        installing into some temporary location under your home dir versus installing into /usr).
        Now run oprofile's configure/make/make install.
        Test operf.

B
-Maynard

>
> -Bow-Nan
>
>
>
>
> On Fri, Apr 5, 2013 at 3:04 PM, Maynard Johnson <maynardj@us.ibm.com <mailto:maynardj@us.ibm.com>> wrote:
>
>     On 04/05/2013 01:27 PM, Bow-Nan Cheng wrote:
>     > Hello,
>     >
>     > Thanks for your quick response!
>     >
>     > I rebooted into the latest stock kernel -- 2.6.32-358.2.1.el6.x86_64
>     >
>     > Recompiled oprofile:
>     > make clean
>     > ./configure
>     > make
>     > sudo make install
>     >
>     > Then tried to re-run operf, and it gave me this error:
>     > Your kernel's Performance Events Subsystem does not support your processor type. Please use the opcontrol command instead of operf.
>     >
>     > cat /proc/cpuinfo says my processor is:
>     > model: 23
>     > model name: Intel(R) Core(TM)2 Duo CPU     T9900  @ 3.06GHz
>     >
>     > Which.. according to this site: http://web.eece.maine.edu/~vweaver/projects/perf_events/support.html should support the perf_event.
>     >
>     > When i run:
>     >
>     > /usr/local/bin/opcontrol --start
>     > It seems to update the /var/lib/oprofile/samples folder once, but then it doesn't touch it again. The result is that "opreport -l" gives the same error about being too strict. Perhaps i'm doing something wrong in my compile or setup?
>     >
>     > Thanks for your help again!
>     Do you have the CentOS 6.3 kernel headers package installed -- i.e., with version that matches your running kernel?
>
>     Try 'strace operf --help > out 2>&1' and search for a syscall. You should see a number associated with that syscall entry in the strace output which should match the number for the '#define __NR_perf_event_open' in unistd.h (will need to grep /usr/include for '__NR_perf_event_open' to find the actual unistd.h file that has this #define).
>
>     It may also be instructive to cd into oprofile/pe_profiling and do the following:
>             'touch operf.cpp'
>             'make'
>             Copy the g++ command onto the command line, add '-save-temps', then run 'make' again.
>             Look at the operf.ii file that is generated by -save-temps option and look for "asm/unistd".  What's the pathname?
>
>     -Maynard
>
>
>     >
>     >
>     >
>     >
>     > On Fri, Apr 5, 2013 at 1:48 PM, Maynard Johnson <maynardj@us.ibm.com <mailto:maynardj@us.ibm.com> <mailto:maynardj@us.ibm.com <mailto:maynardj@us.ibm.com>>> wrote:
>     >
>     >     On 04/05/2013 12:22 PM, Bow-Nan Cheng wrote:
>     >     > Hello,
>     >     >
>     >     > I'm having a bit of difficulty getting oprofile to correctly profile my custom kernel module (or even profile the current kernel in general). I'm running CentOS 6.3 with a custom 3.2 kernel. Here's the process i used to install oprofile-0.9.8:
>     >     >
>     >     > ./configure --with-kernel=/home/username/linux-3.2.32
>     >     > make
>     >     > sudo make install
>     >     >
>     >     > Then i run operf to profile the entire system:
>     >     >
>     >     > sudo /usr/local/bin/operf -s
>     >     >
>     >     > However, when i check the oprofile_data/current folder, it's empty and the "opreport" command throws a "no samples found" error
>     >     >
>     >     > Any suggestions? I'm sure its just something small..
>     >     The first thing I would suggest is to boot back to your stock CentOS 6.3 kernel and build oprofile without the --with-kernel configure option. I'm quite certain that 6.3 uses a kernel version that supports operf, so as long as you have the CentOS kernel headers package installed, oprofile (with operf) will build just fine.  So try that, and see how operf runs.  Try operf in both single app mode and system-wide mode (see operf man page -- and since you're building it yourself, you'll need to do: 'export MANPATH=<oprof-install-dir>/share/man:$MANPATH').
>     >
>     >     If operf built with the stock CentOS kernel works OK, then you may be running into the bug with the '--with-kernel' option that another user recently encountered.  If so, apply my patch that I posted to the list on Wednesday of this week -- subject "[PATCH] Fix broken --with-kernel configure option".
>     >
>     >     Good luck!
>     >
>     >     -Maynard
>     >     >
>     >     >
>     >     > ------------------------------------------------------------------------------
>     >     > Minimize network downtime and maximize team effectiveness.
>     >     > Reduce network management and security costs.Learn how to hire
>     >     > the most talented Cisco Certified professionals. Visit the
>     >     > Employer Resources Portal
>     >     > http://www.cisco.com/web/learning/employer_resources/index.html
>     >     >
>     >     >
>     >     >
>     >     > _______________________________________________
>     >     > oprofile-list mailing list
>     >     > oprofile-list@lists.sourceforge.net <mailto:oprofile-list@lists.sourceforge.net> <mailto:oprofile-list@lists.sourceforge.net <mailto:oprofile-list@lists.sourceforge.net>>
>     >     > https://lists.sourceforge.net/lists/listinfo/oprofile-list
>     >     >
>     >
>     >
>
>