OProfile

Hello,

I'm unsure how oprofile behaves when the location of the EIP during
kernel monitoring falls in an "unusual" VMA. By unsual I mean one whose
address is not found neither in the list of absolute memory addresses
avaliable in the vmlinux ELF file, nor from the memory addresses
allocated when a module is insmodded. That would be memory that is
manually kmalloc'ed and code placed in it is executed.

I'm using oprofile and I expect to see some EIP hits inside such unusual
addresses. 'opreport -d' does not list any of those. Is this simply
because opreport does not have that symbol information, in which case I
need to find a way to make that available to opreport ? Are there any
data that opreport discards when showing a report, and if so a way to
still report that ?

Thanks,
Kristis

Re: opreport-ing an EIP in a VMA that does not match a symbol From: William Cohen <wcohen@nc...> - 2004-11-05 21:56 Kristis Makris wrote: > Hello, > > I'm unsure how oprofile behaves when the location of the EIP during > kernel monitoring falls in an "unusual" VMA. By unsual I mean one whose > address is not found neither in the list of absolute memory addresses > avaliable in the vmlinux ELF file, nor from the memory addresses > allocated when a module is insmodded. That would be memory that is > manually kmalloc'ed and code placed in it is executed. > > I'm using oprofile and I expect to see some EIP hits inside such unusual > addresses. 'opreport -d' does not list any of those. Is this simply > because opreport does not have that symbol information, in which case I > need to find a way to make that available to opreport ? Are there any > data that opreport discards when showing a report, and if so a way to > still report that ? > > Thanks, > Kristis For user-space sample if they don't fall into something that maps to a file they are discarded in the oprofile kernel driver and don't even make it to the daemon. You should be able to get the number discarded in /dev/oprofile/status/samples_lost_no_mapping. Thus, for things like JITed code, you will not see any samples. For a kernel/module space sample the sample will make it to the daemon. However, the daemon will not find a suitable mapping and discard the sample. -Will

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: Kristis Makris <kristis.makris@as...> - 2004-11-05 23:19 Attachments: typeo_internals.patch      > For user-space sample if they don't fall into something that maps to a > file they are discarded in the oprofile kernel driver and don't even > make it to the daemon. You should be able to get the number discarded > in /dev/oprofile/status/samples_lost_no_mapping. Thus, for things like > JITed code, you will not see any samples. # cat /dev/oprofile/status/samples_lost_no_mapping cat: /dev/oprofile/status/samples_lost_no_mapping: No such file or directory Am I missing something ? I never get a /dev/oprofile entry, even though: # cat /proc/devices |grep -i oprof 252 oprof > For a kernel/module space sample the sample will make it to the daemon. > However, the daemon will not find a suitable mapping and discard the sample. This is for kernel space JITed code. I need to change this behavior. Where do I start ? Is opd_kernel.c:opd_reread_module_info() the place I should do that ? Is there some /proc entry I can use to manually trigger a module reread from userspace ? Also, please apply the attached.

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: Kristis Makris <kristis.makris@as...> - 2004-11-05 23:39 > However, the daemon will not find a suitable mapping and discard the sample. I effectively need an interface that will let me describe an additional module mapping from userspace. I'm guessing this belongs to the daemon. I need to run: struct kernel_image * image; image = opd_create_module(my_own_module_name, start_address, end_address); ...where I am responsible for determining the correct values of start_address and end_address. Any suggestions how I should do this before proceeding blind-folded ?

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: John Levon <levon@mo...> - 2004-11-07 17:02 On Fri, Nov 05, 2004 at 04:38:29PM -0700, Kristis Makris wrote: > > However, the daemon will not find a suitable mapping and discard the sample. > > I effectively need an interface that will let me describe an additional > module mapping from userspace. I'm guessing this belongs to the daemon. > I need to run: > > struct kernel_image * image; > image = opd_create_module(my_own_module_name, start_address, > end_address); > > ...where I am responsible for determining the correct values of > start_address and end_address. > > Any suggestions how I should do this before proceeding blind-folded ? First, if you're still working with 2.4, I wouldn't bother. Improvements like this should only be for the 2.6 kernel OProfile. Anyway, sounds like you need a generalisation of the 'module' concept to generic kernel mappings. This can be as simple as renaming the interface opd_create_kmapping() or whatever, or might require more significant reworking. I don't know how you intend to export the mapping addresses to userspace? I would not be particularly fond of injecting this into the /dev/oprofile/buffer event stream (beyond creating a module event as it does now). regards john

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: John Levon <levon@mo...> - 2004-11-07 17:22 On Fri, Nov 05, 2004 at 04:17:01PM -0700, Kristis Makris wrote: > Also, please apply the attached. Applied. A changelog entry would be cool next time :) regards john

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: Kristis Makris <kristis.makris@as...> - 2004-11-07 17:43 > First, if you're still working with 2.4, I wouldn't bother. Improvements > like this should only be for the 2.6 kernel OProfile. I'm working with 2.4. > Anyway, sounds like you need a generalisation of the 'module' concept to > generic kernel mappings. This can be as simple as renaming the interface > opd_create_kmapping() or whatever, or might require more significant > reworking. Using opd_create_module() I should be able to add to the list of mapping addresses. But how can I do so while the daemon is running ? > I don't know how you intend to export the mapping addresses to > userspace? I would not be particularly fond of injecting this into the I don't necessarily need to export the mapping addresses to userspace. I need a mechanism of adding to the list of mapping addresses. If I understand this correctly, the information I need has already been collected by the kernel module, and the daemon simply discards EIP hits within the vma I'm interested in, simply because there's no 'module' description for it. If I can run something like: opcontrol --start --add-mapping=my_first_module_name:0xc7770000:0xc777FFFF,my_second_module_name:0xc8880000,0xc888FFFF then I should be able to inform the daemon that there is an additional module mapping with the above start+end addresses. I see that in oprofiled.c USR1 and USR2 are used strictly for start/stop, so I don't see another way of communicating this information to the daemon (maybe there is). > userspace? I would not be particularly fond of injecting this into the > /dev/oprofile/buffer event stream (beyond creating a module event as > it does now). What's the preferable way of communicating with the daemon ? Or should I not communicate with the daemon but directly with the kernel module through something in /dev/oprofile/* ?

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: John Levon <levon@mo...> - 2004-11-07 18:39 On Sun, Nov 07, 2004 at 10:43:45AM -0700, Kristis Makris wrote: > > First, if you're still working with 2.4, I wouldn't bother. Improvements > > like this should only be for the 2.6 kernel OProfile. > > I'm working with 2.4. Note I won't accept any patches that don't work with 2.6 kernel OProfile. > > Anyway, sounds like you need a generalisation of the 'module' concept to > > generic kernel mappings. This can be as simple as renaming the interface > > opd_create_kmapping() or whatever, or might require more significant > > reworking. > > Using opd_create_module() I should be able to add to the list of mapping > addresses. But how can I do so while the daemon is running ? Well, by making changes to oprofiled so that it picks this information from wherever you've placed it. > opcontrol --start > --add-mapping=my_first_module_name:0xc7770000:0xc777FFFF,my_second_module_name:0xc8880000,0xc888FFFF > then I should be able to inform the daemon that there is an additional > module mapping with the above start+end addresses. I see that in > oprofiled.c USR1 and USR2 are used strictly for start/stop, so I don't > see another way of communicating this information to the daemon (maybe > there is). If you only need to do this at start time, then add some oprofiled code that reads /var/lib/oprofile/mappings file or something. If you need to do it dynamically, then it's time to add some IPC to oprofiled (most likely a named pipe). regards john

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: Kristis Makris <kristis.makris@as...> - 2004-11-07 20:14 > Note I won't accept any patches that don't work with 2.6 kernel > OProfile. That's a fair response. > If you only need to do this at start time, then add some oprofiled code > that reads /var/lib/oprofile/mappings file or something. If you need to > do it dynamically, then it's time to add some IPC to oprofiled (most > likely a named pipe). I could get away with using it only at start time. But since I'm going to spend time on this anyway, I might as well go the IPC route rather than a command-line arguments hack that will be useless in the future for other people. I just wanted to know what you'd prefer. A named pipe sounds good to me. Thanks for the feedback.

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: John Levon <levon@mo...> - 2004-11-07 21:33 On Sun, Nov 07, 2004 at 01:14:30PM -0700, Kristis Makris wrote: > > If you only need to do this at start time, then add some oprofiled code > > that reads /var/lib/oprofile/mappings file or something. If you need to > > do it dynamically, then it's time to add some IPC to oprofiled (most > > likely a named pipe). > > I could get away with using it only at start time. But since I'm going > to spend time on this anyway, I might as well go the IPC route rather > than a command-line arguments hack that will be useless in the future > for other people. I just wanted to know what you'd prefer. A named pipe > sounds good to me. Thanks for the feedback. Talking of IPC, it's worth reading the discussions about Java support (the more recent discussions are generally the more useful ones). They need something similar: not required now, but useful to keep in mind when adding the "I can talk to oprofiled" named pipe. regards john

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: Kristis Makris <kristis.makris@as...> - 2004-11-07 23:09 > Talking of IPC, it's worth reading the discussions about Java support > (the more recent discussions are generally the more useful ones). They > need something similar: not required now, but useful to keep in mind > when adding the "I can talk to oprofiled" named pipe. Thanks for the heads up. I tracked those down. In order to dynamically accept control commands, a thread must wait in some loop for data available from the pipe. I will need to run an extra thread that waits in this loop. Is this acceptable ? I'm thinking pthreads here. Otherwise I can only read data from the pipe in the opd_do_read() loop, which will happen only when data from the device are available, instead of when the user sends data to the pipe.

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: John Levon <levon@mo...> - 2004-11-08 00:46 On Sun, Nov 07, 2004 at 04:09:50PM -0700, Kristis Makris wrote: > In order to dynamically accept control commands, a thread must wait in > some loop for data available from the pipe. I will need to run an extra > thread that waits in this loop. Is this acceptable ? I don't think so. I'd need to have an extraordinarily strong argument for making oprofiled multi-threaded, and I don't believe that this is it. > pthreads here. Otherwise I can only read data from the pipe in the > opd_do_read() loop, which will happen only when data from the device are > available, instead of when the user sends data to the pipe. Why can't we use select()? You'd need to open the named pipe read-write, and we'd probably need to add select() for the kernel device driver, but it seems this should work reasonably well? Or it might turn out that we should have a UNIX domain socket... regards john

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: Kristis Makris <kristis.makris@as...> - 2004-11-08 18:46 > > opd_do_read() loop, which will happen only when data from the device are > > available, instead of when the user sends data to the pipe. > > Why can't we use select()? You'd need to open the named pipe read-write, > and we'd probably need to add select() for the kernel device driver, but > it seems this should work reasonably well? I was unsure if adding select() on the common path was a good idea. Calling select() each time would guarantee a system call. Instead, the thread would be blocked until data were available on the pipe, since it would be sleeping most of the time. In any case, I would not object to changing the IPC mechanism. It's your call. But anyway, I have IPC using pthreads working, and I was able to add module mappings by calling opd_create_module(). However, opreport won't show any hits on the jitted section I described. How can I be sure that such kernel hits made it all the way to the daemon ? I'm testing this on a P4 and I am collecting GLOBAL_POWER_EVENTS with the lowest counter possible (3000). I'm 100% sure that the jitted code runs. 3000 appears to be the highest precision I can get and just don't see any hits reported. This is available at http://www.public.asu.edu/~makrists/oprofile/oprofile.ipc/ if anybody would like to have a look. BTW: It's pretty laborious to make diffs when make distclean does not clean everything.

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: John Levon <levon@mo...> - 2004-11-08 19:25 On Mon, Nov 08, 2004 at 11:46:30AM -0700, Kristis Makris wrote: > I was unsure if adding select() on the common path was a good idea. > Calling select() each time would guarantee a system call. Instead, the I don't understand. We already have a system call there, namely read(). > But anyway, I have IPC using pthreads working, and I was able to add > module mappings by calling opd_create_module(). However, opreport won't > show any hits on the jitted section I described. How can I be sure that > such kernel hits made it all the way to the daemon ? I'm testing this on Use verbose option and look in oprofiled.log. Check the sample file is getting created. john

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: Kristis Makris <kristis.makris@as...> - 2004-11-10 00:11 > > Calling select() each time would guarantee a system call. Instead, the > > I don't understand. We already have a system call there, namely read(). We would need to call select() to determine which fd is ready, and then read() on it. That's 2 system calls. > Use verbose option and look in oprofiled.log. Check the sample file is Thanks. I was able to add mappings during runtime by executing: "echo add_mapping sys_brk_jit 0xdac4b5e0 0xdac4b610 > /var/lib/oprofile/oprofiled.ipc" I had a hard time debugging this, since calling opd_create_module() within opd_ipc_commands.c resulted in the 2.6 version being called, while all other modules where loaded using the liblegacy(2.4) version. I had to rename opd_create_module() accordingly. I'm not sure how you had this layed out, but it seemed that both 2.4 and 2.6 oprofiled functionality were both compiled in the same binary. According to the latest version of the patch, I now see the following: opd_handle_module_sample: eip=0xdac4b5e0, counter=0 legacy: Looking for eip 0xdac4b5e0, within 'sys_brk_jit', start=0xdac4b5e0, end=0xdac4b610 No image for sampled module sys_brk_jit It looks like module->image is NULL, which results in the "No Image" error message. What should I do with module->image to get a sample stored for opreport to report ? Is this just a matter of calling opd_get_image() when collecting a jit sample ? If so, what should I specify as tid+tgid? How do I get those ?

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: Kristis Makris <kristis.makris@as...> - 2004-11-13 20:50 John, I was able to have performance counters on kernel JITed code stored by oprofiled and reported by opreport. Instead of output like: c013fd6c 51 0.0597 vmlinux path_walk c013fd6c 27 52.9412 c013fd6e 7 13.7255 c013fd73 1 1.9608 c013fd7f 10 19.6078 c013fd81 1 1.9608 c013fd86 5 9.8039 opreport at least shows: 00000000 53 0.0621 sys_brk_jit (no symbols) 00000002 4 7.5472 00000012 6 11.3208 0000001f 8 15.0943 00000023 1 1.8868 00000024 7 13.2075 0000002a 27 50.9434 In the above example, sys_brk_jit is the symbol of the JIT code, instead of a module name, but I don't know how to flag it that way. I simply call the module name the same as the symbol name. If you are interested in having this merged with OProfile, I can cleanup the patch as required, and test with 2.6. If not, I'll stop harassing you. Again, the latest version of this work is available at: http://www.public.asu.edu/~makrists/oprofile/oprofile.ipc/ Thanks for your help, Kristis

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: John Levon <levon@mo...> - 2004-11-15 21:55 On Sat, Nov 13, 2004 at 01:50:32PM -0700, Kristis Makris wrote: > opreport at least shows: > > 00000000 53 0.0621 sys_brk_jit (no symbols) Cool, I'm glad you could get useful info out of OProfile for what you need. I don't think the patch is ready for general use, though. Both with stuff we've discussed already, and the fact that these symbols are appearing as the module name. To properly support dynamic code you'd have to add a "fake binary" facility similar to that discussed at length in the Java threads. That might turn out to be a fair amount of work. (Also, btw, you really shouldn't ever be using pthreads without thinking carefully about your locking scheme...) regards john

Re: [ham] Re: opreport-ing an EIP in a VMA that does not match a symbol From: Kristis Makris <kristis.makris@as...> - 2004-11-15 22:22 > I don't think the patch is ready for general use, though. Both with > stuff we've discussed already, and the fact that these symbols are > appearing as the module name. To properly support dynamic code you'd I agree! If anyone is in the same kernel JIT shoes, feel free to point them my way. > have to add a "fake binary" facility similar to that discussed at length > in the Java threads. That might turn out to be a fair amount of work. Yes, I browsed through them and it appears to be a significant amount of work. When that happens (by others), work to add IPC can be extracted from this patch. > (Also, btw, you really shouldn't ever be using pthreads without thinking > carefully about your locking scheme...) Indeed. Unless you have a term paper to write ;) Cheers!

609 messages have been excluded from this view by a project administrator.