From: Daniel G. <gr...@su...> - 2012-06-25 21:46:15
|
I'm using oprofile to debug a C++ application that has occasional "hiccups" where the processing loop slows drastically down for 100ms or so. I'd like to find out what is actually going on during that slowdown, but it happens so intermittently that the relevant samples would be overwhelmed by the rest of the data. Possibilities include some Linus system task, a rogue thread within the application, or a resource contention issue. I'd like to be able to turn sample collection on/off from within my application, triggered by my "oh, no! that was a problem!" detector. But, the only way I've seen to control oprofile is by using the opcontrol control shell script; I could call that, but the process would be too slow to catch the problem. I've taken a look at the oprofile Eclipse plugin's source code, but it's mixed into thousands of source files, and is written in Java, so that wasn't fruitful. Any suggestions on techniques that could be used to identify the cause of a brief, occasional disruption? Thanks, Dan -- Daniel T. Griscom gr...@su... Suitable Systems http://www.suitable.com/ 1 Centre Street, Suite 204 (781) 665-0053 Wakefield, MA 01880-2400 |
From: Oestman, F. <Fre...@me...> - 2012-06-26 08:05:36
|
Hi Daniel, > I'm using oprofile to debug a C++ application that has occasional > "hiccups" where the processing loop slows drastically down for 100ms > or so. I'd like to find out what is actually going on during that > slowdown, but it happens so intermittently that the relevant samples > would be overwhelmed by the rest of the data. Possibilities include > some Linus system task, a rogue thread within the application, or a > resource contention issue. Did you consider to use a different tool, such as the LTTng kernel/user-space tracer? Then you would get timestamped detail data, not only a statistical overview. You could zoom in on your problem interval and see what the scheduler etc. are doing, what thread is running on what core and so on. Oprofile is an excellent profiling tool, but for some problems you need other tools. > I'd like to be able to turn sample collection on/off from within my > application, triggered by my "oh, no! that was a problem!" detector. > But, the only way I've seen to control oprofile is by using the > opcontrol control shell script; I could call that, but the process > would be too slow to catch the problem. LTTng has such an API, but you only need it if you have severe resource restrictions on your target system. The data is always detailed and timestamped. Cheers, Fredrik Östman |
From: Daniel G. <gr...@su...> - 2012-06-26 14:01:18
|
At 8:05 AM +0000 6/26/12, Oestman, Fredrik wrote: >Hi Daniel, > > >> I'm using oprofile to debug a C++ application that has occasional >> "hiccups" where the processing loop slows drastically down for 100ms >> or so. I'd like to find out what is actually going on during that >> slowdown, but it happens so intermittently that the relevant samples >> would be overwhelmed by the rest of the data. Possibilities include >> some Linus system task, a rogue thread within the application, or a >> resource contention issue. > >Did you consider to use a different tool, such as the LTTng >kernel/user-space tracer? > >Then you would get timestamped detail data, not only a statistical >overview. You could zoom in on your problem interval and see what >the scheduler etc. are doing, what thread is running on what core >and so on. > >Oprofile is an excellent profiling tool, but for some problems you >need other tools. Thank you. I'll look into LTTng. Thanks, Dan -- Daniel T. Griscom gr...@su... Suitable Systems http://www.suitable.com/ 1 Centre Street, Suite 204 (781) 665-0053 Wakefield, MA 01880-2400 |