From: Yaoping R. <yr...@cs...> - 2004-04-27 22:01:16
|
Hi, Could anyone let me know how to fix this problem. I use Oprofile to monitor a high demand workload. What's the default buffer-size? Do I need to increase it? The following is the error message I noticed: Apr 27 17:28:17 typhoon kernel: oprofile: Detected overflow of size 15041. You must increase the module buffer size with Apr 27 17:28:17 typhoon kernel: opcontrol --setup --bufer-size= or reduce the interrupt frequency Thanks - Yaoping Ruan |
From: Philippe E. <ph...@wa...> - 2004-04-28 05:46:34
|
On Tue, 27 Apr 2004 at 18:00 +0000, Yaoping Ruan wrote: > Hi, > > Could anyone let me know how to fix this problem. I use Oprofile to > monitor a high demand workload. What's the default buffer-size? Do I > need to increase it? The following is the error message I noticed: > > Apr 27 17:28:17 typhoon kernel: oprofile: Detected overflow of size > 15041. You must increase the module buffer size with > Apr 27 17:28:17 typhoon kernel: opcontrol --setup --bufer-size= or > reduce the interrupt frequency sampling rateis too high, default buffer size is 65536 entry (32 byte per entry on x86), you have to shutdown oprofile and either decrease sampling rate or increase buffer size. regards, Phil P.S. I needed to accept manually your mail from the spam filter, no idea why the filter trigerred for your mail, the header seems fine. John any idea ? (message was "suspicious header") |
From: John L. <le...@mo...> - 2004-04-28 10:22:33
|
On Wed, Apr 28, 2004 at 07:52:17AM +0000, Philippe Elie wrote: > P.S. I needed to accept manually your mail from the spam filter, no idea > why the filter trigerred for your mail, the header seems fine. > John any idea ? (message was "suspicious header") Probably he sent HTML (which explains why I never saw the post either - my spam filter junks all HTML) john |
From: Yaoping R. <yruan@CS.Princeton.EDU> - 2004-04-28 15:15:18
|
Hi, well, it is hard to believe it is HTML. My email was written in Netscape Inbox with plain text only. This message is from "pine". Hope it can get through. Back to my question, thanks for the answer. Just confirm that the buffer is the event buffer explained in the Internal, not the CPU buffer, right? I notice "--buffer-size" option for opcontrol. I assume it is for the event buffer. Then is it possible that CPU buffer also overflows? And how can I decrease sampling rate? Is it set by increasing the overflow "count" value for each event? > Probably he sent HTML (which explains why I never saw the post either - > my spam filter junks all HTML) Thanks a lot -Yaoping |
From: Philippe E. <ph...@wa...> - 2004-04-28 18:50:15
|
On Wed, 28 Apr 2004 at 11:15 +0000, Yaoping Ruan wrote: > Hi, > well, it is hard to believe it is HTML. My email was written in Netscape > Inbox with plain text only. This message is from "pine". Hope it can get > through. right that's why I didn't undertand the rejection, anyway if it occur again I'll look deeper. > Back to my question, thanks for the answer. Just confirm that the buffer > is the event buffer explained in the Internal, not the CPU buffer, right? > I notice "--buffer-size" option for opcontrol. I assume it is for the > event buffer. Then is it possible that CPU buffer also overflows? And how I think you use a 2.4 module, the only relevant parameter is --buffer-size and --note-table-size (note table size can overflow if there is a lot of fork/exec/mmap). > can I decrease sampling rate? Is it set by increasing the overflow "count" > value for each event? yes, to decrease sampling rate you must increase count with $ opcontrol --event=XXX:counts, the ~root/.oprofile/daemonrc file contains your setting which are persistent over different profiling session, perhaps during experimentation you setup count to a very low value and don't re set it ? regards, Phil |
From: Yaoping R. <yruan@CS.Princeton.EDU> - 2004-04-28 19:09:10
|
Hi PHil: Thanks for your response. > I think you use a 2.4 module, the only relevant parameter is --buffer-size and > --note-table-size (note table size can overflow if there is a lot of fork/exec/mmap). I also tried 2.6 kernel, I guess I also need to increase --buffer-size, right? Does every overflow give out a warning message? I noticed that sometimes it doesn't but the results indicate overflow actually ever happened. > yes, to decrease sampling rate you must increase count with > $ opcontrol --event=XXX:counts, the ~root/.oprofile/daemonrc file > contains your setting which are persistent over different profiling > session, perhaps during experimentation you setup count to a very > low value and don't re set it ? well, I did reset before each measurement and setup counts much larger than the default value. e.g. 1000000 for global_power_events, and 10000 for ITLB_REFERENCE. But since I measure 8 events simultaneously, it is unclear to me that if the kernel buffer holds all of the events together or not? Also another related question is that how frequent the user space daemon reads out the kernel buffer? is it interrupt based or time based? Sorry for so many questions, but I really couldn't find any clue from the manual, internal, or previous discussion. I do admit that my experiments are very high demand which may have not been tested before. Thanks a lot -Yaoping |
From: Philippe E. <ph...@wa...> - 2004-04-28 19:30:34
|
On Wed, 28 Apr 2004 at 15:09 +0000, Yaoping Ruan wrote: > Hi PHil: > > Thanks for your response. > > > I think you use a 2.4 module, the only relevant parameter is --buffer-size and > > --note-table-size (note table size can overflow if there is a lot of fork/exec/mmap). > > I also tried 2.6 kernel, I guess I also need to increase --buffer-size, > right? Does every overflow give out a warning message? I noticed that > sometimes it doesn't but the results indicate overflow actually ever > happened. on 2.6 there is no such message through printk but rather in the file /var/lib/oprofile/oprofiled.log with a dump of statistics at end of run or through /dev/oprofile/stats/* for older version (I don't remember when I added the stats dump in oprofiled.log) > > > yes, to decrease sampling rate you must increase count with > > $ opcontrol --event=XXX:counts, the ~root/.oprofile/daemonrc file > > contains your setting which are persistent over different profiling > > session, perhaps during experimentation you setup count to a very > > low value and don't re set it ? > > well, I did reset before each measurement and setup counts much larger > than the default value. e.g. 1000000 for global_power_events, and 10000 > for ITLB_REFERENCE. But since I measure 8 events simultaneously, it is > unclear to me that if the kernel buffer holds all of the events together > or not? Buffer on 2.4/2.6 is per cpu, events are mixed in the same buffer. > > Also another related question is that how frequent the user space daemon > reads out the kernel buffer? is it interrupt based or time based? behavior is different on 2.6/2.4 - 2.4: when reaching a given threshold on number of used entry in buffer we wake up a task to flush samples to daemon, the watermark is defined as min(buffer_size / 8, 8192) - 2.6: there is a two level buffer, per cpu buffer are flushed to a high level buffer by a timer (and also when an exit() call occur iirc). I don't remember the details on how the high level buffer is flushed but iirc we never got problem from it. Only recent version of the driver and user space tool allow to setup the per cpu buffer size through --cpu-buffer-size (added 2004-01-27), I don't remember which kernel version you need) > Sorry for so many questions, but I really couldn't find any clue from the > manual, internal, or previous discussion. I do admit that my experiments > are very high demand which may have not been tested before. You must experiment a bit to find a right setting., remember to look at $ dmesg and /var/lib/oprofile/oprofiled.log (or /dev/oprofile/stats/*) regards, Phil |
From: Yaoping R. <yruan@CS.Princeton.EDU> - 2004-04-28 22:20:26
|
Hi Phil: I am afraid the mechanism of 2.4 and 2.6 need to be switched. I noticed there's watershed under /dev/oprofile on 2.6 but not 2.4. Also, with my experiments, there's event buffer overflow under 2.4, which implies the kernel buffer is not flushed based on watermark trigger. But on 2.6, there's no event_lost_overflow, but there's sample_lost_overflow under /dev/oprofile/stats/cpu0. behavior is different on 2.6/2.4 - 2.4: when reaching a given threshold on number of used entry in buffer we wake up a task to flush samples to daemon, the watermark is defined as min(buffer_size / 8, 8192) - 2.6: there is a two level buffer, per cpu buffer are flushed to a high level buffer by a timer (and also when an exit() call occur iirc). I don't remember the details on how the high level buffer is flushed but iirc we never got problem from it. You must experiment a bit to find a right setting., remember to look at $ dmesg and /var/lib/oprofile/oprofiled.log (or /dev/oprofile/stats/*) How can I validate the measurement data? no event_lost_overflow and no sample_lost_overflow? (sample_lost_overflow appears under per CPU directory on 2.6) Thanks again -Yaoping |
From: Philippe E. <ph...@wa...> - 2004-04-29 02:58:09
|
On Wed, 28 Apr 2004 at 18:20 +0000, Yaoping Ruan wrote: > > Hi Phil: > > I am afraid the mechanism of 2.4 and 2.6 need to be switched. I noticed > there's watershed > under /dev/oprofile on 2.6 but not 2.4. Also, with my experiments, there's > event buffer > overflow under 2.4, which implies the kernel buffer is not flushed based > on watermark > trigger. for 2.4, I don't said the buffer is flushed when the watermark is reached but than we wake up a task doing that, sample collection is done under NMI context, a lot of things are forbidden under NMI (more than under normal hardware IRQ context), if the watermark is too small compared to the sampling rate the buffer will overflow before the task is started by the scheduler, the problem is similar in 2.6 but implemented in a different way. On 2.4 the report of sample lost is done under the name /proc/sys/dev/oprofile/buffer_overflow. > But on 2.6, there's no event_lost_overflow, but there's > sample_lost_overflow > under /dev/oprofile/stats/cpu0. # ls /dev/oprofile/stats/cpu0/ sample_lost_overflow sample_lost_task_exit sample_received # ls /dev/oprofile/stats/ cpu0 cpu1 event_lost_overflow sample_lost_no_mapping sample_lost_no_mm name under cpu?/* account the per cpu buffer, name under /dev/oprofile/stats account the event buffer http://oprofile.sourceforge.net/doc/internals/core-structure.html /dev/oprofile/buffer_size and /dev/oprofile/buffer_watershed tweak the event buffer not the per cpu buffer, there is no per cpu buffer watershed. > behavior is different on 2.6/2.4 > > - 2.4: when reaching a given threshold on number of used entry in > buffer we wake up a task to flush samples to daemon, the watermark is > defined as min(buffer_size / 8, 8192) > > - 2.6: there is a two level buffer, per cpu buffer are flushed to a > high level buffer by a timer (and also when an exit() call occur iirc). > I don't remember the details on how the high level buffer is flushed > but iirc we never got problem from it. > > > > You must experiment a bit to find a right setting., remember to look at > $ dmesg and /var/lib/oprofile/oprofiled.log (or /dev/oprofile/stats/*) > > How can I validate the measurement data? no event_lost_overflow and no > sample_lost_overflow? (sample_lost_overflow appears under per CPU > directory on 2.6) yes, for event_lost_overflow you can tweak both buffer_size and buffer_watershed; for sample_lost_overflow you can tweak the cpu buffer size. And for the two sort of overlow you can decrease sampling rate. Phil |