Thank you very much!!!

Now I understand! I really appreciate your help!

Thanasis Petsas

On Sun, Sep 19, 2010 at 11:11 PM, Maucci, Cyrille <cyrille.maucci@hp.com> wrote:
Hello,
 
Those reports below look fine.
When your system is idle, a few samples are reported : ~ 50000 samples for 15 minutes.
When your system is loaded, loads of samples are reported: more than 5 millions samples for 5 minutes.
 
When your system is idle, most of the CPU time is spent in kernel which is expected.
 
So to reply your question,
 
My question is: Why the 80-85% of the CPU time is used from
the kernel (no-vmlinux) since no program is running in this machine?
when the system is idle, it does not consume a lot of CPU time, and the few CPU time it consumes is in the kernel, which is normal (handling interrupts, select(), ...).
 
I hope that is clear.
++Cyrille

 

From: Thanasis [mailto:skizofrenito@gmail.com]
Sent: Sunday, September 19, 2010 10:08 PM

To: Maucci, Cyrille
Cc: oprofile-list@lists.sourceforge.net
Subject: Re: 80-85% of CPU time is used from the kernel in a machine that no program is running. Why?

This is for 15 minutes without any program running by me:

root@gen1:~# opreport --exclude-dependent --demangle=smart --symbols | head
warning: /no-vmlinux could not be found.
CPU: Core 2, speed 2659.99 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        app name                 symbol name
49300    85.4879  no-vmlinux               (no symbols)
5624      9.7522  python2.5                (no symbols)
1357      2.3531  libc-2.8.90.so           (no symbols)
466       0.8081  ld-2.8.90.so             (no symbols)
205       0.3555  oprofiled                (no symbols)
136       0.2358  libglib-2.0.so.0.1800.2  (no symbols)
132       0.2289  libdbus-1.so.3.4.0       (no symbols)
...

And this is for 5 minutes with running snort in this machine. In an other machine
that it was connected with a Gigabit network I was replaying (with tcp replay) a trace
file with data from the university network with traffic rate 434 Mbps:

root@gen1:~# opreport --exclude-dependent --demangle=smart --symbols | head
warning: /no-vmlinux could not be found.
CPU: Core 2, speed 2659.99 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        app name                 symbol name
2773054  40.7946  no-vmlinux               (no symbols)
2117696  31.1536  snort                    bnfaSearch
212398    3.1246  snort                    detection_option_node_evaluate
143720    2.1143  snort                    AVLFind
86869     1.2779  libc-2.8.90.so           (no symbols)
69232     1.0185  snort                    DecodeTCP
68631     1.0096  snort                    ProcessTcp
...

 The average overall usage of the 4 CPUs was around: 28 %,
and for the Snort, the CPU usage that top output was 100%.



On Sun, Sep 19, 2010 at 10:14 PM, Maucci, Cyrille <cyrille.maucci@hp.com> wrote:
When you report
 
samples  %        app name                 symbol name
29322    85.6042  no-vmlinux               (no symbols)

what was the duration of the oprofile capture?
 
Having 85% of the cpu time of an idle server spent in the kernel is not a suprise to anyone I guess.
The surprise would be if oprofile was reporting a high cpu usage unlike other tools.
 
 
++Cyrille
 
 

From: Thanasis [mailto:skizofrenito@gmail.com]
Sent: Sunday, September 19, 2010 9:10 PM

To: Maucci, Cyrille
Cc: oprofile-list@lists.sourceforge.net
Subject: Re: 80-85% of CPU time is used from the kernel in a machine that no program is running. Why?



On Sun, Sep 19, 2010 at 9:35 PM, Maucci, Cyrille <cyrille.maucci@hp.com> wrote:
Hello,
 
first, you should run top or sar with refresh every 1 sec, during a long period, first when oprofile is not started and also when it is running.
For top, you should hit the '1' key to see the per-CPU breakdown.
 
I did it. All the CPUs seem to be idle. I can see only 1% usage from top or sshd some times
and 2% from oprofile.. Nothing more.
 
Then you should look not only at the CPUs but the hottest processes.
 
++Cyrille
 


From: skizofrenito@gmail.com [mailto:skizofrenito@gmail.com] On Behalf Of Petsas Athanasios
Sent: Sunday, September 19, 2010 8:33 PM
To: Maucci, Cyrille
Cc: oprofile-list@lists.sourceforge.net
Subject: Re: 80-85% of CPU time is used from the kernel in a machine that no program is running. Why?





On Sun, Sep 19, 2010 at 9:22 PM, Maucci, Cyrille <cyrille.maucci@hp.com> wrote:
Hello,
 
why do you say no programis running?
Didn't you mean that yourself had launched no program, but have you already looked at what an 'idle' system does?
 

Yes, I mean that I have not launched any program
 
From my perspective, performance troubleshooting is a drill down approach and oprofile is already a low-level thing.
Before running oprofile, have you looked your idle system at a much higher level, like what does top or sar report?

Yes I have done it. The top shows that all the cores of the machine are idle:



top - 14:28:30 up 87 days,  3:59,  1 user,  load average: 0.00, 0.00, 0.00
Tasks:  90 total,   1 running,  89 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2052152k total,  1117572k used,   934580k free,   260452k buffers
Swap:  6008268k total,    14464k used,  5993804k free,   445756k cached
 
You would probably be surprised.
 
++Cyrille
 

From: Thanasis [mailto:skizofrenito@gmail.com]
Sent: Sunday, September 19, 2010 8:12 PM
To: oprofile-list@lists.sourceforge.net
Subject: 80-85% of CPU time is used from the kernel in a machine that no program is running. Why?

Hello,

I made a profiler in a machine with ubuntu os where no program was running, I
started it, and after aome minutes I dumped the statistics. The results of the
oreport are:

root@gen1:~# opreport --exclude-dependent --demangle=smart --symbols | head
warning: /no-vmlinux could not be found.                                  
CPU: Core 2, speed 2659.99 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        app name                 symbol name
29322    85.6042  no-vmlinux               (no symbols)
2993      8.7379  python2.5                (no symbols)
991       2.8932  libc-2.8.90.so           (no symbols)
330       0.9634  ld-2.8.90.so             (no symbols)
124       0.3620  oprofiled                (no symbols)
92        0.2686  libdbus-1.so.3.4.0       (no symbols)
90        0.2628  libglib-2.0.so.0.1800.2  (no symbols)
...
...

I run the profiler again:

root@gen1:~# opreport --exclude-dependent --demangle=smart --symbols | head
warning: /no-vmlinux could not be found.
CPU: Core 2, speed 2659.99 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        app name                 symbol name
16227    79.4040  no-vmlinux               (no symbols)
2978     14.5723  python2.5                (no symbols)
671       3.2834  libc-2.8.90.so           (no symbols)
240       1.1744  ld-2.8.90.so             (no symbols)
60        0.2936  libdbus-1.so.3.4.0       (no symbols)
47        0.2300  libglib-2.0.so.0.1800.2  (no symbols)
39        0.1908  oprofiled                (no symbols)
...
...

My question is: Why the 80-85% of the CPU time is used from
the kernel (no-vmlinux) since no program is running in this machine?

Anyone that can help me please?

Thank you in advance,
Thanasis Petsas