Re: [webmin-l] Hundreds of cron /usr/bin/monitor.sh notifications followed by server crash

[John H. <web...@ew...>]

pr...@pr... wrote:
> From: Prodos
> To: John Hinton
>
> Dear John,
>
> Howdy! Thanks for your reply ....
>
>
>
>   
>> Is this under Sendmail in logwatch?
>>     
>
>
> Yes! Here is how it goes ...
>
>
> [Below: From my logwatch]
>   
>>  --------------------- sendmail Begin ------------------------ 
>>
>>
>>
>> 12 messages returned after 4 hours
>>
>> 169 unidentified unknown users
>>
>> Connections Rejected due to load average::
>>     Load Avg 12: 111 Times(s)
>>     Load Avg 13: 51 Times(s)
>>     Load Avg 131: 1 Times(s)
>>     Load Avg 14: 59 Times(s)
>>     Load Avg 15: 41 Times(s)
>>     Load Avg 16: 23 Times(s)
>>     Load Avg 17: 24 Times(s)
>>     Load Avg 18: 19 Times(s)
>>     Load Avg 19: 14 Times(s) 
>>     
>
>
> [Above: From my logwatch.]
>   
Something is causing very high server loads. Your Sendmail is set to 
stop accepting email when the load hits 12, which is a pretty good place 
to let it do that. You need to find out what is causing these high 
server loads.. Generally server loads of 3 are OK, with spikes up from 
there being acceptable. I personally like to have my systems run below 1 
during normal load times. Things that will run server loads up are 
oftentimes email related. SpamAssassin, ClamAV and the various 
spam/virus filtering systems. When high loads of incoming email hit, 
serverloads rise on a normal system. If a spammer sends out a bulk 
message that might be hitting a bunch of your domains at once and 
perhaps many email addresses good or bad for those domains, loads go up.

But, these loads are by no means limited to mail systems.

Most Redhat flavors have an application called top. From the command 
line you just enter top. It will display server loads, disk IO, 
processor and user loads and a list of processes which are active. It 
refreshes at a set interval. I think it defaults to like 5 seconds. This 
gives you an overview of what's happening on a system. You can leave it 
running for a while, to see if loads are just hitting spikes or if they 
are constantly high. Also, you can see if it's twenty mail functions or 
most of the important processes that might be eating up resources.

Another program available in Redhat flavors is sar. It is not installed 
by default, but I normally do install it. This program takes a snapshot 
reading every ten minutes and holds the output for the current day, 
which is a good way to establish normal operation.

The reason I pointed to email to start with, is because oftentimes it is 
a compromised server which is sending spam and loads spike.

Within Webmin, you can go to System -> Running Processes and use the 
various sort functions to see whats eating your resources. Note that as 
you switch the order of display, webmin will show up at the top as it is 
running to create that new display. Also, at the top of that page CPU 
load averages are shown for the last 1 minute, 5 minutes and 15 minutes. 
You may need to check this with some frequency to see those loads 
change.. again, to determine if these are just spikes or if the load 
stays high.

Best,
John Hinton