Re: [webmin-l] Hundreds of cron /usr/bin/monitor.sh notifications followed by server crash

[Kris D. <kd...@vi...>]

pr...@pr... wrote:
> [John:]
> 
>> If so, you have a very bad email 
>>problem going on... or something using up all the resources and Sendmail 
>>is gracefully allowing the other stuff to run....

> Sorry, I don't know what that means.

Literally, it means exactly what is says.  Either something is flooding 
sendmail with SMTP connections, or some other process or set of 
processes is/are hogging CPU/memory/disk (pick any one or more of the 
three - you'll need to investigate further) causing sendmail's load 
limits to activate and stop it from attempting to process more mail.

>>What does top or sar show for load averages over time?

> Ehm ... sorry, I don't know what that means.

Again, exactly what it says.  top is one program you can use in a shell 
to watch what processes are running on your system in realtime.  sar is 
probably another;  I'm not familiar with it.

> Is that information in my logwatch? I couldn't see it.

Logwatch reports are WAY too late and far too condensed to be much use 
in tracking down what's actually happening.  (Among other things, all of 
the timestamps are stripped off, so you can't tell *when* something 
happened.)

> I can forward you the complete logwatch email privately if you like.
> But it's a long S.O.B. - and of course I don't want to waste your time.
> 
> Or do I find this information within my WEBMIN?

This isn't really a problem that can be usefully solved within Webmin, 
unless there's a Java-based monitoring module I don't know about. 
(Possible, but unlikely.)  You need to watch what's happening in 
realtime, without the filtering that logwatch needs to do so you don't 
just get a raw copy of all of your logs for the day.

Your basic problem is that for some reason, your server is trying to do 
too much at once for some indefinite amount of time, causing sendmail's 
load average restrictions to activate.

In order to get useful data, you'll have to sit and watch what's 
happening on your server in realtime - even digging through log files 
will only give you a partial picture.

Some things you can do to see what's going on are:
- Manually examine the mail log to see if you're getting flooded with 
SMTP connections around the times sendmail's load restrictions activate
- Check your other logs for the times sendmail has started rejecting 
connections to see what's going on with other processes
- Watch active processes with top running in a shell window.  A second 
window showing memory usage instead of CPU might be helpful;  most 
problems I've encountered like this were due to memory starvation rather 
than CPU limits.  (Adding swap will NOT help, it just delays the 
inevitable, and often makes the problem worse when it shows up.)

Given that your server is crashing, you may need to temporarily disable 
one or more services as a brute-force way to discover which one is 
getting "attacked" (whether this is a real attack is another question).

-kgd