Menu
▾
▴
webadmin-list
|
From: <pr...@pr...> - 2006-09-06 07:17:16
|
From: Prodos To: John Hinton Dear John, Howdy! Thanks for your reply .... > Is this under Sendmail in logwatch? Yes! Here is how it goes ... [Below: From my logwatch] > --------------------- sendmail Begin ------------------------ > > > > 12 messages returned after 4 hours > > 169 unidentified unknown users > > Connections Rejected due to load average:: > Load Avg 12: 111 Times(s) > Load Avg 13: 51 Times(s) > Load Avg 131: 1 Times(s) > Load Avg 14: 59 Times(s) > Load Avg 15: 41 Times(s) > Load Avg 16: 23 Times(s) > Load Avg 17: 24 Times(s) > Load Avg 18: 19 Times(s) > Load Avg 19: 14 Times(s) [Above: From my logwatch.] [John:] > If so, you have a very bad email > problem going on... or something using up all the resources and Sendmail > is gracefully allowing the other stuff to run.... Sorry, I don't know what that means. > > What does top or sar show for load averages over time? > Ehm ... sorry, I don't know what that means. Is that information in my logwatch? I couldn't see it. I can forward you the complete logwatch email privately if you like. But it's a long S.O.B. - and of course I don't want to waste your time. Or do I find this information within my WEBMIN? Many thanks for your help. Best Wishes, PRODOS http://prodos.thinkertothinker.com Have a hug. Read Atlas Shrugged. |
|
From: <pr...@pr...> - 2006-09-06 23:40:24
|
From: Prodos To: John Hinton Dear John, Good morning. Thank you for your detailed notes (below). I'll do my best to see if I can follow them when I get back this afternoon. By the way, over the last 12 hours my server crashed twice (!!) Question: Would you or someone else on this list be willing to enter my WEBMIN and have a look around to see if the problem can be located and fixed? If so, how much would you charge, please? Thanks very much for your help. Best Wishes, PRODOS http://prodos.thinkertothinker.com Have a hug. Read Atlas Shrugged. > -------- Original Message -------- > Subject: Re: [webmin-l] Hundreds of cron /usr/bin/monitor.sh > notifications followed by server crash > From: John Hinton <web...@ew...> > Date: Wed, September 06, 2006 11:19 pm > To: Webmin users list <web...@li...> > > pr...@pr... wrote: > > From: Prodos > > To: John Hinton > > > > Dear John, > > > > Howdy! Thanks for your reply .... > > > > > > > > > >> Is this under Sendmail in logwatch? > >> > > > > > > Yes! Here is how it goes ... > > > > > > [Below: From my logwatch] > > > >> --------------------- sendmail Begin ------------------------ > >> > >> > >> > >> 12 messages returned after 4 hours > >> > >> 169 unidentified unknown users > >> > >> Connections Rejected due to load average:: > >> Load Avg 12: 111 Times(s) > >> Load Avg 13: 51 Times(s) > >> Load Avg 131: 1 Times(s) > >> Load Avg 14: 59 Times(s) > >> Load Avg 15: 41 Times(s) > >> Load Avg 16: 23 Times(s) > >> Load Avg 17: 24 Times(s) > >> Load Avg 18: 19 Times(s) > >> Load Avg 19: 14 Times(s) > >> > > > > > > [Above: From my logwatch.] > > > Something is causing very high server loads. Your Sendmail is set to > stop accepting email when the load hits 12, which is a pretty good place > to let it do that. You need to find out what is causing these high > server loads.. Generally server loads of 3 are OK, with spikes up from > there being acceptable. I personally like to have my systems run below 1 > during normal load times. Things that will run server loads up are > oftentimes email related. SpamAssassin, ClamAV and the various > spam/virus filtering systems. When high loads of incoming email hit, > serverloads rise on a normal system. If a spammer sends out a bulk > message that might be hitting a bunch of your domains at once and > perhaps many email addresses good or bad for those domains, loads go up. > > But, these loads are by no means limited to mail systems. > > Most Redhat flavors have an application called top. From the command > line you just enter top. It will display server loads, disk IO, > processor and user loads and a list of processes which are active. It > refreshes at a set interval. I think it defaults to like 5 seconds. This > gives you an overview of what's happening on a system. You can leave it > running for a while, to see if loads are just hitting spikes or if they > are constantly high. Also, you can see if it's twenty mail functions or > most of the important processes that might be eating up resources. > > Another program available in Redhat flavors is sar. It is not installed > by default, but I normally do install it. This program takes a snapshot > reading every ten minutes and holds the output for the current day, > which is a good way to establish normal operation. > > The reason I pointed to email to start with, is because oftentimes it is > a compromised server which is sending spam and loads spike. > > Within Webmin, you can go to System -> Running Processes and use the > various sort functions to see whats eating your resources. Note that as > you switch the order of display, webmin will show up at the top as it is > running to create that new display. Also, at the top of that page CPU > load averages are shown for the last 1 minute, 5 minutes and 15 minutes. > You may need to check this with some frequency to see those loads > change.. again, to determine if these are just spikes or if the load > stays high. > > Best, > John Hinton > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > - > Forwarded by the Webmin mailing list at web...@li... > To remove yourself from this list, go to > http://lists.sourceforge.net/lists/listinfo/webadmin-list |
|
From: <pr...@pr...> - 2006-09-09 06:44:40
|
From: Prodos Good afternoon. First of all, many thanks to John Hinton who helped me out with this problem both on this list and off-list. Since getting this problem fixed is beyond my (pathetically meagre) technical expertise I ended up having to hire a Linux expert to do the remaining trouble-shooting and fixing. It looks like John was spot on with his diagnosis. There was some program on one of my sites that was allowing some idiot to use it for spamming and crashing my server - which cost me a lot of time and money that I couldn't really afford to spend. Grrrrrr. [Much seething and cussing.] Well, back to work now. The Late Crocodile Man taught us to keep smiling. It's the Aussie thing to do! =]:-) Best Wishes, PRODOS http://prodos.thinkertothinker.com Have a hug. Read Atlas Shrugged. |
|
From: John H. <web...@ew...> - 2006-09-06 13:19:26
|
pr...@pr... wrote: > From: Prodos > To: John Hinton > > Dear John, > > Howdy! Thanks for your reply .... > > > > >> Is this under Sendmail in logwatch? >> > > > Yes! Here is how it goes ... > > > [Below: From my logwatch] > >> --------------------- sendmail Begin ------------------------ >> >> >> >> 12 messages returned after 4 hours >> >> 169 unidentified unknown users >> >> Connections Rejected due to load average:: >> Load Avg 12: 111 Times(s) >> Load Avg 13: 51 Times(s) >> Load Avg 131: 1 Times(s) >> Load Avg 14: 59 Times(s) >> Load Avg 15: 41 Times(s) >> Load Avg 16: 23 Times(s) >> Load Avg 17: 24 Times(s) >> Load Avg 18: 19 Times(s) >> Load Avg 19: 14 Times(s) >> > > > [Above: From my logwatch.] > Something is causing very high server loads. Your Sendmail is set to stop accepting email when the load hits 12, which is a pretty good place to let it do that. You need to find out what is causing these high server loads.. Generally server loads of 3 are OK, with spikes up from there being acceptable. I personally like to have my systems run below 1 during normal load times. Things that will run server loads up are oftentimes email related. SpamAssassin, ClamAV and the various spam/virus filtering systems. When high loads of incoming email hit, serverloads rise on a normal system. If a spammer sends out a bulk message that might be hitting a bunch of your domains at once and perhaps many email addresses good or bad for those domains, loads go up. But, these loads are by no means limited to mail systems. Most Redhat flavors have an application called top. From the command line you just enter top. It will display server loads, disk IO, processor and user loads and a list of processes which are active. It refreshes at a set interval. I think it defaults to like 5 seconds. This gives you an overview of what's happening on a system. You can leave it running for a while, to see if loads are just hitting spikes or if they are constantly high. Also, you can see if it's twenty mail functions or most of the important processes that might be eating up resources. Another program available in Redhat flavors is sar. It is not installed by default, but I normally do install it. This program takes a snapshot reading every ten minutes and holds the output for the current day, which is a good way to establish normal operation. The reason I pointed to email to start with, is because oftentimes it is a compromised server which is sending spam and loads spike. Within Webmin, you can go to System -> Running Processes and use the various sort functions to see whats eating your resources. Note that as you switch the order of display, webmin will show up at the top as it is running to create that new display. Also, at the top of that page CPU load averages are shown for the last 1 minute, 5 minutes and 15 minutes. You may need to check this with some frequency to see those loads change.. again, to determine if these are just spikes or if the load stays high. Best, John Hinton |
|
From: Kris D. <kd...@vi...> - 2006-09-06 14:51:02
|
pr...@pr... wrote: > [John:] > >> If so, you have a very bad email >>problem going on... or something using up all the resources and Sendmail >>is gracefully allowing the other stuff to run.... > Sorry, I don't know what that means. Literally, it means exactly what is says. Either something is flooding sendmail with SMTP connections, or some other process or set of processes is/are hogging CPU/memory/disk (pick any one or more of the three - you'll need to investigate further) causing sendmail's load limits to activate and stop it from attempting to process more mail. >>What does top or sar show for load averages over time? > Ehm ... sorry, I don't know what that means. Again, exactly what it says. top is one program you can use in a shell to watch what processes are running on your system in realtime. sar is probably another; I'm not familiar with it. > Is that information in my logwatch? I couldn't see it. Logwatch reports are WAY too late and far too condensed to be much use in tracking down what's actually happening. (Among other things, all of the timestamps are stripped off, so you can't tell *when* something happened.) > I can forward you the complete logwatch email privately if you like. > But it's a long S.O.B. - and of course I don't want to waste your time. > > Or do I find this information within my WEBMIN? This isn't really a problem that can be usefully solved within Webmin, unless there's a Java-based monitoring module I don't know about. (Possible, but unlikely.) You need to watch what's happening in realtime, without the filtering that logwatch needs to do so you don't just get a raw copy of all of your logs for the day. Your basic problem is that for some reason, your server is trying to do too much at once for some indefinite amount of time, causing sendmail's load average restrictions to activate. In order to get useful data, you'll have to sit and watch what's happening on your server in realtime - even digging through log files will only give you a partial picture. Some things you can do to see what's going on are: - Manually examine the mail log to see if you're getting flooded with SMTP connections around the times sendmail's load restrictions activate - Check your other logs for the times sendmail has started rejecting connections to see what's going on with other processes - Watch active processes with top running in a shell window. A second window showing memory usage instead of CPU might be helpful; most problems I've encountered like this were due to memory starvation rather than CPU limits. (Adding swap will NOT help, it just delays the inevitable, and often makes the problem worse when it shows up.) Given that your server is crashing, you may need to temporarily disable one or more services as a brute-force way to discover which one is getting "attacked" (whether this is a real attack is another question). -kgd |
|
From: <da...@so...> - 2006-09-06 16:10:59
|
Hey, thanks for the heads up on the "top" function. I didn't know about it and it's really useful. It even breaks out the process load across each of the CPUs so you can keep a status. Sometimes monitoring someone else's question is useful ;-) >> we return to your regularly scheduled programing<< Date sent: Wed, 06 Sep 2006 10:50:56 -0400 From: Kris Deugau <kd...@vi...> Organization: ViaNet Internet Solutions To: Webmin users list <web...@li...> Subject: Re: [webmin-l] Hundreds of cron /usr/bin/monitor.sh notifications followed by server crash Send reply to: Webmin users list <web...@li...> <mailto:web...@li...?subject=unsubscribe> <mailto:web...@li...?subject=subscribe> [ Double-click this line for list subscription options ] pr...@pr... wrote: > [John:] > >> If so, you have a very bad email >>problem going on... or something using up all the resources and Sendmail >>is gracefully allowing the other stuff to run.... > Sorry, I don't know what that means. Literally, it means exactly what is says. Either something is flooding sendmail with SMTP connections, or some other process or set of processes is/are hogging CPU/memory/disk (pick any one or more of the three - you'll need to investigate further) causing sendmail's load limits to activate and stop it from attempting to process more mail. >>What does top or sar show for load averages over time? > Ehm ... sorry, I don't know what that means. Again, exactly what it says. top is one program you can use in a shell to watch what processes are running on your system in realtime. sar is probably another; I'm not familiar with it. > Is that information in my logwatch? I couldn't see it. Logwatch reports are WAY too late and far too condensed to be much use in tracking down what's actually happening. (Among other things, all of the timestamps are stripped off, so you can't tell *when* something happened.) > I can forward you the complete logwatch email privately if you like. > But it's a long S.O.B. - and of course I don't want to waste your time. > > Or do I find this information within my WEBMIN? This isn't really a problem that can be usefully solved within Webmin, unless there's a Java-based monitoring module I don't know about. (Possible, but unlikely.) You need to watch what's happening in realtime, without the filtering that logwatch needs to do so you don't just get a raw copy of all of your logs for the day. Your basic problem is that for some reason, your server is trying to do too much at once for some indefinite amount of time, causing sendmail's load average restrictions to activate. In order to get useful data, you'll have to sit and watch what's happening on your server in realtime - even digging through log files will only give you a partial picture. Some things you can do to see what's going on are: - Manually examine the mail log to see if you're getting flooded with SMTP connections around the times sendmail's load restrictions activate - Check your other logs for the times sendmail has started rejecting connections to see what's going on with other processes - Watch active processes with top running in a shell window. A second window showing memory usage instead of CPU might be helpful; most problems I've encountered like this were due to memory starvation rather than CPU limits. (Adding swap will NOT help, it just delays the inevitable, and often makes the problem worse when it shows up.) Given that your server is crashing, you may need to temporarily disable one or more services as a brute-force way to discover which one is getting "attacked" (whether this is a real attack is another question). -kgd ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as- us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 - Forwarded by the Webmin mailing list at webadmin- li...@li... To remove yourself from this list, go to http://lists.sourceforge.net/lists/listinfo/webadmin-list ****************************************************************************** *** Domains by SolvNet http://solvnetdomains.com Be a domain reseller and make cash http://domainmiddleman.com Domain name registration and hosting solutions. http://solvnethosting.com |
