Re: [webmin-l] Webmin / Perl / bw.pl consuming CPU and memory.

[Jamie C. <jca...@we...>]

On 5/Sep/2007 15:53 Jaron Parsons wrote ..
> Jamie Cameron wrote:
> > On 5/Sep/2007 14:34 Jaron Parsons wrote ..
> >   
> >> Jamie Cameron wrote:
> >>     
> >>> On 5/Sep/2007 12:21 Jaron Parsons wrote ..
> >>>   
> >>>       
> >>>> I keep having an issue with Perl eating up all of the CPU.  It starts
> >>>> with 90% in one process, then 45% in two and 30% in three and so on...
> >>>> over time there will be several of these processes eating up the CPU.
> >>>> I 
> >>>> am not sure if it is the cause or not, but something is causing this
> >>>> system to lock up over the time of a week, forcing me to hit the reset
> >>>> button.  During the time that the system is running, it is at full
> CPU
> >>>> load, and everything responds very sluggishly.   I first assumed 
> >>>> hardware, so i have replaced nearly every piece in the server.  No
> change.
> >>>>
> >>>> I am running FreeBSD 6.2-Stable
> >>>> Webmin 1.360
> >>>> virtual min 3.45
> >>>>
> >>>> here is a look at the cpu hog processes from "top" two hours after
> a
> >>>>         
> >> reboot:
> >>     
> >>>> 29597 root               1 126    0 45020K 44172K RUN      5:14  46.09%
> >>>> perl5.8.7
> >>>> 22337 root               1 126    0 45076K 44184K RUN     30:30 45.56%
> >>>> perl5.8.7
> >>>>
> >>>> here is the what ps shows for those processes:
> >>>> USER   PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
> >>>> root 22337 45.4  4.5 45076 44184  ??  R     1:00PM  
> >>>> 31:08.91/usr/bin/perl/usr/local/lib/webmin-1.360/virtual-server/bw.pl
> >>>> (perl5.
> >>>> root 29597 45.8  4.5 45020 44172  ??  R     2:00PM   5:43.12 
> >>>> /usr/bin/perl /usr/local/lib/webmin-1.360/virtual-server/bw.pl (perl5.
> >>>>
> >>>> Any ideas out there?
> >>>> any more information i should look towards?
> >>>> it looks like bw.pl is the always the perl file it is executing. and
> >>>> throughout the day, the process do die off, and new ones are created,
> >>>>         
> >> so
> >>     
> >>>> the process ID's change constantly. 
> >>>>
> >>>> If this is the wrong place to ask this, please point me to the correct
> >>>> location.
> >>>>     
> >>>>         
> >>> Hi Jaron,
> >>>
> >>> The bw.pl program is used by Virtualmin to scan mail and webserver
> logs,
> >>> in order to work out how much bandwidth each domain has comsumed. Normally
> >>> it runs pretty quickly, as each domains' web log file is rotated daily
> >>> by a logrotate configuration entry, so it doesn't have to scan through
> >>>       
> >> months
> >>     
> >>> of old data.
> >>>
> >>> How large are the web and mail log files on your system? You can find
> >>>       
> >> them
> >>     
> >>> under domain home directories at logs/access_log ,  and at /var/log/maillog
> >>>       
> >> or
> >>     
> >>> mail.log .
> >>>
> >>>  - Jamie
> >>>
> >>>
> >>> -------------------------------------------------------------------------
> >>> This SF.net email is sponsored by: Splunk Inc.
> >>> Still grepping through log files to find problems?  Stop.
> >>> Now Search log events and configuration files using AJAX and a browser.
> >>> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> >>> -
> >>> Forwarded by the Webmin mailing list at web...@li...
> >>> To remove yourself from this list, go to
> >>> http://lists.sourceforge.net/lists/listinfo/webadmin-list
> >>>   
> >>>       
> >> Jamie,
> >> thanks for the reply.
> >> after investigating.. seems that might be the whole issue.  This system
> >> is several years old now, and it was running some VERY early versions
> of
> >> virtualmin when first setup, the newer domains are setup correctly,
> >> however the older ones seem to have the access_log files under /var/log
> >> and they are not setup to rotate.  some are around 200 megs or more.
> >> upon further investigation, it does not appear that logrotate is working
> >> on the other domains either.  they are all HUGE i found one that was
> >> nearly 3GB  OUCH.  ill try to get logrotate setup. 
> >> would it be better to just let logrotate fix the issue from here, or
> to
> >> just delete all the huge access_log files and logrotate should handle
> >> for the future?
> >> when i setup logrotate will all the domains pickup the settings 
> >> automatically, or am i going to need to retouch each domain?
> >> The maillog files are rotating properly it seems, just the access_log
> >> that are causing issues.
> >>     
> >
> > What I would recommend is getting logrotate setup, then running the command
> :
> >
> > logrotate -f /etc/logrotate.conf
> >
> > to force a rotation, which should clear those huge log files. It will
> also 
> > show any error messages if logrotate is not working for some reason.
> >
> >  - Jamie
> I found that logrotate was partially setup, but not running due to the
> folder /var/lib not existing on the system for the logrotate.status.  So
> i created this folder. it now runs however, most of the long term 
> domains are not setup with logrotate enabled.  I changed this in a 
> couple and it is fine now, but is there anyway to set all the domains to
> enabled all at once?

Yes - you can use the enable-feature.pl script in the Virtualmin base directory.
On a Redhat-ish system this is /usr/libexec/webmin/virtual-server, while on Debian
it would be /usr/share/webmin/virtual-server . You can run it as :

./enable-feature.pl --all-domains --logrotate

 - Jamie