Thread: [Secureideas-base-user] re: Base has become unuseable. (Jacob, Raymond A Jr) some success
Brought to you by:
secureideas,
sinukas
From: Jacob, R. A J. <ray...@na...> - 2006-04-04 18:28:37
|
Message: 3 Date: Tue, 28 Mar 2006 16:54:01 -0500 From: "Jacob, Raymond A Jr" <ray...@na...> To: <sec...@li...> Subject: [Secureideas-base-user] re: Base has become unuseable. From: Michael Stone <mstone@ma...>=3D20 Re: Base has become unuseable =3D20 2006-03-28 12:05 =3D20 On Tue, Mar 28, 2006 at 12:19:24PM -0500, you wrote: >Problem: Base has become unuseable. >When I try to look at alerts from base_main.php, the operation times = =3D out. >Any suggestions? Install the latest base. Even then you're beyond what's comfortable with = =3D mysql & the default base schema. >> I intrepet from your answer that combination of mysql and the default = =3D base >> schema can not handle databases over 2GB? Is that a correct =3D assumption? >> Or are you saying that I need a faster CPU? Mike Stone --__--__-- I turned on slow-query-log and found that when base fired up, the Refresh Stat directive in the html along with the other queries that start when the page opened caused the CPU to Max out. So turned off Refresh Stat and autoupdate cache in the base_conf.php. Checking the machineName-slow.log. I found the query SELECT COUNT(DISTINCT acid_event.ip_src, acid_event.ip_dst, = acid_event.ip_proto) FROM acid_event; took 340 secs(5:40 mins) at a minimum. The updatecache cron job runs = every 5:00 minutes and spikes the CPU to 57% with nothing running. If I can time it right i.e. = use base between updating the acid cache I can get results within a reasonable amount of = time. On a more complex query, if I don't time it right,the query can take up = to 1931 secs(32:11 minutes). The complex query never appears to use more than 20% of the CPU once the = updatecache cron job completes. Off the top of my head there appears to be a need for some kind of job = queueing and caching mechanism in order for BASE to scale beyond one user.=20 r/Raymond |
From: Michael S. <ms...@ma...> - 2006-04-04 18:43:32
|
On Tue, Apr 04, 2006 at 02:28:14PM -0400, Jacob, Raymond A Jr wrote: >I turned on slow-query-log and found that when base fired up, >the Refresh Stat directive in the html along with the other >queries that start when the page opened caused the CPU to Max >out. So turned off Refresh Stat and autoupdate cache in the >base_conf.php. That's definately a prerequisite for large installations. >Checking the machineName-slow.log. I found the query >SELECT COUNT(DISTINCT acid_event.ip_src, acid_event.ip_dst, acid_event.ip_proto) FROM acid_event; >took 340 secs(5:40 mins) at a minimum. Check out main_page_detail and show_summary_stats in base_conf.php; setting them to 0 will avoid those huge queries, which are non-optimizable and fairly useless for a large database. Mike Stone |
From: Joel E. <es...@gm...> - 2006-04-04 19:39:14
|
It is possible to do what you are talking about Raymond. Use the base_maintenance.pl script in the scripts directory and cron it. J On Apr 4, 2006, at 2:43 PM, Michael Stone wrote: > On Tue, Apr 04, 2006 at 02:28:14PM -0400, Jacob, Raymond A Jr wrote: >> I turned on slow-query-log and found that when base fired up, >> the Refresh Stat directive in the html along with the other >> queries that start when the page opened caused the CPU to Max >> out. So turned off Refresh Stat and autoupdate cache in the >> base_conf.php. > > That's definately a prerequisite for large installations. > >> Checking the machineName-slow.log. I found the query >> SELECT COUNT(DISTINCT acid_event.ip_src, acid_event.ip_dst, >> acid_event.ip_proto) FROM acid_event; >> took 340 secs(5:40 mins) at a minimum. > > Check out main_page_detail and show_summary_stats in base_conf.php; > setting them to 0 will avoid those huge queries, which are non- > optimizable and fairly useless for a large database. > Mike Stone > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Secureideas-base-user mailing list > Sec...@li... > https://lists.sourceforge.net/lists/listinfo/secureideas-base-user |
From: Mordread W. <mor...@gm...> - 2006-04-05 06:43:44
|
On 4/4/06, Joel Esler <es...@gm...> wrote: > It is possible to do what you are talking about Raymond. Use the > base_maintenance.pl script in the scripts directory and cron it. We use the following (Debian GNU/Linux): 1. /etc/cron.d/base_update: */1 * * * * root cd /var/www/xxx/base/ && /usr/bin/php4 "/var/www/xxx/base/base_maintenance.php" > /dev/null 2>&1 --> this will cron the base_maintenance script every minutes (you may need to change this to 5 minutes if the script take longer to execute with your installation...). 2. /var/www/xxx/base/base_maintenance.php I've also add these lines in base_maintenance.php in order to update DNS cache also (we can't pass variables with PHP CLI): if ( $submit =3D=3D "Update Alert Cache" ) ... else { UpdateAlertCache($db); UpdateDNSCache($db); } |