RE: [Perfparse-users] database size....

[James O. <jo...@re...>]

A little more information on this issue as well... since nagios 2 beta
crashes on the kill -HUP signal, I turned off the log rotation via
perfparse.  It appears after some testing last night that perfparse is not
correctly calculating hashes on my system.  I upgraded to 100.6, dropped the
existing database, and used perfparse to load the current log file.  It
stopped (it was running under the daemon shell script) halfway through and I
restarted perfparse.  It then said that it was moving to the line that it
had stopped on, after which it said that the hash was incorrect and started
reloading the data from the beginning of the file.  I suspect that this is a
regular occurrence because over the last month a 1.5 million line logfile
generated 62 million rows in the perfdata_service_bin table.

I am using perfparse 100.6 on fedora core 2 if that helps any...

James

-----Original Message-----
From: Jeff Scott [mailto:je...@sk...] 
Sent: Wednesday, September 29, 2004 9:14 AM
To: James Ochs; per...@li...
Subject: Re: [Perfparse-users] database size....

I'm in a simular situation, but more servers....looking at about 600  
servers eventually, and I'd like to store about a year worth of data for  
post analysis, and trending.  What kind of storage am I going to need?   
I'm looking at 86mil rows inserted per month....ouch.  Also, has anyone  
every used Oracle with Perfparse?  What would be involved in writing it  
modular to use any database backend?


On Tue, 28 Sep 2004 17:21:49 -0700, James Ochs <jo...@re...>  
wrote:

> Hi all,
> I sent an email to the list about a month ago on this issue and haven't
> gotten a response.
> I have perfparse and am tracking about 80 servers and about 10 or so
> monitors per server, all running at five minute intervals.  I have 30  
> days
> worth of data in the database.
> Currently the perfdata_service_raw table is 45M rows and over 800M.  The
> perfdata_service_bin table is 62M rows and about 1.6G (this is due to not
> purging for a long time before I got the deletion policies working, I  
> think
> its about 900M of actual data).
> Due to the number of rows the perfparse-db-purge actually crashes mysql  
> due
> to running out of buffer space... I have upped this to 16M and it still
> crashes.
> I'd also like to be able to monitor trends in services over a longer  
> period
> of time, like say the last month, the last year, the last 5 years  
> similar to
> the way mrtg does, but with the dataset the way it is that is currently  
> not
> feasible.
> How much data are other people retaining?  Has anyone else run into  
> similar
> issues?  Does anyone have a solution?
> Thanks,
> James