RE: [Perfparse-users] DB Size grows very fast since v101.01

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi Ben,

I just checked your point with the dublicated entries. And it's just as y=
ou thought, the mechanism to remember the file location is not working he=
re. I got all the data inserted 6x.
When I run the perfparse-log2db tool without options, it seems as if it p=
arses over the whole logfile...and I already wondered why every parse las=
t so long *g*
Think I gotta write a skript, that sets the perlog to "" after each run..=
=2E

Regards,
Philipp

> -----Original Message-----
> From: Ben Clewett [mailto:Be...@cl...]=20
> Sent: Friday, October 22, 2004 12:38 PM
> To: Sand Philipp; 'per...@li...'
> Subject: Re: [Perfparse-users] DB Size grows very fast since v101.01
>=20
> Philipp,
>=20
> Lets look at the data...
>=20
> You have 3730416 rows of raw data in ~three days.  This is therefore
> ~5118 services a minute.  You say you have ~2000 metrics.  If=20
> every one of your services is posting every minute, each one=20
> having average 0.5 metrics, these figures could therefore be correct.
>=20
> If not, there is something wrong.  My first guess is that the=20
> log file is not being deleted between parses, and the=20
> mechanism to remember the file location is not working. =20
> Therefore most of your data is duplicates.  (Which are not=20
> checked for in the current version.)  Please look at the=20
> documentation, there is information in there about the two=20
> ways of deleting the log file between runs.  Also check you=20
> log, anything in it indicates some problem.
>=20
> Don't worry too much if this is the case.  When the data gets=20
> older that
>   it's deletion policy, all data and duplicates will be deleted.
>=20
> If these figures are correct, read on:
>=20
> There are ways of reducing the size, which we have to address now.
>=20
> First off, are you using both the binary (metric data) and=20
> raw (plugin output)?  If not, use of these two flags on parse:
> 	--no-raw-data
> 	--no-bin-data
>=20
> Next we have to look at your deletion policies.  The data=20
> should grow to the size of the deletion policies and no more.
>=20
> First thing to do here is to set the policy to 'Never' for=20
> any data you do not want to import.  PP will now ignore it.
>=20
> Second, decide on a limit for your data, set that limit, and=20
> ensure the perfparse-db-purge is running nightly.
>=20
> The data will now grow to a fixed size.  You know from your=20
> table below that one days data is:
>=20
> Raw:	506 / 3 =3D 167 MB / day
> Bin:	723 / 3 =3D 241 MB / day
>=20
> If, say, you don't want any raw data and you do want 30 days=20
> data for some metrics, 10 days for others, and none for the=20
> rest.  It should be easy to calculate your max. size.  Or=20
> work backwards from the space you have to see how much data=20
> you can store.
>=20
> Hope this is of some use,
>=20
> Regards
>=20
> Ben
>=20
>=20
>=20
> Sand Philipp wrote:
>=20
> > No, table space isn't define as ~3GB, it's growing=20
> constantly. Since my first mail, the database size has=20
> increased to 3,4GB...
> > Here's some Data out of the mysql database:
> > mysql> SHOW TABLE STATUS from nagios like 'perfdata_service_bin';
> > | Name                 | Type   | Row_format | Rows    |=20
> Avg_row_length | Data_length | Max_data_length | Index_length=20
> | Data_free | Auto_increment | Create_time | Update_time |=20
> Check_time | Create_options | Comment =20
> > | perfdata_service_bin | InnoDB | Dynamic    | 7005182 |   =20
>         104 |   731906048 |            NULL |   1695547392 | =20
>        0 |        6971161 | NULL        | NULL        | NULL =20
>      |                | InnoDB free: 4096 kB; (`host_name`=20
> `service_description`) REFER=20
> `nagios/perfdata_service`(`host_name` `service_description`);=20
> (`host_name` `service_description` `metric`) REFER=20
> `nagios/perfdata_service_metric`(`host_name`=20
> `service_description` `metric`) |=20
> >=20
> > So the Data lenght of the perfdata_service_bin is already=20
> at 730MB and the index size is 1.6GB ... In 3Days!
> >=20
> > Here's the output for the raw table:
> > | perfdata_service_raw | InnoDB | Dynamic    | 3730416 |   =20
>         135 |   506462208 |            NULL |    401457152 | =20
>        0 |        3611239 | NULL        | NULL        | NULL =20
>      |                | InnoDB free: 4096 kB; (`host_name`=20
> `service_description`) REFER=20
> `nagios/perfdata_service`(`host_name` `service_description`) |
> >=20
> > The serviceperf.log has only 164MB of size and hasn't been=20
> deleted since Tuesday!
> >=20
> > Regards, Philipp
> >=20
> >=20
> >>-----Original Message-----
> >>From: Ben Clewett [mailto:Be...@cl...]
> >>Sent: Friday, October 22, 2004 11:13 AM
> >>To: Sand Philipp
> >>Cc: per...@li...
> >>Subject: Re: [Perfparse-users] DB Size grows very fast since v101.01
> >>
> >>Phillipp,
> >>
> >>Nothing new added, sorry.
> >>
> >>The innodb table space is not linear.  For historical=20
> reasons it uses=20
> >>large files of a fixed size.  The size is defined in the my.cfg (I=20=

> >>believe).  My guess is that your table space size in this file is=20
> >>defined as ~3GB.  Or there is some other similar logical reason.
> >>
> >>Please check as well your log files.  These can grow faster=20
> than the=20
> >>table space.
> >>
> >>Regards, Ben.
> >>
> >>
> >>Sand Philipp wrote:
> >>
> >>
> >>>Hello,
> >>>
> >>>I created a brand new innodb Database since upgrading to
> >>
> >>101.01. In this db I insert data for ~300 hosts ~2000 metrics.=20
> >>
> >>>This Database has been created this Tuesday (10/19/2004)
> >>
> >>and it's already 3GB huge today (10/22/2004). With version 100 and=20=

> >>earlier I had a database with the same amount of hosts and=20
> metrics and=20
> >>had a Database size of about 8GB, but this had been running=20
> for a few=20
> >>months!
> >>
> >>>Had there been some fields modifications in the database?
> >>>
> >>>Regards,
> >>>Philipp
> >>>
> >>>_____________________________
> >>>
> >>>Philipp Sand
> >>>OC-CC-TEC-SYS
> >>>
> >>>SYCOR GmbH
> >>>Heinrich-von-Stephan-Stra=DFe 1-5
> >>>D - 37073 G=F6ttingen
> >>>
> >>>Telefon	+49 (0) 551 / 490 - 0
> >>>Telefax	+49 (0) 551 / 490 - 232468
> >>>
> >>>phi...@sy...
> >>>www.sycor.de
> >>>------------------------------------------------
> >>>
> >>>
> >>>
> >>>************************************************
> >>>
> >>>sycor plastics - die neue Branchenl=F6sung f=FCr die=20
> Kunststoffindustrie
> >>>
> >>>www.sycor-plastics.de
> >>>
> >>>************************************************
> >>>
> >>>
> >>>
> >>>
> >>>Diese E-Mail ist vertraulich und kann dar=FCber hinaus
> >>
> >>pers=F6nliche Informationen beinhalten. Wenn Sie nicht der=20
> >>bestimmungsgem=E4=DFe Empf=E4nger sind, l=F6schen Sie bitte die E-Mai=
l und=20
> >>deren Anh=E4nge sofort und benachrichtigen Sie uns dar=FCber. Die Fir=
ma=20
> >>sycor willigt in keine Vertr=E4ge oder vertragliche=20
> Verpflichtungen ein=20
> >>oder =FCbermittelt rechtsverbindliche Angebote, die in Form=20
> von E-Mail=20
> >>versandt werden, sofern dies nicht ausdr=FCcklich in=20
> schriftlicher Form=20
> >>zwischen den Parteien vereinbart wurde.
> >>
> >>>This e-mail is confidential and may contain personal and/or
> >>
> >>privileged information. If you are not the intended=20
> recipient please=20
> >>delete this e-mail and all attachments immediately and=20
> inform us. The=20
> >>company sycor does not agree with contracts or contract obligations=20=

> >>sent by e-mail, neither do we transmit legally binding offers by=20
> >>e-mail, unless this is not expressly agreed upon between=20
> the parties=20
> >>and documented in written form.
> >>
> >>>
> >>>
> >>>-------------------------------------------------------
> >>>This SF.net email is sponsored by: IT Product Guide on=20
> >>>ITManagersJournal Use IT products in your business? Tell us
> >>
> >>what you
> >>
> >>>think of them. Give us Your Opinions, Get Free ThinkGeek Gift=20
> >>>Certificates! Click to find out more=20
> >>>http://productguide.itmanagersjournal.com/guidepromo.tmpl
> >>>_______________________________________________
> >>>Perfparse-users mailing list
> >>>Per...@li...
> >>>https://lists.sourceforge.net/lists/listinfo/perfparse-users
> >>>
> >>
> >>
> >=20
> >=20
> > ************************************************
> >=20
> > sycor plastics - die neue Branchenl=F6sung f=FCr die Kunststoffindust=
rie
> >=20
> > www.sycor-plastics.de
> >=20
> > ************************************************
> >=20
> >=20
> >=20
> >=20
> > Diese E-Mail ist vertraulich und kann dar=FCber hinaus=20
> pers=F6nliche Informationen beinhalten. Wenn Sie nicht der=20
> bestimmungsgem=E4=DFe Empf=E4nger sind, l=F6schen Sie bitte die=20
> E-Mail und deren Anh=E4nge sofort und benachrichtigen Sie uns=20
> dar=FCber. Die Firma sycor willigt in keine Vertr=E4ge oder=20
> vertragliche Verpflichtungen ein oder =FCbermittelt=20
> rechtsverbindliche Angebote, die in Form von E-Mail versandt=20
> werden, sofern dies nicht ausdr=FCcklich in schriftlicher Form=20
> zwischen den Parteien vereinbart wurde.
> >=20
> > This e-mail is confidential and may contain personal and/or=20
> privileged information. If you are not the intended recipient=20
> please delete this e-mail and all attachments immediately and=20
> inform us. The company sycor does not agree with contracts or=20
> contract obligations sent by e-mail, neither do we transmit=20
> legally binding offers by e-mail, unless this is not=20
> expressly agreed upon between the parties and documented in=20
> written form.
> >=20
> >=20
> >=20
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IT Product Guide on=20
> > ITManagersJournal Use IT products in your business? Tell us=20
> what you=20
> > think of them. Give us Your Opinions, Get Free ThinkGeek Gift=20
> > Certificates! Click to find out more=20
> > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > _______________________________________________
> > Perfparse-users mailing list
> > Per...@li...
> > https://lists.sourceforge.net/lists/listinfo/perfparse-users
> >=20
>=20
>=20

************************************************

sycor plastics - die neue Branchenl=F6sung f=FCr die Kunststoffindustrie

www.sycor-plastics.de

************************************************

Diese E-Mail ist vertraulich und kann dar=FCber hinaus pers=F6nliche Info=
rmationen beinhalten. Wenn Sie nicht der bestimmungsgem=E4=DFe Empf=E4nge=
r sind, l=F6schen Sie bitte die E-Mail und deren Anh=E4nge sofort und ben=
achrichtigen Sie uns dar=FCber. Die Firma sycor willigt in keine Vertr=E4=
ge oder vertragliche Verpflichtungen ein oder =FCbermittelt rechtsverbind=
liche Angebote, die in Form von E-Mail versandt werden, sofern dies nicht=
 ausdr=FCcklich in schriftlicher Form zwischen den Parteien vereinbart wu=
rde.

This e-mail is confidential and may contain personal and/or privileged in=
formation. If you are not the intended recipient please delete this e-mai=
l and all attachments immediately and inform us. The company sycor does n=
ot agree with contracts or contract obligations sent by e-mail, neither d=
o we transmit legally binding offers by e-mail, unless this is not expres=
sly agreed upon between the parties and documented in written form.