Menu
▾
▴
Re: [Perfparse-users] DB Size grows very fast since v101.01
|
From: Ben C. <Be...@cl...> - 2004-10-22 10:38:11
|
Philipp, Lets look at the data... You have 3730416 rows of raw data in ~three days. This is therefore=20 ~5118 services a minute. You say you have ~2000 metrics. If every one=20 of your services is posting every minute, each one having average 0.5=20 metrics, these figures could therefore be correct. If not, there is something wrong. My first guess is that the log file=20 is not being deleted between parses, and the mechanism to remember the=20 file location is not working. Therefore most of your data is=20 duplicates. (Which are not checked for in the current version.) Please=20 look at the documentation, there is information in there about the two=20 ways of deleting the log file between runs. Also check you log,=20 anything in it indicates some problem. Don't worry too much if this is the case. When the data gets older that=20 it's deletion policy, all data and duplicates will be deleted. If these figures are correct, read on: There are ways of reducing the size, which we have to address now. First off, are you using both the binary (metric data) and raw (plugin=20 output)? If not, use of these two flags on parse: --no-raw-data --no-bin-data Next we have to look at your deletion policies. The data should grow to=20 the size of the deletion policies and no more. First thing to do here is to set the policy to 'Never' for any data you=20 do not want to import. PP will now ignore it. Second, decide on a limit for your data, set that limit, and ensure the=20 perfparse-db-purge is running nightly. The data will now grow to a fixed size. You know from your table below=20 that one days data is: Raw: 506 / 3 =3D 167 MB / day Bin: 723 / 3 =3D 241 MB / day If, say, you don't want any raw data and you do want 30 days data for=20 some metrics, 10 days for others, and none for the rest. It should be=20 easy to calculate your max. size. Or work backwards from the space you=20 have to see how much data you can store. Hope this is of some use, Regards Ben Sand Philipp wrote: > No, table space isn't define as ~3GB, it's growing constantly. Since my= first mail, the database size has increased to 3,4GB... > Here's some Data out of the mysql database: > mysql> SHOW TABLE STATUS from nagios like 'perfdata_service_bin'; > | Name | Type | Row_format | Rows | Avg_row_length= | Data_length | Max_data_length | Index_length | Data_free | Auto_increm= ent | Create_time | Update_time | Check_time | Create_options | Comment =20 > | perfdata_service_bin | InnoDB | Dynamic | 7005182 | 104= | 731906048 | NULL | 1695547392 | 0 | 6971= 161 | NULL | NULL | NULL | | InnoDB fr= ee: 4096 kB; (`host_name` `service_description`) REFER `nagios/perfdata_s= ervice`(`host_name` `service_description`); (`host_name` `service_descrip= tion` `metric`) REFER `nagios/perfdata_service_metric`(`host_name` `servi= ce_description` `metric`) |=20 >=20 > So the Data lenght of the perfdata_service_bin is already at 730MB and = the index size is 1.6GB ... In 3Days! >=20 > Here's the output for the raw table: > | perfdata_service_raw | InnoDB | Dynamic | 3730416 | 135= | 506462208 | NULL | 401457152 | 0 | 3611= 239 | NULL | NULL | NULL | | InnoDB fr= ee: 4096 kB; (`host_name` `service_description`) REFER `nagios/perfdata_s= ervice`(`host_name` `service_description`) | >=20 > The serviceperf.log has only 164MB of size and hasn't been deleted sinc= e Tuesday! >=20 > Regards, Philipp >=20 >=20 >>-----Original Message----- >>From: Ben Clewett [mailto:Be...@cl...]=20 >>Sent: Friday, October 22, 2004 11:13 AM >>To: Sand Philipp >>Cc: per...@li... >>Subject: Re: [Perfparse-users] DB Size grows very fast since v101.01 >> >>Phillipp, >> >>Nothing new added, sorry. >> >>The innodb table space is not linear. For historical reasons=20 >>it uses large files of a fixed size. The size is defined in=20 >>the my.cfg (I believe). My guess is that your table space=20 >>size in this file is defined as ~3GB. Or there is some other=20 >>similar logical reason. >> >>Please check as well your log files. These can grow faster=20 >>than the table space. >> >>Regards, Ben. >> >> >>Sand Philipp wrote: >> >> >>>Hello, >>> >>>I created a brand new innodb Database since upgrading to=20 >> >>101.01. In this db I insert data for ~300 hosts ~2000 metrics.=20 >> >>>This Database has been created this Tuesday (10/19/2004)=20 >> >>and it's already 3GB huge today (10/22/2004). With version=20 >>100 and earlier I had a database with the same amount of=20 >>hosts and metrics and had a Database size of about 8GB, but=20 >>this had been running for a few months! >> >>>Had there been some fields modifications in the database? >>> >>>Regards, >>>Philipp >>> >>>_____________________________ >>> >>>Philipp Sand >>>OC-CC-TEC-SYS >>> >>>SYCOR GmbH >>>Heinrich-von-Stephan-Stra=DFe 1-5 >>>D - 37073 G=F6ttingen >>> >>>Telefon +49 (0) 551 / 490 - 0 >>>Telefax +49 (0) 551 / 490 - 232468 >>> >>>phi...@sy... >>>www.sycor.de >>>------------------------------------------------ >>> >>> >>> >>>************************************************ >>> >>>sycor plastics - die neue Branchenl=F6sung f=FCr die Kunststoffindustr= ie >>> >>>www.sycor-plastics.de >>> >>>************************************************ >>> >>> >>> >>> >>>Diese E-Mail ist vertraulich und kann dar=FCber hinaus=20 >> >>pers=F6nliche Informationen beinhalten. Wenn Sie nicht der=20 >>bestimmungsgem=E4=DFe Empf=E4nger sind, l=F6schen Sie bitte die=20 >>E-Mail und deren Anh=E4nge sofort und benachrichtigen Sie uns=20 >>dar=FCber. Die Firma sycor willigt in keine Vertr=E4ge oder=20 >>vertragliche Verpflichtungen ein oder =FCbermittelt=20 >>rechtsverbindliche Angebote, die in Form von E-Mail versandt=20 >>werden, sofern dies nicht ausdr=FCcklich in schriftlicher Form=20 >>zwischen den Parteien vereinbart wurde. >> >>>This e-mail is confidential and may contain personal and/or=20 >> >>privileged information. If you are not the intended recipient=20 >>please delete this e-mail and all attachments immediately and=20 >>inform us. The company sycor does not agree with contracts or=20 >>contract obligations sent by e-mail, neither do we transmit=20 >>legally binding offers by e-mail, unless this is not=20 >>expressly agreed upon between the parties and documented in=20 >>written form. >> >>> >>> >>>------------------------------------------------------- >>>This SF.net email is sponsored by: IT Product Guide on=20 >>>ITManagersJournal Use IT products in your business? Tell us=20 >> >>what you=20 >> >>>think of them. Give us Your Opinions, Get Free ThinkGeek Gift=20 >>>Certificates! Click to find out more=20 >>>http://productguide.itmanagersjournal.com/guidepromo.tmpl >>>_______________________________________________ >>>Perfparse-users mailing list >>>Per...@li... >>>https://lists.sourceforge.net/lists/listinfo/perfparse-users >>> >> >> >=20 >=20 > ************************************************ >=20 > sycor plastics - die neue Branchenl=F6sung f=FCr die Kunststoffindustri= e >=20 > www.sycor-plastics.de >=20 > ************************************************ >=20 >=20 >=20 >=20 > Diese E-Mail ist vertraulich und kann dar=FCber hinaus pers=F6nliche In= formationen beinhalten. Wenn Sie nicht der bestimmungsgem=E4=DFe Empf=E4n= ger sind, l=F6schen Sie bitte die E-Mail und deren Anh=E4nge sofort und b= enachrichtigen Sie uns dar=FCber. Die Firma sycor willigt in keine Vertr=E4= ge oder vertragliche Verpflichtungen ein oder =FCbermittelt rechtsverbind= liche Angebote, die in Form von E-Mail versandt werden, sofern dies nicht= ausdr=FCcklich in schriftlicher Form zwischen den Parteien vereinbart wu= rde. >=20 > This e-mail is confidential and may contain personal and/or privileged = information. If you are not the intended recipient please delete this e-m= ail and all attachments immediately and inform us. The company sycor does= not agree with contracts or contract obligations sent by e-mail, neither= do we transmit legally binding offers by e-mail, unless this is not expr= essly agreed upon between the parties and documented in written form. >=20 >=20 >=20 > ------------------------------------------------------- > This SF.net email is sponsored by: IT Product Guide on ITManagersJourna= l > Use IT products in your business? Tell us what you think of them. Give = us > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out = more > http://productguide.itmanagersjournal.com/guidepromo.tmpl > _______________________________________________ > Perfparse-users mailing list > Per...@li... > https://lists.sourceforge.net/lists/listinfo/perfparse-users >=20 |
