perfparse-users

RE: [Perfparse-users] Help Needed

[Sand P. <Phi...@sy...>]

> When the current versions are stable, we will be looking to=20
> convert the binary table in the PP database.  This will=20
> probably take our product up to version 1.0.  I need some=20
> advise from users about the best method.
>=20
> The advantages of the change are:
> - A range for the Warn and Crit values, rather than just a=20
> single value.
> - Smaller storage space needed for same data.
> - Faster access to data, bigger graphs possible.
> - A scalable solution where new information can be added easily.
Sounds good, how will the tow tables look like?

> Option 1.
> ---------
>=20
> Copy the old table to new tables and delete old table.
>=20
> Pro:
>    Fast.
>    Easy to program & test =3D> low in bugs.
> Con:
>    Will double size of MySQL database during conversion.
>      This space is very hard to recover.

I think I would prefere that one. We're storing the performance Data for
long time (estimated 3years deletion policy)
So over long terms our db will grow anyway :) Userers who store Data for
shortterms could easyly create a new Database for perfparse "1.0" and
delete the old Database when it's not needed anymore...

> Option 2.
> ---------
>=20
> Copy all data to an external text files.  Delete old table. =20
> Copy back to new tables.  Delete text files.
>=20
> Pro:
>    Fast.
>    No increase in MySQL table space.
> Con:
>    Harder to program & test =3D> bugs possible.
>    Text file size is far larger than table, which is binary.
>    Need database size again in temporary storage for text=20
> dump of table.
After messing around with mysqldum(b)p this week, i think it's to risky
to do it that way...but if you can make sure, that no data will be lost
during that progress, it would be nice, too. But I think you should
better use this time to put in some other features (like multiple
metrics in one graph :) )


> Option 3.
> ---------
>=20
> Copy over a line at a time and delete line from old table.
>=20
> Pro:
>    No extra space needed.  No increase in MySQL table space.
>    Easy to program & test =3D> low in bugs.
> Con:
>    Deleting line by line is slow.  Conversion may take several weeks.
I think thats no alternative...as you said, that would last a few weeks
for a few million lines...
Before doing that, I personally would start with a new database ...

> Finally, a plee to anybody who knows some MySQL and a little=20
> 'c'.  Can you spare some time to help develop and/or test a=20
> conversion program.=20
> You time will be welcome, and you will be credited as a developer.
I now a little sql and can read a little c ... But we're just in a ADS
Migration Process here, so I don't have very much time...but when I find
some time here, i'll try to help...But don't schedule me :)


************************************************

sycor plastics - die neue Branchenl=F6sung f=FCr die Kunststoffindustrie

www.sycor-plastics.de

************************************************




Diese E-Mail ist vertraulich und kann dar=FCber hinaus pers=F6nliche Info=
rmationen beinhalten. Wenn Sie nicht der bestimmungsgem=E4=DFe Empf=E4nge=
r sind, l=F6schen Sie bitte die E-Mail und deren Anh=E4nge sofort und ben=
achrichtigen Sie uns dar=FCber. Die Firma sycor willigt in keine Vertr=E4=
ge oder vertragliche Verpflichtungen ein oder =FCbermittelt rechtsverbind=
liche Angebote, die in Form von E-Mail versandt werden, sofern dies nicht=
 ausdr=FCcklich in schriftlicher Form zwischen den Parteien vereinbart wu=
rde.

This e-mail is confidential and may contain personal and/or privileged in=
formation. If you are not the intended recipient please delete this e-mai=
l and all attachments immediately and inform us. The company sycor does n=
ot agree with contracts or contract obligations sent by e-mail, neither d=
o we transmit legally binding offers by e-mail, unless this is not expres=
sly agreed upon between the parties and documented in written form.

RE: [Perfparse-users] Help Needed

[James O. <jo...@re...>]

As far as recovering database space from using option one, I've =
successfully
used the optimize table function in mysqlcheck to compact the tables =
and
reclaim space.  It runs relatively quickly, but you do need to have the
database quiet (which I imaging you would while upgrading perfparse =
anyway
;)

Also, one thing that I did using the latest version of mysql was to set =
up
innodb to create separate files for each table.  This makes it much =
easier
to reclaim space on a per table basis, and it keeps me from having a =
massive
file for the innodb database... now I just have big files for the
perfparse_service_bin table ;)

James

-----Original Message-----
From: Sand Philipp [mailto:Phi...@sy...]=20
Sent: Tuesday, October 26, 2004 4:07 AM
To: Ben Clewett
Cc: per...@li...
Subject: RE: [Perfparse-users] Help Needed

> When the current versions are stable, we will be looking to=20
> convert the binary table in the PP database.  This will=20
> probably take our product up to version 1.0.  I need some=20
> advise from users about the best method.
>=20
> The advantages of the change are:
> - A range for the Warn and Crit values, rather than just a=20
> single value.
> - Smaller storage space needed for same data.
> - Faster access to data, bigger graphs possible.
> - A scalable solution where new information can be added easily.
Sounds good, how will the tow tables look like?

> Option 1.
> ---------
>=20
> Copy the old table to new tables and delete old table.
>=20
> Pro:
>    Fast.
>    Easy to program & test =3D> low in bugs.
> Con:
>    Will double size of MySQL database during conversion.
>      This space is very hard to recover.

I think I would prefere that one. We're storing the performance Data =
for
long time (estimated 3years deletion policy)
So over long terms our db will grow anyway :) Userers who store Data =
for
shortterms could easyly create a new Database for perfparse "1.0" and
delete the old Database when it's not needed anymore...

> Option 2.
> ---------
>=20
> Copy all data to an external text files.  Delete old table. =20
> Copy back to new tables.  Delete text files.
>=20
> Pro:
>    Fast.
>    No increase in MySQL table space.
> Con:
>    Harder to program & test =3D> bugs possible.
>    Text file size is far larger than table, which is binary.
>    Need database size again in temporary storage for text=20
> dump of table.
After messing around with mysqldum(b)p this week, i think it's to risky
to do it that way...but if you can make sure, that no data will be lost
during that progress, it would be nice, too. But I think you should
better use this time to put in some other features (like multiple
metrics in one graph :) )


> Option 3.
> ---------
>=20
> Copy over a line at a time and delete line from old table.
>=20
> Pro:
>    No extra space needed.  No increase in MySQL table space.
>    Easy to program & test =3D> low in bugs.
> Con:
>    Deleting line by line is slow.  Conversion may take several weeks.
I think thats no alternative...as you said, that would last a few weeks
for a few million lines...
Before doing that, I personally would start with a new database ...

> Finally, a plee to anybody who knows some MySQL and a little=20
> 'c'.  Can you spare some time to help develop and/or test a=20
> conversion program.=20
> You time will be welcome, and you will be credited as a developer.
I now a little sql and can read a little c ... But we're just in a ADS
Migration Process here, so I don't have very much time...but when I =
find
some time here, i'll try to help...But don't schedule me :)


************************************************

sycor plastics - die neue Branchenl=F6sung f=FCr die =
Kunststoffindustrie

www.sycor-plastics.de

************************************************




Diese E-Mail ist vertraulich und kann dar=FCber hinaus pers=F6nliche
Informationen beinhalten. Wenn Sie nicht der bestimmungsgem=E4=DFe =
Empf=E4nger
sind, l=F6schen Sie bitte die E-Mail und deren Anh=E4nge sofort und
benachrichtigen Sie uns dar=FCber. Die Firma sycor willigt in keine =
Vertr=E4ge
oder vertragliche Verpflichtungen ein oder =FCbermittelt =
rechtsverbindliche
Angebote, die in Form von E-Mail versandt werden, sofern dies nicht
ausdr=FCcklich in schriftlicher Form zwischen den Parteien vereinbart =
wurde.

This e-mail is confidential and may contain personal and/or privileged
information. If you are not the intended recipient please delete this =
e-mail
and all attachments immediately and inform us. The company sycor does =
not
agree with contracts or contract obligations sent by e-mail, neither do =
we
transmit legally binding offers by e-mail, unless this is not expressly
agreed upon between the parties and documented in written form.



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on =
ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give =
us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out =
more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Perfparse-users mailing list
Per...@li...
https://lists.sourceforge.net/lists/listinfo/perfparse-users

RE: [Perfparse-users] Help Needed

[Yves M. <yme...@li...>]

> Also, one thing that I did using the latest version of mysql was to set=
 up
> innodb to create separate files for each table.  This makes it much eas=
ier
> to reclaim space on a per table basis, and it keeps me from having a ma=
ssive
> file for the innodb database... now I just have big files for the
> perfparse_service_bin table ;)

James,
would you mind writing a short document about how you did that, for inclu=
sion in the
install guide ?
xml docbook is preferred if you know how (OpenOffice.org can output that)=
, but html, and
plain text are accepted too :)

Yves

--=20
- Homepage    - http://ymettier.free.fr - http://www.logicacmg.com -
- GPG key     - http://ymettier.free.fr/gpg.txt                    -
- Maitretarot - http://www.nongnu.org/maitretarot/                 -
- Perfparse   - http://perfparse.sf.net/                           -

Re: [Perfparse-users] Help Needed

[Ben C. <Be...@cl...>]

Sand Philipp wrote:

> Sounds good, how will the tow tables look like?

As well as some incremental changes, the main change is based around two 
observations:

1. With the current table most data is duplicated.  Warn, Crit, Min, 
Max. are the same for thousands of rows at a time.  When the Warn and 
Crit become a range, they will increase size from 4 to 10 bytes. 
Therefore 28 bytes of duplicate information in every row.

2. Secondly, the key structure contains the metric.  Defined by 
VARCHAR(75), VARCHAR(75), VARCHAR(75).  So there is also a maximum of 
225 more bytes of duplicate data in every row.

These two problems are countered using two changes.

First a new table for all the peripheral duplicate information is 
defined.  Warn, Max etc.  There will only be a new record when there is 
a change to these values.  Linked back to the main table by an integer 
foreign key.  Reducing the field size by 24 bytes.  Quite a lot if you 
have 10e+8 rows.  If you like Entity Relationship Diagrams:

+---------------+     +---------------+
|  bin values   |>----| bin extras    |
+---------------+     +---------------+

The second change will be to assign every Metric a unique ID.  A MID. 
These will auto-increment from 1.  Each value will be indexed by this 
MID and not the massive index above.  Therefore saving another load of 
space.

The schema is not yet defined.  (Discussion should really be in the 
devel news group, but just for interested...)  But the value table will 
be something like:

TABLE perfdata_bin (
	mid UNSIGNED INT NOT NULL,
	ctime DATETIME NOT NULL,
	PRIMARY KEY (mid, ctime),
	value DOUBLE,
	extra_data UNSIGNED INT NOT NULL
) type=innodb;

You can see that the 'value' is now DOUBLE and not FLOAT.  This allows 
date/time data to be safely stored, as well as other data to which an 
extremely accurate comparison is needed.  NULL as the value can also be 
used.  This will imply a missing data sample and reflected by a gap in 
the graph.  Although not yet supported by the plugin performance output 
format.  You can also see that duplicate data is now, thankfully, 
impossible!

The extra table being:

TABLE perfdata_bin_extra (
	mid UNSIGNED INT NOT NULL,
	extra_data UNSIGNED INT AUTO_INCREMENT NOR NULL,
	PRIMARY KEY (mid, extra_data),
	...

When I get a moment, a more complete specification will be posted to the 
perfparse-devel news group.

I hope this gives you some idea,

Regards, Ben.

[Perfparse-users] New version 0.102.1

[Ben C. <Be...@cl...>]

New version 0.102.1 released.

Release of new tools, including the PerfParse Daemon, perfparsed.  See 
documentation for more details.  With thanks to Yves for excellent 
programming.

This also contains bug-fixes to previous version, including warn and 
critical storage issue.

Please report any issues.  All feedback welcome.

Regards,

Ben.

[Perfparse-users] Re: [Perfparse-devel] New version 0.102.1

[Yves M. <yme...@li...>]

> New version 0.102.1 released.

Some last minute bug fixed was tested only on Linux. It does not compile =
on Solaris
(isinf does not exist).
I'm working on that issue and Ben will probably release some 0.102.2 vers=
ion today. If
you can compile 0.102.1, there is no reason to wait for 0.102.2.


Here are some more details about 0.102 branch :

0.101.1 was a major rewrite of the code of the parser and few bugs were r=
eported.
0.102.1 takes benefit of the new code to provide a lot of new features, i=
ncluding : -
perfparse-log2file that outputs the perfdata to a file (rotation possible=
) -
perfparse-log2socket to connect to a server and send the perfdata through=
 the network -
perfparsed, a daemon that can read from a pipe (native in nagios-2.0) or =
a network
socket.
- perfparse-log2db runs in the same way as in 0.101.1.

perfparse-log2* binaries each enable one storage module.
perfparsed can enable any. Here are the modules :
- mysql : the one that you already use in perfparse-0.101.1. Some bug fix=
ed. -
output_file [new] : writes the data to a file. File rotation is possible =
with one file
per day.
- output_socket [new] : writes the data into a socket. The server can be =
perfparsed with
the socket server enabled.
- print : this is an example for developers, but is disabled by default.

Documentation is in docbook format in doc/ and a html file was generated =
there too.
Check the documentation for the new possibilities that perfparsed gives y=
ou !

Last thing : the undocumented --show_config in perfparse-log2* and perfpa=
rsed make them
print the configuration option in a format compatible with the configurat=
ion file.
Please check your default options :)

Yves



--=20
- Homepage    - http://ymettier.free.fr - http://www.logicacmg.com -
- GPG key     - http://ymettier.free.fr/gpg.txt                    -
- Maitretarot - http://www.nongnu.org/maitretarot/                 -
- Perfparse   - http://perfparse.sf.net/                           -

[Perfparse-users] New mailing list

[Ben C. <Be...@cl...>]

Dear PerfParse,

I have set up a new mailing list:

per...@li...

This will be live 6 - 24 hours from this posting.

This will be used to announce new versions and vital information, such 
as bug-fixes.  For those members publishing development versions, 
alternate PHP front ends, other tools, conversions to other databases, 
please post here.

The existing news groups will be kept as-is:

users	All general discussion and support.
devel	Technical discussion about the development of PP.

Regards, Ben.

[Perfparse-users] New version 0.104.1

[Ben C. <BCl...@pe...>]

Dear users,

We are happy to release the next version 0.104.1.  This has many 
interesting additions:

- dynamic libraries for perfparse-log2* and perfparsed

- perfparse-log2any can load more than one storage module; the other 
perfparse-log2* are links, shortcuts to perfparse-log2any with one 
module only.

- Perfparse is now internationalized, with a first French translation to 
be improved and others to be done (contributions welcome)

- perfgraph.cgi renamed as perfparse.cgi. To avoid confusions, upgrades 
should remove their old perfparse.cgi.

I will repeat the last change:  perfgraph.cgi is now renamed to 
perfparse.cgi.  Please delete perfgraph.cgi and change your links.

Contributions will be particularly appreciated on:

- translations (including improving the French one)
- helping Garry on the documentation
- porting perfparse on other systems than GNU/Linux and Solaris
- making storage modules for other databases or RRD files

Regards, PP Development team.