perfparse-devel

[Perfparse-devel] Submitting work

[Tim W. <tim...@sc...>]

Hi,

I've made some additions to storage_mysql.c to provide a summary table of
raw data (availability data) to perfparse. I also added two tables,
perfdata_summary_raw and perfdata_summary_raw_data. It's rather
straightforward stuff, so I guess it should be easy to incorporate into the
official version (if you think it's useful).
I followed Ben's ideas for the summary table, so that it can contain summary
for different 'epochs' (periods).

Things on the TODO list:
- deletion policies: haven't read the docs on this yet. Are they still
uptodate?
- make summaries optional
- defaults (every service now gets a default summary with a epoch of 1 day)
- cgi report (-> I don't intend to write one. I don't really like c code
that printf's HTML ;))

And finally an important question: How do I submit my changes? I've never
used sourceforge (as a developer) before. For the moment, I just attached
the files to this email...

Feedback always welcome!

Regards,
Tim

Re: [Perfparse-devel] Submitting work

[Yves M. <yme...@li...>]

Hi Tim,


> Hi,
>
> I've made some additions to storage_mysql.c to provide a summary table =
of

Thanks for contributing :)
I will let Ben accept or reject the patch after reviewing it.

> And finally an important question: How do I submit my changes? I've nev=
er
> used sourceforge (as a developer) before. For the moment, I just attach=
ed
> the files to this email...

I don't know if Ben uses sourceforge. I think that the best way is to :
1/ send a patch (not the full file)
2/ compress your patchs and files (with respect to those who still have l=
ow speed access
to internet)
3/ describe what it does (you did it : good point :)
4/ add one line in ChangeLog : we are sure not to forget to add it. Don't=
 forget to
specify your initials :)
5/ add your name in the "Last Modified" line at the beginning. Without it=
, we consider
that you give the copyright of your work to the author of the file

Well, that's a lot to think. If you do only 1/, 2/ and 3/, this is correc=
t. 4/ and 5/
are rather for you, not for the project :)

We prefer a patch because it's easier to
- patch the current development version and not the current stable releas=
e :)
- see the changes

OK, you asked and I gave you a full (but personnal) answer to your questi=
on :)

For both Ben and you :
- I made the patch. Find it attached.
- the coding style is the same as the rest. Very good.
- the patch applies successfully on my latest development version
(http://pagesperso.laposte.net/ymettier/perfparse-devel/)
- I have not tried to understand what it does. Ben, you have to check tha=
t :)

I suggest that someone cat mysql_summary_raw.sql >> mysql_create.sql

And I don't think this will be accepted for next version because it is al=
ready ready and
Ben can release it today evening if he has time.
And there is something missing in your changes : perfparse-db-tool has to=
 be updated to
create your 2 new tables.

Maybe Ben will accept it for 0.102.2 or 0.103.1 ?

If Ben accepts, I can also accept next patches to include them in my deve=
lopment versions.
Well, when perfparse-0.102.0ym5.tar.gz can be found on
http://pagesperso.laposte.net/ymettier/perfparse-devel/, Ben can take it =
and release
0.102.1 :)

Yves




--=20
- Homepage    - http://ymettier.free.fr - http://www.logicacmg.com -
- GPG key     - http://ymettier.free.fr/gpg.txt                    -
- Maitretarot - http://www.nongnu.org/maitretarot/                 -
- Perfparse   - http://perfparse.sf.net/                           -

Re: [Perfparse-devel] Submitting work

[Ben C. <Ben...@ro...>]

Hi,

I saw my name mentioned eight times, let me add to this discussion :)

Love the code, nice work.

I totally agree about submitting patches.  There is a facility to do 
this on http://perfparse.sf.net.  I have not used this, but this might 
guarantee all of us can see the patch and gives you somewhere to post 
it.  If there are more than a few of us, might be time to us it.

In the future there will be development and live versions.  Right now I 
ask you to submit a new patch against each new release version until 
it's ready.  If you think this is a good idea!  Hopefully this will not 
be for too much time.

I think it may be too late for the current version.  I am in discussion 
with Yves about releasing over weekend.

Before this is included in the release version, we need to think about 
making the provided facilities concurrent with the existing services.

Let me list these below and we can comment on whether these are needed 
for release or can wait:

1/

We need to ensure that we have the control option on the parsing, to set 
whether the user gets this data or not.  On current methods, I suggest a:
	--no-raw-summary-data
Flag.

At the moment the single --no-raw-data flag will block both raw and raw 
summary. :)

2/

The deletion policies can wait a while.  After all, the point of these 
files is for long term storage.  But this must come at some time.

3/

You are using a fixed epoch of one day.  You comment about whether this 
is configurable.  If you decide it should be, we need a way of defining 
the used epochs, and delete those that are not needed any more.

4/

Reports.  Do you want any CGI reports?  Do you want to copy the ones 
that exist, or write your own, or would you like me to write them for 
you.  The final option may be the slowest :)

Anything else?

Anyway, busy time for me, I'll look over your code and make any coding 
comments directly with you.

Regards, Ben.




Yves Mettier wrote:

> Hi Tim,
> 
> 
> 
>>Hi,
>>
>>I've made some additions to storage_mysql.c to provide a summary table of
> 
> 
> Thanks for contributing :)
> I will let Ben accept or reject the patch after reviewing it.
> 
> 
>>And finally an important question: How do I submit my changes? I've never
>>used sourceforge (as a developer) before. For the moment, I just attached
>>the files to this email...
> 
> 
> I don't know if Ben uses sourceforge. I think that the best way is to :
> 1/ send a patch (not the full file)
> 2/ compress your patchs and files (with respect to those who still have low speed access
> to internet)
> 3/ describe what it does (you did it : good point :)
> 4/ add one line in ChangeLog : we are sure not to forget to add it. Don't forget to
> specify your initials :)
> 5/ add your name in the "Last Modified" line at the beginning. Without it, we consider
> that you give the copyright of your work to the author of the file
> 
> Well, that's a lot to think. If you do only 1/, 2/ and 3/, this is correct. 4/ and 5/
> are rather for you, not for the project :)
> 
> We prefer a patch because it's easier to
> - patch the current development version and not the current stable release :)
> - see the changes
> 
> OK, you asked and I gave you a full (but personnal) answer to your question :)
> 
> For both Ben and you :
> - I made the patch. Find it attached.
> - the coding style is the same as the rest. Very good.
> - the patch applies successfully on my latest development version
> (http://pagesperso.laposte.net/ymettier/perfparse-devel/)
> - I have not tried to understand what it does. Ben, you have to check that :)
> 
> I suggest that someone cat mysql_summary_raw.sql >> mysql_create.sql
> 
> And I don't think this will be accepted for next version because it is already ready and
> Ben can release it today evening if he has time.
> And there is something missing in your changes : perfparse-db-tool has to be updated to
> create your 2 new tables.
> 
> Maybe Ben will accept it for 0.102.2 or 0.103.1 ?
> 
> If Ben accepts, I can also accept next patches to include them in my development versions.
> Well, when perfparse-0.102.0ym5.tar.gz can be found on
> http://pagesperso.laposte.net/ymettier/perfparse-devel/, Ben can take it and release
> 0.102.1 :)
> 
> Yves
> 
> 
> 
> 

Re: [Perfparse-devel] Submitting work

[Yves M. <yme...@li...>]

> Hi,
>
> I saw my name mentioned eight times, let me add to this discussion :)

Respect to our leader :)

> I totally agree about submitting patches.  There is a facility to do
> this on http://perfparse.sf.net.  I have not used this, but this might
> guarantee all of us can see the patch and gives you somewhere to post
> it.  If there are more than a few of us, might be time to us it.

OK, so I will apply the patch for 0.103.0ymX versions and I and Tim will =
work together
to have something for 0.103.1 :)

> In the future there will be development and live versions.  Right now I

This is not future but present :)
Development versions are here :
http://pagesperso.laposte.net/ymettier/perfparse-devel/

Well, you mean official development versions :)

[...]

> Before this is included in the release version, we need to think about
> making the provided facilities concurrent with the existing services.
>
> Let me list these below and we can comment on whether these are needed
> for release or can wait:
>
> 1/
>
> We need to ensure that we have the control option on the parsing, to se=
t
> whether the user gets this data or not.  On current methods, I suggest =
a:
> 	--no-raw-summary-data
> Flag.
>
> At the moment the single --no-raw-data flag will block both raw and raw
> summary. :)

This will be a important point. When --no-raw-data is enabled (that means=
 the feature is
disabled - I hate negative options) the storage_module is not even called=
. So the
discussion will be :
do we ask the module to test --no-raw-data, or do we add a callback for s=
ummary ?

In the discussion, we will also have to think how to have raw data in one=
 module and no
raw data in another module.

This is not very simple and the impact can be large :)

> 2/
>
> The deletion policies can wait a while.  After all, the point of these
> files is for long term storage.  But this must come at some time.

I don't agree. I have a small database here. And I get a huge amount of p=
erf data.
This is not tuned (still testing nagios), but it's good to test nagios an=
d perfparse
limits :)
Well, this is my problem.
But my problem is also that I have to keep only 2 or 3 days of data.

Yves
--=20
- Homepage    - http://ymettier.free.fr - http://www.logicacmg.com -
- GPG key     - http://ymettier.free.fr/gpg.txt                    -
- Maitretarot - http://www.nongnu.org/maitretarot/                 -
- Perfparse   - http://perfparse.sf.net/                           -

Re: [Perfparse-devel] Submitting work

[Ben C. <Be...@cl...>]

Yves Mettier wrote:

>>Hi,
>>
>>I saw my name mentioned eight times, let me add to this discussion :)
> 
> 
> Respect to our leader :)

Thanks!  There are no leaders, I was just first.  Besides which there is 
not a great deal of my original code there :)

Ben

[Perfparse-devel] Summary work.

[Ben C. <Be...@cl...>]

Hi Tim,

Forgive me if I repeat anything, three way conversations, three time 
zones, one Friday afternoon!

One small fix: Can you make your delete policy default set to 'host'?

I had a look at the code.  This will be a great addition to PP.  I want 
to get this available to all users as soon as we can!

I think we are agreed that it cannot be included currently as there is 
no way to control the data, and no way to view it.  So lets sort this 
out....

-----------------------------------

The deletion policies are no haste.

The CGI code to define them needs amending.  (This code is a picture of 
a tortured mind on another Friday afternoon, rather than crafted coding 
Yves consistently contributes.)  Since you have followed the definition 
of the policies exactly, we only need to copy some code, change the 
table names, and a new menu option.

My first question Tim, do you want to have a look and see what you 
think?  Code is cgi_del_policy.c.  Need to copy:
	void displayDeletePolicyBin()
To new function:
	void displayDeletePolicySummaryBin()
As I said, ignore all the logic, just change the table names and all 
should work.  This is linked into the menu in perfgraph.c.

-----------------------------------

Then the deletion policies need to be actioned.  Again, copy similar 
code in the perfparse-db-purge code.  This should be a very simple 
operation.

-----------------------------------

Next, the --no-raw-data flag.

Yves suggested this could be another callback to store the summary data. 
  This worries me:
1. The call back will be identical to the existing one.
2. As we add more ways of analyzing data, more and more callbacks will 
be needed.

I wonder as a an alternate option with one callback:  Whether the 
attached modules should have their own option set, which is added to the 
main option set and displayed together when --help is shown, set by the 
same code.  But this is complex for a small problem :)

In the short term, I suggest making the option array visible to modules. 
   Sorry, another global variable.  They can read them and action them 
as they see fit.  In this case we can now have two flags:
	--no-raw-data
	--no-raw-summary-data
Which can be viewed in the function to store lines of data.

What do you all think?

This is one for my self or Yves to hack.

-----------------------------------

There needs to be a function to automatically amend users tables, and 
update the schema number.  See upgrade.c.  No magic there.  Also need to 
amend the mysql_create.sql and mysql_delete.sql.

This can be released now as this does not effect functioning of the 
current versions.  As I have done with the binary summary tables.

-----------------------------------

Finally the display of data.  I can think of various ways of displaying:

-	Report of activity for a day with summary.
-	Line graph showing percentage activity of each state.
-	Pie chart of summary....
-	Exported CSV or XML reports.

Tim, what do you want to see?

-----------------------------------

Regards, Ben.




Tim Wuyts wrote:

> Hi,
> 
> I've made some additions to storage_mysql.c to provide a summary table of
> raw data (availability data) to perfparse. I also added two tables,
> perfdata_summary_raw and perfdata_summary_raw_data. It's rather
> straightforward stuff, so I guess it should be easy to incorporate into the
> official version (if you think it's useful).
> I followed Ben's ideas for the summary table, so that it can contain summary
> for different 'epochs' (periods).
> 
> Things on the TODO list:
> - deletion policies: haven't read the docs on this yet. Are they still
> uptodate?
> - make summaries optional
> - defaults (every service now gets a default summary with a epoch of 1 day)
> - cgi report (-> I don't intend to write one. I don't really like c code
> that printf's HTML ;))
> 
> And finally an important question: How do I submit my changes? I've never
> used sourceforge (as a developer) before. For the moment, I just attached
> the files to this email...
> 
> Feedback always welcome!
> 
> Regards,
> Tim

Re: [Perfparse-devel] Summary work.

[Yves M. <yme...@li...>]

Hi all :)

I answer only where my comments are needed :)


> Next, the --no-raw-data flag.
>
> Yves suggested this could be another callback to store the summary data=
.
>   This worries me:
> 1. The call back will be identical to the existing one.
> 2. As we add more ways of analyzing data, more and more callbacks will
> be needed.
>
> I wonder as a an alternate option with one callback:  Whether the
> attached modules should have their own option set, which is added to th=
e
> main option set and displayed together when --help is shown, set by the
> same code.  But this is complex for a small problem :)

I have not followed the whole discussion about summary data. Maybe a new =
callback is not
necessary.

> In the short term, I suggest making the option array visible to modules=
.

This is already the case. Check where the DB_user and DB_pwd come from :)

>    Sorry, another global variable.  They can read them and action them
> as they see fit.  In this case we can now have two flags:
> 	--no-raw-data
> 	--no-raw-summary-data
> Which can be viewed in the function to store lines of data.
>
> What do you all think?

This is maybe the good way to do it. I just say "maybe" because I have no=
t followed the
whole discussion :)

> This is one for my self or Yves to hack.

config_file.c and libpp_common.h
And one additionnal line in perfparsed.c and perfparse-log2any.c to say t=
hat we use it
(used only when printing help)


> -----------------------------------
>
> Finally the display of data.  I can think of various ways of displaying=
:
>
> -	Report of activity for a day with summary.
> -	Line graph showing percentage activity of each state.
> -	Pie chart of summary....
> -	Exported CSV or XML reports.
>
> Tim, what do you want to see?

About displaying, in the far future (I hope to be there anyways), I sugge=
st a library to
compute the data, and binaries that use the library to produce files (com=
mand line
tools), html (CGI), and other good things.
This is for the far future, probably after perfparse-1.0.
The reason why I say it is if you can code with that if mind, it will may=
be be easier to
move the existing code in that library one day ?

Yves
--=20
- Homepage    - http://ymettier.free.fr - http://www.logicacmg.com -
- GPG key     - http://ymettier.free.fr/gpg.txt                    -
- Maitretarot - http://www.nongnu.org/maitretarot/                 -
- Perfparse   - http://perfparse.sf.net/                           -