Re: [Valgrind-developers] run "perf" tests as part of the nightly job

[Florian K. <fl...@ei...>]

On 09/09/2013 11:32 PM, Philippe Waroquiers wrote:
> On Sun, 2013-09-08 at 23:34 +0200, Florian Krohm wrote:

>> What if we run perf every day and send the results to e.g
>> pe...@va...? Whenever mail is received at that address a little
>> script runs that reads the perf results and collects them in some light
>> weight "data base" that we could look at at valgrind.org/perf. 

> So, maybe the valdev archive can be seen as the DB.
> I think you started to write a script to handle archived test results
> and make an html page from it.

We could use the mailing list archive as the "data base". But accessing
it is slow. I ran that script you mentioned and it took minutes to
process 100 mails. Which means that you probably don't want to run it
yourself to figure out whether last night's build regressed performancewise.
Also, accessing the mailing list archive is "by number". E.g. mail
#20000 is a certain build that happened on Sep 8, 2012. But you don't
know that relation. So if you want to know how performance evolved since
a particular date, you either need to do some kind of bisection to
figure out the corresponding email number forat that date or you need to
store this information somewhere.
I guess a nightly cronjob on valgrind.org should do this:
(1) process the valdev emails that came in today (which implies that we
    keep the largest email number that was processed the previous night)
(2) extract the performance data and write it somewhere for later query
(3) create some web page to visualise the performance data (it will use
    the data collected in #2)

> A colleague has started enhancing
> this script (e.g. taking some other ideas from what wine does for
> their night tests results).

Let's see it :)

> 
> Maybe we could have the "conf" allowing to specify:
>   to run perf test y/n
>   the --reps= value
>   with comparison with the previous day y/n

That would be sufficient to do what we envision.

  Florian