Re: [Valgrind-developers] run "perf" tests as part of the nightly job

[Philippe W. <phi...@sk...>]

On Tue, 2013-09-10 at 16:37 +0200, Florian Krohm wrote:
> On 09/09/2013 11:32 PM, Philippe Waroquiers wrote:
> > On Sun, 2013-09-08 at 23:34 +0200, Florian Krohm wrote:
> 
> >> What if we run perf every day and send the results to e.g
> >> pe...@va...? Whenever mail is received at that address a little
> >> script runs that reads the perf results and collects them in some light
> >> weight "data base" that we could look at at valgrind.org/perf. 
> 
> > So, maybe the valdev archive can be seen as the DB.
> > I think you started to write a script to handle archived test results
> > and make an html page from it.
> 
> We could use the mailing list archive as the "data base". But accessing
> it is slow. I ran that script you mentioned and it took minutes to
> process 100 mails. Which means that you probably don't want to run it
> yourself to figure out whether last night's build regressed performancewise.
> Also, accessing the mailing list archive is "by number". E.g. mail
> #20000 is a certain build that happened on Sep 8, 2012. But you don't
> know that relation. So if you want to know how performance evolved since
> a particular date, you either need to do some kind of bisection to
> figure out the corresponding email number forat that date or you need to
> store this information somewhere.
> I guess a nightly cronjob on valgrind.org should do this:
> (1) process the valdev emails that came in today (which implies that we
>     keep the largest email number that was processed the previous night)
> (2) extract the performance data and write it somewhere for later query
> (3) create some web page to visualise the performance data (it will use
>     the data collected in #2)
> 
> > A colleague has started enhancing
> > this script (e.g. taking some other ideas from what wine does for
> > their night tests results).
> 
> Let's see it :)
Some significant work was done some months ago (the script is
now about 1200 lines), including I think performance optimisation as it
maintains a local directory with the messages, and produces an html
page. 
It might not be very easy to summarise the performance
(ie. many tools/many tests for many archs)

> 
> > 
> > Maybe we could have the "conf" allowing to specify:
> >   to run perf test y/n
> >   the --reps= value
> >   with comparison with the previous day y/n
> 
> That would be sufficient to do what we envision.
Ok, will look at this to at least have the nrs started to
be captured.
Script and web page will follow :).

Philippe