Re: [Valgrind-developers] performance tests in perf/ directory

[Philippe W. <phi...@sk...>]

On Wed, 2014-10-29 at 06:19 +0100, Ivo Raisr wrote:


> I would like to address this shortcoming so the results are more
> indicative.
> 
> Although vg_perf offers '--reps' option, it just chooses the best
> running time;
> 
> so it does not help here.
> 
> 
> One possible approach would be to run a test as many times as required
> to cover let's say 1 second during native run. And so many runs will
> be used
> for running instrumented. However this approach will measure mainly
> the
> 
> start program sequence in case of short tests.
For sure, the perf tests are far to be perfect.
A.o., as you say, the perf ratio with native run is heavily skewed by
small variations.
We also just have significant variations in a test (e.g. memcheck)
between multiple runs.
So, would be nice to have something better.
> 
> 
> So I was thinking about different approach where the running time of
> tests will be first
> 
> normalized (increasing or reducing work) so they finish roughly in the
> same time for
> 
> native run. All tests will also accept command line option for the
> number of main iterations
> to be performed. When vg_perf starts, it will measure time required to
> perform 1 iteration
> of a representative test. And according to this measurement, it will
> adjust automatically
> 
> the number of iterations passed to each test so that every native test
> runs roughly let's
> say for 1 second.

This approach will have as disadvantage that we cannot compare
anymore different platforms together.
Also, to keep the performance ratio correct, it means that the
nr of iterations needed will have to be propagated from the
native run to the run under the various tools. The perf tests
will then run even more slowly (e.g. on ppc64, the perf tests takes
about one hour).

So, I am not too sure about what to do.
Typically, "rich" projects similar to valgrind have access to
tests such as "SPEC" and use that to compare to valgrind.
But these tests are to my knowledge not free (both in terms
of free beer and free speech).

Does someone know about good free perf tests ?

> 
> 
> I am writing on valgrind-developers and not submitting a bug report
> because
> 
> I would like to solicit opinions first. And because bug reports seem
> to be unhandled
> and forgotten.
True, but posts on valgrind developers tend also to be forgotten :).

Philippe