|
From: Ivo R. <iv...@iv...> - 2014-10-29 05:19:53
|
I would like to discuss possible changes to the tests in perf/directory so the performance results are more indicative. Currently on my amd64/Linux machine the results are as follows: -- Running tests in perf ---------------------------------------------- bigcode1 valgrind-upstream:0.12s no: 1.8s (14.7x, -----) me: 3.5s (28.8x, -----) bigcode2 valgrind-upstream:0.12s no: 4.0s (33.0x, -----) me: 8.4s (70.4x, -----) bz2 valgrind-upstream:0.56s no: 1.9s ( 3.3x, -----) me: 7.3s (13.1x, -----) fbench valgrind-upstream:0.25s no: 1.1s ( 4.6x, -----) me: 4.3s (17.3x, -----) ffbench valgrind-upstream:0.24s no: 1.1s ( 4.7x, -----) me: 3.3s (13.8x, -----) heap valgrind-upstream:0.10s no: 0.8s ( 7.6x, -----) me: 5.7s (56.6x, -----) heap_pdb4 valgrind-upstream:0.10s no: 0.8s ( 8.0x, -----) me: 9.0s (90.1x, -----) many-loss-records valgrind-upstream:0.01s no: 0.3s (33.0x, -----) me: 1.6s (158.0x, -----) many-xpts valgrind-upstream:0.05s no: 0.4s ( 7.8x, -----) me: 2.1s (42.0x, -----) sarp valgrind-upstream:0.01s no: 0.4s (36.0x, -----) me: 2.5s (249.0x, -----) tinycc valgrind-upstream:0.17s no: 1.7s (10.1x, -----) me: 9.4s (55.2x, -----) -- Finished tests in perf ---------------------------------------------- Please pay attention specially at tests which native execution completes in less than 100 milliseconds. Native running time is so short that calculation of the performance speedup is heavily skewed. In particular there are 2 tests which run in less than 10 milliseconds. In these cases, the performance speedup calculation is pure guesswork. I would like to address this shortcoming so the results are more indicative. Although vg_perf offers '--reps' option, it just chooses the best running time; so it does not help here. One possible approach would be to run a test as many times as required to cover let's say 1 second during native run. And so many runs will be used for running instrumented. However this approach will measure mainly the start program sequence in case of short tests. So I was thinking about different approach where the running time of tests will be first normalized (increasing or reducing work) so they finish roughly in the same time for native run. All tests will also accept command line option for the number of main iterations to be performed. When vg_perf starts, it will measure time required to perform 1 iteration of a representative test. And according to this measurement, it will adjust automatically the number of iterations passed to each test so that every native test runs roughly let's say for 1 second. I am writing on valgrind-developers and not submitting a bug report because I would like to solicit opinions first. And because bug reports seem to be unhandled and forgotten. Let me know, I. |