[Valgrind-developers] run "perf" tests as part of the nightly job

[Philippe W. <phi...@sk...>]

Every night, Valgrind is compiled and regression tested on a bunch of
platform. However, there is no continuous tracking of the performance.
=> it has been suggested to add in the nightly script a run of
the performance tests.
This mail discusses how we could set that up (in particular to decide
how much additional cpu these perf tests can take). 

On a slow computer (pentium 4):
  time perl perf/vg_perf --tools=none,memcheck,cachegrind,callgrind,helgrind,drd,exp-sgcheck,exp-dhat perf
takes 32 minutes.
On a faster computers such as gcc20, and on gcc110 (but without exp-sgcheck, crashing),
it takes about 10 minutes.
See details below.

If we want more reproducable nrs, we have to give a --reps=... arg.
Then to have a comparison with previous day, it further doubles the nr of run.
So if we use e.g. --reps=3, this would mean about 3 hours of cpu
on a slow computer, and 1 hour on faster computers.

So, a few questions:
1. how much --reps ? Is 3 ok ?
2. do we run perf for all tools ?
   or only for non-experiment tools ?
   or for even less tools (only none and memcheck ?)
3. I think we should we output the perf nrs in the short log (sent by mail)
   (after the reg tests results).
   Unclear how to make a shorter summary of these nrs.
4. How to decrease the variations ?
   Idea is to use taskset to force all these tests to run on a single
   cpu, to avoid cpu frequency scaling and process migration to badly
   interact.
5. Any other idea/suggestion/... ?

Philippe


Slow computer (pentium 4)
-- Running  tests in perf ----------------------------------------------
bigcode1 trunk_untouched:0.19s  no: 3.9s (20.3x, -----)  me: 7.0s (36.8x, -----)  ca:22.1s (116.2x, -----)  ca:37.5s (197.4x, -----)  he: 4.8s (25.2x, -----)  dr: 4.7s (24.7x, -----)  ex:13.2s (69.6x, -----)  ex: 6.3s (33.3x, -----)
bigcode2 trunk_untouched:0.20s  no: 9.2s (46.1x, -----)  me:18.7s (93.5x, -----)  ca:31.6s (158.2x, -----)  ca:62.8s (313.9x, -----)  he:12.7s (63.7x, -----)  dr:11.8s (58.9x, -----)  ex:21.8s (108.8x, -----)  ex:13.7s (68.5x, -----)
bz2      trunk_untouched:1.25s  no: 5.1s ( 4.1x, -----)  me:17.8s (14.2x, -----)  ca:77.9s (62.3x, -----)  ca:51.5s (41.2x, -----)  he:38.1s (30.5x, -----)  dr:34.7s (27.8x, -----)  ex:110.2s (88.1x, -----)  ex:23.0s (18.4x, -----)
fbench   trunk_untouched:0.66s  no: 6.0s ( 9.0x, -----)  me:13.2s (19.9x, -----)  ca:27.6s (41.8x, -----)  ca:21.0s (31.8x, -----)  he:14.9s (22.6x, -----)  dr:11.2s (17.0x, -----)  ex:48.0s (72.7x, -----)  ex:10.3s (15.7x, -----)
ffbench  trunk_untouched:2.00s  no: 5.6s ( 2.8x, -----)  me: 9.8s ( 4.9x, -----)  ca:27.3s (13.7x, -----)  ca: 8.6s ( 4.3x, -----)  he:33.7s (16.9x, -----)  dr:13.8s ( 6.9x, -----)  ex:38.4s (19.2x, -----)  ex: 9.6s ( 4.8x, -----)
heap     trunk_untouched:0.23s  no: 2.3s (10.0x, -----)  me:12.8s (55.8x, -----)  ca:25.2s (109.7x, -----)  ca:24.7s (107.3x, -----)  he:31.3s (136.1x, -----)  dr:12.7s (55.1x, -----)  ex:54.9s (238.5x, -----)  ex:31.6s (137.3x, -----)
heap_pdb4 trunk_untouched:0.27s  no: 2.5s ( 9.2x, -----)  me:23.9s (88.6x, -----)  ca:27.4s (101.5x, -----)  ca:26.6s (98.5x, -----)  he:34.9s (129.4x, -----)  dr:14.0s (51.9x, -----)  ex:55.8s (206.7x, -----)  ex:36.4s (134.9x, -----)
many-loss-records trunk_untouched:0.03s  no: 0.8s (25.0x, -----)  me: 3.2s (106.0x, -----)  ca: 4.3s (142.7x, -----)  ca: 3.5s (116.3x, -----)  he: 4.6s (153.0x, -----)  dr: 2.5s (83.0x, -----)  ex:13.7s (456.3x, -----)  ex: 5.2s (171.7x, -----)
many-xpts trunk_untouched:0.09s  no: 1.1s (11.7x, -----)  me: 4.9s (54.3x, -----)  ca:11.1s (122.9x, -----)  ca: 9.6s (106.7x, -----)  he:11.2s (124.7x, -----)  dr: 4.5s (50.6x, -----)  ex:23.9s (265.6x, -----)  ex: 4.8s (52.9x, -----)
sarp     trunk_untouched:0.06s  no: 0.9s (14.8x, -----)  me: 6.2s (103.8x, -----)  ca: 5.8s (96.8x, -----)  ca: 6.0s (100.7x, -----)  he:30.0s (500.2x, -----)  dr: 2.9s (48.8x, -----)  ex:24.0s (400.2x, -----)  ex: 1.3s (21.0x, -----)
tinycc   trunk_untouched:0.43s  no: 3.7s ( 8.6x, -----)  me:26.5s (61.6x, -----)  ca:56.8s (132.2x, -----)  ca:35.8s (83.1x, -----)  he:43.7s (101.7x, -----)  dr:28.0s (65.1x, -----)  ex:101.6s (236.2x, -----)  ex:31.2s (72.5x, -----)
-- Finished tests in perf

gcc110
- Running  tests in perf ----------------------------------------------
bigcode1 trunk_untouched:0.23s  no: 1.9s ( 8.3x, -----)  me: 3.1s (13.6x, -----)  ca: 6.3s (27.5x, -----)  ca:19.1s (83.0x, -----)  he: 1.9s ( 8.3x, -----)  dr: 1.8s ( 7.6x, -----)  ex: 2.4s (10.2x, -----)
bigcode2 trunk_untouched:0.23s  no: 1.6s ( 7.2x, -----)  me: 3.0s (13.1x, -----)  ca: 5.7s (24.8x, -----)  ca:18.5s (80.3x, -----)  he: 2.1s ( 9.0x, -----)  dr: 1.7s ( 7.3x, -----)  ex: 2.3s ( 9.9x, -----)
bz2      trunk_untouched:0.73s  no: 4.5s ( 6.2x, -----)  me:12.0s (16.5x, -----)  ca:26.4s (36.1x, -----)  ca:26.8s (36.6x, -----)  he:14.8s (20.2x, -----)  dr:20.5s (28.1x, -----)  ex:13.1s (17.9x, -----)
fbench   trunk_untouched:0.34s  no: 2.2s ( 6.5x, -----)  me: 5.3s (15.7x, -----)  ca: 8.2s (24.0x, -----)  ca: 8.8s (25.7x, -----)  he: 5.4s (16.0x, -----)  dr: 5.1s (14.9x, -----)  ex: 4.8s (14.0x, -----)
ffbench  trunk_untouched:0.45s  no: 1.1s ( 2.4x, -----)  me: 2.8s ( 6.2x, -----)  ca: 7.7s (17.0x, -----)  ca: 2.6s ( 5.8x, -----)  he: 7.2s (16.0x, -----)  dr: 5.1s (11.4x, -----)  ex: 3.4s ( 7.6x, -----)
heap     trunk_untouched:0.41s  no: 2.4s ( 5.8x, -----)  me: 9.9s (24.1x, -----)  ca:12.9s (31.5x, -----)  ca:13.8s (33.8x, -----)  he:13.3s (32.6x, -----)  dr: 8.1s (19.7x, -----)  ex:17.1s (41.8x, -----)
heap_pdb4 trunk_untouched:0.42s  no: 2.8s ( 6.6x, -----)  me:14.3s (34.0x, -----)  ca:13.4s (32.0x, -----)  ca:14.7s (34.9x, -----)  he:14.6s (34.9x, -----)  dr: 9.0s (21.5x, -----)  ex:20.4s (48.5x, -----)
many-loss-records trunk_untouched:0.03s  no: 0.6s (18.3x, -----)  me: 2.2s (74.3x, -----)  ca: 1.9s (65.0x, -----)  ca: 1.9s (64.7x, -----)  he: 2.1s (70.0x, -----)  dr: 1.5s (49.7x, -----)  ex: 3.3s (109.7x, -----)
many-xpts trunk_untouched:0.07s  no: 0.8s (11.1x, -----)  me: 3.5s (49.3x, -----)  ca: 3.0s (43.4x, -----)  ca: 4.9s (69.6x, -----)  he: 5.2s (74.7x, -----)  dr: 2.3s (32.3x, -----)  ex: 2.8s (39.9x, -----)
sarp     trunk_untouched:0.02s  no: 0.4s (21.0x, -----)  me: 3.9s (192.5x, -----)  ca: 2.0s (101.0x, -----)  ca: 3.0s (151.0x, -----)  he:11.3s (565.5x, -----)  dr: 1.0s (52.0x, -----)  ex: 0.6s (31.5x, -----)
tinycc   trunk_untouched:0.28s  no: 3.1s (11.0x, -----)  me:15.9s (56.7x, -----)  ca:16.5s (59.0x, -----)  ca:17.3s (61.7x, -----)  he:18.9s (67.6x, -----)  dr:12.1s (43.0x, -----)  ex:18.6s (66.4x, -----)
-- Finished tests in perf ----------------------------------------------

== 11 programs, 77 timings =================


gcc20
-- Running  tests in perf ----------------------------------------------
bigcode1 trunk_untouched:0.12s  no: 1.9s (16.2x, -----)  me: 3.7s (30.7x, -----)  ca: 4.8s (39.8x, -----)  ca:16.5s (137.8x, -----)  he: 2.2s (18.4x, -----)  dr: 2.1s (17.4x, -----)  ex: 4.9s (40.9x, -----)  ex: 2.4s (19.8x, -----)
bigcode2 trunk_untouched:0.12s  no: 4.6s (38.5x, -----)  me: 9.5s (79.5x, -----)  ca: 8.8s (73.3x, -----)  ca:34.3s (285.6x, -----)  he: 5.6s (46.8x, -----)  dr: 5.4s (44.7x, -----)  ex: 8.5s (70.8x, -----)  ex: 6.0s (49.9x, -----)
bz2      trunk_untouched:0.70s  no: 2.2s ( 3.2x, -----)  me: 7.0s (10.0x, -----)  ca:13.0s (18.6x, -----)  ca:21.1s (30.1x, -----)  he: 9.2s (13.2x, -----)  dr:14.0s (20.0x, -----)  ex:24.9s (35.5x, -----)  ex: 7.3s (10.5x, -----)
fbench   trunk_untouched:0.28s  no: 1.3s ( 4.6x, -----)  me: 4.0s (14.3x, -----)  ca: 3.9s (13.9x, -----)  ca: 6.3s (22.4x, -----)  he: 3.4s (12.1x, -----)  dr: 2.9s (10.2x, -----)  ex:15.1s (53.9x, -----)  ex: 2.8s ( 9.9x, -----)
ffbench  trunk_untouched:0.25s  no: 1.2s ( 4.6x, -----)  me: 3.3s (13.3x, -----)  ca: 5.1s (20.3x, -----)  ca: 2.4s ( 9.4x, -----)  he: 5.4s (21.5x, -----)  dr: 3.8s (15.3x, -----)  ex: 9.7s (38.8x, -----)  ex: 2.6s (10.4x, -----)
heap     trunk_untouched:0.10s  no: 0.7s ( 7.4x, -----)  me: 5.8s (58.1x, -----)  ca: 4.3s (42.6x, -----)  ca: 7.3s (73.4x, -----)  he: 8.4s (83.6x, -----)  dr: 5.0s (49.5x, -----)  ex:17.5s (174.6x, -----)  ex:10.6s (105.8x, -----)
heap_pdb4 trunk_untouched:0.12s  no: 0.8s ( 6.8x, -----)  me: 9.8s (81.9x, -----)  ca: 4.4s (36.3x, -----)  ca: 7.8s (65.2x, -----)  he:10.5s (87.3x, -----)  dr: 5.4s (44.8x, -----)  ex:16.1s (133.8x, -----)  ex:10.9s (91.2x, -----)
many-loss-records trunk_untouched:0.02s  no: 0.3s (14.0x, -----)  me: 1.4s (71.0x, -----)  ca: 0.8s (39.0x, -----)  ca: 1.1s (57.0x, -----)  he: 1.3s (63.5x, -----)  dr: 1.0s (50.0x, -----)  ex: 4.7s (237.0x, -----)  ex: 1.9s (93.0x, -----)
many-xpts trunk_untouched:0.04s  no: 0.3s ( 8.0x, -----)  me: 2.0s (49.5x, -----)  ca: 1.1s (27.8x, -----)  ca: 3.0s (76.0x, -----)  he: 2.5s (61.5x, -----)  dr: 1.4s (35.2x, -----)  ex: 7.6s (190.0x, -----)  ex: 1.6s (40.5x, -----)
sarp     trunk_untouched:0.03s  no: 0.3s ( 9.3x, -----)  me: 2.4s (81.0x, -----)  ca: 1.0s (34.0x, -----)  ca: 2.0s (66.0x, -----)  he: 7.4s (247.3x, -----)  dr: 0.9s (29.0x, -----)  ex: 6.7s (222.7x, -----)  ex: 0.4s (14.7x, -----)
tinycc   trunk_untouched:0.26s  no: 1.6s ( 6.0x, -----)  me:10.2s (39.3x, -----)  ca: 9.2s (35.3x, -----)  ca:12.6s (48.3x, -----)  he:11.1s (42.8x, -----)  dr: 8.9s (34.3x, -----)  ex:38.5s (147.9x, -----)  ex:12.1s (46.4x, -----)
-- Finished tests in perf ----------------------------------------------

== 11 programs, 88 timings =================