|
From: Philippe W. <phi...@sk...> - 2013-09-12 22:10:50
Attachments:
diffs.txt.gz
|
Find below a patch to have nightly producing perf tests results.
At the end, you find an example mail sent on gcc110.
If no comments, I will commit in a few days.
I wonder about the reproducability of the results:
we might need bigger --reps value (but that will take a lot of cpu)
or we might rather just run perf tests with the new
(and let the html page make the delta)
or we might increase --reps but also decrease the nr of tools
or ....
Marc, I guess the perf output is parsable without too much difficulties,
but of course each "nightly regression owner" might change the setup,
e.g. to run with more (or less) tools, only with valgrind-new, or ...
Note that the patch is such that by default, no perf test are run.
We might decide to instead run the perf tests by default,
and have "nightly regression owner" opt out using the configuration.
Philippe
Index: nightly/bin/nightly
===================================================================
--- nightly/bin/nightly (revision 13542)
+++ nightly/bin/nightly (working copy)
@@ -116,7 +116,14 @@ if [ "${ABT_RUN_REGTEST}" = "" ]; then
ABT_RUN_REGTEST="make regtest"
fi
+if [ "${ABT_PERF_TOOLS}" = "" ]; then
+ ABT_PERF_TOOLS="--tools=none,memcheck,callgrind,helgrind,cachegrind,drd,massif"
+fi
+if [ "${ABT_PERF_REPS}" = "" ]; then
+ ABT_PERF_REPS="--reps=3"
+fi
+
#----------------------------------------------------------------------------
# Check out, build, test
#----------------------------------------------------------------------------
@@ -184,6 +191,14 @@ for logfile in old new ; do
)
done
+# if requested, run regression tests and produce results in perflogfile.out
+if [ "${ABT_PERF}" != "" ]; then
+ cd valgrind-new
+ perl perf/vg_perf ${ABT_PERF_TOOLS} ${ABT_PERF_REPS} ${ABT_PERF} perf > ../perflogfile 2>&1
+ cd ..
+fi
+
+
#----------------------------------------------------------------------------
# Prepare results and send
#----------------------------------------------------------------------------
@@ -257,6 +272,11 @@ if [ $changed != 0 ] ; then
echo >> final
fi
+# add perf results if requested
+if [ "${ABT_PERF}" != "" ]; then
+ cat perflogfile >> final
+fi
+
# Gather up the diffs (at most the first 100 lines for each one) into a
# single file.
MAX_LINES=100
Index: nightly/README.txt
===================================================================
--- nightly/README.txt (revision 13542)
+++ nightly/README.txt (working copy)
@@ -86,6 +86,18 @@ To set up nightly testing for a machine, do the fo
- ABT_JOBS: allows parallel builds -- it's passed as the argument to
"make -j" when building Valgrind and the tests. The default is 1.
+ - ABT_PERF: unset or set to "" mean 'do not run perf tests' (default value)
+ set to "--vg=../valgrind-new" (run perf tests for new tree)
+ set to "--vg=../valgrind-new --vg=../valgrind-old"
+ (run perf tests for "new" and for "24 hours ago",
+ to compare the performances between the 2 trees).
+
+ - ABT_PERF_TOOLS: --tools=.... option of perf/vg_perf.
+ (default value: all non experimental tools)
+
+ - ABT_PERF_REPS: --reps=... option of perf/vg_perf
+ (default value: --reps=3)
+
Note that the appropriate syntax to use in this file will depend on the
shell from which the $DIR/bin/nightly script is run (which in turn may
depend on what shell is used by cron or any similar program).
-------- Forwarded Message --------
From: Philippe Waroquiers <phi...@sk...>
To: phi...@sk...
Subject: (unchanged) 2013-09-12 12:15:48 PDT nightly build (gcc110,
Fedora release 18 (Spherical Cow), ppc64)
Date: Thu, 12 Sep 2013 13:54:22 -0700
valgrind revision: 13544
VEX revision: 2760
C compiler: gcc (GCC) 4.7.2 20121109 (Red Hat 4.7.2-8)
GDB: GNU gdb (GDB) Fedora (7.5.1-37.fc18)
Assembler: GNU assembler version 2.23.51.0.1-7.fc18 20120806
C library: GNU C Library stable release version 2.16
uname -mrs: Linux 3.7.2-204.fc18.ppc64 ppc64
Vendor version: Fedora release 18 (Spherical Cow)
Nightly build on gcc110 ( Fedora release 18 (Spherical Cow), ppc64 )
Started at 2013-09-12 12:15:48 PDT
Ended at 2013-09-12 13:54:21 PDT
Results unchanged from 24 hours ago
Checking out valgrind source tree ... done
Configuring valgrind ... done
Building valgrind ... done
Running regression tests ... failed
Regression test results follow
== 563 tests, 33 stderr failures, 5 stdout failures, 0 stderrB failures, 0 stdoutB failures, 2 post failures ==
memcheck/tests/linux/getregset (stdout)
memcheck/tests/linux/getregset (stderr)
memcheck/tests/ppc64/power_ISA2_05 (stdout)
memcheck/tests/supp_unknown (stderr)
memcheck/tests/varinfo6 (stderr)
memcheck/tests/wrap8 (stdout)
memcheck/tests/wrap8 (stderr)
massif/tests/big-alloc (post)
massif/tests/deep-D (post)
none/tests/ppc32/jm-misc (stdout)
none/tests/ppc32/jm-misc (stderr)
none/tests/ppc64/jm-misc (stdout)
none/tests/ppc64/jm-misc (stderr)
helgrind/tests/annotate_rwlock (stderr)
helgrind/tests/free_is_write (stderr)
helgrind/tests/hg02_deadlock (stderr)
helgrind/tests/hg03_inherit (stderr)
helgrind/tests/hg04_race (stderr)
helgrind/tests/hg05_race2 (stderr)
helgrind/tests/locked_vs_unlocked1_fwd (stderr)
helgrind/tests/locked_vs_unlocked1_rev (stderr)
helgrind/tests/locked_vs_unlocked2 (stderr)
helgrind/tests/locked_vs_unlocked3 (stderr)
helgrind/tests/pth_barrier1 (stderr)
helgrind/tests/pth_barrier2 (stderr)
helgrind/tests/pth_barrier3 (stderr)
helgrind/tests/pth_destroy_cond (stderr)
helgrind/tests/rwlock_race (stderr)
helgrind/tests/tc01_simple_race (stderr)
helgrind/tests/tc05_simple_race (stderr)
helgrind/tests/tc06_two_races (stderr)
helgrind/tests/tc06_two_races_xml (stderr)
helgrind/tests/tc09_bad_unlock (stderr)
helgrind/tests/tc14_laog_dinphils (stderr)
helgrind/tests/tc16_byterace (stderr)
helgrind/tests/tc18_semabuse (stderr)
helgrind/tests/tc19_shadowmem (stderr)
helgrind/tests/tc20_verifywrap (stderr)
helgrind/tests/tc21_pthonce (stderr)
helgrind/tests/tc22_exit_w_lock (stderr)
-- Running tests in perf ----------------------------------------------
-- bigcode1 --
bigcode1 valgrind-new:0.22s no: 1.6s ( 7.1x, -----) me: 2.8s (12.9x, -----) ca:18.1s (82.1x, -----) he: 1.7s ( 7.7x, -----) ca: 5.3s (24.3x, -----) dr: 1.6s ( 7.4x, -----) ma: 2.1s ( 9.6x, -----)
bigcode1 valgrind-old:0.22s no: 1.7s ( 7.7x, -8.3%) me: 3.1s (14.2x, -9.9%) ca:17.7s (80.4x, 2.0%) he: 1.7s ( 7.9x, -1.8%) ca: 5.3s (24.1x, 0.7%) dr: 1.6s ( 7.4x, 0.0%) ma: 2.1s ( 9.6x, 0.0%)
-- bigcode2 --
bigcode2 valgrind-new:0.21s no: 1.7s ( 8.2x, -----) me: 3.0s (14.5x, -----) ca:18.4s (87.5x, -----) he: 2.1s ( 9.9x, -----) ca: 5.5s (26.0x, -----) dr: 1.7s ( 8.0x, -----) ma: 2.3s (10.8x, -----)
bigcode2 valgrind-old:0.21s no: 1.7s ( 8.0x, 2.3%) me: 2.9s (14.0x, 3.6%) ca:17.8s (85.0x, 2.9%) he: 2.1s ( 9.9x, 0.0%) ca: 5.5s (26.0x, -0.2%) dr: 1.7s ( 8.1x, -1.2%) ma: 2.3s (10.8x, 0.0%)
-- bz2 --
bz2 valgrind-new:0.75s no: 4.8s ( 6.4x, -----) me:13.1s (17.4x, -----) ca:26.0s (34.7x, -----) he:15.0s (20.0x, -----) ca:24.8s (33.0x, -----) dr:20.1s (26.9x, -----) ma: 5.0s ( 6.6x, -----)
bz2 valgrind-old:0.75s no: 4.8s ( 6.4x, 0.6%) me:11.8s (15.7x, 9.5%) ca:25.6s (34.1x, 1.8%) he:14.7s (19.5x, 2.3%) ca:24.6s (32.8x, 0.6%) dr:20.0s (26.7x, 0.6%) ma: 4.6s ( 6.2x, 6.5%)
-- fbench --
fbench valgrind-new:0.34s no: 2.1s ( 6.2x, -----) me: 5.2s (15.4x, -----) ca: 8.4s (24.9x, -----) he: 5.2s (15.2x, -----) ca: 7.6s (22.4x, -----) dr: 4.8s (14.2x, -----) ma: 2.1s ( 6.2x, -----)
fbench valgrind-old:0.34s no: 2.1s ( 6.2x, -0.5%) me: 5.3s (15.5x, -0.6%) ca: 8.3s (24.6x, 1.2%) he: 5.2s (15.2x, 0.2%) ca: 7.5s (22.0x, 2.0%) dr: 4.9s (14.5x, -2.3%) ma: 2.1s ( 6.3x, -1.4%)
-- ffbench --
ffbench valgrind-new:0.45s no: 1.0s ( 2.3x, -----) me: 2.5s ( 5.6x, -----) ca: 2.5s ( 5.6x, -----) he: 6.9s (15.3x, -----) ca: 7.0s (15.6x, -----) dr: 5.2s (11.6x, -----) ma: 1.0s ( 2.3x, -----)
ffbench valgrind-old:0.45s no: 1.0s ( 2.3x, 1.0%) me: 2.6s ( 5.7x, -1.6%) ca: 2.5s ( 5.6x, 0.8%) he: 6.8s (15.1x, 1.5%) ca: 7.2s (16.0x, -3.0%) dr: 5.3s (11.9x, -2.1%) ma: 1.0s ( 2.3x, 1.0%)
-- heap --
heap valgrind-new:0.41s no: 2.4s ( 5.9x, -----) me: 9.9s (24.2x, -----) ca:13.3s (32.4x, -----) he:12.8s (31.3x, -----) ca:12.2s (29.7x, -----) dr: 7.9s (19.3x, -----) ma: 8.5s (20.7x, -----)
heap valgrind-old:0.41s no: 2.5s ( 6.0x, -3.3%) me:10.0s (24.4x, -0.6%) ca:13.3s (32.4x, 0.1%) he:13.0s (31.8x, -1.4%) ca:12.1s (29.4x, 1.1%) dr: 7.8s (19.1x, 1.0%) ma: 8.7s (21.1x, -2.1%)
-- heap_pdb4 --
heap_pdb4 valgrind-new:0.43s no: 2.5s ( 5.9x, -----) me:13.8s (32.2x, -----) ca:14.2s (33.0x, -----) he:14.6s (34.0x, -----) ca:13.3s (31.0x, -----) dr: 9.1s (21.1x, -----) ma: 8.7s (20.2x, -----)
heap_pdb4 valgrind-old:0.43s no: 2.5s ( 5.9x, -1.2%) me:14.1s (32.8x, -1.9%) ca:14.2s (33.0x, -0.1%) he:14.5s (33.7x, 1.1%) ca:13.3s (31.0x, 0.3%) dr: 8.8s (20.5x, 3.0%) ma: 8.6s (20.0x, 0.6%)
-- many-loss-records --
many-loss-records valgrind-new:0.03s no: 0.5s (17.7x, -----) me: 2.2s (72.3x, -----) ca: 1.9s (62.7x, -----) he: 2.0s (66.3x, -----) ca: 1.9s (62.0x, -----) dr: 1.5s (50.0x, -----) ma: 1.6s (52.0x, -----)
many-loss-records valgrind-old:0.03s no: 0.5s (17.3x, 1.9%) me: 2.2s (72.7x, -0.5%) ca: 1.9s (62.0x, 1.1%) he: 2.0s (66.7x, -0.5%) ca: 1.9s (62.7x, -1.1%) dr: 1.5s (49.3x, 1.3%) ma: 1.6s (53.7x, -3.2%)
-- many-xpts --
many-xpts valgrind-new:0.06s no: 0.7s (12.3x, -----) me: 3.4s (56.5x, -----) ca: 4.7s (78.3x, -----) he: 5.0s (83.2x, -----) ca: 2.9s (48.2x, -----) dr: 2.2s (37.3x, -----) ma: 2.3s (37.8x, -----)
many-xpts valgrind-old:0.06s no: 0.7s (12.2x, 1.4%) me: 3.4s (57.0x, -0.9%) ca: 4.7s (77.5x, 1.1%) he: 5.1s (84.3x, -1.4%) ca: 2.9s (48.0x, 0.3%) dr: 2.2s (37.3x, 0.0%) ma: 2.4s (39.5x, -4.4%)
-- sarp --
sarp valgrind-new:0.02s no: 0.4s (20.0x, -----) me: 3.6s (179.5x, -----) ca: 2.9s (146.5x, -----) he:10.8s (537.5x, -----) ca: 1.6s (82.0x, -----) dr: 1.0s (49.5x, -----) ma: 0.4s (21.0x, -----)
sarp valgrind-old:0.02s no: 0.4s (19.5x, 2.5%) me: 3.6s (179.0x, 0.3%) ca: 2.9s (146.5x, 0.0%) he:10.9s (546.0x, -1.6%) ca: 1.7s (85.5x, -4.3%) dr: 1.0s (51.0x, -3.0%) ma: 0.4s (21.0x, 0.0%)
-- tinycc --
tinycc valgrind-new:0.27s no: 3.0s (11.0x, -----) me:14.2s (52.6x, -----) ca:17.2s (63.7x, -----) he:18.8s (69.6x, -----) ca:15.6s (57.6x, -----) dr:12.1s (44.8x, -----) ma: 3.8s (14.1x, -----)
tinycc valgrind-old:0.27s no: 3.0s (11.0x, -0.3%) me:14.5s (53.7x, -2.1%) ca:27.6s (102.4x,-60.8%) he:28.2s (104.3x,-50.0%) ca:20.1s (74.6x,-29.4%) dr:14.4s (53.4x,-19.2%) ma: 4.2s (15.7x,-11.5%)
-- Finished tests in perf ----------------------------------------------
== 11 programs, 154 timings =================
|
|
From: marc b. <mar...@gm...> - 2013-09-15 19:21:18
|
Hi Philippe,
Yes this should be easily parsable.
We have to discuss what we want the html generator to compare/generate.
Cheers,
Marc
2013/9/13 Philippe Waroquiers <phi...@sk...>
> Find below a patch to have nightly producing perf tests results.
> At the end, you find an example mail sent on gcc110.
>
> If no comments, I will commit in a few days.
> I wonder about the reproducability of the results:
> we might need bigger --reps value (but that will take a lot of cpu)
> or we might rather just run perf tests with the new
> (and let the html page make the delta)
> or we might increase --reps but also decrease the nr of tools
> or ....
>
> Marc, I guess the perf output is parsable without too much difficulties,
> but of course each "nightly regression owner" might change the setup,
> e.g. to run with more (or less) tools, only with valgrind-new, or ...
>
> Note that the patch is such that by default, no perf test are run.
> We might decide to instead run the perf tests by default,
> and have "nightly regression owner" opt out using the configuration.
>
> Philippe
>
> Index: nightly/bin/nightly
> ===================================================================
> --- nightly/bin/nightly (revision 13542)
> +++ nightly/bin/nightly (working copy)
> @@ -116,7 +116,14 @@ if [ "${ABT_RUN_REGTEST}" = "" ]; then
> ABT_RUN_REGTEST="make regtest"
> fi
>
> +if [ "${ABT_PERF_TOOLS}" = "" ]; then
> +
> ABT_PERF_TOOLS="--tools=none,memcheck,callgrind,helgrind,cachegrind,drd,massif"
> +fi
> +if [ "${ABT_PERF_REPS}" = "" ]; then
> + ABT_PERF_REPS="--reps=3"
> +fi
>
> +
>
> #----------------------------------------------------------------------------
> # Check out, build, test
>
> #----------------------------------------------------------------------------
> @@ -184,6 +191,14 @@ for logfile in old new ; do
> )
> done
>
> +# if requested, run regression tests and produce results in
> perflogfile.out
> +if [ "${ABT_PERF}" != "" ]; then
> + cd valgrind-new
> + perl perf/vg_perf ${ABT_PERF_TOOLS} ${ABT_PERF_REPS} ${ABT_PERF} perf
> > ../perflogfile 2>&1
> + cd ..
> +fi
> +
> +
>
> #----------------------------------------------------------------------------
> # Prepare results and send
>
> #----------------------------------------------------------------------------
> @@ -257,6 +272,11 @@ if [ $changed != 0 ] ; then
> echo >> final
> fi
>
> +# add perf results if requested
> +if [ "${ABT_PERF}" != "" ]; then
> + cat perflogfile >> final
> +fi
> +
> # Gather up the diffs (at most the first 100 lines for each one) into a
> # single file.
> MAX_LINES=100
> Index: nightly/README.txt
> ===================================================================
> --- nightly/README.txt (revision 13542)
> +++ nightly/README.txt (working copy)
> @@ -86,6 +86,18 @@ To set up nightly testing for a machine, do the fo
> - ABT_JOBS: allows parallel builds -- it's passed as the argument to
> "make -j" when building Valgrind and the tests. The default is 1.
>
> + - ABT_PERF: unset or set to "" mean 'do not run perf tests' (default
> value)
> + set to "--vg=../valgrind-new" (run perf tests for new
> tree)
> + set to "--vg=../valgrind-new --vg=../valgrind-old"
> + (run perf tests for "new" and for "24 hours ago",
> + to compare the performances between the 2 trees).
> +
> + - ABT_PERF_TOOLS: --tools=.... option of perf/vg_perf.
> + (default value: all non experimental tools)
> +
> + - ABT_PERF_REPS: --reps=... option of perf/vg_perf
> + (default value: --reps=3)
> +
> Note that the appropriate syntax to use in this file will depend on
> the
> shell from which the $DIR/bin/nightly script is run (which in turn may
> depend on what shell is used by cron or any similar program).
>
>
>
>
> -------- Forwarded Message --------
> From: Philippe Waroquiers <phi...@sk...>
> To: phi...@sk...
> Subject: (unchanged) 2013-09-12 12:15:48 PDT nightly build (gcc110,
> Fedora release 18 (Spherical Cow), ppc64)
> Date: Thu, 12 Sep 2013 13:54:22 -0700
>
> valgrind revision: 13544
> VEX revision: 2760
> C compiler: gcc (GCC) 4.7.2 20121109 (Red Hat 4.7.2-8)
> GDB: GNU gdb (GDB) Fedora (7.5.1-37.fc18)
> Assembler: GNU assembler version 2.23.51.0.1-7.fc18 20120806
> C library: GNU C Library stable release version 2.16
> uname -mrs: Linux 3.7.2-204.fc18.ppc64 ppc64
> Vendor version: Fedora release 18 (Spherical Cow)
>
> Nightly build on gcc110 ( Fedora release 18 (Spherical Cow), ppc64 )
> Started at 2013-09-12 12:15:48 PDT
> Ended at 2013-09-12 13:54:21 PDT
> Results unchanged from 24 hours ago
>
> Checking out valgrind source tree ... done
> Configuring valgrind ... done
> Building valgrind ... done
> Running regression tests ... failed
>
> Regression test results follow
>
> == 563 tests, 33 stderr failures, 5 stdout failures, 0 stderrB failures, 0
> stdoutB failures, 2 post failures ==
> memcheck/tests/linux/getregset (stdout)
> memcheck/tests/linux/getregset (stderr)
> memcheck/tests/ppc64/power_ISA2_05 (stdout)
> memcheck/tests/supp_unknown (stderr)
> memcheck/tests/varinfo6 (stderr)
> memcheck/tests/wrap8 (stdout)
> memcheck/tests/wrap8 (stderr)
> massif/tests/big-alloc (post)
> massif/tests/deep-D (post)
> none/tests/ppc32/jm-misc (stdout)
> none/tests/ppc32/jm-misc (stderr)
> none/tests/ppc64/jm-misc (stdout)
> none/tests/ppc64/jm-misc (stderr)
> helgrind/tests/annotate_rwlock (stderr)
> helgrind/tests/free_is_write (stderr)
> helgrind/tests/hg02_deadlock (stderr)
> helgrind/tests/hg03_inherit (stderr)
> helgrind/tests/hg04_race (stderr)
> helgrind/tests/hg05_race2 (stderr)
> helgrind/tests/locked_vs_unlocked1_fwd (stderr)
> helgrind/tests/locked_vs_unlocked1_rev (stderr)
> helgrind/tests/locked_vs_unlocked2 (stderr)
> helgrind/tests/locked_vs_unlocked3 (stderr)
> helgrind/tests/pth_barrier1 (stderr)
> helgrind/tests/pth_barrier2 (stderr)
> helgrind/tests/pth_barrier3 (stderr)
> helgrind/tests/pth_destroy_cond (stderr)
> helgrind/tests/rwlock_race (stderr)
> helgrind/tests/tc01_simple_race (stderr)
> helgrind/tests/tc05_simple_race (stderr)
> helgrind/tests/tc06_two_races (stderr)
> helgrind/tests/tc06_two_races_xml (stderr)
> helgrind/tests/tc09_bad_unlock (stderr)
> helgrind/tests/tc14_laog_dinphils (stderr)
> helgrind/tests/tc16_byterace (stderr)
> helgrind/tests/tc18_semabuse (stderr)
> helgrind/tests/tc19_shadowmem (stderr)
> helgrind/tests/tc20_verifywrap (stderr)
> helgrind/tests/tc21_pthonce (stderr)
> helgrind/tests/tc22_exit_w_lock (stderr)
>
> -- Running tests in perf ----------------------------------------------
> -- bigcode1 --
> bigcode1 valgrind-new:0.22s no: 1.6s ( 7.1x, -----) me: 2.8s (12.9x,
> -----) ca:18.1s (82.1x, -----) he: 1.7s ( 7.7x, -----) ca: 5.3s (24.3x,
> -----) dr: 1.6s ( 7.4x, -----) ma: 2.1s ( 9.6x, -----)
> bigcode1 valgrind-old:0.22s no: 1.7s ( 7.7x, -8.3%) me: 3.1s (14.2x,
> -9.9%) ca:17.7s (80.4x, 2.0%) he: 1.7s ( 7.9x, -1.8%) ca: 5.3s (24.1x,
> 0.7%) dr: 1.6s ( 7.4x, 0.0%) ma: 2.1s ( 9.6x, 0.0%)
> -- bigcode2 --
> bigcode2 valgrind-new:0.21s no: 1.7s ( 8.2x, -----) me: 3.0s (14.5x,
> -----) ca:18.4s (87.5x, -----) he: 2.1s ( 9.9x, -----) ca: 5.5s (26.0x,
> -----) dr: 1.7s ( 8.0x, -----) ma: 2.3s (10.8x, -----)
> bigcode2 valgrind-old:0.21s no: 1.7s ( 8.0x, 2.3%) me: 2.9s (14.0x,
> 3.6%) ca:17.8s (85.0x, 2.9%) he: 2.1s ( 9.9x, 0.0%) ca: 5.5s (26.0x,
> -0.2%) dr: 1.7s ( 8.1x, -1.2%) ma: 2.3s (10.8x, 0.0%)
> -- bz2 --
> bz2 valgrind-new:0.75s no: 4.8s ( 6.4x, -----) me:13.1s (17.4x,
> -----) ca:26.0s (34.7x, -----) he:15.0s (20.0x, -----) ca:24.8s (33.0x,
> -----) dr:20.1s (26.9x, -----) ma: 5.0s ( 6.6x, -----)
> bz2 valgrind-old:0.75s no: 4.8s ( 6.4x, 0.6%) me:11.8s (15.7x,
> 9.5%) ca:25.6s (34.1x, 1.8%) he:14.7s (19.5x, 2.3%) ca:24.6s (32.8x,
> 0.6%) dr:20.0s (26.7x, 0.6%) ma: 4.6s ( 6.2x, 6.5%)
> -- fbench --
> fbench valgrind-new:0.34s no: 2.1s ( 6.2x, -----) me: 5.2s (15.4x,
> -----) ca: 8.4s (24.9x, -----) he: 5.2s (15.2x, -----) ca: 7.6s (22.4x,
> -----) dr: 4.8s (14.2x, -----) ma: 2.1s ( 6.2x, -----)
> fbench valgrind-old:0.34s no: 2.1s ( 6.2x, -0.5%) me: 5.3s (15.5x,
> -0.6%) ca: 8.3s (24.6x, 1.2%) he: 5.2s (15.2x, 0.2%) ca: 7.5s (22.0x,
> 2.0%) dr: 4.9s (14.5x, -2.3%) ma: 2.1s ( 6.3x, -1.4%)
> -- ffbench --
> ffbench valgrind-new:0.45s no: 1.0s ( 2.3x, -----) me: 2.5s ( 5.6x,
> -----) ca: 2.5s ( 5.6x, -----) he: 6.9s (15.3x, -----) ca: 7.0s (15.6x,
> -----) dr: 5.2s (11.6x, -----) ma: 1.0s ( 2.3x, -----)
> ffbench valgrind-old:0.45s no: 1.0s ( 2.3x, 1.0%) me: 2.6s ( 5.7x,
> -1.6%) ca: 2.5s ( 5.6x, 0.8%) he: 6.8s (15.1x, 1.5%) ca: 7.2s (16.0x,
> -3.0%) dr: 5.3s (11.9x, -2.1%) ma: 1.0s ( 2.3x, 1.0%)
> -- heap --
> heap valgrind-new:0.41s no: 2.4s ( 5.9x, -----) me: 9.9s (24.2x,
> -----) ca:13.3s (32.4x, -----) he:12.8s (31.3x, -----) ca:12.2s (29.7x,
> -----) dr: 7.9s (19.3x, -----) ma: 8.5s (20.7x, -----)
> heap valgrind-old:0.41s no: 2.5s ( 6.0x, -3.3%) me:10.0s (24.4x,
> -0.6%) ca:13.3s (32.4x, 0.1%) he:13.0s (31.8x, -1.4%) ca:12.1s (29.4x,
> 1.1%) dr: 7.8s (19.1x, 1.0%) ma: 8.7s (21.1x, -2.1%)
> -- heap_pdb4 --
> heap_pdb4 valgrind-new:0.43s no: 2.5s ( 5.9x, -----) me:13.8s (32.2x,
> -----) ca:14.2s (33.0x, -----) he:14.6s (34.0x, -----) ca:13.3s (31.0x,
> -----) dr: 9.1s (21.1x, -----) ma: 8.7s (20.2x, -----)
> heap_pdb4 valgrind-old:0.43s no: 2.5s ( 5.9x, -1.2%) me:14.1s (32.8x,
> -1.9%) ca:14.2s (33.0x, -0.1%) he:14.5s (33.7x, 1.1%) ca:13.3s (31.0x,
> 0.3%) dr: 8.8s (20.5x, 3.0%) ma: 8.6s (20.0x, 0.6%)
> -- many-loss-records --
> many-loss-records valgrind-new:0.03s no: 0.5s (17.7x, -----) me: 2.2s
> (72.3x, -----) ca: 1.9s (62.7x, -----) he: 2.0s (66.3x, -----) ca: 1.9s
> (62.0x, -----) dr: 1.5s (50.0x, -----) ma: 1.6s (52.0x, -----)
> many-loss-records valgrind-old:0.03s no: 0.5s (17.3x, 1.9%) me: 2.2s
> (72.7x, -0.5%) ca: 1.9s (62.0x, 1.1%) he: 2.0s (66.7x, -0.5%) ca: 1.9s
> (62.7x, -1.1%) dr: 1.5s (49.3x, 1.3%) ma: 1.6s (53.7x, -3.2%)
> -- many-xpts --
> many-xpts valgrind-new:0.06s no: 0.7s (12.3x, -----) me: 3.4s (56.5x,
> -----) ca: 4.7s (78.3x, -----) he: 5.0s (83.2x, -----) ca: 2.9s (48.2x,
> -----) dr: 2.2s (37.3x, -----) ma: 2.3s (37.8x, -----)
> many-xpts valgrind-old:0.06s no: 0.7s (12.2x, 1.4%) me: 3.4s (57.0x,
> -0.9%) ca: 4.7s (77.5x, 1.1%) he: 5.1s (84.3x, -1.4%) ca: 2.9s (48.0x,
> 0.3%) dr: 2.2s (37.3x, 0.0%) ma: 2.4s (39.5x, -4.4%)
> -- sarp --
> sarp valgrind-new:0.02s no: 0.4s (20.0x, -----) me: 3.6s (179.5x,
> -----) ca: 2.9s (146.5x, -----) he:10.8s (537.5x, -----) ca: 1.6s
> (82.0x, -----) dr: 1.0s (49.5x, -----) ma: 0.4s (21.0x, -----)
> sarp valgrind-old:0.02s no: 0.4s (19.5x, 2.5%) me: 3.6s (179.0x,
> 0.3%) ca: 2.9s (146.5x, 0.0%) he:10.9s (546.0x, -1.6%) ca: 1.7s
> (85.5x, -4.3%) dr: 1.0s (51.0x, -3.0%) ma: 0.4s (21.0x, 0.0%)
> -- tinycc --
> tinycc valgrind-new:0.27s no: 3.0s (11.0x, -----) me:14.2s (52.6x,
> -----) ca:17.2s (63.7x, -----) he:18.8s (69.6x, -----) ca:15.6s (57.6x,
> -----) dr:12.1s (44.8x, -----) ma: 3.8s (14.1x, -----)
> tinycc valgrind-old:0.27s no: 3.0s (11.0x, -0.3%) me:14.5s (53.7x,
> -2.1%) ca:27.6s (102.4x,-60.8%) he:28.2s (104.3x,-50.0%) ca:20.1s
> (74.6x,-29.4%) dr:14.4s (53.4x,-19.2%) ma: 4.2s (15.7x,-11.5%)
> -- Finished tests in perf ----------------------------------------------
>
> == 11 programs, 154 timings =================
>
>
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. Consolidate legacy IT systems to a single system of record for IT
> 2. Standardize and globalize service processes across IT
> 3. Implement zero-touch automation to replace manual, redundant tasks
> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
> _______________________________________________
> Valgrind-developers mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-developers
>
>
|
|
From: Florian K. <fl...@ei...> - 2013-09-15 20:05:55
|
On 09/13/2013 12:10 AM, Philippe Waroquiers wrote: > Find below a patch to have nightly producing perf tests results. > At the end, you find an example mail sent on gcc110. > > If no comments, I will commit in a few days. > I wonder about the reproducability of the results: > we might need bigger --reps value (but that will take a lot of cpu) > or we might rather just run perf tests with the new > (and let the html page make the delta) > or we might increase --reps but also decrease the nr of tools > or .... > I'd say, let Marc's script take care of producing the delta to previous night and perhaps also the delta to some baseline result. That way we can increase the reps value without taking longer to run. One suggestion I'd have would be to output the --reps value at the beginning of the perf results. That way it can be extracted and shown on the webpage. Florian |
|
From: Philippe W. <phi...@sk...> - 2013-09-17 15:01:12
|
On Sun, 2013-09-15 at 22:05 +0200, Florian Krohm wrote: > On 09/13/2013 12:10 AM, Philippe Waroquiers wrote: > > Find below a patch to have nightly producing perf tests results. > > At the end, you find an example mail sent on gcc110. > > > > If no comments, I will commit in a few days. > > I wonder about the reproducability of the results: > > we might need bigger --reps value (but that will take a lot of cpu) > > or we might rather just run perf tests with the new > > (and let the html page make the delta) > > or we might increase --reps but also decrease the nr of tools > > or .... > > > > I'd say, let Marc's script take care of producing the delta to previous > night and perhaps also the delta to some baseline result. That way we > can increase the reps value without taking longer to run. Effectively, Marc's script will be able to do that (as long as the perf tests are run for new. Running also for old and doing the comparison with make perf will not arm, if cpu is can be consumed). > > One suggestion I'd have would be to output the --reps value at the > beginning of the perf results. That way it can be extracted and shown on > the webpage. Good idea. Also I am thinking to prefix with "time", e.g. time perl perf/vg_perf xxxxxx >> ../perflogfile 2>&1 to get the real/user/sys of the full run of the perf tests. Philippe |