|
From: <sv...@va...> - 2005-12-15 19:52:41
|
Author: njn Date: 2005-12-15 19:52:37 +0000 (Thu, 15 Dec 2005) New Revision: 258 Log: update projects to account for recent work done on performance. Modified: trunk/help/projects.html Modified: trunk/help/projects.html =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- trunk/help/projects.html 2005-12-14 14:14:26 UTC (rev 257) +++ trunk/help/projects.html 2005-12-15 19:52:37 UTC (rev 258) @@ -26,43 +26,11 @@ =20 <h2>Software Infrastructure</h2> =20 -<h3>Profiling Valgrind</h3> -<p>Valgrind used to have a built-in tick-based profiler, but it stopped -working about two years ago and we've had no way of profiling it since, -because neither gprof nor gcov work with Valgrind.</p> +<h3>Improving the Performance Testing Suite</h3> +<p>We have a growing suite of performance benchmarks in the perf/ direct= ory, +which would benefit from more programs. We have a mix of two kinds of +programs:</p> =20 -<p>Recently self-hosting started working well enough to use Cachegrind o= n -Valgrind (see the README_DEVELOPERS file for how to do this), but no -thorough performance investigation has yet been done. This would be ver= y -valuable because Valgrind's speed is the single biggest cause of user -complaints.</p> - -<p>The only problem with using Cachegrind on Valgrind is that it's hard = to -profile big programs due to the double-slowdown. If you know of another -profiler that will work with Valgrind (OProfile perhaps?) that could be -helpful. (Added August 27, 2005; updated November 11, 2005)</p> - - -<h3>Performance regression testing</h3> -<p>We currently have some scripts to run the regression tests nightly on -a range of machines. This is very useful for spotting correctness -regressions. Equally useful would be a system for spotting performance -regressions (or improvements).</p> - -<p>This would involve running the Valgrind tools on a given suite of -programs, and recording how long they take to run. Or, perhaps better -would be recording how much slower than normal the programs run under -Valgrind; that metric would be more robust if the compiler or system -libraries on the test machine changed.</p> - -<p>The nightly measurements should be kept and ideally there would be a -system for producing graphs that show the performance changes over time. -You'd have to specify somehow where the previous measurements would be -stored, perhaps that would be a command line argument to the script.</p> - -<p>Choosing the programs for the test suite would be challenging. -Ideally we'd have a mix of two kinds of programs:</p> - <ol> <li><p>Real programs. Ones like the SPEC2000 benchmarks would be ideal, but they are not free.</p></li> @@ -74,10 +42,31 @@ performance bug in Valgrind's heap allocator.</p></li> </ol> =20 +The programs should be fairly small (preferably a few thousand lines at +most). Any new programs added should add something to the suite that th= e +current programs don't provide. Programs showing performance bugs (ie. +cases where Valgrind does very badly) are particularly welcome. +(Added August 27, 2005; updated December 15, 2005)</p> + + +<h3>Performance regression testing</h3> +<p>We currently have some scripts to run the regression tests nightly on +a range of machines. This is very useful for spotting correctness +regressions. Equally useful would be an automated system for spotting +performance regressions (or improvements).</p> + +<p>It would be nice to have scripts to run the performance benchmarks ev= ery +night. The nightly measurements should be kept and ideally there would = be a +system for producing graphs that show the performance changes over time. +You'd have to specify somehow where the previous measurements would be +stored, perhaps that would be a command line argument to the script.</p> + <p>The scripts in nightly/ for doing the nightly regression tests would = be the -right place to start on this. (Added August 27, 2005)</p> +right place to start on this. (Added August 27, 2005; updated December= 15, +2005)</p> =20 =20 + <h3>Regression test brittleness</h3> <p>Valgrind's regression test suite (run with "make regtest") is extreme= ly useful. The scripts in nightly/ are used on various test machines to |