> Because the recompilation time of one patched php is nearly 20 seconds, so if the
> sum of degradation time is more than 20 seconds, then our method of indirecting
> function will perorm worse than native regression testing.
You must expect performance under Valgrind to be worse than any normal kind of
test harness for regression testing.
not a substitute for normal kinds of testing, or is it a general-purpose
testing framework. Its value is that it can find errors that normal testing won't,
errors that leave the code producing correct results most of the time, but wasting
memory, using uninitialized variables whose contents are unpredictable, or other
errors depending on the tool that you use.
For example, the products I work on have slowdowns of x20 to x30 under Valgrind.
I don't attempt to run daily testing with Valgrind, because it would be far too slow.
Instead, as the development phase of each release comes (there are two per year)
to an end, and pre-release maintenance gets under way, I run all of the appropriate
testing once under Valgrind. This takes several weeks, and finds (some types of)
coding errors made by the developers during the development of the release. Those
get fixed in the release.
This method is not perfect — running all the testing every day under Valgrind would
be preferable. But the amount of hardware it would require (perhaps 50 modern
Linux machines) compared to the number of problem it would find (perhaps one per
moth) makes the perfect solution quite impractical.
Using Valgrind has considerably reduced the numbers of errors reported by users
that we can't reproduce, which is well worthwhile. That has been achieved using
one or two machines.