> Because the recompilation time of one patched php is nearly 20 seconds, so if the

> sum of degradation time is more than 20 seconds, then our method of indirecting

> function will perorm worse than native regression testing.

 

You must expect performance under Valgrind to be worse than any normal kind of

test harness for regression testing.

 

Valgrind is not a substitute for normal kinds of testing, or is it a general-purpose

testing framework. Its value is that it can find errors that normal testing won't,

errors that leave the code producing correct results most of the time, but wasting

memory, using uninitialized variables whose contents are unpredictable, or other

errors depending on the tool that you use.

 

For example, the products I work on have slowdowns of x20 to x30 under Valgrind.

I don't attempt to run daily testing with Valgrind, because it would be far too slow.

Instead, as the development phase of each release comes (there are two per year)

to an end, and pre-release maintenance gets under way, I run all of the appropriate

testing once under Valgrind. This takes several weeks, and finds (some types of)

coding errors made by the developers during the development of the release. Those

get fixed in the release.

 

This method is not perfect — running all the testing every day under Valgrind would

be preferable. But the amount of hardware it would require (perhaps 50 modern

Linux machines) compared to the number of problem it would find (perhaps one per

moth) makes the perfect solution quite impractical.

 

Using Valgrind has considerably reduced the numbers of errors reported by users

that we can't reproduce, which is well worthwhile. That has been achieved using

one or two machines.  

 

--

John Dallman