Re: [Valgrind-users] Using valgrind from a test suite

[Olly B. <ol...@su...>]

On Fri, Mar 28, 2003 at 12:20:44AM +0000, Nicholas Nethercote wrote:
> On Wed, 26 Mar 2003, Olly Betts wrote:
> 
> > I had a brief discussion with Julian back in April 2002 about using
> > valgrind as part of an automated testsuite for a library.
> 
> > My idea was to add a call or two so that the testsuite could be run
> > under valgrind and the test harness could find out about problems
> > from valgrind and fail the test in question.
> 
> > The testsuite needs to be able to test if memory has been leaked (so it
> > can fail that test) and it would be useful to be able to totally
> > suppress valgrind's output for that check when running in the
> > testsuite's terse mode - it would just print "test7: FAIL (MEMORY LEAK)"
> > or something similar.
> >
> > And if a test has failed with a memory leak or valgrind error, we want
> > to be able to reset valgrind's counters so we can move onto the next
> > test and not have the same problem re-reported.
> 
> >   int VALGRIND_SILENT(int silent);
> >
> >     If silent is non-zero, suppress *all* output from valgrind.  Returns
> >     the setting prior to the call.
> >
> >   VALGRIND_READ_COUNTERS(int *errors, int *lost, int *possiblylost,
> > 			 int *reachable);
> >
> >     Read counters into pointed to variables.  A pointer can be zero if
> >     you aren't interested in a particular value.
> >
> >   VALGRIND_RESET_COUNTERS
> >
> >     Zeros error and leak counters.
> 
> I don't entirely understand what you want to do.

Sorry, I'll try to elaborate.  Let me know if anything is still not
clear...
 
> Does your entire test suite run as a single process under a single
> invocation of Valgrind?  If so do you use --trace-children for the
> individual tests?

No, the testsuite consists of over 300 tests, grouped into 3 programs.
Each test is a function.  Putting each test in a separate program isn't
feasible - when static linking is in use it would inflate the disk space
requirements to a ludicrous extent.

We could potentially run each test program many times, running one test
function each time.

> I assume your three functions are client requests...  would they be
> inserted into your test harness program, or the test programs themselves?

The test harness is linked into each test program, and calls the test
functions one after another, reporting any which fail (or are skipped
if they can't be run for some reason).  The test harness will also
catch unexpected exceptions and signals.

The 3 client requests would be used in that test harness.

> I guess you want to do something like: run valgrind in silent mode, after
> each test read the counters to see if any leaks occurred and if so print
> out the "terse" failure message, then reset them before going onto the
> next test.

Indeed.  And if the testsuite was run in "verbose" mode (which you
generally use with it running just one test function) it wouldn't put
valgrind into silent mode.

> I would have thought it better for your test harness to run each test
> program under Valgrind separately, maybe using --quiet (although leak
> reports still get printed), and then maybe parse the output to see if any
> leaks occurred.  Then counters wouldn't need to be reset at all.  But
> maybe I'm missing something about what you want to do.

That would require us to run the test program once for each test
function.  Doable, but it feels rather clumsy.  I also worry about the
speed - just running the test programs under valgrind is an excuse for
a teabreak.  If we rerun under valgrind for each testcase that'll incur
valgrind retranslating the common code (test harness and library) 300+
extra times...

Cheers,
    Olly