[Valgrind-users] Using valgrind from a test suite

[Olly B. <ol...@su...>]

I had a brief discussion with Julian back in April 2002 about using
valgrind as part of an automated testsuite for a library.  The testsuite
code could exercise features of the library and use valgrind to detect
accesses to uninitialised data, memory leaks, mismatched calls to
malloc/new/..., etc.

My particular interest is the GPLed search engine library I work on
(http://www.xapian.org/) which is mostly C++.  We've tried catching
memory leaks using an LD_PRELOAD library to wrap malloc and free, and
other tricks to intercept new and delete, but it doesn't work well -
various STL implementations do things which mess it up, and we've
eventually given up on that approach.  And of course we can't catch
accesses to uninitialised data.  The LD_PRELOAD approach would probably
work for a C library, but the using valgrind still offers benefits.

This would only work on x86 Linux, but that's not a big problem for
libraries like ours with almost no platform specific code.  On platforms
without valgrind, or on x86 Linux without valgrind installed, the test
suite would just not notice memory leaks.

My idea was to add a call or two so that the testsuite could be run
under valgrind and the test harness could find out about problems
from valgrind and fail the test in question.

Julian asked me to propose an interface.  I made a first stab at one and
things stalled there.  I contacted Julian and Nick a few months ago.
Nick responded briefly, but I imagine they're both swamped with mail.

I'm happy to try to implement this myself, but I'd like to settle
on an interface first to avoid wasted work.  I thought it would be
worthwhile to post this to the list so at least other valgrind users
can comment on the interface.

The intervening months have at least allowed me to mull this over, and
my envisioned interface is now rather simpler and more widely useful
than either of my previous suggestions.

The testsuite needs to be able to test if memory has been leaked (so it
can fail that test) and it would be useful to be able to totally
suppress valgrind's output for that check when running in the
testsuite's terse mode - it would just print "test7: FAIL (MEMORY LEAK)"
or something similar.

And if a test has failed with a memory leak or valgrind error, we want
to be able to reset valgrind's counters so we can move onto the next
test and not have the same problem re-reported.

My revised proposed interface adds three calls:

  int VALGRIND_SILENT(int silent);

    If silent is non-zero, suppress *all* output from valgrind.  Returns
    the setting prior to the call.

  VALGRIND_READ_COUNTERS(int *errors, int *lost, int *possiblylost,
			 int *reachable);

    Read counters into pointed to variables.  A pointer can be zero if
    you aren't interested in a particular value.

  VALGRIND_RESET_COUNTERS

    Zeros error and leak counters.

Cheers,
    Olly