|
From: Dave N. <dc...@us...> - 2006-02-13 20:41:36
|
Is there a set of standard test scenarios that you've run Valgrind under other than the regression test suite? IBM is interested in making sure that Valgrind on the IBM PPC platforms is as robust as possible so I was thinking that it would be worthwhile looking for other test applications, e.g. mozilla, mail programs, standard linux utilities (calculator, spreadsheet, editors). What applications of this sort have already been run on PPC? Do you have any expected results that I can compare against when running the same application on an IBM PPC platform? Are there other applications that have only been run on x86 that I might want to try? What might I expect in trying to assess whether or not valgrind is working as expected? Should I expect to have to create lots of suppressions to limit the volume of spurious/known errors from valgrind? Are most of the suppressions I'm going to want included in glibc-*.supp? Are there other suppression files for known problems for other applications? |
|
From: Julian S. <js...@ac...> - 2006-02-14 11:33:18
|
> Is there a set of standard test scenarios that you've run Valgrind under > other than the regression test suite? The only other one used is to run the test suite for the GNU Scientific Library v 1.6 on Valgrind. http://www.gnu.org/software/gsl. GSL 1.6 is 252000 lines of C, is very portable, and has an excellent test suite of over 100000 tests. I've found it effective in finding obscure instruction emulation bugs in both integer and FP code, and much of the reliability of vex compilation pipeline comes from testing with GSL. To make running the tests easy I made the script auxprogs/gsl16test. This is not run as part of the automated nightly builds, though. Perhaps it should. Even the quickest run of a gsl test takes about an hour. Apart from that .. nothing. Informally, we try to ensure that OpenOffice and Mozilla work prior to each release, but this is not a very methodical test. One thing to point out is that tests which run as part of the overnight builds are much more effective at keeping the system reliable than is a few one-off tests, eg you running OOo or Mozilla or whatever from time to time. Bottom line is, if you want functionality to work reliably, you need to have it tested every night, on all platforms. The instruction-emulation aspects of testing Valgrind are fairly well covered. What is weaker is our testing of simulation of threads, signals, syscalls, etc. At some point in the past Jeremy Fitzhardinge was using an open source POSIX conformance suite to test the signals, threads, etc, aspects of Valgrind. That sounds like a very useful thing. In my view it would be an effective way to ensure the kind of robustness you are looking for - more effective than running standard Linux utilities, since running standard utilities gives us no real idea of what functionality is tested and what isn't. Plus we can't easily integrate that kind of thing into the test suite. J |
Julian Seward wrote: >>Is there a set of standard test scenarios that you've run Valgrind under >> other than the regression test suite? > > > The only other one used is to run the test suite for the GNU Scientific > Library v 1.6 on Valgrind. http://www.gnu.org/software/gsl. ... The internal testsuite for glibc C runtime library can be adapted. Nearly all the tests (some hundreds) run .../elf/ld-linux.so.2 explicitly, and normally this is a symlink to the real executable .../elf/ld.so. So just sever the link, and make .../elf/ld-linux.so.2 be an executable shell script that runs valgrind .../elf/ld.so "$@" . The raw coverage is somewhat good, the coverage of typical usage is so-so. Most individual features get tested, but in stylized scenarios that can avoid the interactions that may be present in user code. I've run it a few times, and produced a false positive complaint from memcheck ("don't care" with partial undefined is not tracked correctly), an occasional hang in thread code, and some "uninit" bugs in glibc. -- |