|
From: Bart V. A. <bar...@gm...> - 2008-02-25 12:05:57
|
On Mon, Feb 25, 2008 at 4:31 AM, Tom Hughes <th...@cy...> wrote: > > Nightly build on dellow ( x86_64, Fedora 8 ) started at 2008-02-25 03:10:06 GMT > Results differ from 24 hours ago > ... > exp-drd/tests/pth_barrier (stderr) > ... While all exp-drd regression tests ran fine on my development system, the pth_barrier regression test failed on all systems the nightly build ran on. The result of this regression test must be that a single data race is be reported. Depending on how the Valgrind core scheduled the threads of this test, the data race is either reported in thread 2 or in thread 3. Furthermore, the details of the call stacks printed depend on the glibc library used during the test. So I would like to change this test such that only the number of errors is compared during the regression tests instead of the full output. I would like to have a facility to suppress the output of the error messages generated by a tool while keeping the error count summary. Should I implement such functionality in the Valgrind core or in the exp-drd tool ? Bart. |
|
From: Nicholas N. <nj...@cs...> - 2008-02-25 21:31:29
|
On Mon, 25 Feb 2008, Bart Van Assche wrote: > While all exp-drd regression tests ran fine on my development system, > the pth_barrier regression test failed on all systems the nightly > build ran on. The result of this regression test must be that a single > data race is be reported. Depending on how the Valgrind core scheduled > the threads of this test, the data race is either reported in thread 2 > or in thread 3. Furthermore, the details of the call stacks printed > depend on the glibc library used during the test. So I would like to > change this test such that only the number of errors is compared > during the regression tests instead of the full output. I would like > to have a facility to suppress the output of the error messages > generated by a tool while keeping the error count summary. Should I > implement such functionality in the Valgrind core or in the exp-drd > tool ? For this specific problem, it would be easier to use a filter script for the test output, to remove the information you don't want. See the comment at the top of tests/vg_regtest, particular the stuff about "stderr_filter"/"stdout_filter". Nick |
|
From: Bart V. A. <bar...@gm...> - 2008-02-26 19:15:37
|
On Mon, Feb 25, 2008 at 10:31 PM, Nicholas Nethercote <nj...@cs...> wrote: > For this specific problem, it would be easier to use a filter script for the > test output, to remove the information you don't want. See the comment at > the top of tests/vg_regtest, particular the stuff about > "stderr_filter"/"stdout_filter". Thanks for the hint, by this time I have specified a custom stderr filter for some of the tests. By the way, does the patch below make sense or is the comment in tests/vg_regtest.in correct as it is now ? Index: tests/vg_regtest.in =================================================================== --- tests/vg_regtest.in (revision 7477) +++ tests/vg_regtest.in (working copy) @@ -61,7 +61,7 @@ # Note that filters are necessary for stderr results to filter out things that # always change, eg. process id numbers. # -# Expected stdout (filtered) is kept in <test>.stdout.exp[0-9]* (can be more +# Expected stdout (filtered) is kept in <test>.stdout.exp* (can be more # than one expected output). It can be missing if it would be empty. Expected # stderr (filtered) is kept in <test>.stderr.exp*. There must be at least # one stderr.exp* file. Bart. |
|
From: Nicholas N. <nj...@cs...> - 2008-02-26 21:27:36
|
On Tue, 26 Feb 2008, Bart Van Assche wrote: > By the way, does the patch below make sense or is the comment in > tests/vg_regtest.in correct as it is now ? > > Index: tests/vg_regtest.in > =================================================================== > --- tests/vg_regtest.in (revision 7477) > +++ tests/vg_regtest.in (working copy) > @@ -61,7 +61,7 @@ > # Note that filters are necessary for stderr results to filter out things that > # always change, eg. process id numbers. > # > -# Expected stdout (filtered) is kept in <test>.stdout.exp[0-9]* (can be more > +# Expected stdout (filtered) is kept in <test>.stdout.exp* (can be more > # than one expected output). It can be missing if it would be empty. Expected > # stderr (filtered) is kept in <test>.stderr.exp*. There must be at least > # one stderr.exp* file. The comment is correct. For example, in memcheck/tests/ we have both: sh-mem-random.stdout.exp sh-mem-random.stdout.exp64 For 32- and 64-bit machines Nick |
|
From: Bart V. A. <bar...@gm...> - 2008-02-27 07:15:23
|
On Tue, Feb 26, 2008 at 10:27 PM, Nicholas Nethercote <nj...@cs...> wrote: > On Tue, 26 Feb 2008, Bart Van Assche wrote: > > > By the way, does the patch below make sense or is the comment in > > tests/vg_regtest.in correct as it is now ? > > > > Index: tests/vg_regtest.in > > =================================================================== > > --- tests/vg_regtest.in (revision 7477) > > +++ tests/vg_regtest.in (working copy) > > @@ -61,7 +61,7 @@ > > # Note that filters are necessary for stderr results to filter out things that > > # always change, eg. process id numbers. > > # > > -# Expected stdout (filtered) is kept in <test>.stdout.exp[0-9]* (can be more > > +# Expected stdout (filtered) is kept in <test>.stdout.exp* (can be more > > # than one expected output). It can be missing if it would be empty. Expected > > # stderr (filtered) is kept in <test>.stderr.exp*. There must be at least > > # one stderr.exp* file. > > The comment is correct. For example, in memcheck/tests/ we have both: > > sh-mem-random.stdout.exp > sh-mem-random.stdout.exp64 > > For 32- and 64-bit machines Maybe I should have made myself more clear. The comment in tests/vg_regtest.pl.in explains that stdout.exp files only can have a numeric suffix ([0-9]*), while stderr.exp files can have any suffix (*). If I understood the tests/vg_regtest.pl.in sourcecode correctly, both stdout.exp and stderr.exp files can now have any suffix. Is this correct ? Bart. |
|
From: Nicholas N. <nj...@cs...> - 2008-02-27 18:10:58
|
On Wed, 27 Feb 2008, Bart Van Assche wrote: > Maybe I should have made myself more clear. The comment in > tests/vg_regtest.pl.in explains that stdout.exp files only can have a > numeric suffix ([0-9]*), while stderr.exp files can have any suffix > (*). If I understood the tests/vg_regtest.pl.in sourcecode correctly, > both stdout.exp and stderr.exp files can now have any suffix. Is this > correct ? Ah, yes. I misread your first email. I've updated that comment and some related ones. Thanks. Nick |
|
From: Julian S. <js...@ac...> - 2008-02-27 10:52:28
|
> Maybe I should have made myself more clear. The comment in > tests/vg_regtest.pl.in explains that stdout.exp files only can have a > numeric suffix ([0-9]*), while stderr.exp files can have any suffix > (*). If I understood the tests/vg_regtest.pl.in sourcecode correctly, > both stdout.exp and stderr.exp files can now have any suffix. Is this > correct ? I thought I only changed the stderr.exp to allow stderr.exp*, and left the others (stdout.exp, post.exp) unchanged. But I can't remember now. A bit of poking around with svn ann / svn log might make it clearer. J |
|
From: Bart V. A. <bar...@gm...> - 2008-02-27 11:01:55
|
On Wed, Feb 27, 2008 at 11:49 AM, Julian Seward <js...@ac...> wrote: > > > Maybe I should have made myself more clear. The comment in > > tests/vg_regtest.pl.in explains that stdout.exp files only can have a > > numeric suffix ([0-9]*), while stderr.exp files can have any suffix > > (*). If I understood the tests/vg_regtest.pl.in sourcecode correctly, > > both stdout.exp and stderr.exp files can now have any suffix. Is this > > correct ? > > I thought I only changed the stderr.exp to allow stderr.exp*, and > left the others (stdout.exp, post.exp) unchanged. But I can't > remember now. A bit of poking around with svn ann / svn log might > make it clearer. The lines that recognize stderr.exp* were last changed by user 'sewardj' and the lines that recognize stdout.exp* were last changed by user 'nethercote'. Nick, is it possible that you added support for stdout.exp* but did not update the documentation in tests/vg_regtest.in ? Bart. |
|
From: Ashley P. <api...@co...> - 2008-02-27 11:24:28
|
On Wed, 2008-02-27 at 11:49 +0100, Julian Seward wrote: > > Maybe I should have made myself more clear. The comment in > > tests/vg_regtest.pl.in explains that stdout.exp files only can have a > > numeric suffix ([0-9]*), while stderr.exp files can have any suffix > > (*). If I understood the tests/vg_regtest.pl.in sourcecode correctly, > > both stdout.exp and stderr.exp files can now have any suffix. Is this > > correct ? > > I thought I only changed the stderr.exp to allow stderr.exp*, and > left the others (stdout.exp, post.exp) unchanged. But I can't > remember now. A bit of poking around with svn ann / svn log might > make it clearer. When I was playing with post-processing of valgrind output a year or so ago I did get as far as being able to run the regression tests through my script which has the potential to make the tests simpler and more robust. The basic scheme was to run valgrind in xml mode and have a output filter to convert from xml back to "standard" output. The primary purpose of this was for parallel jobs, to have a single output file per job rather than per process but it also allowed you to pick and choose which output you wanted to see, it became easy to suppress things like line number and thread id's if required. Potentially each test could have a single sample output file and a file listing which elements of the output should be invariant. The code is basically there, the problem I had was a performance one, for the input size I wanted XML::Simple was taking in excess of a minute to parse the data, this wouldn't be a problem for the tests however. Of course all this might not help drd although it would be nice to see the xml output extended to allow higher level tools to be built on top of valgrind. Ashley, |
|
From: Konstantin S. <kon...@gm...> - 2008-02-27 13:05:07
|
My $0.02 In my variant of helgrind I have a client request saying that we expect a race on a particular address. If the race is detected, the tool is silent. But if the race is *not* detected, the tool complains at the end. This is not a substitute for the current testing scheme; it is quite orthogonal. The current scheme tests that the output matches the golden file (modulo addresses, tids, etc). My scheme is not applicable to all kinds of errors that valgrind tools may emit and does not check the exact output. But it is immune to changes in the test code and compiler optimizations (inlining). And it does not require any golden file which is useful when the test suite is under development. --kcc On Wed, Feb 27, 2008 at 2:24 PM, Ashley Pittman <api...@co...> wrote: > > On Wed, 2008-02-27 at 11:49 +0100, Julian Seward wrote: > > > Maybe I should have made myself more clear. The comment in > > > tests/vg_regtest.pl.in explains that stdout.exp files only can have a > > > numeric suffix ([0-9]*), while stderr.exp files can have any suffix > > > (*). If I understood the tests/vg_regtest.pl.in sourcecode correctly, > > > both stdout.exp and stderr.exp files can now have any suffix. Is this > > > correct ? > > > > I thought I only changed the stderr.exp to allow stderr.exp*, and > > left the others (stdout.exp, post.exp) unchanged. But I can't > > remember now. A bit of poking around with svn ann / svn log might > > make it clearer. > > When I was playing with post-processing of valgrind output a year or so > ago I did get as far as being able to run the regression tests through > my script which has the potential to make the tests simpler and more > robust. > > The basic scheme was to run valgrind in xml mode and have a output > filter to convert from xml back to "standard" output. The primary > purpose of this was for parallel jobs, to have a single output file per > job rather than per process but it also allowed you to pick and choose > which output you wanted to see, it became easy to suppress things like > line number and thread id's if required. Potentially each test could > have a single sample output file and a file listing which elements of > the output should be invariant. > > The code is basically there, the problem I had was a performance one, > for the input size I wanted XML::Simple was taking in excess of a minute > to parse the data, this wouldn't be a problem for the tests however. > > Of course all this might not help drd although it would be nice to see > the xml output extended to allow higher level tools to be built on top > of valgrind. > > Ashley, > > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Valgrind-developers mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-developers > |
|
From: Nicholas N. <nj...@cs...> - 2008-02-27 18:14:30
|
On Wed, 27 Feb 2008, Konstantin Serebryany wrote: > My $0.02 > In my variant of helgrind I have a client request saying that we > expect a race on a particular address. > If the race is detected, the tool is silent. But if the race is *not* > detected, the tool complains at the end. > > This is not a substitute for the current testing scheme; it is quite > orthogonal. > > The current scheme tests that the output matches the golden file > (modulo addresses, tids, etc). > > My scheme is not applicable to all kinds of errors that valgrind tools > may emit and does not check the exact output. > But it is immune to changes in the test code and compiler > optimizations (inlining). > And it does not require any golden file which is useful when the test > suite is under development. Such a test is doable within the vg_regtest framework. The .stderr.exp file will be empty (well, there'll be two blank lines in it). If the tool complains, the .stderr.out file won't match. Or have I misunderstood what you are saying? Nick |