|
From: Jeremy F. <ja...@lc...> - 2005-06-13 22:41:45
|
Hi guys, We're planning on enhancing our daily-autobuild process to include some daily automated testing also, and it would be cool if we could have the unit tests run under valgrind so that any valgrind-detectable errors show up in the daily auto-test results. In order to implement this, I think we would need to somehow tell valgrind the following: - If any valgrind-errors are detected, immediately exit with an error code - If, at program-exit time, any memory leaks are detected, exit with an error code Is there any way to tell valgrind to behave like that? If not, could a flag of some sort be added to implement this? Currently it appears that errors messages are printed to stderr; I suppose our build script could be made to parse the stderr output but that would make things more complicated than I would like to have them. Thanks, Jeremy Friesner Level Control Systems |
|
From: Tom H. <to...@co...> - 2005-06-13 23:02:13
|
In message <200...@lc...>
Jeremy Friesner <ja...@lc...> wrote:
> In order to implement this, I think we would need to somehow tell valgrind
> the following:
>
> - If any valgrind-errors are detected, immediately exit with an error code
> - If, at program-exit time, any memory leaks are detected, exit with an error
> code
>
> Is there any way to tell valgrind to behave like that? If not, could a
> flag of some sort be added to implement this? Currently it appears that
> errors messages are printed to stderr; I suppose our build script could
> be made to parse the stderr output but that would make things more
> complicated than I would like to have them.
The exit status returned by valgrind is the exit status of the program
being run by valgrind - if valgrind were to replace that with it's own
exit status then you wouldn't know what the original exit status of the
program was.
I think most people doing this just grep the valgrind output for non-zero
error counts.
Tom
--
Tom Hughes (to...@co...)
http://www.compton.nu/
|
|
From: Nicholas N. <nj...@cs...> - 2005-06-13 23:15:00
|
On Tue, 14 Jun 2005, Tom Hughes wrote: >> In order to implement this, I think we would need to somehow tell valgrind >> the following: >> >> - If any valgrind-errors are detected, immediately exit with an error code >> - If, at program-exit time, any memory leaks are detected, exit with an error >> code >> >> Is there any way to tell valgrind to behave like that? If not, could a >> flag of some sort be added to implement this? Currently it appears that >> errors messages are printed to stderr; I suppose our build script could >> be made to parse the stderr output but that would make things more >> complicated than I would like to have them. > > The exit status returned by valgrind is the exit status of the program > being run by valgrind - if valgrind were to replace that with it's own > exit status then you wouldn't know what the original exit status of the > program was. > > I think most people doing this just grep the valgrind output for non-zero > error counts. Better is to run with -q and use the client requests like VALGRIND_COUNT_ERRORS and VALGRIND_COUNT_LEAKS within the tests. Others have certainly done this -- they may have more advice. N |
|
From: Bob R. <bo...@br...> - 2005-06-14 12:32:19
|
On Mon, Jun 13, 2005 at 06:14:53PM -0500, Nicholas Nethercote wrote: > On Tue, 14 Jun 2005, Tom Hughes wrote: > > >>In order to implement this, I think we would need to somehow tell valgrind > >>the following: > >> > >>- If any valgrind-errors are detected, immediately exit with an error code > >>- If, at program-exit time, any memory leaks are detected, exit with an > >>error > >>code > >> > >>Is there any way to tell valgrind to behave like that? If not, could a > >>flag of some sort be added to implement this? Currently it appears that > >>errors messages are printed to stderr; I suppose our build script could > >>be made to parse the stderr output but that would make things more > >>complicated than I would like to have them. > > > >The exit status returned by valgrind is the exit status of the program > >being run by valgrind - if valgrind were to replace that with it's own > >exit status then you wouldn't know what the original exit status of the > >program was. > > > >I think most people doing this just grep the valgrind output for non-zero > >error counts. > > Better is to run with -q and use the client requests like > VALGRIND_COUNT_ERRORS and VALGRIND_COUNT_LEAKS within the tests. Others > have certainly done this -- they may have more advice. Yes, this is the approach we take and it works wonderfully. Bob Rossi |
|
From: <jp...@mu...> - 2005-06-13 23:10:16
|
What about questionable leaks? I'd assert that a little 'more complicated' in the script can buy you a lot more useful information. If you don't actually ask to attach gdb, print suppressions, or somesuch, I think it reliably just runs to completion. On our project, a python unit test script runs a slew of test programs without and with valgrind. Python re.compile calls then extract the error count and various lost bytes and prints a pretty table. http://www.freeelectron.org/svn/fe/branch/20050511/test/bld.py Executable Tests: norm sg ex grnd er leak indir poss reach supp xDL_Loader.exe ................[PASS] 0 0 [PASS] 0 0000 0000 0000 0000 0188 xUnitTest.exe .................[PASS] 0 0 [PASS] 0 0000 0000 0000 0000 0039 xException.exe ................[PASS] 0 0 [PASS] 0 0000 0000 0000 0008 0167 xPlatform.exe .................[PASS] 0 0 [PASS] 0 0000 0000 0000 0000 0039 xList.exe .....................[PASS] 0 0 [PASS] 0 0000 0000 0000 0000 0041 [...] -- Jason Weber On Mon, June 13, 2005 3:38 pm, Jeremy Friesner said: > Hi guys, > > We're planning on enhancing our daily-autobuild process to include some > daily > automated testing also, and it would be cool if we could have the unit > tests > run under valgrind so that any valgrind-detectable errors show up in the > daily auto-test results. > > In order to implement this, I think we would need to somehow tell valgrind > the > following: > > - If any valgrind-errors are detected, immediately exit with an error code > - If, at program-exit time, any memory leaks are detected, exit with an > error > code > > Is there any way to tell valgrind to behave like that? If not, could a > flag > of some sort be added to implement this? Currently it appears that errors > messages are printed to stderr; I suppose our build script could be made > to > parse the stderr output but that would make things more complicated than I > would like to have them. > > Thanks, > Jeremy Friesner > Level Control Systems > > > ------------------------------------------------------- > This SF.Net email is sponsored by: NEC IT Guy Games. How far can you > shotput > a projector? How fast can you ride your desk chair down the office luge > track? > If you want to score the big prize, get to know the little guy. > Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20 > _______________________________________________ > Valgrind-users mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-users > |
|
From: Mathieu M. <mma...@ny...> - 2005-06-14 00:22:26
|
Jeremy,
This is already done with cmake/ctest. Results from both valgrind or
purify are processed and sent to Dart. For example you can have a look
at:
http://public.kitware.com/dashboard.php?name=cmake
Down in the page there is a table: [Dynamic Analysis]
http://www.cmake.org/Testing/Sites/smallfry.kitwareout/Linux-gcc-
snapshot/20050613-0557-Experimental/DynamicAnalysis.html
My 2 cents
Mathieu
On Jun 13, 2005, at 6:38 PM, Jeremy Friesner wrote:
> Hi guys,
>
> We're planning on enhancing our daily-autobuild process to include
> some daily
> automated testing also, and it would be cool if we could have the unit
> tests
> run under valgrind so that any valgrind-detectable errors show up in
> the
> daily auto-test results.
>
> In order to implement this, I think we would need to somehow tell
> valgrind the
> following:
>
> - If any valgrind-errors are detected, immediately exit with an error
> code
> - If, at program-exit time, any memory leaks are detected, exit with
> an error
> code
>
> Is there any way to tell valgrind to behave like that? If not, could
> a flag
> of some sort be added to implement this? Currently it appears that
> errors
> messages are printed to stderr; I suppose our build script could be
> made to
> parse the stderr output but that would make things more complicated
> than I
> would like to have them.
>
> Thanks,
> Jeremy Friesner
> Level Control Systems
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: NEC IT Guy Games. How far can you
> shotput
> a projector? How fast can you ride your desk chair down the office
> luge track?
> If you want to score the big prize, get to know the little guy.
> Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users
>
|
|
From: Stefan K. <en...@ho...> - 2005-06-14 09:51:12
|
Hi Jeremy, if you want have a look at http://www.buzztard.org/index.php/Testing http://www.buzztard.org/files/guadec2005_advanced_unit_testing.pdf in there we presented some ideas of unit test integration with valgrind. This list was quite helpful to get it working (thanks again). I'd like to here back from you, if you find better ways. Stefan > Hi guys, > > We're planning on enhancing our daily-autobuild process to include some daily > automated testing also, and it would be cool if we could have the unit tests > run under valgrind so that any valgrind-detectable errors show up in the > daily auto-test results. > > In order to implement this, I think we would need to somehow tell valgrind the > following: > > - If any valgrind-errors are detected, immediately exit with an error code > - If, at program-exit time, any memory leaks are detected, exit with an error > code > > Is there any way to tell valgrind to behave like that? If not, could a flag > of some sort be added to implement this? Currently it appears that errors > messages are printed to stderr; I suppose our build script could be made to > parse the stderr output but that would make things more complicated than I > would like to have them. > > Thanks, > Jeremy Friesner > Level Control Systems > > > ------------------------------------------------------- > This SF.Net email is sponsored by: NEC IT Guy Games. How far can you shotput > a projector? How fast can you ride your desk chair down the office luge track? > If you want to score the big prize, get to know the little guy. > Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20 > _______________________________________________ > Valgrind-users mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-users |
|
From: Daniel V. <vei...@re...> - 2005-06-14 10:53:44
|
On Tue, Jun 14, 2005 at 11:50:45AM +0200, Stefan Kost wrote: > Hi Jeremy, > > if you want have a look at > http://www.buzztard.org/index.php/Testing > http://www.buzztard.org/files/guadec2005_advanced_unit_testing.pdf > in there we presented some ideas of unit test integration with valgrind. > This list was quite helpful to get it working (thanks again). > I'd like to here back from you, if you find better ways. > > Stefan > > >Hi guys, > > > >We're planning on enhancing our daily-autobuild process to include some > >daily automated testing also, and it would be cool if we could have the > >unit tests run under valgrind so that any valgrind-detectable errors show > >up in the daily auto-test results. For the record I have "make valgrind" targets which is just a "make tests" but running under the control of valgrind, it's precious. The only big problem I have is that a fair amount of my regression tests are driven by python scripts and valgrind doesn't grok the garbage collector of recent python versions. Daniel -- Daniel Veillard | Red Hat Desktop team http://redhat.com/ vei...@re... | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ |
|
From: Nicholas N. <nj...@cs...> - 2005-06-14 13:21:59
|
On Tue, 14 Jun 2005, Daniel Veillard wrote:
> For the record I have "make valgrind" targets which is just a
> "make tests" but running under the control of valgrind, it's precious.
> The only big problem I have is that a fair amount of my regression
> tests are driven by python scripts and valgrind doesn't grok the
> garbage collector of recent python versions.
Why would you run the python scripts under Valgrind? One possibility is
to use a C wrapper that looks (very roughly) like this:
#include "valgrind.h"
int main(argc, argv)
{
int n = VALGRIND_COUNT_ERRORS(...)
... execute C/C++ program specified on command line ...
int n2 = VALGRIND_COUNT_ERRORS(...)
return n2 - n;
}
If the exit code is non-zero, you have some errors in the test. You could
invoke this from your Python script for each test. I'm sure there are
other ways of achieving this.
Nick
|
|
From: Daniel V. <vei...@re...> - 2005-06-14 14:51:21
|
On Tue, Jun 14, 2005 at 08:21:07AM -0500, Nicholas Nethercote wrote:
> On Tue, 14 Jun 2005, Daniel Veillard wrote:
>
> > For the record I have "make valgrind" targets which is just a
> >"make tests" but running under the control of valgrind, it's precious.
> >The only big problem I have is that a fair amount of my regression
> >tests are driven by python scripts and valgrind doesn't grok the
> >garbage collector of recent python versions.
>
> Why would you run the python scripts under Valgrind?
Because the testing is done from the python bindings, that high
level scripting is needed to make advanced tests, and that doing them
in shell is less potable and harder than from python. One instance of
a python test for example go through 20,000+ individual tests from
a big testsuite.
> to use a C wrapper that looks (very roughly) like this:
>
> #include "valgrind.h"
> int main(argc, argv)
> {
> int n = VALGRIND_COUNT_ERRORS(...)
>
> ... execute C/C++ program specified on command line ...
>
> int n2 = VALGRIND_COUNT_ERRORS(...)
>
> return n2 - n;
> }
No that would not work. Part of the tests are simple that I can use
directly a C based command (xmllint/xsltproc) from shell then that's
fine, but there are tests which are way too complex for that approach.
It used to work from Python, it doesn't anymore with recent python
versions. The only option would be to rewrite all this Python code in
C, it's not trivial, takes quite a bit of time, and makes maintainance
heavy. Apparently there is no simple solution to this Python problem
short pf recompiling Python with a special option (urgh...).
Another problem I'm facing is for gamin. It seems valgrind 2.x doesn't
handle the dnotify kernel API, basically if run under valgrind the gamin
server (gam_server) never get the exception raised by the kernel to alert
the user code of changes on the monitored directories. Usually this will
be carried by signal 33 or 34 (Real Time I think).
Except that I'm a very happy valgrind user :-)
Daniel
--
Daniel Veillard | Red Hat Desktop team http://redhat.com/
vei...@re... | libxml GNOME XML XSLT toolkit http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
|