|
From: Nicholas N. <nj...@cs...> - 2008-02-06 21:11:31
|
[Forwarding this from the valgrind-developers list in response to a query
from Robert Lieske about coverage tools]]
---------- Forwarded message ----------
Date: Mon, 4 Feb 2008 10:06:34 +1100 (EST)
From: Nicholas Nethercote <nj...@cs...>
To: Valgrind Developers <val...@li...>
Subject: [Valgrind-developers] Experimental Valgrind coverage tool
Hi,
I've written an experimental Valgrind coverage tool, called VCov. To try it
out, do this:
svn co svn://svn.valgrind.org/valgrind/branches/VCOV
Then build in the usual way (see the README file). To run it, use 'valgrind
--tool=exp-vcov <program>'.
It's pretty simple. It just records, for each line of source code, how many
instructions that were derived from that line were executed. It aggregates
data from multiple executions. It all goes in a file called "vcov.out".
To interpret the data, run this command:
perl exp-vcov/vc_annotate vcov.out
The output gives total, per-file and per-line coverage. An example:
--------------------------------------------------------------------------------
Total coverage
--------------------------------------------------------------------------------
100.0% ( 9 of 9 lines)
--------------------------------------------------------------------------------
Per-file coverage (files with the most unexecuted lines are shown first)
--------------------------------------------------------------------------------
100.0% ( 3 of 3 lines): /home/njn/grind/trunk9/b.h
100.0% ( 6 of 6 lines): /home/njn/grind/trunk9/a.c
--------------------------------------------------------------------------------
/home/njn/grind/trunk9/b.h
--------------------------------------------------------------------------------
-: 1:__inline__ int foo(void)
4: 2:{
2: 3: return 0;
4: 4:}
--------------------------------------------------------------------------------
/home/njn/grind/trunk9/a.c
--------------------------------------------------------------------------------
-: 1:#include <stdio.h>
-: 2:#include "b.h"
-: 3:
-: 4:int main(int argc, char* argv[])
20: 5:{
4: 6: if (argc < 2)
5: 7: printf("hello, world\n");
-: 8: else
4: 9: printf("hello, extended world\n");
-: 10:
2: 11: return foo();
4: 12:}
--------------------------------------------------------------------------------
Total coverage
--------------------------------------------------------------------------------
88.9% ( 8 of 9 lines)
--------------------------------------------------------------------------------
Per-file coverage (files with the most unexecuted lines are shown first)
--------------------------------------------------------------------------------
83.3% ( 5 of 6 lines): /home/njn/grind/trunk9/a.c
100.0% ( 3 of 3 lines): /home/njn/grind/trunk9/b.h
--------------------------------------------------------------------------------
/home/njn/grind/trunk9/b.h
--------------------------------------------------------------------------------
-: 1:__inline__ int foo(void)
2: 2:{
1: 3: return 0;
2: 4:}
--------------------------------------------------------------------------------
/home/njn/grind/trunk9/a.c
--------------------------------------------------------------------------------
-: 1:#include <stdio.h>
-: 2:#include "b.h"
-: 3:
-: 4:int main(int argc, char* argv[])
10: 5:{
2: 6: if (argc < 2)
5: 7: printf("hello, world\n");
-: 8: else
#####: 9: printf("hello, extended world\n");
-: 10:
1: 11: return foo();
2: 12:}
Non-executable lines are preceded with '-'. Lines that are executable but
not executed are preceded with '######', so they are obvious.
There are lots of shortcomings, but it gets the general idea across. Some
things worth noting:
- Files not compiled with -g won't have any results.
- You can use the --fresh=yes option to overwrite any prior coverage data.
- I plan to add the ability to change the name of the output file.
- Results with --trace-children will be suspect, as there is no file locking
done yet on the output file.
- I hope to add some kind of branch coverage info, at least for the
encountered branches.
- If your program has a file for which no code is executed, it won't get
included in the output.
- I plan to add the ability to be more selective about which files get
annotated (currently every file that has been executed and that can be
found is annotated).
Any comments are welcome.
Nick
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Valgrind-developers mailing list
Val...@li...
https://lists.sourceforge.net/lists/listinfo/valgrind-developers
|
|
From: Oliver G. <ol...@gm...> - 2008-02-06 23:19:32
|
Nicholas Nethercote schrieb: > [Forwarding this from the valgrind-developers list in response to a query > from Robert Lieske about coverage tools]] > > ---------- Forwarded message ---------- > Date: Mon, 4 Feb 2008 10:06:34 +1100 (EST) > From: Nicholas Nethercote <nj...@cs...> > To: Valgrind Developers <val...@li...> > Subject: [Valgrind-developers] Experimental Valgrind coverage tool > > Hi, > > I've written an experimental Valgrind coverage tool, called VCov. To try it > out, do this: > > svn co svn://svn.valgrind.org/valgrind/branches/VCOV > > Then build in the usual way (see the README file). To run it, use 'valgrind > --tool=exp-vcov <program>'. > > It's pretty simple. It just records, for each line of source code, how many > instructions that were derived from that line were executed. It aggregates > data from multiple executions. It all goes in a file called "vcov.out". [...] Hi, that sounds quite like I was looking for. But one thing that I've wondered about in particular when using Valgrind for coverage is how it works with C++ templates? If a template is never instantiated, how can a coverage tool detect that there is source code that is never executed? Is that possible somehow? Regards, Oliver Gerlich |
|
From: Nicholas N. <nj...@cs...> - 2008-02-07 00:34:34
|
On Thu, 7 Feb 2008, Oliver Gerlich wrote: > that sounds quite like I was looking for. But one thing that I've wondered > about in particular when using Valgrind for coverage is how it works with C++ > templates? If a template is never instantiated, how can a coverage tool > detect that there is source code that is never executed? Is that possible > somehow? My tool just records which instructions get executed, and then maps that back to the source code lines according to the debug info. It also identifies source lines mentioned in the debug info which weren't executed. So I don't know the answer to your question, it depends entirely how GCC produces its debug info for templates, but I suspect the lines of code within the template would end up marked as unexecuted. Nick |
|
From: Oliver G. <ol...@gm...> - 2008-02-07 10:33:52
Attachments:
coverage-template-test.C
|
Nicholas Nethercote wrote: > On Thu, 7 Feb 2008, Oliver Gerlich wrote: > >> that sounds quite like I was looking for. But one thing that I've >> wondered about in particular when using Valgrind for coverage is how >> it works with C++ templates? If a template is never instantiated, how >> can a coverage tool detect that there is source code that is never >> executed? Is that possible somehow? > > My tool just records which instructions get executed, and then maps that > back to the source code lines according to the debug info. It also > identifies source lines mentioned in the debug info which weren't > executed. So I don't know the answer to your question, it depends > entirely how GCC produces its debug info for templates, but I suspect > the lines of code within the template would end up marked as unexecuted. > > Nick Thanks for the explanation! I had tried to build a coverage tool myself which parses callgrind/cachegrind output files; but I have not found a way to get it working correctly for this "not-instantiated template" case... That's why I'm interested whether someone has found solution for this. Attached is an example where my coverage tool fails: while it explicitly marks templateFunction3 as not-executed, it doesn't mark templateFunction2 like that. You can see at http://img509.imageshack.us/img509/9682/cgctemplateexample1na6.png how it looks. Also, because of this, the "68% executed" number is wrong as well. Regards, Oliver Gerlich |
|
From: Josef W. <Jos...@gm...> - 2008-02-07 12:50:14
|
On Thursday 07 February 2008, Oliver Gerlich wrote: > way to get it working correctly for this "not-instantiated template" > case... That's why I'm interested whether someone has found solution for > this. When the compiler does not instantiate a template, there is no code generated for that template, and there can not be any debug info for not existing code. Thus, any coverage tool purely based on the binary and its debug info has no way to know about the existance of template code without instantiation. To make this work, you need additional information from the compiler. Does gcov cover this issue? Josef |
|
From: Tom H. <to...@co...> - 2008-02-07 12:59:42
|
In message <200...@gm...>
Josef Weidendorfer <Jos...@gm...> wrote:
> On Thursday 07 February 2008, Oliver Gerlich wrote:
>> way to get it working correctly for this "not-instantiated template"
>> case... That's why I'm interested whether someone has found solution for
>> this.
>
> When the compiler does not instantiate a template, there is
> no code generated for that template, and there can not be any debug info
> for not existing code.
>
> Thus, any coverage tool purely based on the binary and its debug info
> has no way to know about the existance of template code without instantiation.
But equally the code can never be executed because it doesn't
exist. So you know what percentage of the code that actually
exists in your shipping executable is tested, you just don't
know about hypothetical code that might be created some day if
somebody instantiates a template differently.
Tom
--
Tom Hughes (to...@co...)
http://www.compton.nu/
|