|
From: Nicholas N. <nj...@cs...> - 2008-02-03 23:09:59
|
Hi, I've written an experimental Valgrind coverage tool, called VCov. To try it out, do this: svn co svn://svn.valgrind.org/valgrind/branches/VCOV Then build in the usual way (see the README file). To run it, use 'valgrind --tool=exp-vcov <program>'. It's pretty simple. It just records, for each line of source code, how many instructions that were derived from that line were executed. It aggregates data from multiple executions. It all goes in a file called "vcov.out". To interpret the data, run this command: perl exp-vcov/vc_annotate vcov.out The output gives total, per-file and per-line coverage. An example: -------------------------------------------------------------------------------- Total coverage -------------------------------------------------------------------------------- 100.0% ( 9 of 9 lines) -------------------------------------------------------------------------------- Per-file coverage (files with the most unexecuted lines are shown first) -------------------------------------------------------------------------------- 100.0% ( 3 of 3 lines): /home/njn/grind/trunk9/b.h 100.0% ( 6 of 6 lines): /home/njn/grind/trunk9/a.c -------------------------------------------------------------------------------- /home/njn/grind/trunk9/b.h -------------------------------------------------------------------------------- -: 1:__inline__ int foo(void) 4: 2:{ 2: 3: return 0; 4: 4:} -------------------------------------------------------------------------------- /home/njn/grind/trunk9/a.c -------------------------------------------------------------------------------- -: 1:#include <stdio.h> -: 2:#include "b.h" -: 3: -: 4:int main(int argc, char* argv[]) 20: 5:{ 4: 6: if (argc < 2) 5: 7: printf("hello, world\n"); -: 8: else 4: 9: printf("hello, extended world\n"); -: 10: 2: 11: return foo(); 4: 12:} -------------------------------------------------------------------------------- Total coverage -------------------------------------------------------------------------------- 88.9% ( 8 of 9 lines) -------------------------------------------------------------------------------- Per-file coverage (files with the most unexecuted lines are shown first) -------------------------------------------------------------------------------- 83.3% ( 5 of 6 lines): /home/njn/grind/trunk9/a.c 100.0% ( 3 of 3 lines): /home/njn/grind/trunk9/b.h -------------------------------------------------------------------------------- /home/njn/grind/trunk9/b.h -------------------------------------------------------------------------------- -: 1:__inline__ int foo(void) 2: 2:{ 1: 3: return 0; 2: 4:} -------------------------------------------------------------------------------- /home/njn/grind/trunk9/a.c -------------------------------------------------------------------------------- -: 1:#include <stdio.h> -: 2:#include "b.h" -: 3: -: 4:int main(int argc, char* argv[]) 10: 5:{ 2: 6: if (argc < 2) 5: 7: printf("hello, world\n"); -: 8: else #####: 9: printf("hello, extended world\n"); -: 10: 1: 11: return foo(); 2: 12:} Non-executable lines are preceded with '-'. Lines that are executable but not executed are preceded with '######', so they are obvious. There are lots of shortcomings, but it gets the general idea across. Some things worth noting: - Files not compiled with -g won't have any results. - You can use the --fresh=yes option to overwrite any prior coverage data. - I plan to add the ability to change the name of the output file. - Results with --trace-children will be suspect, as there is no file locking done yet on the output file. - I hope to add some kind of branch coverage info, at least for the encountered branches. - If your program has a file for which no code is executed, it won't get included in the output. - I plan to add the ability to be more selective about which files get annotated (currently every file that has been executed and that can be found is annotated). Any comments are welcome. Nick |
|
From: Benjamin M. <be...@me...> - 2008-02-04 18:45:15
|
On Feb 4, 2008, at 12:06 AM, Nicholas Nethercote wrote: > Hi, > > I've written an experimental Valgrind coverage tool, called VCov. > To try it > out, do this: > > svn co svn://svn.valgrind.org/valgrind/branches/VCOV > > Then build in the usual way (see the README file). To run it, use > 'valgrind > --tool=exp-vcov <program>'. > > It's pretty simple. It just records, for each line of source code, > how many > instructions that were derived from that line were executed. It > aggregates > data from multiple executions. It all goes in a file called > "vcov.out". > Does it do anything callgrind can't do? Several other people (including me) have written similar tools only to realize after the fact that callgrind files are good enough and can even include jumps if you pass --collect-jumps=yes to the args. By letting callgrind collect the data you can spend your time creating the value add of the tool, source integration, report generating etc, without worrying about any of the valgrind stuff. You e-mailed me about the valgrind-tools that I wrote (http:// benjamin-meyer.blogspot.com/2007/12/valgrind-callgrind-tools-part-3- code.html) . Was there any particular reason why it wasn't good enough, I know it isn't perfect, but were you able to get it working at least? -Benjamin Meyer |
|
From: Nicholas N. <nj...@cs...> - 2008-02-04 21:41:28
|
On Mon, 4 Feb 2008, Benjamin Meyer wrote: >> I've written an experimental Valgrind coverage tool, called VCov. > > Does it do anything callgrind can't do? Several other people > (including me) have written similar tools only to realize after the > fact that callgrind files are good enough and can even include jumps > if you pass --collect-jumps=yes to the args. By letting callgrind > collect the data you can spend your time creating the value add of > the tool, source integration, report generating etc, without worrying > about any of the valgrind stuff. On the tool (data gathering) side: - It's faster than Callgrind. - It aggregates data from multiple executions into a single output file, which makes it much easier to use. - It records info about lines of code that are executable but were not executed. This allows the post-processing tool to be simpler, eg. no language-specific parsing is necessary. - The code is much simpler. Callgrind is almost 12,000 lines of code. VCov is 750. Separating them also makes it easier to change VCov if it needs capabilities that don't fit into Callgrind. > You e-mailed me about the valgrind-tools that I wrote (http:// > benjamin-meyer.blogspot.com/2007/12/valgrind-callgrind-tools-part-3- > code.html) . Was there any particular reason why it wasn't good > enough, I know it isn't perfect, but were you able to get it working > at least? I haven't tried it. I had VCov in its current state before you wrote about your tool, and I promised someone that I would make VCov available in a way that lets him use it. It is on a branch, so it won't necessarily end up in the distribution. Nick |
|
From: Josef W. <Jos...@gm...> - 2008-02-05 21:05:53
|
Hi Nick, On Monday 04 February 2008, Nicholas Nethercote wrote: > I've written an experimental Valgrind coverage tool, called VCov. To try it > out, do this: Cool. Especially the debug info iterators are a very nice addition to the tool interface. I did not check it out yet, but will do soon. How similar is the format for VCov to cachegrind's? I suppose this only needs a further "event" for a source line: whether there is debug info or not. Ah, I just saw the description before parse_buffer(). Do you only output lines where debug info is available? So this is implicit... Josef |
|
From: Nicholas N. <nj...@cs...> - 2008-02-05 23:02:40
|
On Tue, 5 Feb 2008, Josef Weidendorfer wrote: > How similar is the format for VCov to cachegrind's? I suppose this > only needs a further "event" for a source line: whether there is debug info > or not. It's simpler than Cachegrind's. There's no function-level information, so no "fn=" lines. Also, it only records one number per line, and there is no description in the file of what that number means. I thought about reusing the Cachegrind format. I didn't for two main reasons: - The output from cg_annotate isn't quite right -- it doesn't compute coverage percentages, and it doesn't highlight unexecuted lines sufficiently, and we don't need function-level information. Since cg_annotate doesn't do the right thing, the motivation for using the same format is diminished. - I have to parse it in the tool itself (ie. in C) so simpler makes that easier. It's not set in stone, but I felt the reasons against using Cachegrind's format were stronger than the reasons for. > Ah, I just saw the description before parse_buffer(). Do you only output > lines where debug info is available? Yes. Nick |
|
From: Josef W. <Jos...@gm...> - 2008-02-06 14:38:06
|
On Wednesday 06 February 2008, Nicholas Nethercote wrote: > On Tue, 5 Feb 2008, Josef Weidendorfer wrote: > > > How similar is the format for VCov to cachegrind's? I suppose this > > only needs a further "event" for a source line: whether there is debug info > > or not. > > It's simpler than Cachegrind's. There's no function-level information, so > no "fn=" lines. Actually, I have no idea what people want here. Is the function name really totally uninteresting here? Neverless, would you say that "fn=" lines are mandatory for the cachegrind format? It should still load without, and in the absence of such lines, it should say "function name unknown" (ie. "???"). > Also, it only records one number per line, and there is no > description in the file of what that number means. I think it would be quite easy to support this simple format with KCachegrind. Would you mind to add (1) some kind of magic at start for autodetection and (2) provision for possible format changes. Something like a fixed header "creator:VCov\nversion:1\n". Both is quite important for integration with external tools; internal to the VG distribution, it can be ignored on the reading side. > I thought about reusing the Cachegrind format. I didn't for two main > reasons: > > - The output from cg_annotate isn't quite right -- it doesn't compute > coverage percentages, and it doesn't highlight unexecuted lines > sufficiently, and we don't need function-level information. Since > cg_annotate doesn't do the right thing, the motivation for using the same > format is diminished. That's right. However, I personally think that there is already a positive effect when merging/filtering tools could be reused. Josef > - I have to parse it in the tool itself (ie. in C) so simpler makes that > easier. > > It's not set in stone, but I felt the reasons against using Cachegrind's > format were stronger than the reasons for. > > > Ah, I just saw the description before parse_buffer(). Do you only output > > lines where debug info is available? > > Yes. > > Nick > |