|
From: Bryan M. <om...@br...> - 2006-08-04 20:37:47
|
Fellow Valgrinders, please check my small web page for the details: http://www.brainmurders.eclipse.co.uk/covergrind.html >From the web page: ------------------ Covergrind is a code coverage tool for the excellent Valgrind framework. Rather than having to link against strange libraries or compile in various support, Covergrind uses run time instrumentation in order to work out which parts of your source code are traversed. There have been comments made about how trivial it would be to use the Valgrind framework to construct such a tool but I wasn't able to find one so here is mine to share. ------------------ Basically does what it says - run your program with Covergrind and it will output a list of filenames and line number ranges in a simple man/machine readable format so you can check your code coverage. I am hoping to put a simple GUI together to read the output and colour source files accordingly but I haven't got there yet (I normally do low level and device driver stuff so go easy on me :P). The main advantage of this tool (beyond the standard Valgrind advantages of not having to mess around with your source code or build system) is that it is lightweight compared to Memcheck or Callgrind, mainly because it is only providing a single tiny set of functionality. A secondary advantage is that if you are set up to Valgrind your code, you are also set up to Covergrind it... Only tested on x86_64 so far but may well work on other platforms. Feel free to give it a spin and post your experiences to the list. As ever, I welcome your comments and bug reports. Bryan "Brain Murders" Meredith ps. Julian/Nick - happy to change the name if you are uncomfortable with it for ANY reason. |
|
From: Nicholas N. <nj...@cs...> - 2006-08-05 08:51:50
Attachments:
vcov.tar.bz2
|
On Fri, 4 Aug 2006, Bryan Meredith wrote: > please check my small web page for the details: > http://www.brainmurders.eclipse.co.uk/covergrind.html > >> From the web page: > ------------------ > Covergrind is a code coverage tool for the excellent Valgrind framework. > Rather than having to link against strange libraries or compile in > various support, Covergrind uses run time instrumentation in order to > work out which parts of your source code are traversed. There have been > comments made about how trivial it would be to use the Valgrind > framework to construct such a tool but I wasn't able to find one so here > is mine to share. > ------------------ You don't explain how to use it -- what does the output look like? If you run a program multiple times, are the coverage results merged? Also, does it give percentage coverage? Those latter two things are the tricky part of writing a coverage tool with Valgrind. I wrote one a while ago ("vcov") which I never fully tested and released properly. I attach the code in case you find it useful. vcov does the merging from multiple runs ok. It also does the percentage coverage calculation, but it's not perfect -- because you have to know how many lines of executable code each file has, and that's only available from the debug info, and I found that some versions of GCC produced debug info that was unreliable. Benoit Peccatte wrote a similar tool ("cover"), I'm not sure how advanced it is, you can get it here http://b.peccatte.free.fr/cover-0.03.tgz. Nick |
|
From: Bryan M. <om...@br...> - 2006-08-05 11:52:54
|
Nick, thanks as ever for the quick reply. I have inserted some comments below so that the context is intact. Bryan "Brain Murders" Meredith Nicholas Nethercote wrote: > On Fri, 4 Aug 2006, Bryan Meredith wrote: > >> please check my small web page for the details: >> http://www.brainmurders.eclipse.co.uk/covergrind.html >> >>> From the web page: >> ------------------ >> Covergrind is a code coverage tool for the excellent Valgrind framework. >> Rather than having to link against strange libraries or compile in >> various support, Covergrind uses run time instrumentation in order to >> work out which parts of your source code are traversed. There have been >> comments made about how trivial it would be to use the Valgrind >> framework to construct such a tool but I wasn't able to find one so here >> is mine to share. >> ------------------ > > You don't explain how to use it -- what does the output look like? You need to follow the link in order to download it. Details are available there of how to run it. I admit there is no sample output on the web page - I shall add that on the next revision although I had hoped that the omission was an incentive for people to try it out... :D It looks like this: ==4464== Covergrind-beta-01, A code coverage tool.. ==4464== Copyright (C) 2006, and GNU GPL'd, by Bryan Meredith. ==4464== Using LibVEX rev 1631, a library for dynamic binary translation. ==4464== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP. ==4464== Using valgrind-3.3.0.SVN, a dynamic binary instrumentation framework. ==4464== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al. ==4464== For more details, rerun with: -v ==4464== ==4464== COVERGRIND FILES START 0:/home/bryan/covergrind/coregrind/m_trampoline.S 1:/home/bryan/c++/qt-x11-opensource-src-4.1.4/examples/widgets/analogclock/../../../include/QtCore/../../src/corelib/global/qglobal.h 2:/home/bryan/c++/qt-x11-opensource-src-4.1.4/examples/widgets/analogclock/../../../include/QtCore/../../src/corelib/thread/qatomic.h . . (snipped) . 302:/usr/src/packages/BUILD/glibc-2.4/cc-nptl/csu/crti.S 303:/usr/src/packages/BUILD/glibc-2.4/cc-nptl/csu/crtn.S COVERGRIND FILES END COVERGRIND LINES START 0:155-155 0:161-161 1:754-754 1:1538-1538 2:74-75 2:75-75 3:77-77 4:93-94 . . (snipped) . 302:11-14 302:17-18 302:25-26 303:9-10 303:16-17 COVERGRIND LINES END As you can see, each file is given an identifier. These identifiers are then used to show the line coverage information in order to help keep the file size down. I hope you will agree that it is easily parsable by man or machine. The above is the output from running the QT "analogclock" example application (C++). We (at work) are not too interested in how many times a line range has been hit but there is partial support in the Covergrind source for it and I will probably end up implementing it fully. I would envisage adding the hit count to the end of the line range. Actually, I just noticed that it might be a good idea to output the name of the executable under test as well. > If > you run a program multiple times, are the coverage results merged? In the interests of keeping things simple (and hence lightweight and fast) merging of results would be done by the GUI (long way of saying No). > Also, does it give percentage coverage? Again, in the GUI, with both the source file and line coverage information available, it will be simpler to generate any required metrics (another long way of saying No). > Those latter two things are the > tricky part of writing a coverage tool with Valgrind. I appear to have side-stepped both of them. :P Callgrind has a wonderful GUI for post processing and combining run data so this is not setting any precedent. > I wrote one a > while ago ("vcov") which I never fully tested and released properly. I > attach the code in case you find it useful. Thanks. I shall have a look. > > vcov does the merging from multiple runs ok. It also does the > percentage coverage calculation, but it's not perfect -- because you > have to know how many lines of executable code each file has, and that's > only available from the debug info, and I found that some versions of > GCC produced debug info that was unreliable. If I understand this properly, my method differs from this. By outputting source code line number ranges, the source itself can be annotated by the GUI. With suitable parsing of the source to ignore comment only and whitespace only lines, the resulting statistics should be quite accurate for the more common coding styles. Like gcov, one statement per line styles will produce better results. I felt that the best approach was to just output source file line ranges in a fairly dumb fashion and let another tool post process the data. Because the post processing tool (my proposed GUI) would not be in such a constrained environment, there are many more implementation options available, working with the original source file rather than just the debug information being the most important IMO. > > Benoit Peccatte wrote a similar tool ("cover"), I'm not sure how > advanced it is, you can get it here > http://b.peccatte.free.fr/cover-0.03.tgz. Is it still active? I have no problem picking up someone else's work and submitting patches - I am old enough to have gotten over "not written here" syndrome more than two decades ago - but I will only do that if there are signs of active development and I can't say that I have seen anything on the list for a long time. This will be alive for some time yet as I have written it to use at work. It is also simple enough that others can get right in and hack on it if they wish (and I would like to at least hear about it if they do). > > Nick > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > ------------------------------------------------------------------------ > > _______________________________________________ > Valgrind-developers mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-developers |
|
From: Patrick O. <pat...@in...> - 2006-08-07 14:48:33
|
On Sat, 2006-08-05 at 12:52 +0100, Bryan Meredith wrote: > > Also, does it give percentage coverage? > > Again, in the GUI, with both the source file and line coverage > information available, it will be simpler to generate any required > metrics (another long way of saying No). Actually this might be the right approach to handle a problem that no other coverage tool that I am aware of handles right: if you link the same object code, say from a static library, into different executables and then run those multiple times, then merging coverage information about the original object code can be very hard. Now assuming that the same source code is compiled differently into different object files (think Linux and Windows, with and without debugging enabled, etc.) and then executed the object file based approach completely fails - basically you have to merge information about the source code in this case, as you suggest. The drawback (and I suppose that was what Nicholas was pointing out) is then that you don't have information about number of code lines compiled into the object files or executables, so you have to fall back to less reliable methods of source code analysis to have a baseline for the percentage of covered lines of code. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. |
|
From: Bryan M. <om...@br...> - 2006-08-07 19:59:17
|
Patrick, thanks for the input. If the compiled object is available, there should be nothing to stop some "offline" analysis to determine the executable lines within it. It would be better if all of this could automagically be done by a GUI or some other pre/post processing tool rather than as a set of manual steps, but done this way, ALL of the relevant information would be available to perform whatever analysis you wish. The other advantage is that this wouldn't impact the size or speed of the executable under test and you could also cull this information from the "other" ;) OS if you really wanted to do the comparison. Bryan "Brain Murders" Meredith Patrick Ohly wrote: > On Sat, 2006-08-05 at 12:52 +0100, Bryan Meredith wrote: >>> Also, does it give percentage coverage? >> Again, in the GUI, with both the source file and line coverage >> information available, it will be simpler to generate any required >> metrics (another long way of saying No). > > Actually this might be the right approach to handle a problem that no > other coverage tool that I am aware of handles right: if you link the > same object code, say from a static library, into different executables > and then run those multiple times, then merging coverage information > about the original object code can be very hard. Now assuming that the > same source code is compiled differently into different object files > (think Linux and Windows, with and without debugging enabled, etc.) and > then executed the object file based approach completely fails - > basically you have to merge information about the source code in this > case, as you suggest. > > The drawback (and I suppose that was what Nicholas was pointing out) is > then that you don't have information about number of code lines compiled > into the object files or executables, so you have to fall back to less > reliable methods of source code analysis to have a baseline for the > percentage of covered lines of code. > |
|
From: Brad H. <br...@fr...> - 2007-06-10 18:05:07
Attachments:
covergrind-2007-06-10.patch
vcover-merge.py
|
On Saturday 05 August 2006 06:37, Bryan Meredith wrote: > Fellow Valgrinders, > > please check my small web page for the details: > http://www.brainmurders.eclipse.co.uk/covergrind.html Bryan, Thanks for writing this. I found Beta 2 on your website (which has a different output format), but it appeared to be a little bit-rotted. I've updated the patch. Since I don't actually understand anything about valgrind, it could well be complete crackrock. I also started on the path to doing something with the output - find enclosed a small python script that allows you to merge multiple runs (perhaps different unit tests, each exercising a different part of the code), and to exclude files (e.g. system headers, test harness). The output format is the same as the input format. I plan to do a script that actually does some annotation of the files, but don't have time right now. Brad |