coverage for parallel runs
Brought to you by:
pspjuth
Menu
▾
▴
1 Attachments
#8 coverage for parallel runs
Thanks for the quick fixes...
What I have been playing with is to make the coverage stuff work in an environment where a c++ tool with TCL shell (and some TCL scripts as part of the tool) runs 10000 or so testcases, and as much in parallel as compute resources permit. This works fine with the standard C++ coverage tools, but nagelfar sources the current coverage at the beginning of the test and writes out the new coverage after the test.
The simple idea was to write the coverage file with "incr" instead of "set", and then only source the coverage data during exit, immediately before writing the new one. This unfortunately will create corrupted files in some race conditions (which I'm sure to hit in my big testing environment), so I also added some file based locking. And this has been working for me with a considerable number of tests in parallel without any file corruption, and with sensible coverage numbers (lines executed once per tool run get covered around 10000 times)...
Attached is the patch that did it for me - of course, it is specific to my environment, so it simply uses TCL 8.6. In addition, for reasons too complicated to explain, I cannot use source during exit - therefore the strange eval/cat part (and tcllib is installed in my environment).
Anyway, maybe a more generic version of this can make it to your tool...
BTW, I also changed the markup generation to report the number of times a line has been hit - this is sometimes also very valuable information.
This is now in the repository, with some additions to make variable
coverage work as well, and some refactoring while I was poking around.
Can you try this version?
Also, I made the new markup with stats optional using -markupfull flag.
Great! Should be able to give it a try next weekend...
One final point (users are never satisfied...)
It would be nice if I could tell the instrumentation to store the absolute path of the file being instrumented in the _i file such that the logs land in the same directory as the source file was instrumented. Background is a standard "make install" flow - files get copied from a checkout directory to an installation directory (where there are typically no write rights when the application is run) - so coverage data should end up in the checkout dir, not the installation dir.
We currently simply have the installation dir writable during testing for the TCL coverage...
Added -idir option to be used with -instrument and -markup*.
This redirects _i _log and _m files to a dir.
Does that match what you wanted?
Hi Peter,
idir seems to do the trick - I got the log files in the checkout dir in a manual test run. I can comment on the full parallel coverage run this or next weekend.
On the idir side, there is a remaining issue. Some TCL flies log to the installation dir, most to the selected idir. Reason seems to be that when appending to dumpList,
set thisScript [file normalize [file join [pwd] [info script]]]
is used, so the path where the file was sourced. And I rename at least the first file to source from the instrumented .tcl_i to .tcl so that the instrumented version gets sourced. After that, it seems the file from idir gets sourced by the overloaded source (so the logs gets "autoamtically" stored to idir by the fact that the source command re-directed to the idir).
I guess the "intended" use of idir is to use it in dumpList if it is not "" - at least this always does the job in case idir is set:
file join $idir [file tail [info script]]
One more caveat:
we tried to run the markup step on an old machine with tcl8.4. That causes the problem
can't read "::instrument::log(file.tcl,95)": no such variable
which seems to be due to
"Starting with the Tcl 8.5 release, the variable varName passed to incr may be unset"
So the incremental coverage feature officially broke 8.4 compatibility for the markup step. Maybe a good time to require 8.5?