RE: [Cobertura-devel] questions on merge fcn and report comparison

[Mark D. <Mar...@sa...>]

> Hello, 
> I haven't had a chance to play with the merge function that 
> combines .ser files but before I start, could someone tell me 
> if these are viable uses of the function? 
> In our usage, we are using different systems to instrument 
> files and run tests and are copying the ser file from system 
> to system 

Yes, they are both viable uses of the function.  I should point out that=
 I haven't actually used the merge task, and I'm not sure how well it wo=
rks.

> Scenario 1: 
> Assume I have compiled, instrumented 5 class files and 
> created ser file A. 
> I run tests on ser A (which is updated with the hits on line 
> coverage- call this A-mod) 
> I then create and instrument 2 more class files in same 
> package and ser file B is created. 
> I will then run tests that involve all 7 class files. 
> 
> Do I merge ser A-mod and B before running the test? If not, I 
> can't have 2 different files named obertura.ser on my test 
> machines? How do I setup the .ser files? 
> Or when I intrument the 2 class files must I get A-mod file 
> and use the " 
> -Dnet.sourceforge.cobertura.rawcoverage.dir=3DA-mod"   option? 

You could merge A-mod and B between tests run, but it might be easier to=
 use the "-Dnet.sourceforge.cobertura.rawcoverage.dir=3DA-mod" option wh=
en running the second set of tests.  In CVS (and in Cobertura 1.3, when =
it's released) you will be able to specify the location of the ser file =
in the instrument and reporting tasks, as well.

> Scenario 2:
> I have compiled, instrumented 5 class files and created ser file A. 
> I run tests on ser A (which is updated with the hits on line 
> coverage- call this A-mod) 
> I modify the source for class 1  of the 5 original classes 
> that leaves preexisting code the same but adds new methods. 
> I modify the source for class 2  of the 5 original classes 
> that changes preexisting code. 
> I reinstrument the two classes that are changed. 
> I then run different tests that touch all 5 classes. 
> 
> Is there any steps that I can do that preserves and merges 
> the output from the first run of tests with the second run? 
> In particular 
>         - a way to preserve coverage results for classes 3-5 
> which remained the same 
>         - a way to perserve coverage for class 1 
> I  would be pleased but amazed that cobertura could merge 
> test output for class 2 (I don't see how  correlation be 
> made) and if this is possible  which level of source would I 
> use for the report function? 

You ask hard questions :-)

"a way to preserve coverage results for classes 3-5 which remained the s=
ame"
This should be possible.  I think it will work as long as you only re-in=
strument the changed classes.  If you instrument a class, the coverage c=
ounters are reset to 0 (at least, that's how it SHOULD work).

"a way to perserve coverage for class 1" and
"a way to perserve coverage for class 2"
Neither of these are currently possible.  Whenever a class changes the o=
ld coverage data is made invalid.  This stems from the fact that Cobertu=
ra uses the line number of the source file to keep track of which instru=
ctions were run.  If the line numbers change then we have no way of know=
ing if our data is still accurate.  To fix this I guess we could store t=
he line number as an offset from the start of a method, and then also ke=
ep some sort of hashcode for each method to use in determining if the me=
thod has changed.

> On a different subject, once reports are made is there any 
> easy way to compare two sets and indicate changes ? For 
> instance right now I'm baseline about 900 classes and we 
> would like to improve our regression tests on the same code.  
> After some test changes I will produce another report for 
> these 900 classes. 
> For a smaller set I could just visually compare the 2 
> reports, but given the size I need a way to compare results. 
> We are planning on repeating the regression at several points 
> so this analysis would happen again.  We don't have a target 
> in mind yet  so check function is not applicable but we would 
> like to know we are improving and not going down in coverage. 
>  Any thoughts on how to do this would be appreciated. 

I don't know of a way currently to do that.  A change-over-time display =
would be nice.  I suppose you could create some Java code that takes a s=
et of ser files or a set of xml reports and creates a pretty web page sh=
owing just how awesome your tests have become.  And you could probably a=
lso create an ant task that lets you create the report.

-Mark