Handling instrumented artifacts in Maven
Brought to you by:
vlad_r
Menu
▾
▴
Another quote from Chad:
"I would like to start a thread to discuss the problem of including instrumented artifacts in the deployables created by Maven. I'd like to understand how Dilum views this, if he deploys the instrumented code, or how he avoids it (maybe I don't fully understand how the plugin currently works). Also, I don't know how you avoid this in Ant builds, maybe you can comment on this (since I only use Maven anymore)."
I can speak for ANT case myself. It does not seem to be a problem, really. The options are:
(a) you use emmarun when that works for you
(b) you have two output locations, one for normal .class files and another one for instrumented .class files ("copy" instr mode). You use the latter one for testing (it's added to the start of the run classpath), but ship only the normal classes.
(c) you instrument in-place ("overwrite" instr mode) but then you just remember that your output is instrumented for coverage, so you never ship that. You do a clean "release" build to create the release artifacts.
My ANT examples that ship with full EMMA zip show how an ANT user would do it, I believe. ANT is lower level than Maven and gives you more control, there is generally no problem in choosing one of (a)-(c) above.
I have no idea how to achieve the same flexibility in Maven. What puzzles me, other tools like Clover and jcoverage have the same problem (Clover adds instrumentation to the sources, jcoverage is very similar to EMMA), so how do their plugins work?
Vlad
OK, here's my take on those three with respect to Maven
(a) you use emmarun when that works for you
This is the approach I used in the EMMA maven plugin I wrote (although I used emmajava to run the unit test suite, but the same basic approach using on-the-fly instrumentation)
(b) you have two output locations, one for normal .class files and another one for instrumented .class files ("copy" instr mode). You use the latter one for testing (it's added to the start of the run classpath), but ship only the normal classes.
This sounds good, but this is where maven's "standards" can cause you problems. Maven and it's plugins REALLY like to find all classes in target/classes and target/test-classes. It takes careful overriding and hacking to get them put somewhere else, in my experience.
(c) you instrument in-place ("overwrite" instr mode) but then you just remember that your output is instrumented for coverage, so you never ship that. You do a clean "release" build to create the release artifacts.
This is also good, but again, maven can make it difficult to do a clean in the middle of a build, although I mostly encountered this problem with a WAR file being locked and unable to deleted. I actually think this is a problem in the underlying Ant task that maven is using, I haven't had time to dig into it in detail, but it has caused me problems other places besides EMMA.
Anyway, thanks for the response. I'll be looking into these as alternatives and report back my progress here.
-- Chad
Wouldn't this work ?
1) Copy classes to target/temp/classes before instrumentation
2) Instrument and test
3) Copy target/temp/classes to target/classes
I have the feeling that some preGoal/postGoal would dothe trick.
My 2 cents,
J-F
Good idea, that might work. What goals would you hook on to? The emma goals?
On second thought, this approach might limit the flexibility and robustness. For example, the user can run anything they want to obtain coverage data, not just tests via the test goal. Also, they may be doing something custom to build their deployables, not just a standard jar/war/ear. I want to support any usage of maven, and maven lets you be very flexible (in some cases).
Having a hook to copy the classes out to target/temp/classes would be easy (just do it before emma instruments.
However, knowing when to copy them BACK would be the hard part.
I think it might be an acceptable workaround to allow the user to EXPLICITLY (but easily) do it themselves, via a "emma:restore-uninstrumented-classes" goal. It should probably be accompanied by a property to enable/disable the copying, to avoid the performance hit in situations where people don't care (which would be most cases I think except when they are deploying).
This would work for me.
Feedback welcome,
Chad
Just a quick note with a gotcha using this approach. If you have any resources that need to go into the target/classes directory (like *.properties or *.xml in your classpath), maven uses the java:jar-resources goal to copy them from src/main/resources (or wherever). However, this happens after java:compile, but before emma:instrument. So if you use a java:compile postGoal to copy the uninstrumented classes, the copy will not have any resources. Then, when you copy the uninstrumented classes back to create a jar/war, your resources are wiped out of the target/classes dir.
The solution I used was to put a <attainGoal name="java:jar-resources"> in my postGoal to java:compile. This causes the goal to be executed before emma:instrument, rather than after.
The upcoming rewrite of the Maven Plugin will allow you to easily specify whether you want to instrument in-place, or compile to a different location. It accomplishes this by overriding the maven properties that control the dir locations
-- Chad
My rationale was based on the assumption that 90% of emma users instrument code during unit tests.
In this context, restoring classes before JAR would be OK.
The way I use Maven, WAR and EAR are not concerned. They do not contain sources. They just assemble JARs (from other projects) and descriptors.
I think too that offering a 'restore' feature that can be enabled via a property is the best solution to this problem. This should satisfy everyone.
J-F
Yes, but think of integration tests. Users may want to build a WAR or EAR (which calls the jar goal internally) to deploy and perform integration testing in a container. I do this all the time.
In this case, they WOULD want instrumented code in a jar. That's why I think the explicit "uninstrument" goal would be best.
-- Chad
Wth the 'uninstrument' target, the default behavior is to instrument classes.
I still think that classes should only be instrumented for testing purpose (either at unit or integration level).
I think too that default behavior should be to generate production artefacts (uninstrumented), and not testing artefacts.
What do you think about this ?
Regards,
J-F
I'm not sure what you mean by this sentence: "Wth the 'uninstrument' target, the default behavior is to instrument classes. "
As for the default being to generate non-instrumented code, I think these are what the current properties and on/off targets are for.
My understanding is that if there is an 'uninstrument' target to call explicitly, it is because classes are instrumented by default.
Have I misunderstood your solution ?
Maybe it is worth explaining it again shortly step by step.
J-F
Sure, I'll try to explain. Keep in mind I'm still trying to fully understand how the plugin works and was intended to work.
First, there are the following properties which control whether emma instruments product and test classes, respectively:
maven.emma.mode.product=replace_classes | none
maven.emma.mode.testing=replace_classes | none
There are also these goals which can turn the instrumentation hooks on or off "on the fly":
emma:on
emma:off
Note that these currently only seem to affect "product" instrumentation, in other words, even if you call emma:off, test classes will still be instrumented. Also, if you call emma:on with the props NOT enabled, instrumentation of test classes will NOT be enabled. Not sure if this is the best behavior, but that's the way it is now (I'll probably try to fix this).
So, the user currently has to explicitly choose to perform instrumentation, either by setting the props or calling the emma:on (ignoring for now the problems with emma:on).
This means that the "uninstrument" goal (as I envision it) could be used if the user ALREADY has instrumented their classes to perform some tests and get emma reports, but now wants to have uninstrumented classes available so they can build a deployable without instrumented classes, in the same build run.
Make any sense?
Thanks,
Chad
Thanks for info. I now better understand the solution.
I must confess that for me, emma:on/off switches are not really intuitive. Their name reflects more properties than targets.
I tried using them this week-end, and I did not find a natural way to integrate them in my builds.
Maybe we should consider getting rid of them and refactor the plugin.
J-F
yes, I agree that the current plugin is a "good start", but is non-intuitive in many ways. For example, I'd prefer to use "true|false" instead of "replace_classes" and "none".
Any refactoring suggestions would be welcome - but I'd start a new thread for them. Or, even better, submit an enhancement request with a patch.
Any refactorings can be made backward compatible, or at a minimum, we can deprecate current usage with very clear instructions on how to use a new approach.
Also, there are still some things I want to do to clean up the current code before starting refactoring (see bug list).
-- Chad
"replace_classes" was intended to be a *strategy*. The intention was to have a design that allows us to implement various other strategies in the future. These include "switch_for_test" (this will restore the original classes after test), "instrument_war", and whatever else.
There are two reasons for going with named strategies rather than exposing a whole bunch of properties that control behavior at a very fine granularity.
1. a strategy can be made comprehensive and self consistent, while doing the same with a host of variables makes this much harder. Also described as "less state space."
2. a strategy hints at a *pattern* of use, while its difficult to convey your intention to others using a slew of behavioral parameters.
Having said that, I agree that the usability should support "on", "true" and interpret those as some default strategy.
Furthermore, I admit that the rigidness of maven's build sequence makes developing interesting strategies more cumbersome. For example, maven doesn't allow one to assemble a jar and then run tests against it. (As an aside, this is a deficiency that spans beyond emma integration.) Not allowing the flexibility of running unit tests on the assembled binary means that the strategy "dual_binary" where both an instrumented and clean jar are shipped is a whole lot more complicated. So perhaps the strategy idea is out of place in a maven world.
Thanks for the clarifications. The direction and implementation of the plugin makes more sense now.
I agree that maven makes multi-step build processes difficult. You can usually overcome this with custom goals. However, the bigges problem I've encountered here is that many plugins (war, jar) insist on running the tests themselves. Unless you are very careful, this results in the tests being run multiple times, or even worse, endless loops. I've learned to hack around this with "enabletests" and "disabletests" goals (I use a similar approach in some of the new plugin code).
I subscribed to the maven2 list. Hopefully when I get time, I can bring up these issues and try to get them addressed in a more elegant way in maven2.
Also, even with a strategy approach, I think it is very good to have sensible "defaults" which work out of the box, as well as reasonable true/false aliases for the defaults. It's the classic tradeoff between pleasing newbies and powerusers at the same time.
If you continue following the mailing list and defect/enhancement list, please feel free to provide any feedback. I'll try to discuss any major changes here and get feedback before I do them.
Currently, I'm trying to find the time to investigate this before I do anything new:
http://sourceforge.net/tracker/index.php?func=detail&aid=1166107&group_id=108932&atid=651897
Thanks,
Chad
I agree with you Chad. That is exactly the problem I face: tests are executed by 'jar' target.
I have no experience with Clover, but if they have found a nice way to integrate their tool with Maven, I think we should use it as a starting point.
Regarding strategies, don't you think that strategy should be based on activity and not type of classes?
I mean having a strategy for unit testing ('test' target) and one for build ('jar' target) rather than one for production and one for test classes.
The strategy for testing could then be 'replace_classes' and for building 'restore_classes'.
BTW, I actually find no need for restoring test classes. I really don't care whether they are instrumented or not. It is for production classes that I want control.
J-F
Having a test vs. build strategy instead of production classes vs. test classes strategy is interesting.
However, I still fear this may not be flexible enough. There are different types of tests, unit vs. integration, etc. Users may want to treat these differently.
I think this gets to the point that perhaps the strategy pattern is not applicable here. From the GOF book, "Define a family of algorithms ... make them interchangable ... lets the algorithm vary independently"
However, this implies that there is a SINGLE thing to do, and MULTIPLE ways to do it. In our case, there are MULTIPLE different tasks (perform unit tests, perform integration tests, build app for testing, build app for deployment), and MULTIPLE ways to do them (instrument and leave instrumented, instrument and de-instrument, do nothing).
So, I'm not sure what would be the best alternative, but I'm becoming convinced that the current implementation isn't really flexible enough.
Having said that, I think that Dilum's (hope that's right) has a very good point that we should avoid "exposing a whole bunch of properties that control behavior at a very fine granularity". I'm just not sure the strategy pattern is the way to approach that...
Comments are welcome...
Thanks,
Chad
I thought about the "strategy" approach more, and I've got a question. You said:
"replace_classes" was intended to be a *strategy*. The intention was to have a design that allows us to implement various other strategies in the future. These include "switch_for_test" (this will restore the original classes after test), "instrument_war", and whatever else.
Did you intend to allow multiple strategies during the same build session? If so, how? Comma delimited mutiple strategies, perhaps? For example, a user may want to "switch_for_test" and "instrument_war" during the same build session.
-- Chad
The emma:on, emma:off are modelled around the clover plug-in. They allow you to have the instrumentation off by default, but enable coverage for a particular run. These allow you to do things like:
maven clean emma:on test
The Mina project (http://directory.apache.org/subprojects/network/mina/) nicely handles this problem.
It maintains separate target/classes.emma and target/test-classes.emma directories containing enhanced classes.
It would be nice to incorporate the changes in maven.xml in the plugin itself.
Cheers,
J-F
Of possible interest, but I think Maven already compiles actuall test harness files for JUnit into target/test-classes. I put my instrumented classes in target/coverage-classes, and have reports generated in target/coverage-reports.
...Chad
That is even better, indeed.
I guess you had to make your own jelly scripts. Could you share them ?
Best regards,
j-F