Hi Vlad,
we want to use EMMA for our test coverage analysis and also define a quality gate (like "test coverage of all components must be at least 90%").
But there are modules with a relatively big fraction of code in catch-blocks, which cannot be tested by us, because we cannot force these exceptions
(there are other people with another test environment, where such things are tested, but these tests do not count for our quality gate).
Therefore it would be VERY IMPORTANT for us, to be able to switch off coverage analysis for catch-blocks.
Do you think this could be implemented (in "instr" or even better in "report" phase)? And when?
Kind regards, gcey
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I can't speak for vlad's roadmap of EMMA, so I can't tell you whether ignoring certain types of basic blocks goes against his philosophy.
Having said that, I strongly believe that you should not have complicated code inside a catch block. Instead, I believe anything more complicated that exception wrapping should be delegated to specialized handler method.
You've indicated that you're unable to produce some of the exception conditions in your (unit?) testing environment. This is very often the case. Furthermore, some exceptions are near impossible to simulate anywhere. However, there are different things to test for in exception handling:
1. whether you're handling the important exceptions, and whether you're handling it at the right scope. Basically whether you've got the right try-catch-finally's at the right places in the call stack
2. whether you recover from the exception appropriately. For example, the code may be responsible for releasing a resource, or discarding a session, or performing a "retry". Basically, what sorts of actions are performed within the catch { } and finally { } blocks. With my suggestion, this translates to what exception handling methods are invoked in the blocks.
3. exactly what happens during the error handling. For example, you may want to test that when a session discard is performed, that all the necessary operations are performed, and that the appropriate invariants and post conditions are satisfied. This translates to the actual implementations of the exception handling methods.
Getting #1 right involves establishing clear logical layers, building well designed families of exceptions, and extracting base (abstract) classes that hide any complexities. And this involves code-reviews.
One can address some of #2 with code-review too, but it really does have to be tested. In your case, this will be tested by the other test environment.
However, #3 is squarely in your hands. By extracting the actual meat of your catch { } blocks into separate methods, You should be able to unit-test the implementation. Simply make them package private methods, or use an inner-class strategy to expose the test handler methods to your unit tests.
I hope I've expressed this opinion clearly. Please me know if you have any questions, or have opposing views to this.
Thanks,
-d./
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I generally agree with Dilum. I think that code should be tested proportionally to the size of business/infrastructure value that code provides. In this case, it sounds like the fraction of error handling part of your code is unusually large. It seems that there is nontrivial value in handling errors correctly and thus there should be a nontrivial amount of effort spent on testing it.
I also think code should be designed for testability from the get go. It is often overestimated how difficult is might be to reproduce exception scenarios. There are a number of techniques for that:
- mock objects/dynamic proxies/various flavors of Factory and Builder patterns could be exploited to run testcases against test versions of components, versions that could be programmed to simulate any number of error scenarios;
- fault injection (e.g., instructions that throw pre-programmed exceptions) could be used to simulate abnormal execution paths in product code. AspectJ or any handy bytecode manipulation library may be the right tool for this;
Because inferring semantics from source code is easier than from bytecode, it may require changing EMMA design so that is can incorporate source code parsing. This is substantial effort and right now seems to belong to some very remote future given the many smaller and more urgently requested features. As always, this is a tradeoff between what people want and we can do in our spare time.
Vlad.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Vlad,
we want to use EMMA for our test coverage analysis and also define a quality gate (like "test coverage of all components must be at least 90%").
But there are modules with a relatively big fraction of code in catch-blocks, which cannot be tested by us, because we cannot force these exceptions
(there are other people with another test environment, where such things are tested, but these tests do not count for our quality gate).
Therefore it would be VERY IMPORTANT for us, to be able to switch off coverage analysis for catch-blocks.
Do you think this could be implemented (in "instr" or even better in "report" phase)? And when?
Kind regards, gcey
gcey,
I can't speak for vlad's roadmap of EMMA, so I can't tell you whether ignoring certain types of basic blocks goes against his philosophy.
Having said that, I strongly believe that you should not have complicated code inside a catch block. Instead, I believe anything more complicated that exception wrapping should be delegated to specialized handler method.
You've indicated that you're unable to produce some of the exception conditions in your (unit?) testing environment. This is very often the case. Furthermore, some exceptions are near impossible to simulate anywhere. However, there are different things to test for in exception handling:
1. whether you're handling the important exceptions, and whether you're handling it at the right scope. Basically whether you've got the right try-catch-finally's at the right places in the call stack
2. whether you recover from the exception appropriately. For example, the code may be responsible for releasing a resource, or discarding a session, or performing a "retry". Basically, what sorts of actions are performed within the catch { } and finally { } blocks. With my suggestion, this translates to what exception handling methods are invoked in the blocks.
3. exactly what happens during the error handling. For example, you may want to test that when a session discard is performed, that all the necessary operations are performed, and that the appropriate invariants and post conditions are satisfied. This translates to the actual implementations of the exception handling methods.
Getting #1 right involves establishing clear logical layers, building well designed families of exceptions, and extracting base (abstract) classes that hide any complexities. And this involves code-reviews.
One can address some of #2 with code-review too, but it really does have to be tested. In your case, this will be tested by the other test environment.
However, #3 is squarely in your hands. By extracting the actual meat of your catch { } blocks into separate methods, You should be able to unit-test the implementation. Simply make them package private methods, or use an inner-class strategy to expose the test handler methods to your unit tests.
I hope I've expressed this opinion clearly. Please me know if you have any questions, or have opposing views to this.
Thanks,
-d./
I generally agree with Dilum. I think that code should be tested proportionally to the size of business/infrastructure value that code provides. In this case, it sounds like the fraction of error handling part of your code is unusually large. It seems that there is nontrivial value in handling errors correctly and thus there should be a nontrivial amount of effort spent on testing it.
I also think code should be designed for testability from the get go. It is often overestimated how difficult is might be to reproduce exception scenarios. There are a number of techniques for that:
- mock objects/dynamic proxies/various flavors of Factory and Builder patterns could be exploited to run testcases against test versions of components, versions that could be programmed to simulate any number of error scenarios;
- fault injection (e.g., instructions that throw pre-programmed exceptions) could be used to simulate abnormal execution paths in product code. AspectJ or any handy bytecode manipulation library may be the right tool for this;
- a very modest amount of hackery will allow you to write Java code that throws an arbitrary exception from any method, even if it's a checked exception never declared by that method (see http://www.javaworld.com/javaworld/javaqa/2003-02/02-qa-0228-evilthrow.html for an example).
Such techniques definitely go beyond "unit" testing, but they will be worth it if correct error handling is worth something to you.
---
Now, to make some comments about EMMA's point of view on this. We think that "semantic code exclusion" is generally a good feature to have. The current thinking about it and links to previous discussions are summarized in this RFE: http://sourceforge.net/tracker/index.php?func=detail&aid=986234&group_id=108932&atid=651900
Because inferring semantics from source code is easier than from bytecode, it may require changing EMMA design so that is can incorporate source code parsing. This is substantial effort and right now seems to belong to some very remote future given the many smaller and more urgently requested features. As always, this is a tradeoff between what people want and we can do in our spare time.
Vlad.
Is it possible to move catch blocks into a separate category? They do need a special care and a flat metrics does not show a full picture.
- Alexey.
I think, that the Feature Request
http://sourceforge.net/tracker/index.php?func=detail&aid=986234&group_id=108932&atid=651900
should be implemented very soon, a it would solve this problem and various others too.