Out of curiosity, I decided to run coverage on this function
public void testSomething(String s) {
s.trim();
}
The test case was just to call the funtion passing in a value. I was thinking that possibly, emma would say 50% covered, as I have not tested passing in null which would throw a nullpointerexception.
This would actually has the potential to catch alot of mistakes in the field. The only problem is if it is guarded with a
if(s == null)
throw new IllegalARgumentException("some message");
then it will always be marked as not covered. I think the real thing testers are trying to cover is the method signature. ie. I am trying to cover testSomething(String s) and it is not truly fully covered until it has been tested with all variations of null.
Of course, I am wondering if this is too cumbersome as if you have a 5 parameter method, there are so many variations of null....maybe a method is 100% covered only if each parameter has a test that passes in null and one that does non-null. Do you think this would be useful? It might be interesting to try something like that. It may end up preventing alot of nullpointer bugs in the field for products, but I don't know.
just some thoughs,
dean
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
interesting, I thought I logged in, but the post was anonymous. I found out clicking on the forums link from http://emma.sourceforge.net prevents me from logging in for some odd reason. I have to instead to to emma.sourceforge.net/projects/emma and then from the same forums page at the bottom, I can successfully login. Quite weird.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
never mind, just having trouble with SF in general, finally I am logged in. I don't know why that happened....but I guess it was not emma.sourceforge.net that caused the problem...something else weird.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
<quote>
The test case was just to call the funtion passing in a value. I was thinking that possibly, emma would say 50% covered, as I have not tested passing in null which would throw a nullpointerexception.
</quote>
Why do you expect that (the 50% coverage specifically)? There is an infinity of possible input values and only one value that would cause an NPE on 's'. In general it is hard to automatically infer which input values are "bad".
My personal style is to always make input guards explicit:
if (s == null)
throw new IllegalArgumentException ("null input: s");
if you don't have a testcase that triggers this branch, EMMA will show red for the "throw new" line and this fact will be reflected in line/block coverage numbers. It is then up to you whether you want to add more testcases for this or not. Some might say that the validation code is very straightforward and your testing time is better spent on other red areas.
I like my coding style (of course :) because it makes it very explicit what kind of pre-assertions my methods have. I consider it unprofessional to leave null pointer checks to get triggered by the business logic of a method, possibly deeply inside of it, where the exception stack trace will be confusing to whoever is trying to debug the method.
But a practical problem with this style is that it is very tedious to fully test input validation because of its mechanical/repetitive nature. To this effect, I created a tool based on an extended Java syntax that would allow you to embed pre-assertions like
in any testcase. The extended syntax would then be processed by a source code instrumentation tool and get converted to a set of nested loops that call testSomething() with the given permutations of inputs and assert behavior wrt the exceptions thrown, if any.
Because of a requirement to do source code instrumentation the tool was too cumbersome for the value it provided, so I never open-sourced it.
(However, with the advent of metadata support in Java 1.5 there might be an entirely new way to implement this...)
At the moment I continue to use explicit input guards but I don't expect to have 100% line coverage on them, because the code is too simple to worry about it too much.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Actually, I totally agree and that is my coding style too, and all of my team agrees with doing that type of thing but it is missed all the time. I understand there is infinite possibilities when it comes to passing in a variable, but I was thinking it might be nice to at least test null, something that many seem to forget about. It is not as common as other precondition conditions that just check for valid values. The most common precondition is usually, is this null, if not throw IllegalArgument. Sometimes, you need to verify that a string is one of the following enums or something, but most validations are just that it is not null.
Maybe there is a better way to get people to put these preconds in. everyone on my team agrees we should have them but just forget to add them, and so it doesn't show any lack of code coverage(and maybe it shouldn't).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Out of curiosity, I decided to run coverage on this function
public void testSomething(String s) {
s.trim();
}
The test case was just to call the funtion passing in a value. I was thinking that possibly, emma would say 50% covered, as I have not tested passing in null which would throw a nullpointerexception.
This would actually has the potential to catch alot of mistakes in the field. The only problem is if it is guarded with a
if(s == null)
throw new IllegalARgumentException("some message");
then it will always be marked as not covered. I think the real thing testers are trying to cover is the method signature. ie. I am trying to cover testSomething(String s) and it is not truly fully covered until it has been tested with all variations of null.
Of course, I am wondering if this is too cumbersome as if you have a 5 parameter method, there are so many variations of null....maybe a method is 100% covered only if each parameter has a test that passes in null and one that does non-null. Do you think this would be useful? It might be interesting to try something like that. It may end up preventing alot of nullpointer bugs in the field for products, but I don't know.
just some thoughs,
dean
interesting, I thought I logged in, but the post was anonymous. I found out clicking on the forums link from http://emma.sourceforge.net prevents me from logging in for some odd reason. I have to instead to to emma.sourceforge.net/projects/emma and then from the same forums page at the bottom, I can successfully login. Quite weird.
never mind, just having trouble with SF in general, finally I am logged in. I don't know why that happened....but I guess it was not emma.sourceforge.net that caused the problem...something else weird.
<quote>
The test case was just to call the funtion passing in a value. I was thinking that possibly, emma would say 50% covered, as I have not tested passing in null which would throw a nullpointerexception.
</quote>
Why do you expect that (the 50% coverage specifically)? There is an infinity of possible input values and only one value that would cause an NPE on 's'. In general it is hard to automatically infer which input values are "bad".
My personal style is to always make input guards explicit:
if (s == null)
throw new IllegalArgumentException ("null input: s");
if you don't have a testcase that triggers this branch, EMMA will show red for the "throw new" line and this fact will be reflected in line/block coverage numbers. It is then up to you whether you want to add more testcases for this or not. Some might say that the validation code is very straightforward and your testing time is better spent on other red areas.
I like my coding style (of course :) because it makes it very explicit what kind of pre-assertions my methods have. I consider it unprofessional to leave null pointer checks to get triggered by the business logic of a method, possibly deeply inside of it, where the exception stack trace will be confusing to whoever is trying to debug the method.
But a practical problem with this style is that it is very tedious to fully test input validation because of its mechanical/repetitive nature. To this effect, I created a tool based on an extended Java syntax that would allow you to embed pre-assertions like
testSomething(null, *, *) : throws IllegalArgumentException
in any testcase. The extended syntax would then be processed by a source code instrumentation tool and get converted to a set of nested loops that call testSomething() with the given permutations of inputs and assert behavior wrt the exceptions thrown, if any.
Because of a requirement to do source code instrumentation the tool was too cumbersome for the value it provided, so I never open-sourced it.
(However, with the advent of metadata support in Java 1.5 there might be an entirely new way to implement this...)
At the moment I continue to use explicit input guards but I don't expect to have 100% line coverage on them, because the code is too simple to worry about it too much.
Actually, I totally agree and that is my coding style too, and all of my team agrees with doing that type of thing but it is missed all the time. I understand there is infinite possibilities when it comes to passing in a variable, but I was thinking it might be nice to at least test null, something that many seem to forget about. It is not as common as other precondition conditions that just check for valid values. The most common precondition is usually, is this null, if not throw IllegalArgument. Sometimes, you need to verify that a string is one of the following enums or something, but most validations are just that it is not null.
Maybe there is a better way to get people to put these preconds in. everyone on my team agrees we should have them but just forget to add them, and so it doesn't show any lack of code coverage(and maybe it shouldn't).