#69 Create runner that will fail mocks to test the tests

open
nobody
5
2009-11-11
2009-11-11
David Harkness
No

I wrote a fairly complex unit test today, and when it passed on the first try I wondered if I had coded it properly. Then the thought: "How do I test my test code?"

One way to do that would be to run each test case multiple times, and for each run change one mock return value, expected parameter, or call count restriction in a random way. This could be done in the mocks themselves with integration with the unit testing runner, thought it might require modifying the tests somewhat. The class responsible for making this happen would need to know about all of the mocks and expectations for each individual test.

This of course can be captured as the test runs. The first time through it would capture them while running the normal test. If the test passed, it would rerun the test once for each mock expectations, each time changing a single expectation that should cause the test to fail. The test would "pass" if at least one failure was detected.

It might be complicated to figure out a generic way to fail a mock expectation. For an int return value, you could return a different int. Same for a string. But what if the code your testing only uses the low bit of the number or a few characters from the string? Another option would be to add an optional "failure value" parameter or expectation to each mock expectation. This would handle the case where objects are returned. For example,

$mock->setReturnValue('foo', 5);
$mock->setFailReturnValue('foo', 2);

This would be required only when the mock can't generate its own failure. Some cases are easier, such as call count expectations. Actually, call count expectations only fail to be tested if you don't verify the mocks at the end. Having UnitTestCase auto-verify all mocks at the end of each test would take care of that. EasyMock for Java provides its own subclass of TestCase to do that; could SimpleTest add it to UnitTestCase?

Discussion