Adding Debug output for failing test
Brought to you by:
blep
Menu
▾
▴
Hi,
I'm in a situation where i write a lot of testcases which bodies looks like this:
for (.........) {
CPPUNIT_ASSERT (....);
}
In this case it is really more efficient to add debug printouts to see what is going on. You however do not want to have the printouts when the tests succedes (so i have found myself commenting out debug printouts, which takes up both my and the compilers valuable time).
My idea is to "rerun" those test that fails, but instaed of throwing exeptions when executing CPPUNIT_ASSERT_XXX a debug printout will be made instead. Possibly a CPPUNIT_DEBUGOUT could be added which is only produces output in "rerun" mode in order to be able to output in a readable way.
Furthermore cppunit should/could have better support ofr debugging by allowing to custimize the behaviour when a CPPUNIT_ASSERT fails. Cppunit should be able (e.g. by setting an option) be able to abort, instead of throwing an exception. When running in this mode in an debugger, the debugger will (more or less) stop exactly at the point of failure and allowing for a closer inspection.
I have already started to experiment with the code (cppunit-1.9.10) to include these changes, and i would appreciate if i could get feedback on if this is a good idea and how I should go about to get my changes included in the source code (I guess that this project some process for including new features).
Best regards
Jan
Hi,
it seems to me what you want is a trace, not an assertion. Test code (at least mine) usually relies on the truth of an assertion afterwards. I reckon that test suites would become less robust if using optionally true assertions.
However I would welcome a tracing facility in CppUnit and a way to break at an assertion right before it actually triggers. I got half way towards the first one by using my own TestRunner and the second can be sort-of -accomplished by telling the debugger to break at an exception.
Just my 2pc,
MCM
Hi Martin,
It is not the case that I *always* want a trace, only when an assert fails for a specific test. To clarify my situation se some real code below (with trace printouts currently disabled).
void testArray () {
valarray<int> v = a->array ();
// cout << "\n";
for (size_t r = 0; r < a->rows (); r++) {
for (size_t c = 0; c < a->columns (); c++) {
// cout << v[r*a->columns () + c] << " == " << rowcolval(r, c) << ", ";
CPPUNIT_ASSERT (v[r*a->columns () + c] == rowcolval (r, c));
}
// cout << "\n";
}
// cout << "\n";
}
As you see what gets printed, and what is in the CPPUNIT_ASSERT is very simmilar and lazy programmers should not need to repeat that code (not to mention the possibility for typos, and the need to recompile, when a test breaks).
In any case I had made some progress with adding the functionality described above
First the changes that I'm satisfied with:
diff -r1.1.1.2 Asserter.h
136a137,148
>
> /*! \brief Describes what action to take if an (cppunit) assert fails.
> */
> enum AssertFailAction {report, analyze, abortToDebugger};
>
> static void CPPUNIT_API setAssertFailAction (Asserter::AssertFailAction action)
> { assertFailAction = action; }
> static AssertFailAction CPPUNIT_API getAssertFailAction ()
> { return assertFailAction; }
>
> private:
> static AssertFailAction assertFailAction;
RCS file: /cvsrep/cppunit/cppunit/src/cppunit/Asserter.cpp,v
retrieving revision 1.1.1.2
diff -r1.1.1.2 Asserter.cpp
4c4
<
---
> #include <iostream>
7a8
> Asserter::AssertFailAction Asserter::assertFailAction = Asserter::report;
21c22,29
< throw Exception( message, sourceLine );
---
> switch (assertFailAction) {
> case report: throw Exception( message, sourceLine ); break;
> case analyze: std::cout << message.shortDescription()
> << message.details ();
> break;
> case abortToDebugger: /* print error text */ abort (); break;
> default: /* throw unacconted "unaccounted Assert Fail Action" */ break;
> }
I'm not very satisfied with the need to write to cout in this method, since this should be handled by one of the outputter clsses, but I have not been able to make a good connection.
The other two changes are in:
RCS file: /cvsrep/cppunit/cppunit/include/cppunit/TextOutputter.h,v
retrieving revision 1.1.1.2
diff -r1.1.1.2 TextOutputter.h
55a56,72
> class CPPUNIT_API TextDebugPrintOutputter : public TextOutputter
> {
> typedef TextOutputter inherited;
> public:
> TextDebugPrintOutputter (TestResultCollector *result, std::ostream &stream)
> : TextOutputter (result, stream)
> {}
>
> virtual void printFailure (TestFailure *failure, int failureNumber);
>
> private:
> /// Prevents the use of the copy constructor.
> TextDebugPrintOutputter( const TextOutputter © );
>
> /// Prevents the use of the copy operator.
> void operator =( const TextOutputter © );
> };
and
RCS file: /cvsrep/cppunit/cppunit/src/cppunit/TextOutputter.cpp,v
retrieving revision 1.1.1.2
diff -r1.1.1.2 TextOutputter.cpp
6c6,8
<
---
> #include <cppunit/Test.h>
> #include <cppunit/Asserter.h>
> #include <cppunit/TestResult.h>
60a63,76
> }
>
>
> void TextDebugPrintOutputter::printFailure( TestFailure *failure,
> int failureNumber)
> {
> inherited::printFailure( failure, failureNumber );
> if (!failure->isError()) {
> Asserter::AssertFailAction oldAction = Asserter::getAssertFailAction ();
> Asserter::setAssertFailAction (Asserter::analyze);
> TestResult* result = new TestResult();
> (failure->failedTest())->run( result );
> Asserter::setAssertFailAction (oldAction);
> }
(Memory leak above needs to be corrected).
the final step is to activate the new outputter in the main function:
RCS file: /cvsrep/MatrixModule/MatrixModule/Matrix/test_u/MatrixTest.cpp,v
retrieving revision 1.11
diff -r1.11 MatrixTest.cpp
4c4
<
---
> #include <cppunit/TextOutputter.h>
244a245,246
> runner.setOutputter (new CppUnit::TextDebugPrintOutputter (&runner.result(),
> std::cout));
For ease of reference my current main function can be viewed below:
int main (int argc, char* argv[])
{
CppUnit::TestFactoryRegistry&
registry = CppUnit::TestFactoryRegistry::getRegistry();
CppUnit::TextUi::TestRunner runner;
runner.addTest (registry.makeTest ());
runner.setOutputter (new CppUnit::TextDebugPrintOutputter (&runner.result(),
std::cout));
// bool wasSucessful = runner.run ("", false);
bool wasSucessful = runner.run ();
#ifdef _Windows
return wasSucessful;
#else
return wasSucessful ? 0 : 1;
#endif
}
If you only need the ability to abort you only need to make the changes in Asserter.{cpp,h} and add the folowing in your your main before calling runner.run().
Asserter::setAssertFailAction (Asserter::abort);
Optimally only when an -abortonerror option is set when starting you tests.
I still would apreaciate if someone could give me info on changes to cppunit could be incorporated into the source code on sourceforge (or atleast be considered for inclusion).
Best regards, Jan
Conditional output is OK by me - my problem is that a failed assertion ought to stop execution of the current command sequence, since the remaining code lines rely on the assertion being true. That's what "assertion" means.
As I understand the design, a part of a test which does not rely on the rest to succeed should be it's own test. Code used by more than one test should go into setUp() and tearDown() or just in plain old methods.
I solved the problem of sufficiently verbose textual output (and a few other problems) by writing my own test caller, which looks somewhat like that:
template<typename Fixture>
class CMCMTestCaller : public CppUnit::TestCase {
public:
typedef void (Fixture::*TestMethod)();
CMCMTestCaller(std::string name, TestMethod test, Fixture* fixture)
: TestCase(name), m_pclTestMethod(test), m_pclFixture(fixture) {}
~CMCMTestCaller() {
delete m_pclFixture;
}
void setUp() {
try {
m_pclFixture->setUp();
} catch(const CppUnit::Exception &e) {
FLOG("setUp() failed: " << e.what() << " at line " << e.sourceLine().lineNumber() << " in " << e.sourceLine().fileName() << "." << std::endl << std::flush);
throw;
}
catch(std::exception &e) {
FLOG(getName() << "() ends in setUp() with std::exception : " << e.what() << std::endl << std::flush);
throw;
}
catch(...) {
FLOG(getName() << "() ends in setUp() with unknown exception." << std::endl << std::flush);
throw;
}
}
void tearDown() {
try {
m_pclFixture->tearDown();
}
...catch handlers like in setUp()...
}
void runTest() {
FLOG(getName() << "() starts." << std::endl << std::flush);
try {
(m_pclFixture->*m_pclTestMethod)();
}
...catch handlers like in setUp()... FLOG(getName() << "() ends." << std::endl);
}
std::string toString() {
return "CFCMTestCaller." + getName();
}
private:
Fixture *m_pclFixture;
TestMethod m_pclTestMethod;
};
FLOG is a macro that writes to a globally declared output stream (dirty, I know, but quick :-)
I'm using the GUI TestRunner BTW.
Greetings,
MCM
> ... a lot of testcases ... looks like this:
> for (.........) {
> CPPUNIT_ASSERT (....);
> }
In this case I use the "message"-version of the ASSERT-makro. In the message I show the current value of the loop index.
In a second test-run you can stop at the right loop-run using a conditional breakpoint.
Seems to me a lot simpler than changing (complicating) the CppUnit-Code.
Wolfgang
Hi Martin,
I have not at all considered to wrap the calls in the manner like you do, but I need to give it some more thougt if your method solves any problems for me (actually, reading your code made me realize that the second time i call the run method i need to wrap that call with a try and catch, so thanks for your input.)
I still have the feeling that I have not been able to describe what output should be generated when a test failes (then there is of cource the possibility that you *have* understood, but I have not realized that you desagree with me).
I will try to give what I hope will be a final example.
Consider that you are testing an sorting algorithm. Then the input data and the sorting will happen in the setUp method.
You will then test if the sort was succesfull by writing:
// C.E. code not fed through compiler, may not even compile
void SortTest::testSort ()
{
iter i = s->begin ()
Hi Martin,
I have not at all considered to wrap the calls in the manner like you do, but I need to give it some more thougt if your method solves any problems for me (actually, reading your code made me realize that the second time i call the run method i need to wrap that call with a try and catch, so thanks for your input.)
I still have the feeling that I have not been able to describe what output should be generated when a test failes (then there is of cource the possibility that you *have* understood, but I have not realized that you desagree with me).
I will try to give what I hope will be a final example.
Consider that you are testing an sorting algorithm. Then the input data and the sorting will happen in the setUp method.
You will then test if the sort was succesfull by writing:
// C.E. code not fed through compiler, may not even compile
void SortTest::testSort ()
{
iter i = s->begin ()
Ups,
Managed to press TAB and space - which sent the message prematurely. Complete example follows
// C.E. code not fed through compiler, may not even compile
void SortTest::testSort ()
{
iter i = s->begin ();
double d2 = *i++;
while (i != s->end()) {
double d1 = d2;
d2 = *i++;
CPPUNIT_ASSERT (d1 <= d2);
}
}
Now, *any* of the asserts fail, I want to have a complete list of the values printed (for XML output optimally those invokations of CPPUNIT_ASSERT that do not satisfy the condition should be red).
Supposing that the test above failed additional output from the run above could be:
1 <= 2
2 <= 4
4 <= 5
5 <= 3 // red if output is XML
3 <= 7
In this way, it is easier to see what went wrong, whithout having to write any aditional code (except for the changes in the framework). And as mentioned earlier, trace output is/shold only be generated for tests that have failed.
I hope I have been able to explain the effect I want to achieve by this and I'm looking forward for your reaction (e.g. do you think my head is screwed on right).
Best regards, Jan.