cppunit-devel

[Cppunit-devel] class overview I: defining tests

[Steve M. R. <ste...@vi...>]

In response to Baptiste's message about reorganizing include/, I
expressed a desire to understand the overall structure of CppUnit.  I
have gleaned some information from Baptiste's response and from poring
over the code.  With this summary (in two parts) of what I've learned
so far, I'm hoping to provoke some more discussion.

I'm ignoring the assertions and helper macros, for now.

Suppose I want to incorporate testing into my software.  There are two
fundamental operations that need to be supported by the CppUnit
testing framework.  First, I need to define the tests.  Second, I need
to execute them and report results.  This message deals with classes
of the first category.  A companion message deals with classes for
reporting results.



		class Test
		----------

CppUnit has a small hierarchy of classes rooted at class Test.  Test
is a pure virtual class with no implementation code.  I will assume it
is intended to be an interface class.  By this I mean it exists only
to define the interface that a "test-like" classes must implement.  If
we were writing Java, it would be an interface.

The Test interface has one principal method:

	void run( TestResult* res )
	- run the test and collect the results in res


		class TestCase
		--------------

The concrete class that represents a test should be TestCase.
In fact, however, TestCase is a framework to define one OR MORE tests.
The detailed description of TestCase says it defines the "fixture
to run multiple tests".  

The term "fixture" is bothering me.  The documentation suggests that a
TestCase _is_ a fixture.  If so, why have two terms?  Does the word
"fixture" have a standardized meaning?

TestCase is a subclass of Test, and implements run() as

	setUp()
	runTest()
	tearDown()

According to the "cookbook", a TestCase object is intended as the
basic mechanism for defining a test.  One should subclass TestCase
overriding runTest() appropriately.  The code in runTest() would
typically make a number of assertions.  


		class TestSuite
		---------------

Straightforward collection of Test objects.  The run() method calls
run() on each member of the collection in turn.


		class TestCaller
		----------------

Here is where I really start getting muddled.

A TestCaller is a subclass of TestCase that "provides access to a test
case method".  Although nowhere clarified in the docs, "a test case
method" appears to refer to any member function of a TestCase that
takes no parameters and returns void.

All through the TestCaller documentation, the term "fixture" is
used to mean "a TestCase instance".

As near as I can tell, a TestCaller exists so that one can subclass a
TestCase, define a number of "void foo(void)" methods in it, and then
use a TestCaller to make an individual test case out of each such
method.

I'm confused now as to whether an object of type TestCase should
be a single test case or a set of related test cases.  The name
suggests it is a single test case.  However, the TestCaller class
appears to promote the practice of defining a set of related tests
in a TestCase, and then generating a test case (TestCaller) for each.

In addition, the TestCaller class is able to ignore an "expected"
exception in its runTest() method.  This is a useful facility!
Why is it not available in the base TestCase class?


		classes TestDecorator, RepeatedTest, TestSetUp
		----------------------------------------------

TestDecorator and RepeatedTest are straightforward.

TestSetUp is a mystery, though.  It appears very similar to
the TestCase class in that it wraps the Test::run() method
with virtual setUp() and tearDown() methods.  But it doesn't 
do any catching of exceptions like TestCase does.
What is the purpose of this class?


-Steve

-- 
by Rocket to the Moon,
by Airplane to the Rocket,
by Taxi to the Airport,
by Frontdoor to the Taxi,
by throwing back the blanket and laying down the legs ...
- They Might Be Giants

Re: [Cppunit-devel] class overview I: defining tests

[Baptiste L. <gai...@fr...>]

Quoting "Steve M. Robbins" <ste...@vi...>:

> 		class Test
> 		----------
> 
> CppUnit has a small hierarchy of classes rooted at class Test.  Test
> is a pure virtual class with no implementation code.  I will assume it
> is intended to be an interface class.  By this I mean it exists only
> to define the interface that a "test-like" classes must implement.  If
> we were writing Java, it would be an interface.

  Just to put it out since we haven't yet, Test/TestCase/TestSuite are 
respectively the Component/Leaf/Composite of a Composite pattern.

> 
> The Test interface has one principal method:
> 
> 	void run( TestResult* res )
> 	- run the test and collect the results in res

  Don't dismiss getName(), it is an essential element of "reporting" result.

   
> 		class TestCase
> 		--------------
> 
> The concrete class that represents a test should be TestCase.
> In fact, however, TestCase is a framework to define one OR MORE tests.
> The detailed description of TestCase says it defines the "fixture
> to run multiple tests".  
> 
> The term "fixture" is bothering me.  The documentation suggests that a
> TestCase _is_ a fixture.  If so, why have two terms?  Does the word
> "fixture" have a standardized meaning?
> 
> TestCase is a subclass of Test, and implements run() as
> 
> 	setUp()
> 	runTest()
> 	tearDown()
> 
> According to the "cookbook", a TestCase object is intended as the
> basic mechanism for defining a test.  One should subclass TestCase
> overriding runTest() appropriately.  The code in runTest() would
> typically make a number of assertions.  

  I'm not sure what "fixture" means in english. The dictionnary did not provide 
anything that made sense. The "running" definition I have is that a fixture 
provides "context" and "facility" to run a test. I do not know of any 
standardized meaning, though it's also used in JUnit if I remember well.
  I'll continue the discussion about the TestCase/Fixture in TestCaller...


> 		class TestCaller
> 		----------------
> 
> Here is where I really start getting muddled.
> 
> A TestCaller is a subclass of TestCase that "provides access to a test
> case method".  Although nowhere clarified in the docs, "a test case
> method" appears to refer to any member function of a TestCase that
> takes no parameters and returns void.
> 
> All through the TestCaller documentation, the term "fixture" is
> used to mean "a TestCase instance".
> 
> As near as I can tell, a TestCaller exists so that one can subclass a
> TestCase, define a number of "void foo(void)" methods in it, and then
> use a TestCaller to make an individual test case out of each such
> method.
> 
> I'm confused now as to whether an object of type TestCase should
> be a single test case or a set of related test cases.  The name
> suggests it is a single test case.  However, the TestCaller class
> appears to promote the practice of defining a set of related tests
> in a TestCase, and then generating a test case (TestCaller) for each.
  I finally get why you are so confused. Indeed you pointed out a weird stuff 
in the design.

  To write a unit test, you basically have two ways:
  - subclass TestCase, and override the runTest() method to implement your test.
  - have a class that define a setUp() and a tearOff() methods, and some tests 
methods. Then create a TestCaller to create a TestCase for each test method.

  As you can see, TestCaller does not really need a TestCase. As to why 
TestCase is used, you'll have to go down to the original CppUnit version.
  The fact are (concerning the use of TestCase to create test suite of 
TestCaller):
  - TestCase provides a base class, which define setUp() and tearOff() with 
empty implementation.
  - TestCase defines extraneous methods, such as getName(), runTest()...
  - TestCase is not use as a TestCase.

  Having a base class is a good thing (At least I think so, it make writing the 
macros much easier).
  Having the extraneous methods and a "wrong" use of TestCase is confusing.

  To solve that problem, I would suggest introducing a new base class:

/*! This class represents an abstract fixture used to run test with TestCaller.
 *
 * A TestFixture provides a context and facilities to a set of TestCase
 * created with a TestCaller.
 *
 * setUp() is called to initialize the context before running a test, and
 * tearOff() to clean up after running a test.
 */
class TestFixture
{
public:
  virtual ~TestFixture() {}

  virtual void setUp() {}
  virtual void tearOff() {}
};

Well, the definition need work, but the idea is there. The TestFixture is a 
place where you factor out common piece of code of many test case: object 
initialization, comparison...

We would use the TestFixture class, just like we are using the TestCase class 
to define many tests:

class MyTest : public CppUnit::TestFixture
{
  CPPUNIT_TEST_SUITE( MyTest );
  CPPUNIT_TEST( testSomething );
  CPPUNIT_TEST( testSomethingElse );
  CPPUNIT_TEST_SUITE_END();
  
public:
  void setUp();
  void tearOff();

  void testSomething();
  void testSomethingElse();
};

Hope this help solve the confusing stuff. (Note that JUnit is even weirder in 
that aspect).

> 
> In addition, the TestCaller class is able to ignore an "expected"
> exception in its runTest() method.  This is a useful facility!
> Why is it not available in the base TestCase class?

  It is more than "ignoring": if the expected exception is not caught, the test 
failed. This means that we are implementing a test.
  I believe the base TestCase class should remain "test free" (provide facility 
to run and report result, but to not do any test). But you rise an interesting 
question: should the "exception" expected test be in a TestDecorator rather 
than in TestCaller ? This would also make the TestCaller "test free".


> 		classes TestDecorator, RepeatedTest, TestSetUp
> 		----------------------------------------------
> 
> TestDecorator and RepeatedTest are straightforward.
> 
> TestSetUp is a mystery, though.  It appears very similar to
> the TestCase class in that it wraps the Test::run() method
> with virtual setUp() and tearDown() methods.  But it doesn't 
> do any catching of exceptions like TestCase does.
> What is the purpose of this class?
  To use delegation instead of subclassing to set a testing context.

  Here is an example of use:
  Problem: I need to set a global context each time I run the tests using the 
TestRunner. This could be initializing Thread Local Storage to set up the 
Database connexion, initializing resource such as COM...
  Solution: I provide a way for the user to "wrap" the test being run. The user 
could use TestSetUp to do that.

  Baptiste.
---
Baptiste Lepilleur <gai...@fr...>
http://gaiacrtn.free.fr/index.html
Language: English, French

Re: [Cppunit-devel] class overview I: defining tests

[Steve M. R. <ste...@vi...>]

Hi,

Baptiste: thanks for your comments; they are helping clarify the
CppUnit design in my mind.

Everyone: other comments welcome!



On Tue, Jul 10, 2001 at 03:24:45PM +0200, Baptiste Lepilleur wrote:
> Quoting "Steve M. Robbins" <ste...@vi...>:
> 
> > 		class Test
> > 		----------
> > 
> > CppUnit has a small hierarchy of classes rooted at class Test.  Test
> > is a pure virtual class with no implementation code.  I will assume it
> > is intended to be an interface class.  By this I mean it exists only
> > to define the interface that a "test-like" classes must implement.  If
> > we were writing Java, it would be an interface.
> 
>   Just to put it out since we haven't yet, Test/TestCase/TestSuite are 
> respectively the Component/Leaf/Composite of a Composite pattern.

Thanks!  That's a useful pointer.


> > The Test interface has one principal method:
> > 
> > 	void run( TestResult* res )
> > 	- run the test and collect the results in res
> 
>   Don't dismiss getName(), it is an essential element of "reporting" result.

True.  While writing, I was in the mind frame of someone who wanted to
understand "how do I write a test case for my code".  The method
getName() should have appeared in part II, but I think I forgot to
mention it there, too.


> > 		class TestCase
> > 		--------------
> > 
> > The concrete class that represents a test should be TestCase.
> > In fact, however, TestCase is a framework to define one OR MORE tests.
> > The detailed description of TestCase says it defines the "fixture
> > to run multiple tests".  
> > 
> > The term "fixture" is bothering me.  The documentation suggests that a
> > TestCase _is_ a fixture.  If so, why have two terms?  Does the word
> > "fixture" have a standardized meaning?
> > 
> > TestCase is a subclass of Test, and implements run() as
> > 
> > 	setUp()
> > 	runTest()
> > 	tearDown()
> > 
> > According to the "cookbook", a TestCase object is intended as the
> > basic mechanism for defining a test.  One should subclass TestCase
> > overriding runTest() appropriately.  The code in runTest() would
> > typically make a number of assertions.  
> 
>   I'm not sure what "fixture" means in english. The dictionnary did not provide 
> anything that made sense. The "running" definition I have is that a fixture 
> provides "context" and "facility" to run a test. I do not know of any 
> standardized meaning, though it's also used in JUnit if I remember well.

OK.  I have not looked at JUnit.  Does anyone know whether JUnit originated
the term, or whether it has an antecedent?

I think the term is being used, as you suggest, as a synonym for "framework",
except that it is "smaller" than a framework or a sub-part of a framework.
Something like: "fixture" is to "framework" as "test case" is to "test suite"?


> > 		class TestCaller
> > 		----------------
> > 
> > Here is where I really start getting muddled.
> > 
> > A TestCaller is a subclass of TestCase that "provides access to a test
> > case method".  Although nowhere clarified in the docs, "a test case
> > method" appears to refer to any member function of a TestCase that
> > takes no parameters and returns void.
> > 
> > All through the TestCaller documentation, the term "fixture" is
> > used to mean "a TestCase instance".
> > 
> > As near as I can tell, a TestCaller exists so that one can subclass a
> > TestCase, define a number of "void foo(void)" methods in it, and then
> > use a TestCaller to make an individual test case out of each such
> > method.
> > 
> > I'm confused now as to whether an object of type TestCase should
> > be a single test case or a set of related test cases.  The name
> > suggests it is a single test case.  However, the TestCaller class
> > appears to promote the practice of defining a set of related tests
> > in a TestCase, and then generating a test case (TestCaller) for each.
>
>   I finally get why you are so confused. Indeed you pointed out a weird stuff 
> in the design.
> 
>   To write a unit test, you basically have two ways:
>   - subclass TestCase, and override the runTest() method to implement your test.
>   - have a class that define a setUp() and a tearOff() methods, and some tests 
> methods. Then create a TestCaller to create a TestCase for each test method.
> 
>   As you can see, TestCaller does not really need a TestCase. As to why 
> TestCase is used, you'll have to go down to the original CppUnit version.
>   The fact are (concerning the use of TestCase to create test suite of 
> TestCaller):
>   - TestCase provides a base class, which define setUp() and tearOff() with 
> empty implementation.
>   - TestCase defines extraneous methods, such as getName(), runTest()...
>   - TestCase is not use as a TestCase.
> 
>   Having a base class is a good thing (At least I think so, it make writing the 
> macros much easier).
>   Having the extraneous methods and a "wrong" use of TestCase is confusing.
> 
>   To solve that problem, I would suggest introducing a new base class:
    [ ... TestFixture ... ]

Yes, I think this would clarify the design.  

To reiterate, TestFixture would contain setUp() and tearDown()
methods.  [You keep using "tearOff", but I assume you really mean
"tearDown", yes?]  You would subclass this, define void methods, and
use the latter as a parameter to TestCaller to create test cases.

With TestFixture, it seems that TestCase may be defined as

	class TestCase : public Test, public TestFixture

and supply implementations of the Test methods, like run().



> Hope this help solve the confusing stuff. (Note that JUnit is even weirder in 
> that aspect).

Oh dear!  Maybe I *shouldn't* have a look at JUnit after all.  I'm already
confused enough :-/


> > In addition, the TestCaller class is able to ignore an "expected"
> > exception in its runTest() method.  This is a useful facility!
> > Why is it not available in the base TestCase class?
> 
>   It is more than "ignoring": if the expected exception is not caught, the test 
> failed. This means that we are implementing a test.

True.  I was being sloppy in my description.


>   I believe the base TestCase class should remain "test free" (provide facility 
> to run and report result, but to not do any test). But you rise an interesting 
> question: should the "exception" expected test be in a TestDecorator rather 
> than in TestCaller ? This would also make the TestCaller "test free".

Whatever the mechanism, it just seems to me that a test case defined

    1. by subclassing TestCase, or
    2. by subclassing TestFixture and invoking TestCaller

ought to have the same facilities, including testing for "expected
exceptions".  

If you mean to make a new TestDecorator subclass for expecting
exceptions, akin to the existing RepeatedTest, then yes; to me, that
sounds like the way to procede.  It has the added virtue of simpler
code in the common case, when the proposed decorator is NOT used.


> > 		classes TestDecorator, RepeatedTest, TestSetUp
> > 		----------------------------------------------
> > 
> > TestDecorator and RepeatedTest are straightforward.
> > 
> > TestSetUp is a mystery, though.  It appears very similar to
> > the TestCase class in that it wraps the Test::run() method
> > with virtual setUp() and tearDown() methods.  But it doesn't 
> > do any catching of exceptions like TestCase does.
> > What is the purpose of this class?
>   To use delegation instead of subclassing to set a testing context.
> 
>   Here is an example of use:
>   Problem: I need to set a global context each time I run the tests using the 
> TestRunner. This could be initializing Thread Local Storage to set up the 
> Database connexion, initializing resource such as COM...
>   Solution: I provide a way for the user to "wrap" the test being run. The user 
> could use TestSetUp to do that.

I'm not sure I understand.  Is the set up code to be run once at the beginning
of a test run?  Or is it to be run just before each individual test case?

Do you have a real-life example use of TestSetUp that shows why it is
superior to other means (subclassing?) of achieving the same end?

-Steve


-- 
by Rocket to the Moon,
by Airplane to the Rocket,
by Taxi to the Airport,
by Frontdoor to the Taxi,
by throwing back the blanket and laying down the legs ...
- They Might Be Giants

Re: [Cppunit-devel] class overview I: defining tests

[Baptiste L. <bl...@cl...>]

----- Original Message -----
From: "Steve M. Robbins" <ste...@vi...>
To: <cpp...@li...>
Sent: Wednesday, July 11, 2001 4:42 PM
Subject: Re: [Cppunit-devel] class overview I: defining tests


> Hi,
>
> Baptiste: thanks for your comments; they are helping clarify the
> CppUnit design in my mind.
>
> Everyone: other comments welcome!
> [...]
> > > class TestCase
> > > --------------
> > >
> > > The concrete class that represents a test should be TestCase.
> > > In fact, however, TestCase is a framework to define one OR MORE tests.
> > > The detailed description of TestCase says it defines the "fixture
> > > to run multiple tests".
> > >
> > > The term "fixture" is bothering me.  The documentation suggests that a
> > > TestCase _is_ a fixture.  If so, why have two terms?  Does the word
> > > "fixture" have a standardized meaning?
> > >
> >  [...]
> >   I'm not sure what "fixture" means in english. The dictionnary did not
provide
> > anything that made sense. The "running" definition I have is that a
fixture
> > provides "context" and "facility" to run a test. I do not know of any
> > standardized meaning, though it's also used in JUnit if I remember well.
>
> OK.  I have not looked at JUnit.  Does anyone know whether JUnit
originated
> the term, or whether it has an antecedent?
>
> I think the term is being used, as you suggest, as a synonym for
"framework",
> except that it is "smaller" than a framework or a sub-part of a framework.
> Something like: "fixture" is to "framework" as "test case" is to "test
suite"?

   I don't really see where it relate to "framework", since the only reuse
there is for a fixture is in the few tests you write using that fixture.
Fixture aren't usally designed for extensibiliy... Oh well, as long as we
know what we are refering to when we are talking about Fixture, it will do.

    That term might be specific to "testing". JUnit introduced a framework,
but the way testing is done is just like the "traditionnal" way with
traditionnal tool (I think): set up a context, run a test....

> > > class TestCaller
> > > ----------------
> > >  [...]
> >   To solve that problem, I would suggest introducing a new base class:
>     [ ... TestFixture ... ]
>
> Yes, I think this would clarify the design.
>
> To reiterate, TestFixture would contain setUp() and tearDown()
> methods.  [You keep using "tearOff", but I assume you really mean
> "tearDown", yes?]  You would subclass this, define void methods, and
> use the latter as a parameter to TestCaller to create test cases.

  Yes, that is exactly what I mean. As you may have already noted, I have
some memory trouble ;-). It is indeed tearDown (one of a reason I use
WorkspaceWhiz to generate my test case, otherwise I would never have them
running!).

> With TestFixture, it seems that TestCase may be defined as
>
> class TestCase : public Test, public TestFixture
>
> and supply implementations of the Test methods, like run().

    I guess so. Hope it doesn't confuse people (TestCaller is a TestCase,
and therefore a TestFixture, and it takes another TestFixture in the
constructor!)

> > Hope this help solve the confusing stuff. (Note that JUnit is even
weirder in
> > that aspect).
>
> Oh dear!  Maybe I *shouldn't* have a look at JUnit after all.  I'm already
> confused enough :-/

    Well, the weird stuff is that the TestCaller is the TestCase itself in
their case. They pass the name of the method to call in the constructor of
TestCase (that why you always need to specify such a constructor). On the
other hand, there are lots of good ideas, probably more documentations.
CppUnit is strongly based on JUnit.

> > [...]
> >   I believe the base TestCase class should remain "test free" (provide
facility
> > to run and report result, but to not do any test). But you rise an
interesting
> > question: should the "exception" expected test be in a TestDecorator
rather
> > than in TestCaller ? This would also make the TestCaller "test free".
>
> Whatever the mechanism, it just seems to me that a test case defined
>
>     1. by subclassing TestCase, or
>     2. by subclassing TestFixture and invoking TestCaller
>
> ought to have the same facilities, including testing for "expected
> exceptions".

    Yes, that why I'd like to remove the "test" from the TestCaller, to keep
it "test free" like the TestCase.

> If you mean to make a new TestDecorator subclass for expecting
> exceptions, akin to the existing RepeatedTest, then yes; to me, that
> sounds like the way to procede.  It has the added virtue of simpler
> code in the common case, when the proposed decorator is NOT used.

    Yes that what I mean. It would also make it much easier to handle
specific case. For example, in many of the TestCases I'm writing, I also
need to check if the exception point out the correct "cause", then I need to
clean up the exception by calling a Destroy method. having a decorator that
could be subclassed would make life much easier.

> > > classes TestDecorator, RepeatedTest, TestSetUp
> > > ----------------------------------------------
> > >
> > > TestDecorator and RepeatedTest are straightforward.
> > >
> > > TestSetUp is a mystery, though.  It appears very similar to
> > > the TestCase class in that it wraps the Test::run() method
> > > with virtual setUp() and tearDown() methods.  But it doesn't
> > > do any catching of exceptions like TestCase does.
> > > What is the purpose of this class?
> >   To use delegation instead of subclassing to set a testing context.
> >
> >   Here is an example of use:
> >   Problem: I need to set a global context each time I run the tests
using the
> > TestRunner. This could be initializing Thread Local Storage to set up
the
> > Database connexion, initializing resource such as COM...
> >   Solution: I provide a way for the user to "wrap" the test being run.
The user
> > could use TestSetUp to do that.
>
> I'm not sure I understand.  Is the set up code to be run once at the
beginning
> of a test run?  Or is it to be run just before each individual test case?
>
> Do you have a real-life example use of TestSetUp that shows why it is
> superior to other means (subclassing?) of achieving the same end?

    Not yet. The example I give is what I plan to add some day to
TestRunner. I need to be able to set up the thread environment (the tests
are running in a separate thead).

    TestSetUp is by no mean superior to subclassing. It just that sometime
you can not subclass (TestRunner does not publish the ActiveTest class). I
haven't find any use (at the current time) for TestSetUp when writing unit
test using TestFixture/TestCaller.

    When you are using TestSetUp, the decorated TestCase become a "Strategy"
for TestSetUp. So the usual Strategy vs Template Method arguments enter in
play here (delegation vs subclassing). I guess it would be interesting to
use TestSetUp if you can reuse your TestSetUp for more than one test (for
example setting up some files, or setting up a thread context).

---
Baptiste Lepilleur <gai...@fr...>   http://gaiacrtn.free.fr/index.html
Author of The Text Reformatter, a tool for fanfiction readers and writers.
Language: English, French (Well, I'm French).