UnitTest++

Kim Gr=E4sman wrote:

> The nicest, of course, would be if the framework
> didn't require fixtures to
> implement Setup/Teardown, but only called them
> if they were available. Not
> sure if there's some template trickery to achieve that...?

Another of my ... ramblings decrees that all test suites inherit from
a common base, TestCase, even if they use TEST() not TEST_FIXTURE().

So the "trickery" you request is just a stub implementation of setUp()
and tearDown() in the base. Simple OO.

--=20
  Phlip
  http://c2.com/cgi/wiki?ZeekLand  <-- NOT a blog!!

On 8/26/06, Kim Gr=E4sman <kim@...> wrote:
>
> As it currently stands, if an exception is thrown from the TEST, and the
> fixture destructor also throws an exception, std::terminate will be
> called:
>

Hey Kim-

During one incarnation of UT++ i thought we had that working, the idea bein=
g
something like this:

foreach test
{
     declare test object pointer
     try { test object pointer =3D new test object }
     catch { fixture ctor threw, respond, return }
     try { run test object }
     catch { test execution threw, respond }
     try { delete test object }
     catch { fixture dtor threw, respond, return }
}

That way we can handle and identify crashes/throws as being from the fixtur=
e
setup, the test execution, or the fixture teardown.

If not, maybe we can drop that in.  I'll take a quick look on sunday.

chas

Charles Nicholson wrote:

> That way we can handle and identify crashes/throws as being from
> the fixture
> setup, the test execution, or the fixture teardown.
>
> If not, maybe we can drop that in.  I'll take a quick look on sunday.

Per my research (in this forum), that forces the destructor to have
its own exception handler, if it does anything that could throw (such
as shutdown a hairy CORBA ORB).

That forces each fixtures destructor to have a complete
try-catch-catch(...) in it, just to prevent a double-throw from a
destructor and lights-out.

So fold duplication. Don't duplicate the try-catch-catch in every
destructor of every non-trivial fixture. Provide a tearDown().

-- 
  Phlip
  http://c2.com/cgi/wiki?ZeekLand  <-- NOT a blog!!

On 8/26/06, Phlip <phlip2005@...> wrote:
>
> Per my research (in this forum), that forces the destructor to have
> its own exception handler, if it does anything that could throw (such
> as shutdown a hairy CORBA ORB).


First off, the psuedocode i listed would be part of UT++ so clients wouldn't
have to provide anything more.  This is just a proposed method to prevent
total failure when a fixture dtor crashes- that means a SEH exception or bad
signal slipped through the UT++ framework.

Second, destructors in C++ should never throw, ever.  See Sutter's
"Exceptional C++ Style" for the numerous and non-trivial details.

Third, remember that these are fixture destructors we're talking about, they
should strive to be trivial, ideally being empty or consisting of a few
'delete' lines.


> That forces each fixtures destructor to have a complete
> try-catch-catch(...) in it, just to prevent a double-throw from a
> destructor and lights-out.


Maybe we're talking about totally different things.  Is your statement based
on the current source code at all?  The idea here is that UT++ would wrap
the fixture class dtor call in a try/catch block so that it can provide
useful feedback ("FooFixture crashed in destructor!") rather than suffer a
std::terminate.  Clients wouldn't have to add any custom try/catch blocks at
all.

So fold duplication. Don't duplicate the try-catch-catch in every
> destructor of every non-trivial fixture. Provide a tearDown().


This suggestion you're making is based on a premise that doesn't really make
sense to me.  I'm not sure what to say except that I don't see an explicit
tearDown() being different from a fixture destructor.  In fact, having a
fixture destructor seems superior to tearDown() just because it involves
less of the underlying framework.  Currently, the internal Test class is
totally hidden to normal users, they simply create a test with

TEST(foo)
{
}

Using a fixture also doesn't involve the framework at all:

struct MyFixture
{
public:
  MyFixture() { ... }
  ~MyFixture() { ... }
};

TEST_FIXTURE(MyFixture, foo)
{
}

UT++ will create your fixture before running the test and destroy it after
running the test.  You never have to derive from Test explicitly; we handle
all that machinery.  I view this as superior to involving interal test
framework classes from which you have to derive as well as virtual methods
you have to override.

-charles

On 8/26/06, Charles Nicholson <charles.nicholson@...> wrote:
>
> Second, destructors in C++ should never throw, ever.  See Sutter's
> "Exceptional C++ Style" for the numerous and non-trivial details.
>

OK for all the pedants out there I realized I misspoke here-  technically i
think it's ok for dtors to _throw_ internally as long as they never _emit_
exceptions.

No language lawyers have yet posted to this list but they're always welcome,
especially when it comes to the dark and dusty corner of C++ known as proper
exception handling.

-chas

Charles Nicholson wrote:

> > Second, destructors in C++ should never throw, ever.  See Sutter's
> "Exceptional C++ Style" for the numerous and non-trivial details.
>
> OK for all the pedants out there I realized I misspoke here-  technically i
> think it's ok for dtors to _throw_ internally as long as they never _emit_
> exceptions.
>
> No language lawyers have yet posted to this list but they're always welcome,
> especially when it comes to the dark and dusty corner of C++ known as proper
> exception handling.

After researching this I tossed it on the C++ newsgroup, and didn't
get any complaints. Either the language lawyers agreed with me, or I
bored them to death. People who are capable of reading the ISO C++
Standard are probably not susceptible to the latter situation...

Start with the C++ FAQ. It simply says that you might think of some
extraneous situation where you force a throw from a destructor to be
safe, but for all normal coding you must behave as if exceptions
within destructors are lethal, and should architect around them.

http://www.parashift.com/c++-faq-lite/exceptions.html#faq-17.3

This guideline puts extra burdens on framework authors. A unit test
framework should offer the feature of catching and processing
exceptions from the shut-down activities of a test case.

You are requiring the users of UnitTest++ to write redundant code in
the destructors to catch all exceptions (including logging them and
such as part of the test run), instead of permitting an exception to
throw from a non-destructor.

-- 
  Phlip
  http://c2.com/cgi/wiki?ZeekLand  <-- NOT a blog!!

On 8/26/06, Phlip <phlip2005@...> wrote:
>
> A unit test framework should offer the feature of catching and processing
> exceptions from the shut-down activities of a test case.


Is this not exactly what I'm proposing in this thread in response to Kim's
original observation that UT++ doesn't gracefully handle crashing/throwing
fixture dtors?

You are requiring the users of UnitTest++ to write redundant code in
> the destructors to catch all exceptions (including logging them and
> such as part of the test run)


Only because i haven't had time to fix the bug that Kim informally reported
as the starter of this thread!  Once we wrap fixture dtors in a try/catch
block in the framework, this will be fixed!

instead of permitting an exception to throw from a non-destructor.


I don't understand what this means.  If any code executed during a test case
_aside_ from a fixture ctor or dtor throws, UT++ catches it and reports it
correctly.

chas

We're taking this thread offline until we figure out what's going on here.
I'm convinced this is a miscommunication, but it's not one that needs to
happen in public.  We'll post the results!  :)

On 8/26/06, Charles Nicholson <charles.nicholson@...> wrote:
>
> On 8/26/06, Phlip <phlip2005@...> wrote:
>
> > A unit test framework should offer the feature of catching and
> > processing
> > exceptions from the shut-down activities of a test case.
>
>
> Is this not exactly what I'm proposing in this thread in response to Kim's
> original observation that UT++ doesn't gracefully handle crashing/throwing
> fixture dtors?
>
> You are requiring the users of UnitTest++ to write redundant code in
> > the destructors to catch all exceptions (including logging them and
> > such as part of the test run)
>
>
> Only because i haven't had time to fix the bug that Kim informally
> reported as the starter of this thread!  Once we wrap fixture dtors in a
> try/catch block in the framework, this will be fixed!
>
> instead of permitting an exception to throw from a non-destructor.
>
>
> I don't understand what this means.  If any code executed during a test
> case _aside_ from a fixture ctor or dtor throws, UT++ catches it and reports
> it correctly.
>
> chas
>

Hi Charles,

> During one incarnation of UT++ i thought we had that working, 
> the idea being something like this: 
> 
> foreach test
> {
>      declare test object pointer
>      try { test object pointer = new test object }
>      catch { fixture ctor threw, respond, return }
>      try { run test object }
>      catch { test execution threw, respond } 
>      try { delete test object }
>      catch { fixture dtor threw, respond, return } }
> 
> That way we can handle and identify crashes/throws as being 
> from the fixture setup, the test execution, or the fixture teardown. 

That looks like it should work. You've worked around the issue by creating
the object dynamically, and non-automatic objects don't suffer from
std::terminate if the destructor throws.

If I'm reading the macros correctly, this works for fixtures, because they
are currently instantiated like: 

  Fixture##Name##Helper mt(m_details);
  mt.RunTest(testResults_);

and I suppose making them do this, instead, should work:

  // try/catch blocks around each line omitted for brevity
  Fixture##Name##Helper pmt = new Fixture##Name##Helper(m_details);
  pmt->RunTest(testResults_);
  delete pmt;

Is this how you pictured it?

FWIW, I just came up with a solution that calls Setup/Teardown *if* the
fixture derives from a known base class and not otherwise. That way, users
can choose whether they want "safe" setup/teardown, or if relying on
construction/destruction is OK. It does involve some meta-programming
though, per [1], to decide whether Setup/Teardown should be called.

One benefit of this approach is that it's still OK to create fixtures with
automatic storage, rather than needing dynamic allocation, but then again,
that's not a big issue in my book.

Cheers,
- Kim

[1] http://gotw.ca/gotw/071.htm

Clarification;

> FWIW, I just came up with a solution that calls 
> Setup/Teardown *if* the fixture derives from a known base 
> class and not otherwise. That way, users can choose whether 
> they want "safe" setup/teardown, or if relying on 
> construction/destruction is OK. It does involve some 
> meta-programming though, per [1], to decide whether 
> Setup/Teardown should be called. 

More specifically, the usage I had in mind would be:

 struct ThrowingFixture : UnitTest::TestFixture
 {
   // override Setup too, if you want
   // void Setup() {}

   void Teardown()
   {
      throw "bubble";
   }
 };

 TEST_FIXTURE(ThrowingFixture, TestDoesNotTerminate)
 {
    throw "bobble";
 }

 struct NonThrowingFixture
 {
   NonThrowingFixture() { /* setup */ }
   ~NonThrowingFixture() { /* teardown */ }
 };

 TEST_FIXTURE(NonThrowingFixture, TestWillTerminateIfFixtureDtorThrows)
 {
    throw "bobble";
 }

I think this is pretty elegant, because if you have Teardown code that might
throw, you can explicitly cover for it by deriving from
UnitTest::TestFixture and overriding the explicit Setup/Teardown.

Just throwing out ideas here (pun unintended)...

- Kim

On 8/27/06, Kim Gr=E4sman <kim@...> wrote:
>
> You've worked around the issue by creating the object dynamically, and
> non-automatic objects don't suffer from std::terminate if the destructor
> throws.


Umm, I don't think that's quite right, Kim.  std::terminate() is called by
the runtime if an exception is thrown _during_ the unwinding of the stack
caused by a previous exception.  That's why throwing in destructors is a
huge no-no.  Phlip kindly pointed me at the C++ FAQ here:

http://www.parashift.com/c++-faq-lite/exceptions.html#faq-17.3

So it is 'safe' (read: non-fatal) to have exceptions emit from destructors
as long as you can guarantee that another exception isn't currently being
handled.  UT++ makes this guarantee in the code by nature of the way it's
laid out.  More on that further down the post.

The big issue we have with the implementation I pitched above is memory
leaks.  Consider the following code:

struct Foo
{
  ~Foo() { throw "doh!"; }
  char mem[1024];
};

There will be no memory leaks in the following code:

{  Foo f; }

Because Foo is an auto variable here, when the stack frame unwinds, the
memory is 'freed' in so much as stack memory is ever 'freed'.

But this code will leak:

{ delete new Foo(); }

The C++ Standard does not guarantee that the memory that hosted the Foo
object will be returned to the free store if its dtor emits an exception.

I have changes on my machine that I haven't committed yet that do this:

1.  char* buffer =3D new buffer[sizeof(Fixture)];
2.  try { Fixture* fixture =3D new(buffer) Fixture(); } catch { report ctor
throw, fail, delete[] buffer, return }
3.  fixture->RunTest(); // internally does try/catch
4.  try { fixture->~Fixture(); } catch { report dtor throw, fail }
5.  delete[] buffer;

I don't see how we can be in an exception-driven stack unwinding by the tim=
e
we invoke the Fixture's dtor (line 4) in this case.

Explicitly managing the memory in which the Fixture lives (lines 1 and 5)
allows us to decouple the allocation from the potentially volatile lifetime
of the object.  This means that fixtures can crash or throw in any
interesting ways they like and UT++ will not only catch it but clean up
after the fixture in its wake.

-charles

Hi Charles,

> > You've worked around the issue by creating the object=20
> > dynamically, and non-automatic objects don't suffer from=20
> > std::terminate if the destructor throws.
>
> Umm, I don't think that's quite right, Kim.  std::terminate()=20
> is called by the runtime if an exception is thrown _during_=20
> the unwinding of the stack caused by a previous exception.

My comment was actually backed up by my interpretation of the standard.
15.5.1 on std::terminate:

"
In the following situations exception handling must be abandoned for =
less
subtle error handling techniques:

[snip]
=97 when the destruction of an object during stack unwinding (15.2) =
exits
using an exception, or
=97 when construction or destruction of a non-local object with static =
storage
duration exits using an exception (3.6.2), or
[snip]
"

And 15.2 says, on stack unwinding:

"
The process of calling destructors for automatic objects constructed on =
the
path from a try block to a throw-expression is called =93stack =
unwinding.=94
[Note: If a destructor called during stack unwinding exits with an
exception, terminate is called (15.5.1). So destructors should generally
catch exceptions and not let them propagate out of the destructor. =
=97end
note]
"

So I took stack-unwinding to mean destruction of automatic objects... I =
just
tested allocating the Fixture##Name##Helper in TEST_FIXTURE_EX =
dynamically
(igoring any leaks), and sure enough, std::terminate is no longer =
called!

> But this code will leak:
>=20
> { delete new Foo(); }=20
>=20
> The C++ Standard does not guarantee that the memory that=20
> hosted the Foo object will be returned to the free store if=20
> its dtor emits an exception.

Yep, this looks familiar.

> I have changes on my machine that I haven't committed yet=20
> that do this:=20
> [snip]

You could try changing:

 char buffer =3D new buffer[sizeof(Fixture)];
to
 char buffer[sizeof(Fixture)];

That'll get rid of the complexities in memory management of the
placement-new buffer.

I think I'm still in favor of my optional explicit Setup/Teardown pair, =
but
this looks like it should work as well... I'll try posting a patch =
showing
my idea in context, and we can compare?

Cheers,
- Kim

On 8/27/06, Kim Gr=E4sman <kim@...> wrote:
>
> So I took stack-unwinding to mean destruction of automatic objects... I
> just
> tested allocating the Fixture##Name##Helper in TEST_FIXTURE_EX dynamicall=
y
> (igoring any leaks), and sure enough, std::terminate is no longer called!


I respectfully disagree, but i'm not 100% sure of my position because it's
very subtle.  To requote, the note in 15.2.3 reads:

[Note: If a destructor called during stack unwinding exits with an
exception, terminate is called (15.5.1)].

Note that the language doesn't say that the destructor is called on an auto
or dynamically allocated object, it just says "If a destructor is called
during stack unwinding".  Given how explicit those guys are about their
language I think we have to assume that they are talking about _any_
destructor, auto or dynamic, but the important thing is that it happens
during a stack unwinding that was triggered by a thrown exception.

My line of thinking is also driven by how it's not obvious to me why the
storage of an object would be relevant to the issue at hand- std::terminate
occurs because the runtime is left in a "damned if you do, damned if you
don't" situation.  It's trying to unwind the stack to get to a catch block
and now another throw happens.  It has to unwind to a catch block, but whic=
h
one?  Either way information is lost, so it throws its hands in the air and
gives up. kaboom.  std::terminate().  The reason i linked this

http://www.parashift.com/c++-faq-lite/exceptions.html#faq-17.3

in my previous response was because it nicely describes the condition and
also makes no mention of the storage types of the classes involved.  That i=
n
itself isn't proof of course but it's a strong suggestion.  If there were a
subtlety involved i'd think it'd be mentioned there...

FWIW, the following program dies when executed after a compile-time warning
about a possible leak from a throwing dtor (built with vs2005 and both
regular and structured exception handling flags).  It dies in the exact sam=
e
way if i change either or both of 'f' and 'bar' to be auto variables:

----------------------------------------------------------
#include <iostream>
using namespace std;

struct Bar
{
    ~Bar() { cout << "good night" << endl; throw "hello"; }
};

struct Foo
{
    Foo() { bar =3D new Bar; }
    ~Foo() { delete bar; }

    Bar* bar;
};

int main()
{
    Foo* f;
    try
    {
        f =3D new Foo();
        throw "bagel";
    }
    catch(...) { delete f; }

    return 0;
}
----------------------------------------------------------

Anyway, changing topics here:

char buffer =3D new buffer[sizeof(Fixture)];
> to
> char buffer[sizeof(Fixture)];
>
> That'll get rid of the complexities in memory management of the
> placement-new buffer.


I've read that it's dangerous to do the explicit-object-hosting-in-memory
trick with stack memory because it won't guarantee the alignment that
new/malloc will.

http://www.parashift.com/c++-faq-lite/dtors.html#faq-11.10

says it's bad and the corresponding

http://www.parashift.com/c++-faq-lite/dtors.html#faq-11.14

says it's ok to do with new/malloc.

I think I'm still in favor of my optional explicit Setup/Teardown pair, but
> this looks like it should work as well... I'll try posting a patch showin=
g
> my idea in context, and we can compare?


I'm actually very much against Setup/Teardown as a concept.  They totally
violate RAII, they force our internal object model upon the user (admittedl=
y
in a very small way but still).  They also now give us 2 ways of doing
things, and you have to know to derive your fixture from TestFixture if it
might throw.  That's confusing and not obvious to me.

Unless there's no way to handle fixture destruction safely and without
leaks,  unless it's an absolute last resort, I'd _really_ like to stay away
from setup/teardown methods.

-charles

Hi Phlip,
 
> New question: I forget if I have asked yet how UnitTest++ 
> supports this pattern:
> 
> http://c2.com/cgi/wiki?AbstractTest
> 
> Advanced documentation might end there, because it avoids 
> endlessly duplicated test cases.

I actually tried to come up with Abstract Test in UT++ yesterday. I never
managed. I don't think it can be done (famous last words) without some
pretty decent additions to the macro family.

FWIW,
- Kim

Kim Gr=E4sman wrote:

> I actually tried to come up with Abstract Test in UT++ yesterday. I never
> managed. I don't think it can be done (famous last words) without some
> pretty decent additions to the macro family.

As I suspected, that's a hardship of the existing architecture.

Could the SUITE() macro upgrade to support it?

My parallel-evolved TEST_() macro supports it like this. It works like
UnitTest++'s TEST_FIXTURE(), and it needs a tweak to support Abstract
Tests. So suppose we have a dialog box that localizes to Sanskrit, and
we need to test English and all the other languages. First we switch
our latest test, _checkAllLabels, to constrain English instead of
Sanskrit, and we extract from the fixture TestSanskrit a base class
called TestDialog. This refactor moves the case we will abstract up
the inheritance graph:

    TEST_(TestDialog, _checkAllLabels)
    {
 // stuff here to find each string in the dialog, check its
 // locale is correct, check it contains no MISSING CHARACTER
 // GLYPHs [], check it doesn't overflow the dialog, etc
    }

Now write a new macro that reuses a test case, such as
_checkAllLabels, into any derived suite, using some good old-fashioned
"Diamond Inheritance":

    #define TEST_(suite, target)             \
        struct suite##target:  virtual suite \
        { void runCase(); }                  \
        a##suite##target;                    \
        void suite##target::runCase()

    #define RETEST_(base, suite, target)    \
        struct base##suite##target:         \
            virtual suite,                  \
            virtual base##target  {         \
         void setUp() { suite::setUp(); }   \
         void runCase() { base##target::runCase(); }  \
         void tearDown() { suite::tearDown(); }  \
        } a##base##suite##target;

=85

Then express that macro with three parameters: The base class, the
derived class whose setUp() and tearDown() we need, and one base class
case. The macro reuses that case with the derived class:

    RETEST_(TestDialog, TestSanskrit, _checkAllLabels)

That change required TestSanskrit to inherit TestDialog virtually, to
ensure that suite::setUp() sets up the same m_aDlg member object as
base##target::runCase() tests.

Without virtual inheritance, C++'s multiple inheritance system would
make a UML diagram of our design into a huge V, disconnected at the
top. The TestDialogTestSanskrit_checkAllLabels object would contain
two different TestDialog sub-objects, and these would disagree which
instance of their member variable m_aDlg to test, and which to
localize to Sanskrit.

Future extensions could create a template that abstracts setUp() and
tearDown() across a list of locales. When the time comes to conquer -
oops I mean "support" - the entire world, we should build more
elaborate Abstract Tests, then declare stacks of them, one per target
locale:

RETEST_(TestDialog, TestLocale< LANG_AFRIKAANS >,_checkAllLabels)
RETEST_(TestDialog, TestLocale< LANG_ALBANIAN  >,_checkAllLabels)
RETEST_(TestDialog, TestLocale< LANG_ARABIC    >,_checkAllLabels)
RETEST_(TestDialog, TestLocale< LANG_ARMENIAN  >,_checkAllLabels)
RETEST_(TestDialog, TestLocale< LANG_ASSAMESE  >,_checkAllLabels)
RETEST_(TestDialog, TestLocale< LANG_AZERI     >,_checkAllLabels)
RETEST_(TestDialog, TestLocale< LANG_BASQUE    >,_checkAllLabels)
=85

Their test cases should sweep each window and control, for each
locale. We can add things like a spelling and grammar checker to only
one place in the code.

--=20
  Phlip
  http://c2.com/cgi/wiki?ZeekLand  <-- NOT a blog!!

Hi Phlip,

> My parallel-evolved TEST_() macro supports it like this. It works like
> UnitTest++'s TEST_FIXTURE(), and it needs a tweak to support Abstract
> Tests. So suppose we have a dialog box that localizes to 
> Sanskrit, and we need to test English and all the other 
> languages. First we switch our latest test, _checkAllLabels, 
> to constrain English instead of Sanskrit, and we extract from 
> the fixture TestSanskrit a base class called TestDialog. This 
> refactor moves the case we will abstract up the inheritance graph:
> 
>     TEST_(TestDialog, _checkAllLabels)
> [snip]

I guess the problem is that UT++'s concept of a Fixture is not really
xUnit-compatible. Where an xUnit fixture contains test cases, a UT++ fixture
is a base type of test case.

So there's no traditional polymorphic way of blending fixtures and test
cases together like in xUnit, where you can build an entire suite of test
cases in a fixture, and leave a single creator method abstract, as in
Abstract Test.

Your RETEST_ macro seems to work, but you have to invoke it for every single
test case, not once per fixture/suite, so it doesn't scale so well...

No ideas from me, sorry.

- Kim

>
> I love UnitTest++ and I can successfully run through all of the self tests
> under Cygwin (including the seg fault test).   However, if the offending
> line  (something like  reinterpret_cast< void (*)() >(0)()) is part of just
> a simple test the unit test executable always goes into an infinite loop.
>
> TEST(AlwaysCausesAnInfiniteLoopUnderCygwin)
> {
>     reinterpret_cast< void (*)() >(0)();
>     CHECK(true);
> }
>

Crazy.  I've confirmed it on my end and have no idea what it is.  I don't
really know much about signals so if any generous posix types on this list
wouldn't mind taking a closer look at this one, we'd all appreciate it...

-chas

Kim,
 
Thanks, here's my reply^2:
 
>> 1) "Special" CHECK-family macro  behavior.
>
>How about "Overloading the CHECK macros"? Quotes in  headings look weird to 
me.


Sure! Just another case in related to TDD where naming something is  harder 
than making the thing :)

>> You can provide your own 
>> implementation of  CheckEqual, specialized for the specific 
>> type(s) you care about,  and it'll be used whenever necessary. 
>
>Technically, it shouldn't  be specialized, but overloaded.  See
>http://www.gotw.ca/publications/mill17.htm, for  example.


Thanks, I even went and looked this up to be sure I was correct, but it  
seems I failed. 

> And what's with the  "(break)"?


More emotional content leaking into my writing, derived from the context  in 
which I use the framework.

> What do the maintainers think? Is there any harm in including this  part of
the docs as well?




I'll take the rest of the good suggestions that I didn't specifically reply  
to above and spam you all once more with a fixed version. Then the  powers 
that be are welcome to ignore or include as needed :)

 
-Noah