Re: [Perlunit-users] getting the exit status "right"

[Matthew A. <mc...@us...>]

On Wed, May 18, 2011 at 09:39:44AM -0400, Robert Schweikert wrote:

> Using the unit test framework for some Perl work and am running into
> a bit of trouble.

Hi Robert,

I'm glad to hear it's still useful.

> When a test fails [...] I get an appropriate error message such as
> "not ok ERROR test_...." the exit status of the test is still 0.

I agree that Perlunit has this all "wrong" in v0.25 - test runners
should exit with a non-zero return code.  None of them do.

Unfortunately this used to be implemented in the T:U:TestRunner module
but was downgraded from exit to die, then again to just return,
because it was causing trouble for other code.  I assume this was the
GUI runner, which expects to continue running after the tests have
finished.

There's no test for the text-mode runners' exit code...

I added a bug for this,
  https://sourceforge.net/tracker/?func=detail&aid=3304750&group_id=2653&atid=102653


> -> perl myTest.t
> .....
> ERROR
> ....
> -> echo $?
> 0

While there are no test runners in v0.25 which will return the correct
exit code, you may be able to get the result you need by having one
t/run_tests.t execute all your test classes under HarnessUnit, and run
prove(1) to call this in the usual way.

This way you will be able to use prove's exit code, which *is* correct.

Your invocation "perl myTest.t" above suggests you have started down
this road.  Does that myTest.t emit TAP (test anything protocol)?

The other workaround is to write your own test runner which exits
according to the result of
  $testrunner->start()

...which is what I did some time ago.  You may find the script at
  http://www.t8o.org/~mca1001/cgi/viewcvs/bin/testrunner.pl

useful (I know my viewcvs is broken, but the data is still there).

I have for a long time intended to release that testrunner.pl with
Perlunit, but in the meantime I got a Life & a heap of more urgent
things to do...


> This is not good as I would like make and rpm package creation to
> fail when a test fails. How do I get the unit test framework to set
> the exit status to something other than 0 when an ERROR occurs?

I hope two workarounds and a vague promise of a better future is
enough for now.

My next step is to rescue the code from CVS into Git, since I'm tooled
up for this at work at the moment.  This is hampered by the usual
problem of putting multiple sets of related code into one CVS repo,
and Git not supporting that, but I should have it fixed up in a few
days if nothing else pops up.


Thanks for waking the project up,
-- 
Matthew