perlunit-users

[Perlunit-users] getting the exit status "right"

[Robert S. <rsc...@no...>]

Hi,

Using the unit test framework for some Perl work and am running into a 
bit of trouble. When a test fails, i.e. an assert is triggered and I get 
an appropriate error message such as "not ok ERROR test_...." the exit 
status of the test is still 0.

-> perl myTest.t
.....
ERROR
....
-> echo $?
0

This is not good as I would like make and rpm package creation to fail 
when a test fails. How do I get the unit test framework to set the exit 
status to something other than 0 when an ERROR occurs?

Thanks,
Robert

-- 
Robert Schweikert                           MAY THE SOURCE BE WITH YOU
SUSE-IBM Software Integration Center                   LINUX
Tech Lead
rsc...@no...
rsc...@ca...
781-464-8147

Re: [Perlunit-users] getting the exit status "right"

[Matthew A. <mc...@us...>]

On Wed, May 18, 2011 at 09:39:44AM -0400, Robert Schweikert wrote:

> Using the unit test framework for some Perl work and am running into
> a bit of trouble.

Hi Robert,

I'm glad to hear it's still useful.

> When a test fails [...] I get an appropriate error message such as
> "not ok ERROR test_...." the exit status of the test is still 0.

I agree that Perlunit has this all "wrong" in v0.25 - test runners
should exit with a non-zero return code.  None of them do.

Unfortunately this used to be implemented in the T:U:TestRunner module
but was downgraded from exit to die, then again to just return,
because it was causing trouble for other code.  I assume this was the
GUI runner, which expects to continue running after the tests have
finished.

There's no test for the text-mode runners' exit code...

I added a bug for this,
  https://sourceforge.net/tracker/?func=detail&aid=3304750&group_id=2653&atid=102653


> -> perl myTest.t
> .....
> ERROR
> ....
> -> echo $?
> 0

While there are no test runners in v0.25 which will return the correct
exit code, you may be able to get the result you need by having one
t/run_tests.t execute all your test classes under HarnessUnit, and run
prove(1) to call this in the usual way.

This way you will be able to use prove's exit code, which *is* correct.

Your invocation "perl myTest.t" above suggests you have started down
this road.  Does that myTest.t emit TAP (test anything protocol)?

The other workaround is to write your own test runner which exits
according to the result of
  $testrunner->start()

...which is what I did some time ago.  You may find the script at
  http://www.t8o.org/~mca1001/cgi/viewcvs/bin/testrunner.pl

useful (I know my viewcvs is broken, but the data is still there).

I have for a long time intended to release that testrunner.pl with
Perlunit, but in the meantime I got a Life & a heap of more urgent
things to do...


> This is not good as I would like make and rpm package creation to
> fail when a test fails. How do I get the unit test framework to set
> the exit status to something other than 0 when an ERROR occurs?

I hope two workarounds and a vague promise of a better future is
enough for now.

My next step is to rescue the code from CVS into Git, since I'm tooled
up for this at work at the moment.  This is hampered by the usual
problem of putting multiple sets of related code into one CVS repo,
and Git not supporting that, but I should have it fixed up in a few
days if nothing else pops up.


Thanks for waking the project up,
-- 
Matthew

Re: [Perlunit-users] getting the exit status "right"

[Robert S. <rsc...@no...>]

Hi Matt,

On 05/19/2011 03:50 PM, Matthew Astley wrote:
> On Wed, May 18, 2011 at 09:39:44AM -0400, Robert Schweikert wrote:
>
>> Using the unit test framework for some Perl work and am running into
>> a bit of trouble.
>
> Hi Robert,
>
> I'm glad to hear it's still useful.
>
>> When a test fails [...] I get an appropriate error message such as
>> "not ok ERROR test_...." the exit status of the test is still 0.
>
> I agree that Perlunit has this all "wrong" in v0.25 - test runners
> should exit with a non-zero return code.  None of them do.
>
> Unfortunately this used to be implemented in the T:U:TestRunner module
> but was downgraded from exit to die, then again to just return,
> because it was causing trouble for other code.  I assume this was the
> GUI runner, which expects to continue running after the tests have
> finished.
>
> There's no test for the text-mode runners' exit code...
>
> I added a bug for this,
>    https://sourceforge.net/tracker/?func=detail&aid=3304750&group_id=2653&atid=102653
>

Thanks for filing the bug.

>
>> ->  perl myTest.t
>> .....
>> ERROR
>> ....
>> ->  echo $?
>> 0
>
> While there are no test runners in v0.25 which will return the correct
> exit code, you may be able to get the result you need by having one
> t/run_tests.t execute all your test classes under HarnessUnit, and run
> prove(1) to call this in the usual way.
>
> This way you will be able to use prove's exit code, which *is* correct.

Thanks for this tip. I was not aware of prove. I tested this and changed 
my test setup to use prove to run the tests.

>
> Your invocation "perl myTest.t" above suggests you have started down
> this road.  Does that myTest.t emit TAP (test anything protocol)?

No, I am not using TAP.

Robert

-- 
Robert Schweikert                           MAY THE SOURCE BE WITH YOU
SUSE-IBM Software Integration Center                   LINUX
Tech Lead
rsc...@no...
rsc...@ca...
781-464-8147

[Perlunit-users] dev release 0.25_1325, Re: getting the exit status "right"

[Matthew A. <mc...@us...>]

On Thu, May 19, 2011 at 08:50:03PM +0100, Matthew Astley wrote:

> My next step is to rescue the code from CVS into Git, since I'm
> tooled up for this at work at the moment.

I did that, and revised the website a little.  You may clone from
SourceForge or Github, details are on http://perlunit.sourceforge.net/

I then fixed the release process and made development releases.
0.25_1325 should fix the outstanding cpantesters fails and the
TestRunner.pl exit code.  There are also some old fixes.
  http://search.cpan.org/~mcast/Test-Unit-0.25_1325/
  http://search.cpan.org/src/MCAST/Test-Unit-0.25_1325/ChangeLog

This doesn't address all outstanding bugs, but it does get the visible
(dist) tests back to a green bar on all OS/Perl combinations so far,
  http://matrix.cpantesters.org/?dist=Test-Unit+0.25_1325


In addition to bugs in the shipped code, there is also a problem with
my CPAN permissions - I don't have permission for all the Test::Unit*
modules in the dist, because PAUSE treats them separately and we
didn't know this back then.

Adam has fixed the modules he can, and I think I have contact details
for Christian.


Anyway, I wanted to let you know things are still happening slowly.

-- 
Matthew

Re: [Perlunit-users] dev release 0.25_1325, Re: getting the exit status "right"

[Andrey A. <aa...@ma...>]

Hello,

I'm glad Test::Unit is under development again - it is a great module. 
Please, take a look at

https://rt.cpan.org/Public/Bug/Display.html?id=6758

This significantly slows down execution of test suits. It is noticeable 
even on small suits, please try to comment debug() call out , and you 
will see.

Thanks.

10.06.2011 22:46, Matthew Astley wrote:
> On Thu, May 19, 2011 at 08:50:03PM +0100, Matthew Astley wrote:
>
>> My next step is to rescue the code from CVS into Git, since I'm
>> tooled up for this at work at the moment.
>
> I did that, and revised the website a little.  You may clone from
> SourceForge or Github, details are on http://perlunit.sourceforge.net/
>
> I then fixed the release process and made development releases.
> 0.25_1325 should fix the outstanding cpantesters fails and the
> TestRunner.pl exit code.  There are also some old fixes.
>    http://search.cpan.org/~mcast/Test-Unit-0.25_1325/
>    http://search.cpan.org/src/MCAST/Test-Unit-0.25_1325/ChangeLog
>
> This doesn't address all outstanding bugs, but it does get the visible
> (dist) tests back to a green bar on all OS/Perl combinations so far,
>    http://matrix.cpantesters.org/?dist=Test-Unit+0.25_1325
>
>
> In addition to bugs in the shipped code, there is also a problem with
> my CPAN permissions - I don't have permission for all the Test::Unit*
> modules in the dist, because PAUSE treats them separately and we
> didn't know this back then.
>
> Adam has fixed the modules he can, and I think I have contact details
> for Christian.
>
>
> Anyway, I wanted to let you know things are still happening slowly.
>


-- 
Andrey