simpletest-support — Help, advice, bugs and workarounds.

Re: [Simpletest-support] assertions don't throw?

[Marcus B. <ma...@la...>]

Hi...

Noel Darlow wrote:
> (1) A fail will halt execution of the current test method but other test
> methods will proceed as normal. That sounds OK.

That's the aim. We actually do this already in the case that the code
under test throws an exception. In the 1.0.1 version this was only
enabled if PHP5 was detected.

> Finally, in any option, will tear down code always be run after a
> throw? Left-over bits of fixture can sometimes just be a minor
> nuisance but sometimes they can be more serious, if they might
> contaminate future tests and cause misleading results.

That's the situation now. If the tear down code throws an exception then
that exception is reported, but the next test may be in trouble. Again,
this is the existing behaviour. The only change will be to make teh
assertions use the same exception trapping code. It's a big internal
simplification.

> 
> 
> Noel

yours, Marcus

Re: [Simpletest-support] assertions don't throw?

[Roy S. <ro...@s7...>]

On Nov 2, 2010, at 5:07 AM, T.J.Hunt wrote:

> xUnit implementations typically do 1)
> 
> In pseudo code, I guess it looks like
> 
> foreach ($testClass->getTestMethods() as $testMethod) {
>    $testClass->setUp();
>    try {
>        $testClass->$testMethod();
>        $this->countSuccess();
>    } catch(FailureException $f) {
>        $this->countFailure($f);
>    } catch(Exception $e) {
>        if (!$this->isExpectedException($e)) {
>            $this->countUnexpectedException($e);
>        }
>    } finally {
>        $testClass->tearDown();
>    }
> }
> 

Yup, that's exactly the behavior I would expect.  Looking at it another way:

* All test cases are independent, and can be run in any order.

* Within a test case, all test methods are independent, and can be run in any order.

* Within a test method, the individual assertions depend on each other, must be run in the specified order, and once an assertion fails, it makes no sense to run the rest of the assertions in that method.

--
Roy Smith
roy...@s7...

Re: [Simpletest-support] assertions don't throw?

[Roy S. <ro...@s7...>]

On Nov 2, 2010, at 12:55 AM, Noel Darlow wrote:

> Hi
> 
>> I'm used to test cases where the test case stops executing as soon as
>> the first assertion in it fails.  Typically this is implemented by
>> having the assertions throw AssertionFailure or something like that.
> 
> There are a range of possibile behaviours if exceptions are used. I'm
> not sure if they are all good.
> 
> (1) A fail will halt execution of the current test method but other test
> methods will proceed as normal. That sounds OK.

Yup, this is what I would expect.

> Finally, in any option, will tear down code always be run after a
> throw? Left-over bits of fixture can sometimes just be a minor
> nuisance but sometimes they can be more serious, if they might
> contaminate future tests and cause misleading results.

I would expect tearDown() to run after every test method, whether it passed or failed.  A test case throwing AssertionFailure is simply a failure, so yes, tearDown() should run.  The issue gets a little trickier if a test method throws something other than AssertionFailure, but only marginally.  And, yes, tearDown() should run then too.

The issue of cross-method contamination is why many of these X-Unit packages destroy the TestCase instance and create a new one before each test method.  It's a little less efficient, but eliminates most worries of contamination.  That usually leaves tearDown() nothing to do except for cleaning up external resources.

--
Roy Smith
roy...@s7...

Re: [Simpletest-support] assertions don't throw?

[T.J.Hunt <t.j...@op...>]

xUnit implementations typically do 1)

In pseudo code, I guess it looks like

foreach ($testClass->getTestMethods() as $testMethod) {
    $testClass->setUp();
    try {
        $testClass->$testMethod();
        $this->countSuccess();
    } catch(FailureException $f) {
        $this->countFailure($f);
    } catch(Exception $e) {
        if (!$this->isExpectedException($e)) {
            $this->countUnexpectedException($e);
        }
    } finally {
        $testClass->tearDown();
    }
}

One of the good features of the excellent book xUnit Test Patterns is that the author has clearly used a lot of different unit test frameworks himself, so scattered throughout the book are comments on the similarities and differences between the different frameworks. In fact the book had many good features, and only one bad feature - it is over 800 pages long. However, if you can face an 800 page brick of a book on unit testing, I strongly recommend it. There is a lot of wisdom in there, and it raised my TDD practice to a new level.

Tim.

________________________________________
From: Noel Darlow [ma...@mc...]
Sent: 02 November 2010 04:55
To: sim...@li...
Subject: Re: [Simpletest-support] assertions don't throw?

Hi

> I'm used to test cases where the test case stops executing as soon as
> the first assertion in it fails.  Typically this is implemented by
> having the assertions throw AssertionFailure or something like that.

There are a range of possibile behaviours if exceptions are used. I'm
not sure if they are all good.

(1) A fail will halt execution of the current test method but other test
methods will proceed as normal. That sounds OK.

(2) A fail will halt execution of the current test case but other test
cases will proceed. I'll come back to this.

(3) A fail will halt execution of the entire test suite. This is
something I wouldn't like (although I could live with it). If
there are several problem areas in the code, sometimes the first thing
that fails isn't the most important thing you need to know about and
fix. Also, in practice, I sometimes find I have failing tests which
aren't all that important. It could be a false positive caused
by too-strict custom test code or just a minor niggle which does need
fixed, sometime, but shouldn't be allowed to hold up the really
important stuff you need to verify and upload right now. Halting the
entire test suite on the first fail could hide the information I
really need to know.

In saying that, if it is to be the default behaviour, I could get around
it quite easily just by making a failing-but-unimportant test case
skip.

Any problems with the (2) option, halt the current test case but not
the test suite, would depend on how fine-grained the test cases are.
Less well-factored test cases can be like mini test suites and so could
suffer from similar problems as in (3).

Finally, in any option, will tear down code always be run after a
throw? Left-over bits of fixture can sometimes just be a minor
nuisance but sometimes they can be more serious, if they might
contaminate future tests and cause misleading results.


Noel

------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Simpletest-support mailing list
Sim...@li...
https://lists.sourceforge.net/lists/listinfo/simpletest-support

-- 
The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).

[Simpletest-support] bugs are for logged-in only SF users

[Perrick Penet-A. <pe...@no...>]

Hi all,

For the first time in 5 years, we're facing a spam wave on the bugs 
tracker : I just unchecked the "Allow non-logged-in postings" option. I 
might reconsider enabling it in a few days.

Yours,
	Perrick

Re: [Simpletest-support] assertions don't throw?

[Noel D. <ma...@mc...>]

Hi

> I'm used to test cases where the test case stops executing as soon as
> the first assertion in it fails.  Typically this is implemented by
> having the assertions throw AssertionFailure or something like that.

There are a range of possibile behaviours if exceptions are used. I'm
not sure if they are all good.

(1) A fail will halt execution of the current test method but other test
methods will proceed as normal. That sounds OK.

(2) A fail will halt execution of the current test case but other test
cases will proceed. I'll come back to this.

(3) A fail will halt execution of the entire test suite. This is
something I wouldn't like (although I could live with it). If
there are several problem areas in the code, sometimes the first thing
that fails isn't the most important thing you need to know about and
fix. Also, in practice, I sometimes find I have failing tests which
aren't all that important. It could be a false positive caused
by too-strict custom test code or just a minor niggle which does need
fixed, sometime, but shouldn't be allowed to hold up the really
important stuff you need to verify and upload right now. Halting the
entire test suite on the first fail could hide the information I
really need to know. 

In saying that, if it is to be the default behaviour, I could get around
it quite easily just by making a failing-but-unimportant test case
skip.

Any problems with the (2) option, halt the current test case but not
the test suite, would depend on how fine-grained the test cases are.
Less well-factored test cases can be like mini test suites and so could
suffer from similar problems as in (3).

Finally, in any option, will tear down code always be run after a
throw? Left-over bits of fixture can sometimes just be a minor
nuisance but sometimes they can be more serious, if they might
contaminate future tests and cause misleading results.


Noel

Re: [Simpletest-support] assertions don't throw?

[Jason S. <jsw...@ya...>]

From: Perrick Penet-Avez <pe...@no...>

To: "Help, advice,  bugs and workarounds" 
<sim...@li...>
Sent: Sat, October 30, 2010 3:33:45 AM
Subject: Re: [Simpletest-support] assertions don't throw?

>Done :
>http://simpletest.org/
>http://simpletest.org/en/download.html

>Both pages mention the latest version.

>Now we need to get our focus towards the documentation !

Still a work in progress, but I have uploaded a version of the 1.1alpha docs to 
http://simpletest.org/api/

I need to still clean up some parse errors, docblocks, etc.   The latest version 
of phpdoc did not seem to like the template from sf.net for simpletest, so I 
went back to one of the out of the box themes from phpdoc and started tweaking 
from there.

Regards,
Jason



      

Re: [Simpletest-support] assertions don't throw?

[Marcus B. <ma...@la...>]

Hi...

Roy Smith wrote:
> What does "all versions" mean?  If I test under 5.2.14, is that enough to say "tested with 5.2"?, or are you thinking we need to test with every 5.2.x, for x in [0 .. 14]?

That's what I did last time. I downloaded every PHP version I could find
and had a script run overnight. I'll probably set up the same thing again.

It's still helpful for other people to test though. There is a lot of
variation to be had in different PHP set ups.

> 
> --
> Roy Smith
> roy...@s7...

yours, Marcus

Re: [Simpletest-support] assertions don't throw?

[Jason S. <jsw...@ya...>]

svn head, I run:

sweatje@phil:~/simpletest/test$ php all_tests.php 
all_tests.php
Skip: E_USER_ERROR not tested for PHP 5.2 and above at 
[/home/sweatje/simpletest/test/errors_test.php line 172]
OK
Test cases run: 148/154, Passes: 2780, Failures: 0, Exceptions: 0

sweatje@phil:~/simpletest/test$ php -v
PHP 5.3.0 (cli) (built: Jul  5 2009 22:48:25) 
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2009 Zend Technologies






________________________________
From: Roy Smith <ro...@s7...>
To: Roy Smith <ro...@s7...>
Cc: "Help, advice, bugs and workarounds" 
<sim...@li...>
Sent: Sun, October 31, 2010 9:09:06 AM
Subject: Re: [Simpletest-support] assertions don't throw?

I tried running the 1.1-alpha self-tests.  There seem to be some missing files;

arguments.php
test/arguments_test.php
test/'/recorder_test.php

Once I delete the references to those,  what's remaining passes on my PHP-5.3.2, 
OSX-10.6.4 setup....

test$ php unit_tests.php 
unit_tests.php
Skip: E_USER_ERROR not tested for PHP 5.2 and above at 
[/Users/roy/SimpleTest/simpletest/test/errors_test.php line 172]
Skip: Install 'tidy' php extension to enable html tidy based parser at 
[/Users/roy/SimpleTest/simpletest/test/parsing_test.php line 631]
OK
Test cases run: 122/128, Passes: 1992, Failures: 0, Exceptions: 0


--
Roy Smith
roy...@s7...





------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store 
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Simpletest-support mailing list
Sim...@li...
https://lists.sourceforge.net/lists/listinfo/simpletest-support



      

Re: [Simpletest-support] assertions don't throw?

[Roy S. <ro...@s7...>]

I tried running the 1.1-alpha self-tests.  There seem to be some missing files;

arguments.php
test/arguments_test.php
test/'/recorder_test.php

Once I delete the references to those,  what's remaining passes on my PHP-5.3.2, OSX-10.6.4 setup....

test$ php unit_tests.php 
unit_tests.php
Skip: E_USER_ERROR not tested for PHP 5.2 and above at [/Users/roy/SimpleTest/simpletest/test/errors_test.php line 172]
Skip: Install 'tidy' php extension to enable html tidy based parser at [/Users/roy/SimpleTest/simpletest/test/parsing_test.php line 631]
OK
Test cases run: 122/128, Passes: 1992, Failures: 0, Exceptions: 0


--
Roy Smith
roy...@s7...

Re: [Simpletest-support] assertions don't throw?

[Roy S. <ro...@s7...>]

On Oct 30, 2010, at 6:07 AM, Marcus Baker wrote:
> 
> Actually the biggest problem is testing all the versions of PHP5 under
> E_STRICT. I.ve changed machine since I used to do this regularily. I'm
> having to set up that bunch of infrastructure again.

I'm certainly willing to help with the testing.

What does "all versions" mean?  If I test under 5.2.14, is that enough to say "tested with 5.2"?, or are you thinking we need to test with every 5.2.x, for x in [0 .. 14]?

--
Roy Smith
roy...@s7...

Re: [Simpletest-support] assertions don't throw?

[Marcus B. <ma...@la...>]

Hi...

Perrick Penet-Avez wrote:
> Now we need to get our focus towards the documentation !

Actually the biggest problem is testing all the versions of PHP5 under
E_STRICT. I.ve changed machine since I used to do this regularily. I'm
having to set up that bunch of infrastructure again.

I'm still aiming for compatibility from PHP 5.0.3 to PHP 6.0 if possible.

> 
> Yours,
> 	Perrick

yours, Marcus

Re: [Simpletest-support] assertions don't throw?

[Perrick Penet-A. <pe...@no...>]

Hi all,

> And tomorrow I'll update the website ! Now we have a release, I hope
> we'll be able to keep the momentum going.

Done :
http://simpletest.org/
http://simpletest.org/en/download.html

Both pages mention the latest version.

Now we need to get our focus towards the documentation !

Yours,
	Perrick

Re: [Simpletest-support] assertions don't throw?

[Perrick Penet-A. <pe...@no...>]

> From: Marcus Baker<ma...@la...>
>> Done.
>
> I'll see if I have a machine setup to rebuild docs and publish this weekend if I
> can.

And tomorrow I'll update the website ! Now we have a release, I hope 
we'll be able to keep the momentum going.

Yours,
	Perrick

Re: [Simpletest-support] assertions don't throw?

[Jason S. <jsw...@ya...>]

From: Marcus Baker <ma...@la...>

To: "Help, advice, bugs and workarounds" 
<sim...@li...>
Sent: Fri, October 29, 2010 9:39:40 AM
Subject: Re: [Simpletest-support] assertions don't throw?

>Done.

I'll see if I have a machine setup to rebuild docs and publish this weekend if I 
can.

Regards,
Jason



      

Re: [Simpletest-support] assertions don't throw?

[Marcus B. <ma...@la...>]

Hi...

Perrick Penet-Avez wrote:
>> Oh yeah. Today.
> 
> Excellent news...

Done.

> 9th & 10th of November : my goal is having the translation done by then 
> if the beta is out !

I'll do some fiddling and documentation fixes over the next few days
(prob Monday and Tuesday).

> 
> Yours,
> 	Perrick

yours, Marcus

Re: [Simpletest-support] assertions don't throw?

[Perrick Penet-A. <pe...@no...>]

> Oh yeah. Today.

Excellent news...

>> By the way, I'm really glad I'll be able announce a new version at the
>> French PHP Conference.
>
> When is it?

9th & 10th of November : my goal is having the translation done by then 
if the beta is out !

Yours,
	Perrick

Re: [Simpletest-support] assertions don't throw?

[Marcus B. <ma...@la...>]

Hi...

Perrick Penet-Avez wrote:
> 00:56 : does it mean today or tomorrow ?

Oh yeah. Today.

> By the way, I'm really glad I'll be able announce a new version at the 
> French PHP Conference.

When is it?

> 
> Yours,
> 	Perrick

yours, Marcus

Re: [Simpletest-support] assertions don't throw?

[Perrick Penet-A. <pe...@no...>]

On 29/10/10 00:56, Marcus Baker wrote:
> Hi...
>
> Perrick Penet-Avez wrote:
>> What about turning things around a little bit and releasing an beta
>> version when the core is stable (i.e. now) and then a full point version
>> when the doc and translation is done.
>
> Yeah, I can do that. I'll do it tomorrow.

00:56 : does it mean today or tomorrow ?
By the way, I'm really glad I'll be able announce a new version at the 
French PHP Conference.

Yours,
	Perrick

[Simpletest-support] anyone in Australia ?

[Perrick Penet-A. <pe...@no...>]

Hi folks,

A friend of mine is looking for a presentation / demo of SimpleTest in 
Australia. He is a Python geek working with a bunch of PHP developpers...

Anyone from down-under who would be interested in dealing directly with 
him ? Just drop me an email and I'll reply with his address.

Yours,
	Perrick

PS : by the way, if anyone wants to be on the "Speakers, Coachs and 
Consultants" list, make yourself heard as well...

http://simpletest.org/en/speakers_coaches_consultants.html

Re: [Simpletest-support] assertions don't throw?

[Marcus B. <ma...@la...>]

Hi...

Perrick Penet-Avez wrote:
> What about turning things around a little bit and releasing an beta 
> version when the core is stable (i.e. now) and then a full point version 
> when the doc and translation is done.

Yeah, I can do that. I'll do it tomorrow.

yours, Marcus

Re: [Simpletest-support] assertions don't throw?

[Perrick Penet-A. <pe...@no...>]

Hello,

>> Is there still active development happening?  The last release that I could see was 2-1/2 years ago.
>
> It's all been happening in SVN. I am planning a tarball release real
> soon. Just got to update the last 5 docs pages and get translations done.

What about turning things around a little bit and releasing an beta 
version when the core is stable (i.e. now) and then a full point version 
when the doc and translation is done. That way we'd have :

1.1 alpha => now
1.1 beta => with docs and translations
1.1 => with bug fixes

Otherwise we could also use minor version to get stuff out :

1.0.2 => now (stable core)
1.0.3 => with docs
1.0.4 => with translations
1.0.5 => with bug fixes
1.1 => stable

Yours,
	Perrick

Re: [Simpletest-support] assertions don't throw?

[Marcus B. <ma...@la...>]

Hi...

Roy Smith wrote:
> Is there still active development happening?  The last release that I could see was 2-1/2 years ago.

It's all been happening in SVN. I am planning a tarball release real
soon. Just got to update the last 5 docs pages and get translations done.

> I'd hate to hack up something and then discover a new release comes
along which implements the same idea in an incompatible way.

Unfortunately this is a substantial change to the internals, so it's
almost certain it will have to be implemented as part of the core work.
Feel free to patch your own internal version and we'll see if we can
learn from your interface and keep compatibility. Failing that, perhaps
we can give you plenty of warning when things will change.

> 
> --
> Roy Smith
> roy...@s7...

yours, Marcus

Re: [Simpletest-support] assertions don't throw?

[Roy S. <ro...@s7...>]

On Oct 27, 2010, at 5:24 AM, Marcus Baker wrote:

> Hi...
> 
> Roy Smith wrote:
>> One thing I was a bit surprised to discover is that UnitTestCase::assertTrue(), etc, returns False on failure, instead of throwing.
> 
> This is just history.
> 
> SimpleTest is one of the oldest unit testing frameworks around and goes
> back to befor ethis behaviour was standard. It also predates exception
> support in PHP.

OK, that makes sense.  Thanks for the explanation.


> Throwing exceptions is one of the changes that will be happening after
> the interim 1.1 release, along with other changes to the mechanics.


Is there still active development happening?  The last release that I could see was 2-1/2 years ago.  Assertion exceptions is something that's important to me, so I'm willing to write the code myself and donate it back.  What's the process for doing that?  I'd hate to hack up something and then discover a new release comes along which implements the same idea in an incompatible way.  

--
Roy Smith
roy...@s7...

Re: [Simpletest-support] assertions don't throw?

[Marcus B. <ma...@la...>]

Hi...

Roy Smith wrote:
> One thing I was a bit surprised to discover is that UnitTestCase::assertTrue(), etc, returns False on failure, instead of throwing.

This is just history.

SimpleTest is one of the oldest unit testing frameworks around and goes
back to befor ethis behaviour was standard. It also predates exception
support in PHP.

Throwing exceptions is one of the changes that will be happening after
the interim 1.1 release, along with other changes to the mechanics.

yours, Marcus