Thread: [Simpletest-support] Fail fast?

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Is it possible to configure simpletest to fail fast, that is, after the first assertion failure, the rest of the test is skipped?

Ben

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
On 01/04/2011 11:54 PM, Solwitz, Benjamin wrote:
> Is it possible to configure simpletest to fail fast, that is, after the first assertion failure, the rest of the test is skipped?
Currently not.

There was a discussion on this some time ago so you might want to
consult the list archives for more information.

Regards,
Tarjei
> 
> Ben
> ------------------------------------------------------------------------------
> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
> to consolidate database storage, standardize their database environment, and, 
> should the need arise, upgrade to a full multi-node Oracle RAC database 
> without downtime or disruption
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> Simpletest-support mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simpletest-support

- -- 
Tarjei Huse
Mobil: 920 63 413
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0ktRsACgkQYVRKCnSvzfKnOQCgkUwctJU55V1DAs4c39MEKodt
/BsAoIMn3JihHGyXZrCIQRo3VIbxugRI
=XGnB
-----END PGP SIGNATURE-----

Hi...

Solwitz, Benjamin wrote:
> Is it possible to configure simpletest to fail fast, that is, after the first assertion failure, the rest of the test is skipped?

You can throw an exception instead, but no, there is no fail fast option
right now. I'm seriously thinking of switching to that behaviour as
default though unless I get a lot of objections.

> 
> Ben

yours, Marcus

I'd prefer to have a configuration switch and if that's not possible to keep
the current behavior.

We've been using Simpletest very happily for three years to test a
moderately complex data collection website providing a nested set of forms
for each user.

The argument against fast fail: yesterday, I made some underlying changes in
the database - implementing InnoDB foreign key constraints, where before
we'd relied just on PHP logic. I ran the test suite, and saw a handful of
failures in one of the tests run "in the middle" of the suite. That was
annoying, but it was helpful for my time planning to see that it was just
part of one test file, rather than massive (or scattered) failures
throughout.

With a "fast fail" constraint, I wouldn't have been able to distinguish, up
front, between one failure that in effect hides dozens more (as I work
through the code fixing bugs) or, as in this case, a single failure that
really is the only failure in the test suite.

A "fast fail" option would be useful once I know the extent of the damage,
in that it perhaps speeds up each test run and certainly reduces clutter.
But my workaround is to to have separate test files, each relatively short,
and a "run-all.php" in the directory as well. Once I see the extent of the
problem via run-all, I can focus in on getting each individual test file to
pass.

Chris Berendes

On Wed, Jan 5, 2011 at 3:33 PM, Marcus Baker <ma...@la...> wrote:

> Hi...
>
> Solwitz, Benjamin wrote:
> > Is it possible to configure simpletest to fail fast, that is, after the
> first assertion failure, the rest of the test is skipped?
>
> You can throw an exception instead, but no, there is no fail fast option
> right now. I'm seriously thinking of switching to that behaviour as
> default though unless I get a lot of objections.
>
> >
> > Ben
>
> yours, Marcus
>
>
> ------------------------------------------------------------------------------
> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
> to consolidate database storage, standardize their database environment,
> and,
> should the need arise, upgrade to a full multi-node Oracle RAC database
> without downtime or disruption
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> Simpletest-support mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simpletest-support
>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi, I'd just like to +1 this post.
T
On 01/05/2011 09:59 PM, Christoph Berendes wrote:
> I'd prefer to have a configuration switch and if that's not possible to keep
> the current behavior.
> 
> We've been using Simpletest very happily for three years to test a
> moderately complex data collection website providing a nested set of forms
> for each user.
> 
> The argument against fast fail: yesterday, I made some underlying changes in
> the database - implementing InnoDB foreign key constraints, where before
> we'd relied just on PHP logic. I ran the test suite, and saw a handful of
> failures in one of the tests run "in the middle" of the suite. That was
> annoying, but it was helpful for my time planning to see that it was just
> part of one test file, rather than massive (or scattered) failures
> throughout.
> 
> With a "fast fail" constraint, I wouldn't have been able to distinguish, up
> front, between one failure that in effect hides dozens more (as I work
> through the code fixing bugs) or, as in this case, a single failure that
> really is the only failure in the test suite.
> 
> A "fast fail" option would be useful once I know the extent of the damage,
> in that it perhaps speeds up each test run and certainly reduces clutter.
> But my workaround is to to have separate test files, each relatively short,
> and a "run-all.php" in the directory as well. Once I see the extent of the
> problem via run-all, I can focus in on getting each individual test file to
> pass.
> 
> Chris Berendes
> 
> On Wed, Jan 5, 2011 at 3:33 PM, Marcus Baker <ma...@la...> wrote:
> 
>> Hi...
>>
>> Solwitz, Benjamin wrote:
>>> Is it possible to configure simpletest to fail fast, that is, after the
>> first assertion failure, the rest of the test is skipped?
>>
>> You can throw an exception instead, but no, there is no fail fast option
>> right now. I'm seriously thinking of switching to that behaviour as
>> default though unless I get a lot of objections.
>>
>>>
>>> Ben
>>
>> yours, Marcus
>>
>>
>> ------------------------------------------------------------------------------
>> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
>> to consolidate database storage, standardize their database environment,
>> and,
>> should the need arise, upgrade to a full multi-node Oracle RAC database
>> without downtime or disruption
>> http://p.sf.net/sfu/oracle-sfdevnl
>> _______________________________________________
>> Simpletest-support mailing list
>> Sim...@li...
>> https://lists.sourceforge.net/lists/listinfo/simpletest-support
>>
> ------------------------------------------------------------------------------
> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
> to consolidate database storage, standardize their database environment, and, 
> should the need arise, upgrade to a full multi-node Oracle RAC database 
> without downtime or disruption
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> Simpletest-support mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simpletest-support

- -- 
Tarjei Huse
Mobil: 920 63 413
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0leXEACgkQYVRKCnSvzfJQQgCeL9Nq/aUGb9nIbRmWtnDukXHr
cUkAn0XzyLWTKONmqUECQBCclvx6EZAb
=SkxG
-----END PGP SIGNATURE-----

There was an extensive thread on (almost) this topic a couple of months ago:

http://sourceforge.net/mailarchive/message.php?msg_id=26500668

As much as I'm in favor of test assertions throwing (which is how you implement fail-fast), I do agree with Chris that since this would be a backward incompatible change, it should not be the default. Either have a new set of assertions, or a different base class to inherit from, or a configuration switch, or something.  Then, people like Ben and myself can get the behavior we want, but people like Chris won't have all their existing tests break when they upgrade.

I assume what Ben is talking about is having a test method stop at the first assertion failure.  I would still expect that all the test methods in a test class get run.  At least that's the way I would do it (and is in line with most of the other X-Unit frameworks).

On Jan 5, 2011, at 3:59 PM, Christoph Berendes wrote:

> I'd prefer to have a configuration switch and if that's not possible to keep
> the current behavior.
> 
> We've been using Simpletest very happily for three years to test a
> moderately complex data collection website providing a nested set of forms
> for each user.
> 
> The argument against fast fail: yesterday, I made some underlying changes in
> the database - implementing InnoDB foreign key constraints, where before
> we'd relied just on PHP logic. I ran the test suite, and saw a handful of
> failures in one of the tests run "in the middle" of the suite. That was
> annoying, but it was helpful for my time planning to see that it was just
> part of one test file, rather than massive (or scattered) failures
> throughout.
> 
> With a "fast fail" constraint, I wouldn't have been able to distinguish, up
> front, between one failure that in effect hides dozens more (as I work
> through the code fixing bugs) or, as in this case, a single failure that
> really is the only failure in the test suite.
> 
> A "fast fail" option would be useful once I know the extent of the damage,
> in that it perhaps speeds up each test run and certainly reduces clutter.
> But my workaround is to to have separate test files, each relatively short,
> and a "run-all.php" in the directory as well. Once I see the extent of the
> problem via run-all, I can focus in on getting each individual test file to
> pass.
> 
> Chris Berendes
> 
> On Wed, Jan 5, 2011 at 3:33 PM, Marcus Baker <ma...@la...> wrote:
> 
>> Hi...
>> 
>> Solwitz, Benjamin wrote:
>>> Is it possible to configure simpletest to fail fast, that is, after the
>> first assertion failure, the rest of the test is skipped?
>> 
>> You can throw an exception instead, but no, there is no fail fast option
>> right now. I'm seriously thinking of switching to that behaviour as
>> default though unless I get a lot of objections.
>> 
>>> 
>>> Ben
>> 
>> yours, Marcus
>> 
>> 
>> ------------------------------------------------------------------------------
>> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
>> to consolidate database storage, standardize their database environment,
>> and,
>> should the need arise, upgrade to a full multi-node Oracle RAC database
>> without downtime or disruption
>> http://p.sf.net/sfu/oracle-sfdevnl
>> _______________________________________________
>> Simpletest-support mailing list
>> Sim...@li...
>> https://lists.sourceforge.net/lists/listinfo/simpletest-support
>> 
> ------------------------------------------------------------------------------
> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
> to consolidate database storage, standardize their database environment, and, 
> should the need arise, upgrade to a full multi-node Oracle RAC database 
> without downtime or disruption
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> Simpletest-support mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simpletest-support

--
Roy Smith
roy...@s7...