Re: [Simpletest-support] Fail fast?

[Roy S. <ro...@s7...>]

There was an extensive thread on (almost) this topic a couple of months ago:

http://sourceforge.net/mailarchive/message.php?msg_id=26500668

As much as I'm in favor of test assertions throwing (which is how you implement fail-fast), I do agree with Chris that since this would be a backward incompatible change, it should not be the default. Either have a new set of assertions, or a different base class to inherit from, or a configuration switch, or something.  Then, people like Ben and myself can get the behavior we want, but people like Chris won't have all their existing tests break when they upgrade.

I assume what Ben is talking about is having a test method stop at the first assertion failure.  I would still expect that all the test methods in a test class get run.  At least that's the way I would do it (and is in line with most of the other X-Unit frameworks).


On Jan 5, 2011, at 3:59 PM, Christoph Berendes wrote:

> I'd prefer to have a configuration switch and if that's not possible to keep
> the current behavior.
> 
> We've been using Simpletest very happily for three years to test a
> moderately complex data collection website providing a nested set of forms
> for each user.
> 
> The argument against fast fail: yesterday, I made some underlying changes in
> the database - implementing InnoDB foreign key constraints, where before
> we'd relied just on PHP logic. I ran the test suite, and saw a handful of
> failures in one of the tests run "in the middle" of the suite. That was
> annoying, but it was helpful for my time planning to see that it was just
> part of one test file, rather than massive (or scattered) failures
> throughout.
> 
> With a "fast fail" constraint, I wouldn't have been able to distinguish, up
> front, between one failure that in effect hides dozens more (as I work
> through the code fixing bugs) or, as in this case, a single failure that
> really is the only failure in the test suite.
> 
> A "fast fail" option would be useful once I know the extent of the damage,
> in that it perhaps speeds up each test run and certainly reduces clutter.
> But my workaround is to to have separate test files, each relatively short,
> and a "run-all.php" in the directory as well. Once I see the extent of the
> problem via run-all, I can focus in on getting each individual test file to
> pass.
> 
> Chris Berendes
> 
> On Wed, Jan 5, 2011 at 3:33 PM, Marcus Baker <ma...@la...> wrote:
> 
>> Hi...
>> 
>> Solwitz, Benjamin wrote:
>>> Is it possible to configure simpletest to fail fast, that is, after the
>> first assertion failure, the rest of the test is skipped?
>> 
>> You can throw an exception instead, but no, there is no fail fast option
>> right now. I'm seriously thinking of switching to that behaviour as
>> default though unless I get a lot of objections.
>> 
>>> 
>>> Ben
>> 
>> yours, Marcus
>> 
>> 
>> ------------------------------------------------------------------------------
>> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
>> to consolidate database storage, standardize their database environment,
>> and,
>> should the need arise, upgrade to a full multi-node Oracle RAC database
>> without downtime or disruption
>> http://p.sf.net/sfu/oracle-sfdevnl
>> _______________________________________________
>> Simpletest-support mailing list
>> Sim...@li...
>> https://lists.sourceforge.net/lists/listinfo/simpletest-support
>> 
> ------------------------------------------------------------------------------
> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
> to consolidate database storage, standardize their database environment, and, 
> should the need arise, upgrade to a full multi-node Oracle RAC database 
> without downtime or disruption
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> Simpletest-support mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simpletest-support


--
Roy Smith
roy...@s7...