[Perlunit-users] T:U:Assertion, test framework wars; was Re: PerlUnit Development

[Matthew A. <mc...@us...>]

On Sat, Feb 18, 2006 at 02:14:13AM -0600, Joi Ellis wrote:

> I've created a subclass of TestCase.pm that includes versions of tests
> copied from Test::More as well.
> 
> Test/Unit/MyTestCase.pm

I too have a collection of assertion methods, I keep them in my own
T:U:TestCase subclass and yes it's high time I shared it.

Subclassing is the obvious OO way to extend the code, but it is not
without problems.  To bring our code into the project, the obvious
thing to do is just push the new methods up into a superclass so
they're available to everyone...  unless the subclass has a specific
purpose.  But then if the user wishes to have a subclass for doing
local tricks, e.g. database testing utils, they'll need to do it
twice.

I believe Test::Unit::Assert already contains too much stuff all in
one place, leading to

 - overwhelming docs

 - may give new users the impression that there is far too much to
   learn in one go

 - no sensible place to put even *more* stuff

 - accompanying problems in t/tlib/AssertTest.pm

 - hard to share useful assertion code with other test frameworks

 - could interfere with the user's choice of inheritance pattern

Leaving the file to grow without bound as we find more useful and
exciting tests makes it difficult to manage the code and the method
namespace.


For example, T:U:Assert->assert_deep_equals was "shamelessly pinched
from Test::More and adapted to Test::Unit".  This method has an
assortment of bugs filed against it.

I have applied patches but when I look at all this, I wonder

 - which version of Test::More did it come from? (I haven't
   investigated)

 - is it fixed in Test::More, or is it an artifact of the "adapted to
   Test::Unit"?

 - if I fix or extend assert_deep_equals, can I contribute back to
   Test::More?

I use assert_deep_equals quite a bit myself, but I would prefer to put
some distance between maintenance of it and the rest of the framework.


Solutions so far:

 - make a contrib/ directory containing T:U:TestCase subclasses,
   starting with mine and Joi's, and leave the user to pick among the
   code in the methods or using them as SUPER.  Easy for us, but not
   great for code reuse.

 - use mixin classes to supply[1] assertion methods, possibly in a
   dynamic way.  But eww, mixins.

 - symbol table pokery and/or AUTOLOAD sickness to do...  what,
   dispatch methods by hand?

 - continue growing Assert.pm, maybe I imagined a problem?


There are other test frameworks out there with their own assertion
names, argument styles (including conflicting ones[2]) and neat
tricks.  A bit of competition is healthy, but I'm not interested in
proving "our test framework is better than yours".

It would be much more useful to make it easy for users to try one or
another, including porting existing tests without too much fuss.  It
would also be good to make it possible for testrunners (GUI, automated
continuous integration, continuous testing[3], random testing[4] etc.)
to be shared between frameworks.

On my To Do list are such lofty aims as finding out what else is out
there in test world, and then talking to the maintainers of the other
tools.

A comparison here is what I've seen so far while looking at test
frameworks for Common Lisp.  Each provides a set of features, but
there's nowhere you can go to get all the features (whatever that
might mean).

This leads me to think that the test framework should be doing less,
but providing hooks for other packages to do the rest.  More test
runners, more assertions, more types of test failure (todo, skip, slow
test filtering, test dependency).

Clearly this needs more thought and more input data.  A start would be
collecting people's testcase subclasses in a contrib directory, just
to see what's there.


Matthew  #8-)
-- 
[1] On the user side, this might cause a test to be written

 package MyTest;
 use strict;
 use warnings;
 use base 'Test::Unit::TestCase';
 use Test::Unit::AssertMixin::Foo;
 use Test::Unit::AssertMixin::Bar;
 sub test_with_foo {
   my $self = shift;
   $self->assert_foo( ... );
   $self->assert_bar( ... );
 }

The user may shuffle the incantation for mixins out to a project
module or the suit generator, to reduce the per-class noise.

On the perlunit side, the requirement would be for the mixin classes
to register themselves during &import.


[2] http://sf.net/tracker/index.php?func=detail&aid=407833&group_id=2653&atid=102653

The conflict is against the Java flavour, and it's mostly an artifact
of Perl's argument passing style, but still it's unfortunate:

  JUnit 3.5 has
  assert(message, condition);

  perlUnit .13 has
  assert(condition, message);

Maybe there are others, I haven't looked.


[3] http://pag.csail.mit.edu/continuoustesting/

[4] http://www.accesscom.com/~darius/software/clickcheck.html
http://www.findinglisp.com/blog/2004/10/random-testing.html including
link to a scary "razor sharp html fragments" article