Re: [Perlunit-devel] Enhancements to the TestCase class

[Matthew A. <mc...@us...>]

On Fri, Jun 10, 2011 at 03:18:16PM -0400, Desilets, Alain wrote:

> I have been using Perl-Unit happily for some years now, and over the
> years, I have developed a number of useful helper methods in a
> subclass of TestCase called TestCaseWithHelpers.

Hi Alain,

Thanks for mentioning this to me earlier, and I'm sorry I didn't get
back sooner.


Anyway it makes sense to discuss TestCaseWithHelpers on this list.

I guess anyone who uses a tool like this for long makes extensions or
comfortable tweaks.  I did the same, making intermediate base classes
to add some new assertions and setup code.

After partially extracting mine from some only-relevant-to-employer
code, I put it up at
  https://github.com/mca-wtsi/perlunit-extras

but it isn't in a state where I would recommend anything but fishing
in it for ideas.

Thank you for offering yours.


> Most of those are in the vein of special assertion methods. For
> example:
> 
> assert_string_equals($got_string, $exp_string, $message) [...]
> assert_structures_are_equal($got_struct, $exp_struct, $message) [...]

It sounds like these are an improvement on the similar assertions
Perlunit has.  However, similar code came[1] from Test::More and since
then while Perlunit fixed a few bugs, other test modules gained many
more useful features[2].

Perl's testing tools have moved on since Perlunit was cool, and in my
view it would be most constructive to bring the tools closer together
so (ideally) we can all benefit from the combinations of features we
want, without being swamped by complexity and effort to discover them,
and without breaking compatibility.

The intermediate T:U:TestCase class trick is clean and fairly
powerful.  (Perlunit has the problem that it doesn't get the benefit
of subroutine prototypes, since the assertions are method calls, but
that is a problem for another day.)

For these reasons, and the fact we already have enough assert_*
methods to make a newcomer's head spin, I didn't just pile my methods
into Test::Unit::Assert .


> I would be willing to share those enhancements in the official
> version of Perl-Unit, if it was deemed appropriate. All those
> methods are of course unit tested ;-).

That's great.  If the code isn't already published somewhere we can
point to (as mine is), I would like to put it in a contrib/ directory
so people can benefit when they're ready to see about extending the
framework for themselves.  Does this suit you?  It would mean a small
code fork, of your class (unless you abandon your copy).

Also do you have version history for TestCaseWithHelpers which you
would like to import?  I can accept (RCS), CVS, SVN and Git history,
and maybe others.  Once translated to Git I can extract and sanitise
the history to your liking before publishing it.

Or we can just add your current version as one commit.


> I am also interested in helping out to implement two improvements
> that I would like to add to Perl-Unit:
> 
> - Possibility to filter TestCase or test methods by passing an
>   argument to the test runner with a string that should the TestCase
>   class or the test method should match.

I have a feeling the /^test/ method filter is already configurable.
Also have you looked at t/tlib/RunnerTest.pm ?

(Sorry, my knowledge of this codebase has waned.  I just use the bits
I like, and not recently either.)

I also was thinking recently that some control of the TestSuite would
be useful.  It is the place to put control of the thorny issue "tests
run in random order", and maybe also allow parallel execution[3],
using the "yesterday's weather" model to put the slower tests early.


> - Possibility to ask for a verbose mode where the test runner will
>   print the name of every test method it runs,

If you run under Test::Unit::HarnessUnit you get to see each method
run, although it lacks many useful details like...  the name.

I think also the TkTestRunner give you more interactive information.

>   as well as the number of seconds that the test took to run.

Something like this is done by my extension by set_up, tear_down and
an END block
  https://github.com/mca-wtsi/perlunit-extras/blob/707f941d03f929da6e2f0a38d8afff4b3661b84a/testlib/McaTestCase.pm#L575

because I wanted to know which are those slow tests.  This way has the
advantage that it doesn't have to obtain a final callback from the
runner, but it's not pretty.



Overall I would say I currently prefer "extensible, shipped with some
handy extensions" and better support for other Test::* tools, over
having a huge insular API which nobody uses all of.

First I have to get a release accepted.  There are a couple more bugs
in the various trackers, but they don't have failing tests yet.

I also plan to gather feedback and ideas via the SourceForge
"LimeSurvey" hosted app.  I have some questions for the survey, but
please make suggestions too.


Thanks for reading,
-- 
Matthew

[1]
  http://perlunit.git.sourceforge.net/git/gitweb.cgi?p=perlunit/perlunit;a=commit;h=b03cd777c6a8167d2ff50a6239455995ccb84cc8
  http://perlunit.git.sourceforge.net/git/gitweb.cgi?p=perlunit/perlunit;a=commit;h=dca8fab14735649267f72b8904e63cec61aabc83

[2] For example these are neat, and a bit more polished than anything
    I bothered to write for my tests.  I see them as optional extras
for the developer, but wouldn't choose to require them for build.

  http://search.cpan.org/~ovid/Test-Differences-0.61/lib/Test/Differences.pm#DESCRIPTION
  http://search.cpan.org/~sburke/Test-1.25/lib/Test.pm#ENVIRONMENT

[3] I know prove(1) can already do parallelism, at the granularity of
    t/*.t files.  Having the tests run as separate Perl instances
makes this easier, although I guess you pay at each compile time.