Re: [Perlunit-devel] Enhancements to the TestCase class

[Desilets, A. <Ala...@nr...>]

> > Most of those are in the vein of special assertion methods. For
> > example:
> >
> > assert_string_equals($got_string, $exp_string, $message) [...]
> > assert_structures_are_equal($got_struct, $exp_struct, $message) [...]
> 
> It sounds like these are an improvement on the similar assertions
> Perlunit has.  However, similar code came[1] from Test::More and since
> then while Perlunit fixed a few bugs, other test modules gained many
> more useful features[2].

Hum... it would have been nice if the assert_deep_equals() method had been documented in TestCase. It would have avoided me having to reinvent it. I know it's documented as part of the Assert.pm documentation, but I always assumed that this was a class that was internal to PerlUnit, and that I didn't need to know details of it. I never bothered to look there, because all assertions are called from TestCase instances. 

> For these reasons, and the fact we already have enough assert_*
> methods to make a newcomer's head spin, I didn't just pile my methods
> into Test::Unit::Assert .

Looking at the list of assertions on this page:

http://kobesearch.cpan.org/htdocs/Test-Unit/Assert.pm.html

I see that most of what I implemented in TestCaseWithHelpers is indeed already there.

May I respectfully suggest that you put a very prominent link to the documentation of Assert.pm in the documentation for TestCase.pm? Something like a whole section about checking assertions.

> > I would be willing to share those enhancements in the official
> > version of Perl-Unit, if it was deemed appropriate. All those
> > methods are of course unit tested ;-).
> 
> That's great.  If the code isn't already published somewhere we can
> point to (as mine is), I would like to put it in a contrib/ directory
> so people can benefit when they're ready to see about extending the
> framework for themselves.  Does this suit you?  It would mean a small
> code fork, of your class (unless you abandon your copy).

As it turns out, I think pretty much everything I put in TestCaseWithHelpers is a dupliciation of what already exists in Assert.pm. I will start using those instead.

If I find that there are some things in TestCaseWithHelpers that are still of value, then I will post them somewhere and announce it on this list.

> > I am also interested in helping out to implement two improvements
> > that I would like to add to Perl-Unit:
> >
> > - Possibility to filter TestCase or test methods by passing an
> >   argument to the test runner with a string that should the TestCase
> >   class or the test method should match.
> 
> I have a feeling the /^test/ method filter is already configurable.

If it is, it's not clear from any of the documentation I have seen.

> > Also have you looked at t/tlib/RunnerTest.pm ?

I looked at the code briefly, in a first attempt to implement a filter, but didn't get too far. 

It sounds like there are many undocumented features of PerlUnit. Maybe those who know them should publish them in the official documentation?

> I also was thinking recently that some control of the TestSuite would
> be useful.  It is the place to put control of the thorny issue "tests
> run in random order", and maybe also allow parallel execution[3],
> using the "yesterday's weather" model to put the slower tests early.

Actually, I like the "run in random order" part because it forces me to make sure that there are no dependancies between my tests.

It's only recently that I have felt the need for more control over what tests get executed when. This is specifically because some of my recent tests need to create multiple threads. And there are many Perl modules that die when they are being cloned in a new thread. So I have had to split my TestCases into two separate suites, one that is thread-safe, and one that is not. I would not have to do that if I could tell PerlUnit to execute the non-thread safe tests last, in order to avoid cloning a non-thread safe module.

> > - Possibility to ask for a verbose mode where the test runner will
> >   print the name of every test method it runs,
> 
> If you run under Test::Unit::HarnessUnit you get to see each method
> run, although it lacks many useful details like...  the name.

Right... you get to see a dot for each method that gets executed. In most circumstances that's fine, but in some cases, I see a dot that takes forever to execute, and I want to know which test method is taking so long.

> I think also the TkTestRunner give you more interactive information.

Thx, I'll give it a spin.

But in general, I like to run my tests cases through Eclipse, because it's convenient to have everything accessible from my IDE.

> >   as well as the number of seconds that the test took to run.
> 
> Something like this is done by my extension by set_up, tear_down and
> an END block
>   https://github.com/mca-wtsi/perlunit-> extras/blob/707f941d03f929da6e2f0a38d8afff4b3661b84a/testlib/McaTestCase.pm#L575
> 
> because I wanted to know which are those slow tests.  This way has the
> advantage that it doesn't have to obtain a final callback from the
> runner, but it's not pretty.

I don't know how you implemented this, but my TestCaseWithHelpers has some utilities for doing that. Basically, at the start of a test method, you have to invoke start_timing_test($test_method_name), and in the tear_down(), you have to invoke possibly_print_elapsed_time_for_test().

It works, but is clunky, because you have to spell out the name of the test method to start_timing_test(), and if you rename that method, you have to change the name you pass to start_timing_test(). Also, when you want to time a test method, you have to check in the tear_down() method to see if possibly_print_elapsed_time_for_test() is invoked.

I'm sure there is a better way of doing this by modifying TestRunner, but I haven't investigated further.

Alain