Re: Different types of tests (was Re: [Module::Build] [RFC] author tests)

[David G. <da...@hy...>]

Rob Kinyon wrote:
> Here's the rationale behind my proposal - currently, there's no
> mechanism for specifying different types of tests in either MB or
> EUMM. Essentially, that's what this proposal is all about. I see a lot
> of the confusion arising because different people want different kinds
> of tests and don't conceive of them as "author tests", but as "dist
> tests", "unit tests", "packaging tests", "acceptance tests", "system
> tests", etc ad nauseum.

I think this will mean massive overbuilding of a framework without a 
clear need for that type of functionality.

If we remember where this all started -- it's the desire not to inflict 
"pod tests" in *.t files on end users.  Thanks to CPANTS and Kwalitee -- 
and not helped by Module::Starter::PBP -- there's a ton of cargo-cult 
non-functional tests winding up in distributions.

What's missing in all that cargo-culting is a widely accepted mechanism 
for opting in/out of the tests.  Building something into the toolchain 
-- Module::Build and ExtUtils::MakeMaker -- (and then getting that into 
the Test::Pod docs, maybe CPAN/CPANPLUS, etc.) is just a way of driving 
a standard.

I'd prefer to KISS -- look for the simplest solution that could possibly 
work.  So just because M::B, etc. is a conveient place to push the 
standard doesn't mean that the standard should be (a) overworked or (b) 
too tightly coupled to M::B specifically.

I prefer the environment variable approach because, while it's global -- 
it's one of easiest methods of interprocess communication between a 
build framework, the test harness and test files.

What I'd *really* like to see is the community agree on an environment 
variable that signals that "author" testing (for lack of a better term) 
is desired.  And then I'd like to see Test::Pod, Test::Pod::Coverage, 
etc. look for that environment variable and skip running their tests 
unless it exists.  In other words -- I'd like the skip logic be a 
convention followed by those who write tests rather than something 
hacked up for all the different ways people might write/structure/run tests.

Among other things, the environment variable approach means that the 
selection of tests to run doesn't depend on how up to date a user's M::B 
is. It's a big problem if all the tests are *.t and only the latest M::B 
has the logic for which ones to run.  At least the *.ta approach 
defaults to not running.  On the downside, I think getting people to 
shift to *.ta (and getting Module::Starter, ExtUtils::ModuleMaker, etc. 
to use it) will be a bigger hurdle than changing the example synopsis in 
Test::Pod and the default templates, etc.  (Assuming, of course, that 
those who write such tools, like Andy Lester, agree that opt-in is good 
practice.)

Regards,
David