Fwd: Different types of tests (was Re: [Module::Build] [RFC] author tests)

[Rob K. <rob...@gm...>]

I forgot to cc the list. :-|

Rob

---------- Forwarded message ----------
From: Rob Kinyon <rob...@gm...>
Date: Feb 3, 2006 12:57 PM
Subject: Re: Different types of tests (was Re: [Module::Build] [RFC]
author tests)
To: David Golden <da...@hy...>


On 2/3/06, David Golden <da...@hy...> wrote:
> I think this will mean massive overbuilding of a framework without a
> clear need for that type of functionality.

I don't think so - I'll explain further down.

> If we remember where this all started -- it's the desire not to inflict
> "pod tests" in *.t files on end users.  Thanks to CPANTS and Kwalitee --
> and not helped by Module::Starter::PBP -- there's a ton of cargo-cult
> non-functional tests winding up in distributions.

I don't think they're non-functional tests. They're useful tests and
I, as a user, like seeing that POD::Coverage passes. They're better
described as documentation tests. Whether the maintainer feels that
they should be user acceptance tests is up to the maintainer.

> What's missing in all that cargo-culting is a widely accepted mechanism
> for opting in/out of the tests.  Building something into the toolchain
> -- Module::Build and ExtUtils::MakeMaker -- (and then getting that into
> the Test::Pod docs, maybe CPAN/CPANPLUS, etc.) is just a way of driving
> a standard.
>
> I'd prefer to KISS -- look for the simplest solution that could possibly
> work.  So just because M::B, etc. is a conveient place to push the
> standard doesn't mean that the standard should be (a) overworked or (b)
> too tightly coupled to M::B specifically.

It's not. This isn't a standard - it's a way for users of MB to
differentiate between the various kinds of tests in a way that's
maintainer-specific without any extra coding. If other packaging
systems wish to add this feature, they're more than welcome to do so.

> I prefer the environment variable approach because, while it's global --
> it's one of easiest methods of interprocess communication between a
> build framework, the test harness and test files.

Environment variables aren't supported on all systems in the same way,
plus a number of users have issues with them. Personally, I feel the
usage of environment variables needs to die a very quick death,
replaced with parameters passed upon invocation.

> What I'd *really* like to see is the community agree on an environment
> variable that signals that "author" testing (for lack of a better term)
> is desired.  And then I'd like to see Test::Pod, Test::Pod::Coverage,
> etc. look for that environment variable and skip running their tests
> unless it exists.  In other words -- I'd like the skip logic be a
> convention followed by those who write tests rather than something
> hacked up for all the different ways people might write/structure/run tes=
ts.

You're missing the point. What if I, as the maintainer of my distro,
have a series of tests that require special setup? What about
benchmarking tests? What about tests for a reason that only Damian can
think of?

In other words, the various ways of slicing up the testing strategies
are too numerous for a single agreement to cover them all. You'll
start with one environment variable and end up with 22, each meaning
slight variations of the same thing.

Why not just specify the various taggings (because that's what this is
- a tagging scheme) in one place?

Plus, if you decide that a given test file belongs in a different
testing scheme, you change the Build.PL, not the test file. This means
that the test has more integrity  because it's been touched less.

> Among other things, the environment variable approach means that the
> selection of tests to run doesn't depend on how up to date a user's M::B
> is. It's a big problem if all the tests are *.t and only the latest M::B
> has the logic for which ones to run.  At least the *.ta approach
> defaults to not running.  On the downside, I think getting people to
> shift to *.ta (and getting Module::Starter, ExtUtils::ModuleMaker, etc.
> to use it) will be a bigger hurdle than changing the example synopsis in
> Test::Pod and the default templates, etc.  (Assuming, of course, that
> those who write such tools, like Andy Lester, agree that opt-in is good
> practice.)

This is a straw man and there's a very easy solution - require the
version of MB that provides this feature. This is no different than
the Test::Simple upgrade from 0.46 to 0.47 which had the major API
change. Ever wonder why every Build.PL in existence has:

    build_requires =3D> {
        'Test::More' =3D> '0.47',
    },

This would be a similar API change and would have the exact same
solution. And, if you don't want to use the feature, then you don't
have to require a specific MB version.

I do have another option - I write and maintain Module::Build::Rob
that provides this feature. Except, I don't want to do that because:
    a) I have enough on my plate
    b) The name sucks and I can't think of a better one
    c) I don't want to.

So, maybe this becomes a feature of a MB subclass that is bundled with
the next release of MB. I choose to use it and require people who
install my distros to upgrade. You don't use it and your users don't
have to upgrade to install your distros.

Is that fair?

Rob