module-build-general

Different types of tests (was Re: [Module::Build] [RFC] author tests)

[Rob K. <rob...@gm...>]

I'm reposting the below snippet because I don't think anyone has
seriously considered this option (or, if they have, they haven't
replied back, except for Chris Dolan whose concerns I've addressed and
I have that email below as well.)

Here's the rationale behind my proposal - currently, there's no
mechanism for specifying different types of tests in either MB or
EUMM. Essentially, that's what this proposal is all about. I see a lot
of the confusion arising because different people want different kinds
of tests and don't conceive of them as "author tests", but as "dist
tests", "unit tests", "packaging tests", "acceptance tests", "system
tests", etc ad nauseum.

So, I'm going to expand my proposal below and generalize it to any
kind of test type:

1) The current "./Build test" is really "./Build test
type=3Dacceptance". It's expected that these tests will run on the
user's machine and is the gatekeeper as to whether CPAN(PLUS) will
install or not.
2) The maintainer needs to be able to specify a hierarchy of test
types along with which test files belong under which type.
3) A test file may belong under more than one type.
4) The type "acceptance" will be the default type if no type is
specified. This preserves backward compatibility with the current MB
semantics as well as the expectations of the Perl community at large.
4) The type "full" is a synonym for the type "all".
5) If "build_makefile =3D> 'traditional'" is specified, then the TESTS
parameter for WriteMakefile() will be given an explicity list of all
the testfiles that fall under the "acceptance" type. This preserves
the current EU::MM semantics.
6) "./Build test_foo" will run all the tests under the type "foo".
This is shorthand for "./Build test type=3Dfoo".

If anyone wants, I'll gladly whip up the first iteration of this
functionality this weekend.

Thanks,
Rob

On 2/2/06, Rob Kinyon <rob...@gm...> wrote:
> What about doing the following:
> 1) Create a testfull action (vs. the normal test action - see
> testcover as precedence). This will run all the tests in test_files
> and fulltest_files
> 2) test_files is kept as t/*.t
> 3) fulltest_files is added as an option
>
> If you want to differentiate between regular tests and full tests,
> then you, as the author, are expected to use fulltest_files (and
> possibly test_files) to differentiate. Anything listed in
> fulltest_files (globs permitted) will be EXCLUDED from test_files.
>
> This is, obviously, not quite compatible with EU::MM, but I don't
> think that's a problem.

On 2/2/06, Chris Dolan <ch...@cl...> wrote:
> On Feb 2, 2006, at 9:23 AM, Rob Kinyon wrote:
> > This is, obviously, not quite compatible with EU::MM, but I don't
> > think that's a problem.
>
> That last paragraph is the show-stopper.  If it doesn't work with
> EU::MM then it's DOA.  This is why I initially proposed the *.ta
> suffix, but now prefer the envvar trigger.  The envvar trigger is the
> most backward- and cross-compatible suggestion to date, at the
> expense of making the *.t files a tiny bit more complicated.

It's not INcompatible with EU::MM ... just not completely compatible.
If you use a MB-generated Makefile.PL (as I do), then MB would specify
the test files by hand using the TESTS parameter to WriteMakefile(). I
don't think there's an easy way to add to a new command to EU::MM, but
if there was, then we could add the testfull command there, too.

If you hand-generate your Makefile.PL, then it's your responsability
to make sure that it's equivalent to the Build.PL, as it always has
been.

Re: Different types of tests (was Re: [Module::Build] [RFC] author tests)

[David G. <da...@hy...>]

Rob Kinyon wrote:
> Here's the rationale behind my proposal - currently, there's no
> mechanism for specifying different types of tests in either MB or
> EUMM. Essentially, that's what this proposal is all about. I see a lot
> of the confusion arising because different people want different kinds
> of tests and don't conceive of them as "author tests", but as "dist
> tests", "unit tests", "packaging tests", "acceptance tests", "system
> tests", etc ad nauseum.

I think this will mean massive overbuilding of a framework without a 
clear need for that type of functionality.

If we remember where this all started -- it's the desire not to inflict 
"pod tests" in *.t files on end users.  Thanks to CPANTS and Kwalitee -- 
and not helped by Module::Starter::PBP -- there's a ton of cargo-cult 
non-functional tests winding up in distributions.

What's missing in all that cargo-culting is a widely accepted mechanism 
for opting in/out of the tests.  Building something into the toolchain 
-- Module::Build and ExtUtils::MakeMaker -- (and then getting that into 
the Test::Pod docs, maybe CPAN/CPANPLUS, etc.) is just a way of driving 
a standard.

I'd prefer to KISS -- look for the simplest solution that could possibly 
work.  So just because M::B, etc. is a conveient place to push the 
standard doesn't mean that the standard should be (a) overworked or (b) 
too tightly coupled to M::B specifically.

I prefer the environment variable approach because, while it's global -- 
it's one of easiest methods of interprocess communication between a 
build framework, the test harness and test files.

What I'd *really* like to see is the community agree on an environment 
variable that signals that "author" testing (for lack of a better term) 
is desired.  And then I'd like to see Test::Pod, Test::Pod::Coverage, 
etc. look for that environment variable and skip running their tests 
unless it exists.  In other words -- I'd like the skip logic be a 
convention followed by those who write tests rather than something 
hacked up for all the different ways people might write/structure/run tests.

Among other things, the environment variable approach means that the 
selection of tests to run doesn't depend on how up to date a user's M::B 
is. It's a big problem if all the tests are *.t and only the latest M::B 
has the logic for which ones to run.  At least the *.ta approach 
defaults to not running.  On the downside, I think getting people to 
shift to *.ta (and getting Module::Starter, ExtUtils::ModuleMaker, etc. 
to use it) will be a bigger hurdle than changing the example synopsis in 
Test::Pod and the default templates, etc.  (Assuming, of course, that 
those who write such tools, like Andy Lester, agree that opt-in is good 
practice.)

Regards,
David

Fwd: Different types of tests (was Re: [Module::Build] [RFC] author tests)

[Rob K. <rob...@gm...>]

I forgot to cc the list. :-|

Rob

---------- Forwarded message ----------
From: Rob Kinyon <rob...@gm...>
Date: Feb 3, 2006 12:57 PM
Subject: Re: Different types of tests (was Re: [Module::Build] [RFC]
author tests)
To: David Golden <da...@hy...>


On 2/3/06, David Golden <da...@hy...> wrote:
> I think this will mean massive overbuilding of a framework without a
> clear need for that type of functionality.

I don't think so - I'll explain further down.

> If we remember where this all started -- it's the desire not to inflict
> "pod tests" in *.t files on end users.  Thanks to CPANTS and Kwalitee --
> and not helped by Module::Starter::PBP -- there's a ton of cargo-cult
> non-functional tests winding up in distributions.

I don't think they're non-functional tests. They're useful tests and
I, as a user, like seeing that POD::Coverage passes. They're better
described as documentation tests. Whether the maintainer feels that
they should be user acceptance tests is up to the maintainer.

> What's missing in all that cargo-culting is a widely accepted mechanism
> for opting in/out of the tests.  Building something into the toolchain
> -- Module::Build and ExtUtils::MakeMaker -- (and then getting that into
> the Test::Pod docs, maybe CPAN/CPANPLUS, etc.) is just a way of driving
> a standard.
>
> I'd prefer to KISS -- look for the simplest solution that could possibly
> work.  So just because M::B, etc. is a conveient place to push the
> standard doesn't mean that the standard should be (a) overworked or (b)
> too tightly coupled to M::B specifically.

It's not. This isn't a standard - it's a way for users of MB to
differentiate between the various kinds of tests in a way that's
maintainer-specific without any extra coding. If other packaging
systems wish to add this feature, they're more than welcome to do so.

> I prefer the environment variable approach because, while it's global --
> it's one of easiest methods of interprocess communication between a
> build framework, the test harness and test files.

Environment variables aren't supported on all systems in the same way,
plus a number of users have issues with them. Personally, I feel the
usage of environment variables needs to die a very quick death,
replaced with parameters passed upon invocation.

> What I'd *really* like to see is the community agree on an environment
> variable that signals that "author" testing (for lack of a better term)
> is desired.  And then I'd like to see Test::Pod, Test::Pod::Coverage,
> etc. look for that environment variable and skip running their tests
> unless it exists.  In other words -- I'd like the skip logic be a
> convention followed by those who write tests rather than something
> hacked up for all the different ways people might write/structure/run tes=
ts.

You're missing the point. What if I, as the maintainer of my distro,
have a series of tests that require special setup? What about
benchmarking tests? What about tests for a reason that only Damian can
think of?

In other words, the various ways of slicing up the testing strategies
are too numerous for a single agreement to cover them all. You'll
start with one environment variable and end up with 22, each meaning
slight variations of the same thing.

Why not just specify the various taggings (because that's what this is
- a tagging scheme) in one place?

Plus, if you decide that a given test file belongs in a different
testing scheme, you change the Build.PL, not the test file. This means
that the test has more integrity  because it's been touched less.

> Among other things, the environment variable approach means that the
> selection of tests to run doesn't depend on how up to date a user's M::B
> is. It's a big problem if all the tests are *.t and only the latest M::B
> has the logic for which ones to run.  At least the *.ta approach
> defaults to not running.  On the downside, I think getting people to
> shift to *.ta (and getting Module::Starter, ExtUtils::ModuleMaker, etc.
> to use it) will be a bigger hurdle than changing the example synopsis in
> Test::Pod and the default templates, etc.  (Assuming, of course, that
> those who write such tools, like Andy Lester, agree that opt-in is good
> practice.)

This is a straw man and there's a very easy solution - require the
version of MB that provides this feature. This is no different than
the Test::Simple upgrade from 0.46 to 0.47 which had the major API
change. Ever wonder why every Build.PL in existence has:

    build_requires =3D> {
        'Test::More' =3D> '0.47',
    },

This would be a similar API change and would have the exact same
solution. And, if you don't want to use the feature, then you don't
have to require a specific MB version.

I do have another option - I write and maintain Module::Build::Rob
that provides this feature. Except, I don't want to do that because:
    a) I have enough on my plate
    b) The name sucks and I can't think of a better one
    c) I don't want to.

So, maybe this becomes a feature of a MB subclass that is bundled with
the next release of MB. I choose to use it and require people who
install my distros to upgrade. You don't use it and your users don't
have to upgrade to install your distros.

Is that fair?

Rob