Different types of tests (was Re: [Module::Build] [RFC] author tests)

[Rob K. <rob...@gm...>]

I'm reposting the below snippet because I don't think anyone has
seriously considered this option (or, if they have, they haven't
replied back, except for Chris Dolan whose concerns I've addressed and
I have that email below as well.)

Here's the rationale behind my proposal - currently, there's no
mechanism for specifying different types of tests in either MB or
EUMM. Essentially, that's what this proposal is all about. I see a lot
of the confusion arising because different people want different kinds
of tests and don't conceive of them as "author tests", but as "dist
tests", "unit tests", "packaging tests", "acceptance tests", "system
tests", etc ad nauseum.

So, I'm going to expand my proposal below and generalize it to any
kind of test type:

1) The current "./Build test" is really "./Build test
type=3Dacceptance". It's expected that these tests will run on the
user's machine and is the gatekeeper as to whether CPAN(PLUS) will
install or not.
2) The maintainer needs to be able to specify a hierarchy of test
types along with which test files belong under which type.
3) A test file may belong under more than one type.
4) The type "acceptance" will be the default type if no type is
specified. This preserves backward compatibility with the current MB
semantics as well as the expectations of the Perl community at large.
4) The type "full" is a synonym for the type "all".
5) If "build_makefile =3D> 'traditional'" is specified, then the TESTS
parameter for WriteMakefile() will be given an explicity list of all
the testfiles that fall under the "acceptance" type. This preserves
the current EU::MM semantics.
6) "./Build test_foo" will run all the tests under the type "foo".
This is shorthand for "./Build test type=3Dfoo".

If anyone wants, I'll gladly whip up the first iteration of this
functionality this weekend.

Thanks,
Rob

On 2/2/06, Rob Kinyon <rob...@gm...> wrote:
> What about doing the following:
> 1) Create a testfull action (vs. the normal test action - see
> testcover as precedence). This will run all the tests in test_files
> and fulltest_files
> 2) test_files is kept as t/*.t
> 3) fulltest_files is added as an option
>
> If you want to differentiate between regular tests and full tests,
> then you, as the author, are expected to use fulltest_files (and
> possibly test_files) to differentiate. Anything listed in
> fulltest_files (globs permitted) will be EXCLUDED from test_files.
>
> This is, obviously, not quite compatible with EU::MM, but I don't
> think that's a problem.

On 2/2/06, Chris Dolan <ch...@cl...> wrote:
> On Feb 2, 2006, at 9:23 AM, Rob Kinyon wrote:
> > This is, obviously, not quite compatible with EU::MM, but I don't
> > think that's a problem.
>
> That last paragraph is the show-stopper.  If it doesn't work with
> EU::MM then it's DOA.  This is why I initially proposed the *.ta
> suffix, but now prefer the envvar trigger.  The envvar trigger is the
> most backward- and cross-compatible suggestion to date, at the
> expense of making the *.t files a tiny bit more complicated.

It's not INcompatible with EU::MM ... just not completely compatible.
If you use a MB-generated Makefile.PL (as I do), then MB would specify
the test files by hand using the TESTS parameter to WriteMakefile(). I
don't think there's an easy way to add to a new command to EU::MM, but
if there was, then we could add the testfull command there, too.

If you hand-generate your Makefile.PL, then it's your responsability
to make sure that it's equivalent to the Build.PL, as it always has
been.