Thread: [Module::Build] [RFC] author tests
Status: Beta
Brought to you by:
kwilliams
Menu
▾
▴
module-build-general
|
From: Chris D. <ch...@cl...> - 2006-02-02 03:46:39
|
I propose a new author test action for Module::Build. There is a class of tests that module authors perform that end users are not expected to run. For example code coverage tests, spelling tests, coding style tests, etc. These tests are either prohibitively expensive or complicated or unpredictable for end users to run. I call these "private tests" or author tests. I discussed this concept on the Perl QA email list and on my blog: http://www.chrisdolan.net/talk/index.php/2005/11/14/private- regression-tests/ My personal practice has been to simply add these private tests to MANIFEST.SKIP. However, it was pointed out to me on perl-qa that it would be better for accountability and reproducability if these were in the distro, but would not run by default under "make test" or "Build test" Roughly, I propose that we adopt a standard file extension for author tests, like "t/*.ta", and add an ACTION_authortest to M::B that runs both t/*.t and t/*.ta (which ones first??) For example, I would then rename my pod.t and pod-coverage.t to pod.ta and pod-coverage.ta and end users would not need to bothered running these non-essential tests. Detailed end users, like CPANTS, could choose to run these or could simply acknowledge their existence (for Kwalitee). I am not wed to the t/*.ta convention or the name of the ACTION. What do others think of the concept in general? Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/) |
|
From: David G. <da...@hy...> - 2006-02-02 04:29:32
|
Chris Dolan wrote: > I am not wed to the t/*.ta convention or the name of the ACTION. What > do others think of the concept in general? What if "authortest" instead set an environment variable prior to running the normal "test" action? (Look at "testcover" as an example of this.) Then .t files could take various actions based on the presence of that environment variable. At the most basic -- just skip_all. However, this would also allow perhaps more diagnostic information to be provided for the author if desired. My point is that making it *.ta specific allows only one way of doing author testing. The environment variable way just signals that author testing is being requested and lets the existing test scripts choose how to respond. Regards, David |
|
From: Chris D. <ch...@cl...> - 2006-02-02 04:49:38
|
On Feb 1, 2006, at 10:28 PM, David Golden wrote: > Chris Dolan wrote: >> I am not wed to the t/*.ta convention or the name of the ACTION. >> What do others think of the concept in general? > > What if "authortest" instead set an environment variable prior to > running the normal "test" action? (Look at "testcover" as an > example of this.) > > Then .t files could take various actions based on the presence of > that environment variable. At the most basic -- just skip_all. > However, this would also allow perhaps more diagnostic information > to be provided for the author if desired. > > My point is that making it *.ta specific allows only one way of > doing author testing. The environment variable way just signals > that author testing is being requested and lets the existing test > scripts choose how to respond. That's pretty smart. It certainly requires less coordination, and would work fine with normal EU::MM setups (just manually set the envvar). Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/) |
|
From: Eric W. <scr...@gm...> - 2006-02-02 05:03:57
|
# from David Golden
# on Wednesday 01 February 2006 08:28 pm:
>Chris Dolan wrote:
>> I am not wed to the t/*.ta convention or the name of the ACTION. =A0
>> What do others think of the concept in general?
>
>What if "authortest" instead set an environment variable prior to
>running the normal "test" action?
I guess I'm undecided on this. On the one hand, the *.ta convention is=20
going to break the test juggler with which I've recently been=20
experimenting, but on the other it means there is less 'whoami' code in=20
the tests. An example of my current whoami-based approach is below.
I suppose it would be nice for it to work both with a *.ta typeglob and=20
to set an environment variable. Possibly the environment variable(s)=20
used should be configurable in Build.PL? e.g. it might be handy to=20
have the following sort of control:
# runs all of the *.t* files with all authortest_env =3D> [qw(POD WARN)]
./Build authortest
# just set AUTHORTEST_POD and run all *.t*
./Build authortest POD
# just set AUTHORTEST_WARN and run all *.t*
./Build authortest WARN
But then you might also want Build.PL to configure whether or not *.t=20
tests are run?
=2D------
#!/usr/bin/perl
use Test::More;
eval "use Test::Pod::Coverage 1.04";
plan skip_all =3D>
"Test::Pod::Coverage 1.04 required for testing POD coverage"
if $@;
plan skip_all =3D> '$ENV{TEST_POD_COVERAGE} is not set'
unless((($ENV{USER} || '') eq 'ewilhelm') or =20
exists($ENV{TEST_POD_COVERAGE}));
all_pod_coverage_ok();
=2D------
=2D-Eric
=2D-=20
Issues of control, repair, improvement, cost, or just plain
understandability all come down strongly in favor of open source
solutions to complex problems of any sort.
=2D-Robert G. Brown
=2D--------------------------------------------------
http://scratchcomputing.com
=2D--------------------------------------------------
|
|
From: David G. <da...@hy...> - 2006-02-02 12:49:47
|
Eric Wilhelm wrote: > I guess I'm undecided on this. On the one hand, the *.ta convention is > going to break the test juggler with which I've recently been > experimenting, but on the other it means there is less 'whoami' code in > the tests. An example of my current whoami-based approach is below. My thought was that if someone wants to use *.ta files or a ta/*.t directory, they can still do so by subclassing in Build.PL and having a custom authortest action that includes those files. David |
|
From: Rob K. <rob...@gm...> - 2006-02-02 15:23:42
|
What about doing the following: 1) Create a testfull action (vs. the normal test action - see testcover as precedence). This will run all the tests in test_files and fulltest_files 2) test_files is kept as t/*.t 3) fulltest_files is added as an option If you want to differentiate between regular tests and full tests, then you, as the author, are expected to use fulltest_files (and possibly test_files) to differentiate. Anything listed in fulltest_files (globs permitted) will be EXCLUDED from test_files. This is, obviously, not quite compatible with EU::MM, but I don't think that's a problem. Rob. |
|
From: Chris D. <ch...@cl...> - 2006-02-02 15:53:07
|
On Feb 2, 2006, at 9:23 AM, Rob Kinyon wrote: > What about doing the following: > 1) Create a testfull action (vs. the normal test action - see > testcover as precedence). This will run all the tests in test_files > and fulltest_files > 2) test_files is kept as t/*.t > 3) fulltest_files is added as an option > > If you want to differentiate between regular tests and full tests, > then you, as the author, are expected to use fulltest_files (and > possibly test_files) to differentiate. Anything listed in > fulltest_files (globs permitted) will be EXCLUDED from test_files. > > This is, obviously, not quite compatible with EU::MM, but I don't > think that's a problem. That last paragraph is the show-stopper. If it doesn't work with EU::MM then it's DOA. This is why I initially proposed the *.ta suffix, but now prefer the envvar trigger. The envvar trigger is the most backward- and cross-compatible suggestion to date, at the expense of making the *.t files a tiny bit more complicated. Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/) |
|
From: Rob K. <rob...@gm...> - 2006-02-02 16:02:34
|
On 2/2/06, Chris Dolan <ch...@cl...> wrote: > On Feb 2, 2006, at 9:23 AM, Rob Kinyon wrote: > > This is, obviously, not quite compatible with EU::MM, but I don't > > think that's a problem. > > That last paragraph is the show-stopper. If it doesn't work with > EU::MM then it's DOA. This is why I initially proposed the *.ta > suffix, but now prefer the envvar trigger. The envvar trigger is the > most backward- and cross-compatible suggestion to date, at the > expense of making the *.t files a tiny bit more complicated. It's not INcompatible with EU::MM ... just not completely compatible. If you use a MB-generated Makefile.PL (as I do), then MB would specify the test files by hand using the TESTS parameter to WriteMakefile(). I don't think there's an easy way to add to a new command to EU::MM, but if there was, then we could add the testfull command there, too. If you hand-generate your Makefile.PL, then it's your responsability to make sure that it's equivalent to the Build.PL, as it always has been. Rob |
|
From: Chris D. <ch...@cl...> - 2006-02-02 16:14:53
|
On Feb 2, 2006, at 10:02 AM, Rob Kinyon wrote: > On 2/2/06, Chris Dolan <ch...@cl...> wrote: >> On Feb 2, 2006, at 9:23 AM, Rob Kinyon wrote: >>> This is, obviously, not quite compatible with EU::MM, but I don't >>> think that's a problem. >> >> That last paragraph is the show-stopper. If it doesn't work with >> EU::MM then it's DOA. This is why I initially proposed the *.ta >> suffix, but now prefer the envvar trigger. The envvar trigger is the >> most backward- and cross-compatible suggestion to date, at the >> expense of making the *.t files a tiny bit more complicated. > > It's not INcompatible with EU::MM ... just not completely compatible. > If you use a MB-generated Makefile.PL (as I do), then MB would specify > the test files by hand using the TESTS parameter to WriteMakefile(). I > don't think there's an easy way to add to a new command to EU::MM, but > if there was, then we could add the testfull command there, too. > > If you hand-generate your Makefile.PL, then it's your responsability > to make sure that it's equivalent to the Build.PL, as it always has > been. Sorry, I wasn't clear enough about my objection. Sure, anybody could make this work today with no enhancement to M::B or EU::MM. But if it requires you to modify Makefile.PL or Build.PL in some special way, then it's just more cargo cult and no real community improvement. Look at what happened to Tels recently with his custom YAML creating Makefile.PL on the perl-qa list. He created a custom solution that mirrored the M::B META.yml solution, but when YAML.pm changed, he had to update *all* of his modules instead of just having his users update a single upstream package. I see that as a process failure. I started this thread to look for a consensus way to implement author tests in a more forward-looking way than the pod.t solution has been to date. Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/) |
|
From: Chris D. <ch...@cl...> - 2006-02-02 16:07:52
|
On Feb 2, 2006, at 6:49 AM, David Golden wrote:
> Eric Wilhelm wrote:
>> I guess I'm undecided on this. On the one hand, the *.ta
>> convention is going to break the test juggler with which I've
>> recently been experimenting, but on the other it means there is
>> less 'whoami' code in the tests. An example of my current whoami-
>> based approach is below.
>
> My thought was that if someone wants to use *.ta files or a ta/*.t
> directory, they can still do so by subclassing in Build.PL and
> having a custom authortest action that includes those files.
Oh, sure, everything I've suggested in this thread *could* be done as
Build.PL customization. But the reason I brought this up was to try
to achieve some consensus and find a Best Practice before the Cargo
Cult started by Test::Pod takes a deeper hold.
I like the thought of having this as a feature in M::B because then
it flattens the learning curve for a third party to mess with my
author tests. And as a corollary, I don't have to scratch my head as
much when looking at someone else's author tests.
As a concrete example, take this standard pod.t:
use Test::More;
eval "use Test::Pod 1.14";
plan skip_all => "Test::Pod 1.14 required for testing POD" if $@;
all_pod_files_ok();
and rewrite as a hypothetical author test (I'm not advocating for
"RUN_AUTHOR_TESTS" as a name):
use Test::More;
plan skip_all => "author test" if (!$ENV{RUN_AUTHOR_TESTS});
eval 'use Test::Pod 1.14';
all_pod_files_ok();
This is NOT simpler, but is slightly more standardizable since the
first two lines could be identical for every author test.
Additionally, the first one implies to the user that their Perl
environment may be substandard without Test::Pod when really the lack
of Test::Pod is really a performance benefit to them if they're not a
developer!
Chris
--
Chris Dolan, Software Developer, Clotho Advanced Media Inc.
608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703
vCard: http://www.chrisdolan.net/ChrisDolan.vcf
Clotho Advanced Media, Inc. - Creators of MediaLandscape Software
(http://www.media-landscape.com/) and partners in the revolutionary
Croquet project (http://www.opencroquet.org/)
|
|
From: Tyler M. <ty...@yi...> - 2006-02-02 18:15:52
|
Chris Dolan <ch...@cl...> wrote: > This is NOT simpler, but is slightly more standardizable since the > first two lines could be identical for every author test. > Additionally, the first one implies to the user that their Perl > environment may be substandard without Test::Pod when really the lack > of Test::Pod is really a performance benefit to them if they're not a > developer! If they're not a developer, what are they doing running unit tests in the first place??? End users typically install applications, which either prebundle a whole bunch of perl packages or rely on some package management system (ppm, deb, etc) to separate real dependancies from build dependancies for them and only install what they need. I think in this case Test::Pod should be a build_requires... Your other examples (copyright.t, etc) obviously will only work on your system so it makes sense for those to have some sort of special treatment. - Tyler |
|
From: Chris D. <ch...@cl...> - 2006-02-02 21:03:18
|
On Feb 2, 2006, at 12:11 PM, Tyler MacDonald wrote: > If they're not a developer, what are they doing running unit tests > in the first place??? End users typically install applications, > which either > prebundle a whole bunch of perl packages or rely on some package > management > system (ppm, deb, etc) to separate real dependancies from build > dependancies > for them and only install what they need. The end users may be in a different environment from me. I usually only tests Perl 5.8.6 on OS X and Linux before release, so I rely on other installers to provide bug reports for me. > I think in this case Test::Pod should be a build_requires... Your > other examples (copyright.t, etc) obviously will only work on your > system so > it makes sense for those to have some sort of special treatment. Test::Pod is different from general regression tests. POD should either pass on all systems or fail on all. In what scenario would POD fail on a non-developer system. And even if it did fail, do I want to scare people off from using my code because I have an =item without an =over? No. So, I want pod.t to be a non-core test in my modules. Currently I only include it in MANIFEST because of Kwalitee [1]. Chris [1] However, I am grateful to Kwalitee for getting to adopt Test::Pod and its ilk in the first place! -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/) |
|
From: Christopher H. L. <cl...@ch...> - 2006-02-02 21:06:44
Attachments:
signature.asc
|
Chris Dolan wrote: > On Feb 2, 2006, at 12:11 PM, Tyler MacDonald wrote: >=20 >> If they're not a developer, what are they doing running unit tests= >> in the first place??? End users typically install applications, which >> either >> prebundle a whole bunch of perl packages or rely on some package >> management >> system (ppm, deb, etc) to separate real dependancies from build >> dependancies >> for them and only install what they need. >=20 > The end users may be in a different environment from me. I usually onl= y > tests Perl 5.8.6 on OS X and Linux before release, so I rely on other > installers to provide bug reports for me. >=20 >> I think in this case Test::Pod should be a build_requires... Your >> other examples (copyright.t, etc) obviously will only work on your >> system so >> it makes sense for those to have some sort of special treatment. >=20 > Test::Pod is different from general regression tests. POD should eithe= r > pass on all systems or fail on all. In what scenario would POD fail on= > a non-developer system. =20 Manifying problems...ala troff/pod2html/pod2man issues.... -=3DChris |
|
From: Chris D. <ch...@cl...> - 2006-02-02 21:14:33
|
On Feb 2, 2006, at 3:06 PM, Christopher H. Laco wrote: > Chris Dolan wrote: >> Test::Pod is different from general regression tests. POD should >> either >> pass on all systems or fail on all. In what scenario would POD >> fail on >> a non-developer system. > > Manifying problems...ala troff/pod2html/pod2man issues.... This is news to me. So what do you do with CPAN packages that don't include pod.t and have broken POD? Just fail to install them? It seems to me that to solve manifying problems you need *authors* to run Test::Pod, not end users since the latter have no power to fix the distro. If you are a packager, then you probably have to run Test::Pod by hand anyway since so many packages lack pod.t. Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/) |
|
From: Christopher H. L. <cl...@ch...> - 2006-02-02 21:23:24
Attachments:
signature.asc
|
Chris Dolan wrote: > On Feb 2, 2006, at 3:06 PM, Christopher H. Laco wrote: >=20 >> Chris Dolan wrote: >>> Test::Pod is different from general regression tests. POD should eit= her >>> pass on all systems or fail on all. In what scenario would POD fail = on >>> a non-developer system. >> >> Manifying problems...ala troff/pod2html/pod2man issues.... >=20 > This is news to me. So what do you do with CPAN packages that don't > include pod.t and have broken POD? Just fail to install them? >=20 > It seems to me that to solve manifying problems you need *authors* to > run Test::Pod, not end users since the latter have no power to fix the > distro. If you are a packager, then you probably have to run Test::Pod= > by hand anyway since so many packages lack pod.t. >=20 > Chris > --Chris Dolan, Software Developer, Clotho Advanced Media Inc. > 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 > vCard: http://www.chrisdolan.net/ChrisDolan.vcf >=20 > Clotho Advanced Media, Inc. - Creators of MediaLandscape Software > (http://www.media-landscape.com/) and partners in the revolutionary > Croquet project (http://www.opencroquet.org/) >=20 >=20 >=20 Yes, and No. The author can run pod tests and ensure correct pod. What the packager or end user may not know, and the author can not know, is if the various pod parsing modules on the users system are kosher, or if the end users versions will have issues converting the _correct_ pod into other formats. Pod syntax test can be the first sign of thing being wrong they may leave the user without readable docs assuming an install succeeds... While true, it's not the module authors job to ensure the users system can properly spit out man/html pages from the good pod, it's always nice to know that some OSes version of perl/pod2*/Pod:: is going to cause your dist to fail the make install "manifying pod" portion...and a failing dist comes right back into the authors lap..or the packagers... Sure, it's rare, but it does happen... -=3DChris |
|
From: Ken W. <ke...@ma...> - 2006-02-03 03:47:29
|
On Feb 2, 2006, at 10:07 AM, Chris Dolan wrote: > On Feb 2, 2006, at 6:49 AM, David Golden wrote: > >> Eric Wilhelm wrote: >>> I guess I'm undecided on this. On the one hand, the *.ta convention >>> is going to break the test juggler with which I've recently been >>> experimenting, but on the other it means there is less 'whoami' code >>> in the tests. An example of my current whoami-based approach is >>> below. >> >> My thought was that if someone wants to use *.ta files or a ta/*.t >> directory, they can still do so by subclassing in Build.PL and having >> a custom authortest action that includes those files. > > Oh, sure, everything I've suggested in this thread *could* be done as > Build.PL customization. But the reason I brought this up was to try > to achieve some consensus and find a Best Practice before the Cargo > Cult started by Test::Pod takes a deeper hold. There's already a property called 'recursive_test_files' that would presumably cover this need. In general I like the idea. In specific I don't like the idea of using an environment variable. Could we use some parameter or command-line argument or something like that instead? Environment variables aren't completely portable and they're about as global as one can get, so usually I've tried to avoid using them in M::B. > I like the thought of having this as a feature in M::B because then it > flattens the learning curve for a third party to mess with my author > tests. And as a corollary, I don't have to scratch my head as much > when looking at someone else's author tests. Right, and it becomes a paradigm that people just know about, rather than having to scrutinize some author's custom build system. -Ken |
|
From: Ron S. <ro...@sa...> - 2006-02-03 05:59:20
|
On Thu, 2 Feb 2006 21:47:21 -0600, Ken Williams wrote: Hi Ken > In general I like the idea. In specific I don't like the idea of > using an environment variable. Could we use some parameter or > command-line argument or something like that instead? Environment > variables aren't completely portable and they're about as global as > one can get, so usually I've tried to avoid using them in M::B. The killer for an env var is that I for one install modules via a cgi script on machines where I do /not/ have access to httpd.conf, so I can't ever use Perl(Set)Env to pass personal env vars thru to my scripts. This is the danger of admin-centric proposals. -- Ron Savage ro...@sa... http://savage.net.au/index.html |
|
From: David G. <da...@hy...> - 2006-02-03 12:01:42
|
Ron Savage wrote: > The killer for an env var is that I for one install modules via a cgi script on > machines where I do /not/ have access to httpd.conf, so I can't ever use > Perl(Set)Env to pass personal env vars thru to my scripts. But do you need to run all the author tests, then? David |
|
From: Ron S. <ro...@sa...> - 2006-02-03 23:03:42
|
On Fri, 03 Feb 2006 07:01:33 -0500, David Golden wrote: Hi David > But do you need to run all the author tests, then? No, that's true. There are other issues like: Some modules have tests which fail, but I still= go ahead and install and use the modules anyway. Where does this all end? -- Cheers Ron Savage, ro...@sa... on 4/02/2006 http://savage.net.au/index.html Let the record show: Microsoft is not an Australian company |
|
From: Austin S. <te...@of...> - 2006-02-03 23:35:50
|
On Fri, Feb 03, 2006 at 04:58:28PM +1100, Ron Savage wrote:
> On Thu, 2 Feb 2006 21:47:21 -0600, Ken Williams wrote:
>
> Hi Ken
>
> > In general I like the idea. In specific I don't like the idea of
> > using an environment variable. Could we use some parameter or
> > command-line argument or something like that instead? Environment
> > variables aren't completely portable and they're about as global as
> > one can get, so usually I've tried to avoid using them in M::B.
>
> The killer for an env var is that I for one install modules via a cgi script on
> machines where I do /not/ have access to httpd.conf, so I can't ever use
> Perl(Set)Env to pass personal env vars thru to my scripts.
>
Dumb question: can't you just $ENV{'FOO'}= ... before you run the
tests?
I have many scripts which call config files upon startup to set
environment variables and the like.
You wouldn't be able to run the tests directly, but I'm not sure
that's relevant.
Austin
|
|
From: Ron S. <ro...@sa...> - 2006-02-04 06:54:44
|
On Fri, 3 Feb 2006 15:35:42 -0800, Austin Schutz wrote:
Hi Austin
> Dumb question: can't you just $ENV{'FOO'}=3D ... before you run the
> tests?
You mean from the command line? Sure. But why should I be /forced/ to use=
the
command line?
--
Cheers
Ron Savage, ro...@sa... on 4/02/2006
http://savage.net.au/index.html
Let the record show: Microsoft is not an Australian company
|
|
From: Tyler M. <ty...@yi...> - 2006-02-02 04:36:11
|
Chris Dolan <ch...@cl...> wrote: > There is a class of tests that module authors perform that end users > are not expected to run. For example code coverage tests, spelling > tests, coding style tests, etc. These tests are either prohibitively > expensive or complicated or unpredictable for end users to run. I > call these "private tests" or author tests. I really like this idea. But as you pointed out, it's not just authors that need to worry about running these tests, it's packagers (ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and hackers. I'd suggest we call these "exhaustive" tests. Spelling tests, code coverage tests, testing interaction with CPAN.pm, etc. are things that I think should be done by automated testers, and by people packaging your module for distribution inside another larger system (eg, PPMs or debian packages). So maybe it would make sense if there were certain tests that only run if AUTOMATED_TESTING is set, and/or if the "disttest" action is being run. That said, it would be easy to whip together a Test:: module that only allows these tests to run under these conditions, especially if Module::Build, say, set a "DIST_TEST" environment variable when the disttest action was being run. I already check for AUTOMATED_TESTING to manipulate my test environments to be more exhaustive. Both Module::Build, and perhaps future versions of ExtUtils::MakeMaker (if there will ever be a future version) could depend on this Test:: module (whether or not they use it themselves) so that it's already on everybody's system. It'd be nice and small so I don't see anybody complaining too loudly. Maybe we could call it "Test::Exhaustive"? It's mandate being to SKIP_ALL of a test if you havent somehow (AUTOMATED_TESTING, disttest, etc) made it clear that you wish to exhaustively test this module? > Roughly, I propose that we adopt a standard file extension for author > tests, like "t/*.ta", and add an ACTION_authortest to M::B that runs > both t/*.t and t/*.ta (which ones first??) If they were named differently (i don't think they have to be), I'd want to see them run in lexical order like the current tests, so it'd go "01foo.t", "02bar.ta", "03baz.t". Cheers, Tyler |
|
From: Chris D. <ch...@cl...> - 2006-02-02 15:49:43
|
On Feb 1, 2006, at 10:35 PM, Tyler MacDonald wrote: > Chris Dolan <ch...@cl...> wrote: >> There is a class of tests that module authors perform that end users >> are not expected to run. For example code coverage tests, spelling >> tests, coding style tests, etc. These tests are either prohibitively >> expensive or complicated or unpredictable for end users to run. I >> call these "private tests" or author tests. > > I really like this idea. But as you pointed out, it's not just > authors that need to worry about running these tests, it's packagers > (ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and > hackers. > I'd suggest we call these "exhaustive" tests. No, I disagree. I'm specifically talking about author tests, NOT packager tests. Things like Test::Spelling are pointless and difficult for packagers to execute because Test::Spelling relies on an external aspell or ispell program *and* performs differently in the presence of an author's custom dictionary (mine has "Dolan"; does yours?) These specifically are not exhaustive tests but spit-and-polish tests. To make this less abstract, let me list the specific author tests that I employ for most of my CPAN modules, along with an explanation of why it wouldn't work for a packager * copyright.t - Ensures that there is a "Copyright ".([localtime]-> [5]+1900) somewhere in every .pm file. Will break 11 months from now. * distribution.t - Relies on Test::Distribution, which is not in my prereq list * perlcritic.t - Runs Test::Perl::Critic on all .pm files. Will fail without my specific $HOME/.perlcriticrc and will fail with future, more exhaustive versions of P::C * spelling.t - Runs Test::Spelling. Will fail without my custom dictionary * versionsync.t - Checks that the $VERSION is the same in all bin/* and *.pm files. This test is pointless after release, since it's already been tested before release * pod.t - Checks POD validity. This test is pointless after release, since it's already been tested before release * pod-coverage.t - Checks POD completeness. This test is pointless after release, since it's already been tested before release and one I have not yet employed: * coverage.t - Ensures that Devel::Cover totals are higher than some threshold Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/) |
|
From: Marvin H. <ma...@re...> - 2006-02-02 17:22:24
|
On Feb 2, 2006, at 7:49 AM, Chris Dolan wrote: > I'm specifically talking about author tests, NOT packager tests. +1 on the idea of author tests. I don't have strong opinions about the implementation. For one of my distros, there's a script ./bin/test_valgrind.plx, which runs all the other tests under valgrind (this is an XS distro) and kicks out a log file for me to peruse. Haven't quite figured out how to run it under Test::More, but that's how it ought to work, and it is clearly an author test, not a standard test, since it takes forever and depends upon the presence of valgrind. There's also a ./ bin/test_tidiness.plx, which does a perltidy check on each .pm/.pod file -- which takes a while, depends upon perltidy, and is pointless after release for the same reason that checking pod is pointless after release. If there had been a standardized procedure for me to follow, these scripts would be in a more sensible place than ./bin, and it would be more apparent what they're there for. Also, if there had been a procedure for running all of them, they'd get run more reliably -- on the latest release I remembered to run the valgrind tests, but forgot the perltidy tests. Marvin Humphrey Rectangular Research http://www.rectangular.com/ |
|
From: Tyler M. <ty...@yi...> - 2006-02-02 18:05:43
|
Chris Dolan <ch...@cl...> wrote: > * copyright.t - Ensures that there is a "Copyright ".([localtime]-> > [5]+1900) somewhere in every .pm file. Will break 11 months from now. > * distribution.t - Relies on Test::Distribution, which is not in my > prereq list > * perlcritic.t - Runs Test::Perl::Critic on all .pm files. Will > fail without my specific $HOME/.perlcriticrc and will fail with > future, more exhaustive versions of P::C > * spelling.t - Runs Test::Spelling. Will fail without my custom > dictionary > * versionsync.t - Checks that the $VERSION is the same in all bin/* > and *.pm files. This test is pointless after release, since it's > already been tested before release > * pod.t - Checks POD validity. This test is pointless after > release, since it's already been tested before release > * pod-coverage.t - Checks POD completeness. This test is pointless > after release, since it's already been tested before release > > and one I have not yet employed: > * coverage.t - Ensures that Devel::Cover totals are higher than > some threshold Wow, you really *are* exhaustive. How do you find the time to write any code? ;-) Now that I understand exactly what you mean by "author" tests, here's what I think: Whatever convention you're using, if these tests are only going to work on your system, then they definately shouldn't be in "t". And since there's absolutely no value in these types of tests for anybody else except the module author, there's no real point in having a convention, just stick 'em anywhere that the make/buildfiles will ignore them. - Tyler |
Thanks for helping keep SourceForge clean.
X