perlunit-users

[Perlunit-users] PerlUnit Development

[Joi E. <gy...@vi...>]

Greetings!  I downloaded PerlUnit 0.25 from SourceForge a few months ago
for use in a personal perl project.  I've been spending nearly as much time
hacking on Perl Unit as I have my own project.  

I've worked mostly on the Tk GUI and on the Test::Harness compabilitity issues.
IE my local copy now recognizes mostly-proper TAP format from all the other 
Test::* modules, and it produces mostly proper TAP output as well.  I've fixed some bugs,
too, like HarnessUnit failing to pass any useful diagnostic information back to the GUI.

In all, I've worked on the following classes:

Test/Unit/Exception.pm
Test/Unit/HarnessUnit.pm
Test/Unit/Loader.pm
Test/Unit/Result.pm
Test/Unit/Runner.pm
Test/Unit/TkTestRunner.pm
Test/Unit/UnitHarness.pm

I've created a subclass of TestCase.pm that includes versions of tests
copied from Test::More as well.

Test/Unit/MyTestCase.pm

All this still passes all of the original Perl Unit tests when my custom versions are
loaded.

Is there any interest out there?

-- 
Joi Ellis                    
gy...@vi...

No matter what we think of Linux versus FreeBSD, etc., the one thing I
really like about Linux is that it has Microsoft worried.  Anything
that kicks a monopoly in the pants has got to be good for something.
           - Chris Johnson

Re: [Perlunit-users] PerlUnit Development

[Matthew A. <mc...@us...>]

On Sat, Feb 18, 2006 at 02:14:13AM -0600, Joi Ellis wrote:

> Greetings!  I downloaded PerlUnit 0.25 from SourceForge a few months
> ago for use in a personal perl project.  I've been spending nearly
> as much time hacking on Perl Unit as I have my own project.

Cool!  Thanks for letting us know.

> I've worked mostly on the Tk GUI and on the Test::Harness
> compabilitity issues.  IE my local copy now recognizes mostly-proper
> TAP format from all the other Test::* modules, and it produces
> mostly proper TAP output as well.

For other readers: TAP is "test anything protocol", see
http://use.perl.org/~petdance/journal/22057

> I've fixed some bugs, too, like HarnessUnit failing to pass any
> useful diagnostic information back to the GUI.

It would help the changelog if you could list the bugs fixed, it may
not be possible for me to figure this out from your changes.  Failing
this, we'll just have a slightly vaguely changelog.


> In all, I've worked on the following classes:
> 
> Test/Unit/Exception.pm
> Test/Unit/HarnessUnit.pm
> Test/Unit/Loader.pm
> Test/Unit/Result.pm
> Test/Unit/Runner.pm
> Test/Unit/TkTestRunner.pm
> Test/Unit/UnitHarness.pm

How would you like to send these changes?  In exchange for agreeing to
"the same terms as Perl" you can be listed in AUTHORS.  Further
enthusiasm runs the risk of being added as a SF developer.  8-)

Output of "cvs diff -u" plus name or date of what you started hacking
on would suit me quite well, but it won't include new files.  Or you
can just tarball whatever you have and let me sort it out.

Adding patch ticket(s) on Sourceforge will keep your work in view.  If
your changes can be split by theme, sending several patch tickets with
smaller diffs would probably help me out - if you've done a lot of
work it may take me a while to catch up all of it.


> I've created a subclass of TestCase.pm that includes versions of tests
> copied from Test::More as well.
> 
> Test/Unit/MyTestCase.pm

Thread fork here...

Also, a minor point on naming (sorry).  I think T:U:MyTestCase is
unsuitable for project release because of the convention for naming
things MyFoo in the POD, when intending to suggest that it is the
user's own Foo under discussion.  i.e. while your code is joining the
project you're not a Joe Public user.

Apart from T:U:Assert refactoring, possible solutions for the name: if
T:U:MyTestCase only includes versions of tests copied from Test::More,
it might be best named T:U:MoreTestCase or something like that.  Or if
you wanted some code ownership, T:U:JoiTestCase.


> All this still passes all of the original Perl Unit tests when my
> custom versions are loaded.

That's always good.  Do your changes need additional testing or POD?


There may be an issue here with module dependencies - it isn't fair to
expect the user to install the whole set of "Test::* module that reads
or writes TAP" in order to install Test::Unit.

However, it does make a lot of sense for a perlunit developer to run
tests against them if the project is to support them properly.  It may
even be a good idea to keep output from old versions of such modules,
because we can't assume the user is up-to-date without a dependency,
and testing can be a bit special when it comes to legacy support.

Perhaps it's best to bottle some samples of TAP (!) and distribute
them.  Then make some additional code to require the relevant modules
and bottle up some new TAP.

Testing TAP output against an assortment of code that reads it would
be harder.

OK, this could get complicated.  Perhaps it's best to start with
something, then extend it later.


> Is there any interest out there?

I'm definitely interested, and yes this does mean to the extent of
writing tests or maybe even POD.  9-)

My roundtuitometer informs me that we have patches for TestDecorator
and an XML (Ant compatible?) testrunner, plus at least one slightly
scary bug (is_numeric, rt.cpan.org #16130) in the queue already.

I don't claim to operate a fair queueing algorithm for these things,
but I will listen to requests from other users.


Matthew  #8-)

Re: [Perlunit-users] PerlUnit Development

[Joi E. <gy...@vi...>]

On Sun, 19 Feb 2006, Matthew Astley wrote:

> On Sat, Feb 18, 2006 at 02:14:13AM -0600, Joi Ellis wrote:
> 
> For other readers: TAP is "test anything protocol", see
> http://use.perl.org/~petdance/journal/22057

Even easier would be perldoc Test::Harness:TAP, since Test::Harness
is bundled with current perls and its documented there.

> > I've fixed some bugs, too, like HarnessUnit failing to pass any
> > useful diagnostic information back to the GUI.
> 
> It would help the changelog if you could list the bugs fixed, it may
> not be possible for me to figure this out from your changes.  Failing
> this, we'll just have a slightly vaguely changelog.

Is there a specific bug list you want me to look at?  The work I did
was mostly to make the gui usable for my particular project, but it's 
general enough.

I've looked at the bugs list on cpan and on sourceforge.  The stuff I've
address isn't mentioned in any of those that I can see.  I haven't filed any
bug reports myself.
 
> > In all, I've worked on the following classes:
> > 
> > Test/Unit/Exception.pm
> > Test/Unit/HarnessUnit.pm
> > Test/Unit/Loader.pm
> > Test/Unit/Result.pm
> > Test/Unit/Runner.pm
> > Test/Unit/TkTestRunner.pm
> > Test/Unit/UnitHarness.pm
> 
> How would you like to send these changes?  In exchange for agreeing to
> "the same terms as Perl" you can be listed in AUTHORS.  Further
> enthusiasm runs the risk of being added as a SF developer.  8-)

Whatever you like.  I'm not sure how much time I can devote to it, the
past two weeks I've been home on medical leave from my regular job so I've
had an unusual amount of spare time lately.

I already have one project on source forge, XPCGI.  Haven't touched it in
years, though.  I'm tempted to port the thing to Perl/Tk now. ;)

> Output of "cvs diff -u" plus name or date of what you started hacking
> on would suit me quite well, but it won't include new files.  Or you
> can just tarball whatever you have and let me sort it out.

Well, I got lazy and used RCS to track the changes I've made.  That's
where I got the ChangeLog I sent, with some manual edits to delete
unimportant junk comments.

I could probably generate some patch files but it may take me some time,
I get to go back to work tomorrow morning if the doctor agrees.

> Adding patch ticket(s) on Sourceforge will keep your work in view.  If
> your changes can be split by theme, sending several patch tickets with
> smaller diffs would probably help me out - if you've done a lot of
> work it may take me a while to catch up all of it.

That's a new feature since I last worked with SF.  I presume that's a 
place where outsiders can put their patches?  I could do that.  Or I could
create bug reports at cpan and upload patch files as attachments there, too.

Which do you prefer?

> > I've created a subclass of TestCase.pm that includes versions of tests
> > copied from Test::More as well.
> > 
> > Test/Unit/MyTestCase.pm
> 
> Thread fork here...
> 
> Also, a minor point on naming (sorry).  I think T:U:MyTestCase is
> unsuitable for project release because of the convention for naming
> things MyFoo in the POD, when intending to suggest that it is the
> user's own Foo under discussion.  i.e. while your code is joining the
> project you're not a Joe Public user.

No, I didn't intend MyTestCase to become part of the project, I just used
that because I wanted to keep my extensions separate.  I left it in
Test::Unit for my own organization purposes inside Komodo.

> Apart from T:U:Assert refactoring, possible solutions for the name: if
> T:U:MyTestCase only includes versions of tests copied from Test::More,
> it might be best named T:U:MoreTestCase or something like that.  Or if
> you wanted some code ownership, T:U:JoiTestCase.

I don't care about ownership much.  I added another assert last night,
using code lifted from Test::Warn and rewritten to suit Perl Unit, so now
I can capture and test for warn() and carp() from the method under test.
This isn't tested much yet and I haven't yet written a test suite for it.

> > All this still passes all of the original Perl Unit tests when my
> > custom versions are loaded.
> 
> That's always good.  Do your changes need additional testing or POD?
 
It probably wouldn't hurt me to write test cases that show how 0.25
fails to handle TAP, and make sure I've fixed the tests.

> There may be an issue here with module dependencies - it isn't fair to
> expect the user to install the whole set of "Test::* module that reads
> or writes TAP" in order to install Test::Unit.

No, it isn't.  That sort of thing can be made optional both in the install and
in the tests.  None of the changes I've made to Perl Unit classes
involve new Test::* modules.  Even MyTestCase.pm doesn't require them, it
just steals from them. ;)

I do have a mix of *.t files that use Test::More, Test::Pod, etc, which is
one reason I've been hacking on the Tk gui so much.  The gui can now 
handle output from non-Perl Unit tests in a usable fashion, which it utterly
failed to do in 0.25. 

> However, it does make a lot of sense for a perlunit developer to run
> tests against them if the project is to support them properly.  It may
> even be a good idea to keep output from old versions of such modules,
> because we can't assume the user is up-to-date without a dependency,
> and testing can be a bit special when it comes to legacy support.

> Perhaps it's best to bottle some samples of TAP (!) and distribute
> them.  Then make some additional code to require the relevant modules
> and bottle up some new TAP.

Indeed.  It occurs to me that one could use the tests provided with the
Test::Harness module under t/sample-tests.  I haven't gone to that
extreme yet, though.  I've just been fixing what didn't work for my own
project's tests.

> Testing TAP output against an assortment of code that reads it would
> be harder.

I expect a simple test for this would be to use 'prove' against the
whole test suite.  One might want to look around in Test::Harness's tests
to see if they don't already have a TAP validation test.

> 
> OK, this could get complicated.  Perhaps it's best to start with
> something, then extend it later.
> 
> > Is there any interest out there?
> 
> I'm definitely interested, and yes this does mean to the extent of
> writing tests or maybe even POD.  9-)
> 
> My roundtuitometer informs me that we have patches for TestDecorator
> and an XML (Ant compatible?) testrunner, plus at least one slightly
> scary bug (is_numeric, rt.cpan.org #16130) in the queue already.

I remember reading a bug report somewhere more general, but I don't 
remember exactly where it was.  The gist of the issue that that the 
linux libc/glibc is_numeric routine recognizes one additional format
than other unices do, so it fails the test for that routine.

If you look around on Google you can see a bunch of projects where
this particular linux "enhancement" is generating bug reports.
I'd probably just make the test check to see if its running under a 
linux kernel and adjust the test accordingly.  It's not really a
Perl Unit issue anyway.

-- 
Joi Ellis                    
gy...@vi...

No matter what we think of Linux versus FreeBSD, etc., the one thing I
really like about Linux is that it has Microsoft worried.  Anything
that kicks a monopoly in the pants has got to be good for something.
           - Chris Johnson

Re: [Perlunit-users] PerlUnit Development

[Matthew A. <mc...@us...>]

On Sun, Feb 19, 2006 at 03:24:13PM -0600, Joi Ellis wrote:
> On Sun, 19 Feb 2006, Matthew Astley wrote:
> > On Sat, Feb 18, 2006 at 02:14:13AM -0600, Joi Ellis wrote:

> > It would help the changelog if you could list the bugs fixed, it
> > may not be possible for me to figure this out from your changes.
> 
> Is there a specific bug list you want me to look at?  [...]

No, I understood that you had found new bugs and fixed them.  So they
won't be on any bug list...  I don't need bug reports, I was just
after some text to put in the changelog.

> > Failing this, we'll just have a slightly vaguely changelog.

8-)


> > > In all, I've worked on the following classes:
> > > 
> > > Test/Unit/Exception.pm
> > > Test/Unit/HarnessUnit.pm
> > > Test/Unit/Loader.pm
> > > Test/Unit/Result.pm
> > > Test/Unit/Runner.pm
> > > Test/Unit/TkTestRunner.pm
> > > Test/Unit/UnitHarness.pm

> > How would you like to send these changes?  [...]
> 
> Whatever you like.  I'm not sure how much time I can devote to it,

The first thing to do is get whatever you would like to send into the
SF tracker (or somewhere else) so it doesn't ... fade away.

> Well, I got lazy and used RCS to track the changes I've made.
> That's where I got the ChangeLog I sent, with some manual edits to
> delete unimportant junk comments.

Sent?  I haven't seen your changes yet, did I miss them?  I looked in
the SF patch tracker but can't see anything.

I think this is a perfectly valid approach to tweaking files when you
want version control before you submit the patch.  I guess it's why
Arch, SVK and others are popular.

> I could probably generate some patch files but it may take me some
> time, [...]

That'd be good.  Or just upload a tarball of the RCS ",v" files and
your ChangeLog, or whatever you sent last time.  I can deal with
those.

> > Adding patch ticket(s) on Sourceforge will keep your work in view.
> > If your changes can be split by theme, sending several patch
> > tickets with smaller diffs would probably help me out - if you've
> > done a lot of work it may take me a while to catch up all of it.
> 
> That's a new feature since I last worked with SF.  I presume that's
> a place where outsiders can put their patches?  I could do that.  Or
> I could create bug reports at cpan and upload patch files as
> attachments there, too.
> 
> Which do you prefer?

I prefer the SF tracker.  Actually I would be most happy with a
redirection from rt.cpan.org to the SF tracker, there's no merit in
having two bug systems for one package.  8-(


> > > I've created a subclass of TestCase.pm that includes versions of
> > > tests copied from Test::More as well.
[...]
> No, I didn't intend MyTestCase to become part of the project, I just
> used that because I wanted to keep my extensions separate.  I left
> it in Test::Unit for my own organization purposes inside Komodo.

OK.  If you're willing to contribute it too then perhaps I can pick
stuff out of it as I get to reorganising the assertion code.


> [...] I added another assert last night, using code lifted from
> Test::Warn and rewritten to suit Perl Unit, so now I can capture and
> test for warn() and carp() from the method under test.  This isn't
> tested much yet and I haven't yet written a test suite for it.

Ah yes, I have some like this.  Chaos when Carp.pm loads afterwards.
Sigh.

While the patch is open, feel free to just append another pair of
(file + comment) to it.  I guess I should get another release out soon
after, so you can have 0.26 do what you're expecting.


> > > All this still passes all of the original Perl Unit tests when
> > > my custom versions are loaded.
> > 
> > That's always good.  Do your changes need additional testing or POD?
>  
> It probably wouldn't hurt me to write test cases that show how 0.25
> fails to handle TAP, and make sure I've fixed the tests.

Well if you don't get time, you can just say it's the maintainer's
job.  8-]



> Indeed.  It occurs to me that one could use the tests provided with
> the Test::Harness module under t/sample-tests.

Neat idea, I've added this to my TODO.


> > [...] plus at least one slightly scary bug (is_numeric,
> > rt.cpan.org #16130) in the queue already.

> [...]
> If you look around on Google you can see a bunch of projects where
> this particular linux "enhancement" is generating bug reports.  I'd
> probably just make the test check to see if its running under a
> linux kernel and adjust the test accordingly.  It's not really a
> Perl Unit issue anyway.

True it's not a PU issue, but I think it's important for test
portability - the sort of thing the framework should warn the user
about.


Thanks for the extra info and ideas,

Matthew  #8-)

[Perlunit-users] T:U:Assertion, test framework wars; was Re: PerlUnit Development

[Matthew A. <mc...@us...>]

On Sat, Feb 18, 2006 at 02:14:13AM -0600, Joi Ellis wrote:

> I've created a subclass of TestCase.pm that includes versions of tests
> copied from Test::More as well.
> 
> Test/Unit/MyTestCase.pm

I too have a collection of assertion methods, I keep them in my own
T:U:TestCase subclass and yes it's high time I shared it.

Subclassing is the obvious OO way to extend the code, but it is not
without problems.  To bring our code into the project, the obvious
thing to do is just push the new methods up into a superclass so
they're available to everyone...  unless the subclass has a specific
purpose.  But then if the user wishes to have a subclass for doing
local tricks, e.g. database testing utils, they'll need to do it
twice.

I believe Test::Unit::Assert already contains too much stuff all in
one place, leading to

 - overwhelming docs

 - may give new users the impression that there is far too much to
   learn in one go

 - no sensible place to put even *more* stuff

 - accompanying problems in t/tlib/AssertTest.pm

 - hard to share useful assertion code with other test frameworks

 - could interfere with the user's choice of inheritance pattern

Leaving the file to grow without bound as we find more useful and
exciting tests makes it difficult to manage the code and the method
namespace.


For example, T:U:Assert->assert_deep_equals was "shamelessly pinched
from Test::More and adapted to Test::Unit".  This method has an
assortment of bugs filed against it.

I have applied patches but when I look at all this, I wonder

 - which version of Test::More did it come from? (I haven't
   investigated)

 - is it fixed in Test::More, or is it an artifact of the "adapted to
   Test::Unit"?

 - if I fix or extend assert_deep_equals, can I contribute back to
   Test::More?

I use assert_deep_equals quite a bit myself, but I would prefer to put
some distance between maintenance of it and the rest of the framework.


Solutions so far:

 - make a contrib/ directory containing T:U:TestCase subclasses,
   starting with mine and Joi's, and leave the user to pick among the
   code in the methods or using them as SUPER.  Easy for us, but not
   great for code reuse.

 - use mixin classes to supply[1] assertion methods, possibly in a
   dynamic way.  But eww, mixins.

 - symbol table pokery and/or AUTOLOAD sickness to do...  what,
   dispatch methods by hand?

 - continue growing Assert.pm, maybe I imagined a problem?


There are other test frameworks out there with their own assertion
names, argument styles (including conflicting ones[2]) and neat
tricks.  A bit of competition is healthy, but I'm not interested in
proving "our test framework is better than yours".

It would be much more useful to make it easy for users to try one or
another, including porting existing tests without too much fuss.  It
would also be good to make it possible for testrunners (GUI, automated
continuous integration, continuous testing[3], random testing[4] etc.)
to be shared between frameworks.

On my To Do list are such lofty aims as finding out what else is out
there in test world, and then talking to the maintainers of the other
tools.

A comparison here is what I've seen so far while looking at test
frameworks for Common Lisp.  Each provides a set of features, but
there's nowhere you can go to get all the features (whatever that
might mean).

This leads me to think that the test framework should be doing less,
but providing hooks for other packages to do the rest.  More test
runners, more assertions, more types of test failure (todo, skip, slow
test filtering, test dependency).

Clearly this needs more thought and more input data.  A start would be
collecting people's testcase subclasses in a contrib directory, just
to see what's there.


Matthew  #8-)
-- 
[1] On the user side, this might cause a test to be written

 package MyTest;
 use strict;
 use warnings;
 use base 'Test::Unit::TestCase';
 use Test::Unit::AssertMixin::Foo;
 use Test::Unit::AssertMixin::Bar;
 sub test_with_foo {
   my $self = shift;
   $self->assert_foo( ... );
   $self->assert_bar( ... );
 }

The user may shuffle the incantation for mixins out to a project
module or the suit generator, to reduce the per-class noise.

On the perlunit side, the requirement would be for the mixin classes
to register themselves during &import.


[2] http://sf.net/tracker/index.php?func=detail&aid=407833&group_id=2653&atid=102653

The conflict is against the Java flavour, and it's mostly an artifact
of Perl's argument passing style, but still it's unfortunate:

  JUnit 3.5 has
  assert(message, condition);

  perlUnit .13 has
  assert(condition, message);

Maybe there are others, I haven't looked.


[3] http://pag.csail.mit.edu/continuoustesting/

[4] http://www.accesscom.com/~darius/software/clickcheck.html
http://www.findinglisp.com/blog/2004/10/random-testing.html including
link to a scary "razor sharp html fragments" article