perlunit-devel

[Perlunit-devel] Greetings

[Matthias F. <mf...@hi...>]

Hi folks -- you've all probably seen Christian's and Piers's responses to
my query on the other newsgroup, so I won't repeat all the background
here.  (I thought I remembered seeing something on sourceforge about this
list being the totally dead one and the other one only mostly dead, but
apparently my memory was playing tricks on me.)  Christian and Piers,
thanks to both of you for the fast and helpful answers.  It looks like the
interface changes in progress are mostly things a little searching and
replacing could take care of, so I'm going to press onward with what I'm
doing.

At any rate, I'm really liking PerlUnit in its current form and I'd like
to be of assistance however I can.  For the moment that's probably limited
to enthusiastic feedback and questions, but perhaps in the future I'll
have time to be more constructive if you're looking for more help.

I do have these bits of feedback and suggestions, nothing major.  Bear in
mind that I don't know what you're working on now, although I've been
browsing the CVS tree and I see that you've anticipated at least some of
these things...

1) There's no indication yet (that I noticed) of which methods in an
object are intended to be the public interface and which are for internal
use only.  This is important for what I'm working on, because (for reasons
that I won't go into here) I need to invoke TestSuite::do_run() directly
from a script instead of invoking a TestRunner.  If that method isn't
meant to be public, then I'll need to find a workaround, which may lead to
a feature request.  [Actually I think I've just figured out how I can
avoid using this method directly, but still...]

2) I was going to ask if there were any plans for specialized assert() 
methods, like assert_equals, but I see that this is already the direction
you're moving in, which is great.  In particular, I'd like to put in a
plug for assert_matches(regex, string) and assert_not_matches (or
something).  The scheme for using assert() with a regex as the first
parameter is cool, but to test a non-match you have to use a hack, and in
either case you're limited to the default failure message, which for a
regex is likely to be unhelpful.

---Whooops, I see that the message parameter has been added more recently. 
That's terrific.  It would still be nice to have a not-matches assertion,
though, to avoid the (?!) hack.  I'm doing this kind of test, and I
resorted to the scalar(x=~y) syntax instead because it seemed less ugly. 
(I've just discovered the neat trick in Assert.pm for reusing coderefs as
assertions.  That's cool.)

3) Without turning Perl autoflushing on, you can't see the tests
proceeding in a text-based test.  It would be cool if the suite, or
whichever class does the output, could ensure that $| is nonzero so the
dots get displayed as soon as their written.  (Maybe it could even restore
the original value.)  I'm doing that explicitly in the script I've
written. 

It also occurred to me that a "nice-to-have" would be a verbose option
where it displays the name of each test and the TestCase class as it
progresses.  I don't think JUnit has that either, but I found myself
wanting it yesterday.  (Ooh, I see you've made the debug setting global.
Very nice.)

The SourceForge feature request tracker doesn't seem to be getting much
use but I could put these in there if that would help.  Again, you guys
are doing really good work here.  I was very surprised, incidentally, that
a Perl port wasn't finished long ago, but go figure.

Gotta get back to my real job now.

-- Matthias Ferber
   mf...@hi...

Re: [Perlunit-devel] Greetings

[Piers C. <pdc...@bo...>]

Matthias Ferber <mf...@hi...> writes:

> Hi folks -- you've all probably seen Christian's and Piers's responses to
> my query on the other newsgroup, so I won't repeat all the background
> here.  (I thought I remembered seeing something on sourceforge about this
> list being the totally dead one and the other one only mostly dead, but
> apparently my memory was playing tricks on me.)  Christian and Piers,
> thanks to both of you for the fast and helpful answers.  It looks like the
> interface changes in progress are mostly things a little searching and
> replacing could take care of, so I'm going to press onward with what I'm
> doing.
>
> At any rate, I'm really liking PerlUnit in its current form and I'd like
> to be of assistance however I can.  For the moment that's probably limited
> to enthusiastic feedback and questions, but perhaps in the future I'll
> have time to be more constructive if you're looking for more help.
>
> I do have these bits of feedback and suggestions, nothing major.  Bear in
> mind that I don't know what you're working on now, although I've been
> browsing the CVS tree and I see that you've anticipated at least some of
> these things...
>
> 1) There's no indication yet (that I noticed) of which methods in an
> object are intended to be the public interface and which are for internal
> use only.  This is important for what I'm working on, because (for reasons
> that I won't go into here) I need to invoke TestSuite::do_run() directly
> from a script instead of invoking a TestRunner.  If that method isn't
> meant to be public, then I'll need to find a workaround, which may lead to
> a feature request.  [Actually I think I've just figured out how I can
> avoid using this method directly, but still...]

I'd consider do_run to be a 'semi' private method. You're going to
need to know about it if you're writing a Runner, but most of the time
you're not going to go near it.

>
> 2) I was going to ask if there were any plans for specialized assert() 
> methods, like assert_equals, but I see that this is already the direction
> you're moving in, which is great.  In particular, I'd like to put in a
> plug for assert_matches(regex, string) and assert_not_matches (or
> something).  The scheme for using assert() with a regex as the first
> parameter is cool, but to test a non-match you have to use a hack, and in
> either case you're limited to the default failure message, which for a
> regex is likely to be unhelpful.
>
> ---Whooops, I see that the message parameter has been added more recently. 
> That's terrific.  It would still be nice to have a not-matches assertion,
> though, to avoid the (?!) hack.  I'm doing this kind of test, and I
> resorted to the scalar(x=~y) syntax instead because it seemed less ugly. 
> (I've just discovered the neat trick in Assert.pm for reusing coderefs as
> assertions.  That's cool.)

Thanks. I did the Assert reworking and looking at it now I seem to
have overcomplicated it somewhat, but it's definitely flexible :).

How about:

$self->assert_matches(qr/.../, $value, $message) 
$self->assert_does_not_match(qr/.../, $value, $message) 

assert_matches is almost exactly like assert(qr/.../), but it fails if
it's not passed a pattern as its first argument.

> 3) Without turning Perl autoflushing on, you can't see the tests
> proceeding in a text-based test.  It would be cool if the suite, or
> whichever class does the output, could ensure that $| is nonzero so the
> dots get displayed as soon as their written.  (Maybe it could even restore
> the original value.)  I'm doing that explicitly in the script I've
> written. 

That's something for the Runner class. I would be nice though.

> It also occurred to me that a "nice-to-have" would be a verbose option
> where it displays the name of each test and the TestCase class as it
> progresses.  

Again, that's something to hang off the runner, and it definitely
looks possible.

> I don't think JUnit has that either, but I found myself
> wanting it yesterday.  (Ooh, I see you've made the debug setting global.
> Very nice.)

I'm seriously considering hiding the debug variable from end users and
providing a bunch of ->debug, ->info, ->notice type methods which the
runner can display or not as it chooses and using the debug variable
solely for framework debugging stuff. One of the problems with the old
stack dump stuff for instance was that it was only really of any use
to framework developers, and it was a pig to turn off. 

> The SourceForge feature request tracker doesn't seem to be getting
> much use but I could put these in there if that would help. Again,
> you guys are doing really good work here. I was very surprised,
> incidentally, that a Perl port wasn't finished long ago, but go
> figure.

It's sort of confusing. It's sort of finished, but it is also poorly
documented and much harder to extend than it should be. It's been
usable for a long time now, but it's been, well, ragged 'round the
edges so we need to get those tidied up before we can slap a 1.0 label
on it. 

And then the fun really begins. I want something which will make it
easty for non programmers to specify acceptance tests for instance.
And Test::Unit::Web would be nice, and something to do mock objects,
and...

-- 
Piers

   "It is a truth universally acknowledged that a language in
    possession of a rich syntax must be in need of a rewrite."
         -- Jane Austen?

Re: [Perlunit-devel] Greetings

[Adam S. <ad...@sp...>]

Piers Cawley (pdc...@bo...) wrote:
> Matthias Ferber <mf...@hi...> writes:
> > 1) There's no indication yet (that I noticed) of which methods in an
> > object are intended to be the public interface and which are for internal
> > use only.

Yeah, this is bad.  Eventually all public methods should be documented
by accompanying pod, I guess.

> I'd consider do_run to be a 'semi' private method. You're going to
> need to know about it if you're writing a Runner, but most of the time
> you're not going to go near it.

Agreed.  The fact that PerlUnit is aimed at developers is inevitably
going to blur the lines between private/public.

> > 2) I was going to ask if there were any plans for specialized assert() 
> > methods, like assert_equals, but I see that this is already the direction
> > you're moving in, which is great.  In particular, I'd like to put in a
> > plug for assert_matches(regex, string) and assert_not_matches (or
> > something).  The scheme for using assert() with a regex as the first
> > parameter is cool, but to test a non-match you have to use a hack, and in
> > either case you're limited to the default failure message, which for a
> > regex is likely to be unhelpful.
> >
> > ---Whooops, I see that the message parameter has been added more recently. 
> > That's terrific.  It would still be nice to have a not-matches assertion,
> > though, to avoid the (?!) hack.  I'm doing this kind of test, and I
> > resorted to the scalar(x=~y) syntax instead because it seemed less ugly. 
> > (I've just discovered the neat trick in Assert.pm for reusing coderefs as
> > assertions.  That's cool.)
> 
> Thanks. I did the Assert reworking and looking at it now I seem to
> have overcomplicated it somewhat, but it's definitely flexible :).
> 
> How about:
> 
> $self->assert_matches(qr/.../, $value, $message) 
> $self->assert_does_not_match(qr/.../, $value, $message) 
> 
> assert_matches is almost exactly like assert(qr/.../), but it fails if
> it's not passed a pattern as its first argument.

Sounds good.  I might hack that up now ...

> > 3) Without turning Perl autoflushing on, you can't see the tests
> > proceeding in a text-based test.  It would be cool if the suite, or
> > whichever class does the output, could ensure that $| is nonzero so the
> > dots get displayed as soon as their written.  (Maybe it could even restore
> > the original value.)  I'm doing that explicitly in the script I've
> > written. 
> 
> That's something for the Runner class. I would be nice though.

Committed ...

> > It also occurred to me that a "nice-to-have" would be a verbose option
> > where it displays the name of each test and the TestCase class as it
> > progresses.  
> 
> Again, that's something to hang off the runner, and it definitely
> looks possible.

Yep.  Can you give us an example of the output you'd like to see?  (Or
a patch ;-)

> > I don't think JUnit has that either, but I found myself
> > wanting it yesterday.  (Ooh, I see you've made the debug setting global.
> > Very nice.)
> 
> I'm seriously considering hiding the debug variable from end users and
> providing a bunch of ->debug, ->info, ->notice type methods which the
> runner can display or not as it chooses and using the debug variable
> solely for framework debugging stuff. One of the problems with the old
> stack dump stuff for instance was that it was only really of any use
> to framework developers, and it was a pig to turn off. 

I better summarise the status quo w.r.t. debugging just in case my
recent changes have confused anyone.  First, there is no longer a
DEBUG constant anywhere; it has been replaced with a more unified
approach in Test::Unit::Debug.  This means debugging no longer gets
optimised out by the compile phase, but it's extremely unlikely that
this will cause anything more than a negligible performance hit.
Test::Unit::Debug is another case of "semi-privacy"; you can use it if
you want, but it's only really there for people messing around with
the framework.  On the other hand, if you want to debug your own test
cases, you can now do 

  sub test_case {
    my $self = shift;
    ...
    $self->print('some debugging\n');
    ...
  }

and the debugging will only appear if the test case fails.  This is
currently only supported by TestRunner.pm.  Maybe we should think of
a better name than print().

> And then the fun really begins. I want something which will make it
> easty for non programmers to specify acceptance tests for instance.
> And Test::Unit::Web would be nice, and something to do mock objects,
> and...

Worth chucking these ideas in doc/TODO, even if they are a bit vague
at the moment?

Re: [Perlunit-devel] Greetings

[Adam S. <ad...@sp...>]

Adam Spiers (ad...@sp...) wrote:
> Piers Cawley (pdc...@bo...) wrote:
> > Matthias Ferber <mf...@hi...> writes:
> > How about:
> > 
> > $self->assert_matches(qr/.../, $value, $message) 
> > $self->assert_does_not_match(qr/.../, $value, $message) 
> > 
> > assert_matches is almost exactly like assert(qr/.../), but it fails if
> > it's not passed a pattern as its first argument.
> 
> Sounds good.  I might hack that up now ...

Done.  I actually ended up making substantial improvements to
AssertTest.pm, so the tests therein are even more stringent than
before.  As a result it's even more likely that if I cocked something
up, it's broken 'make test'.  Works for me of course, but please cvs
update and test.  Here's the log message for those not on
perlunit-commits yet:

  - Add new assert_matches() and assert_does_not_match() assertions,
    as requested by Matthias Ferber, and new tests for them of course.

  - Start testing for the correct messages in T::U::Failure objects
    (see check_exception()).  In particular this tests that the optional
    MESSAGE args are processed correctly when reporting failures.

  - Improved behaviour of assert_equals() and assert_not_equals() with
    undefined parameters.

  - Added more stringent tests for assert_equals() and assert_not_equals().
    Some of these are possibly debatable.  I think the assert_equals()
    matrix should be retired in favour of the check_failures() style tests
    used to check assert_not_equals(), since the latter also check the
    failure message and originating file/line (added a #FIXME for this).

  - Fixed reversed got/expected message with $self->ok(sub { 2 + 2 }, 4);
    Spotted this as a bonus of the new check_exception(), yay.

I'm off to Jersey tomorrow morning until Sunday, so I won't be
available until I get back.

Re: [Perlunit-devel] Greetings

[Matthias F. <mf...@hi...>]

On Tue, 11 Dec 2001, Adam Spiers wrote:
> > > It also occurred to me that a "nice-to-have" would be a verbose option
> > > where it displays the name of each test and the TestCase class as it
> > > progresses.  
> > 
> > Again, that's something to hang off the runner, and it definitely
> > looks possible.
> 
> Yep.  Can you give us an example of the output you'd like to see?  (Or
> a patch ;-)

Yeah, sure.  I hadn't thought it through in detail but it shouldn't be
hard to work up a simple patch.  If I find time to do that, what should I
do with it exactly?  Send diffs?

>   sub test_case {
>     my $self = shift;
>     ...
>     $self->print('some debugging\n');
>     ...
>   }
> 
> and the debugging will only appear if the test case fails.  This is
> currently only supported by TestRunner.pm.  Maybe we should think of
> a better name than print().

This sounds pretty cool.  What's the thinking behind making $self->print() 
output only show up when tests fail?  I'm guessing that the idea is that
you can print details of what's going on in the test so that when the test
fails you have to do less digging to find the problem, but it doesn't
clutter up the screen when the test succeeds.  If that's correct I think
it's great.  I wrote some tests in JUnit the other day where I could have
really used this feature: I had to do a bunch of messing around with
StringBuffers to cram a whole lot of data into the failure message.

How about $self->print_on_fail()?


> > And Test::Unit::Web would be nice, and something to do mock objects,
> > and...

Hmm... I'm not sure what Test::Unit::Web is meant to be, but... it so
happens that my first trial use of PerlUnit has been to do some stock web
site testing.  We host a few hundred web sites, the majority of which are
functionally identical but the rest of which differ to varying degrees, so
I took pains to make the code general and reusable.  If that's what
Test::Unit::Web would be about, my base class could possibly make a decent
starting point.  It tests that a page loads successfully within a
customizable number of seconds, and provides some facilities for testing
the contents of the page.  More could easily be added.  Anyone interested?

Matthias

Re: [Perlunit-devel] Greetings

[Piers C. <pdc...@bo...>]

Matthias Ferber <mf...@hi...> writes:

> On Tue, 11 Dec 2001, Adam Spiers wrote:
>> > > It also occurred to me that a "nice-to-have" would be a verbose option
>> > > where it displays the name of each test and the TestCase class as it
>> > > progresses.  
>> > 
>> > Again, that's something to hang off the runner, and it definitely
>> > looks possible.
>> 
>> Yep.  Can you give us an example of the output you'd like to see?  (Or
>> a patch ;-)
>
> Yeah, sure.  I hadn't thought it through in detail but it shouldn't be
> hard to work up a simple patch.  If I find time to do that, what should I
> do with it exactly?  Send diffs?

Diffs would be good. If you start making more and we like them I'll
see about giving you write access to the CVS repository.

>>   sub test_case {
>>     my $self = shift;
>>     ...
>>     $self->print('some debugging\n');
>>     ...
>>   }
>> 
>> and the debugging will only appear if the test case fails.  This is
>> currently only supported by TestRunner.pm.  Maybe we should think of
>> a better name than print().
>
> This sounds pretty cool.  What's the thinking behind making $self->print() 
> output only show up when tests fail?  I'm guessing that the idea is that
> you can print details of what's going on in the test so that when the test
> fails you have to do less digging to find the problem, but it doesn't
> clutter up the screen when the test succeeds.  

Yup. Remember that the print message gets sent to the
Listener/TestRunner. If you want a more verbose style you can easily
(must check that statement) subclass the TestRunner and change that
behaviour. 

> If that's correct I think it's great. I wrote some tests in JUnit
> the other day where I could have really used this feature: I had to
> do a bunch of messing around with StringBuffers to cram a whole lot
> of data into the failure message.
>
> How about $self->print_on_fail()?

Nope. Deciding when/whether to print the message is a function of the
runner not of the testcase. C<$self->send_diagnostic(...)> anybody?
Hmm...

>> > And Test::Unit::Web would be nice, and something to do mock objects,
>> > and...
>
> Hmm... I'm not sure what Test::Unit::Web is meant to be, but... it
> so happens that my first trial use of PerlUnit has been to do some
> stock web site testing. We host a few hundred web sites, the
> majority of which are functionally identical but the rest of which
> differ to varying degrees, so I took pains to make the code general
> and reusable. If that's what Test::Unit::Web would be about, my base
> class could possibly make a decent starting point. It tests that a
> page loads successfully within a customizable number of seconds, and
> provides some facilities for testing the contents of the page. More
> could easily be added. Anyone interested?

Ooh, yes please. That's pretty much exactly what I was thinking of. 

-- 
Piers

   "It is a truth universally acknowledged that a language in
    possession of a rich syntax must be in need of a rewrite."
         -- Jane Austen?

Re: [Perlunit-devel] Greetings

[Matthias F. <mf...@hi...>]

On Wed, 12 Dec 2001, Piers Cawley wrote:
> Diffs would be good. If you start making more and we like them I'll
> see about giving you write access to the CVS repository.
....
> > provides some facilities for testing the contents of the page. More
> > could easily be added. Anyone interested?
> 
> Ooh, yes please. That's pretty much exactly what I was thinking of. 

Cool. :-)  I'm in serious deadline hell for the next couple of weeks, but
I'll try to find time to sneak this stuff out the door.

I think send_diagnostics() isn't bad.  Taking print out of the name is
probably a good idea, I think it's misleading (at least it misled me).
Maybe diagnostic_msg()?  Or maybe $self->annotate(), in the sense that
it's not part of the main output, and the runner controls how much
annotation you want (e.g., never, only on fail, or always)?  Just some
additional brainstorming.

Matthias

Re: [Perlunit-devel] Greetings

[Piers C. <pdc...@bo...>]

Matthias Ferber <mf...@hi...> writes:

> On Wed, 12 Dec 2001, Piers Cawley wrote:
>> Diffs would be good. If you start making more and we like them I'll
>> see about giving you write access to the CVS repository.
> ....
>> > provides some facilities for testing the contents of the page.
>> > More could easily be added. Anyone interested?
>> 
>> Ooh, yes please. That's pretty much exactly what I was thinking of. 
>
> Cool. :-) I'm in serious deadline hell for the next couple of weeks,
> but I'll try to find time to sneak this stuff out the door.

Whenever.

> I think send_diagnostics() isn't bad.  Taking print out of the name is
> probably a good idea, I think it's misleading (at least it misled me).
> Maybe diagnostic_msg()?  Or maybe $self->annotate(), in the sense that
> it's not part of the main output, and the runner controls how much
> annotation you want (e.g., never, only on fail, or always)?  Just some
> additional brainstorming.

Hmm... annotate definitely sounds possible.

-- 
Piers

   "It is a truth universally acknowledged that a language in
    possession of a rich syntax must be in need of a rewrite."
         -- Jane Austen?

Re: [Perlunit-devel] Greetings

[Adam S. <ad...@sp...>]

Piers Cawley (pdc...@bo...) wrote:
> Matthias Ferber <mf...@hi...> writes:
> > On Wed, 12 Dec 2001, Piers Cawley wrote:
> >> Diffs would be good. If you start making more and we like them I'll
> >> see about giving you write access to the CVS repository.
> > ....
> >> > provides some facilities for testing the contents of the page.
> >> > More could easily be added. Anyone interested?
> >> 
> >> Ooh, yes please. That's pretty much exactly what I was thinking of. 

Yes please indeed!  We're desperate for that kind of thing here too.

> > Cool. :-) I'm in serious deadline hell for the next couple of weeks,
> > but I'll try to find time to sneak this stuff out the door.
> 
> Whenever.
> 
> > I think send_diagnostics() isn't bad.  Taking print out of the name is
> > probably a good idea, I think it's misleading (at least it misled me).
> > Maybe diagnostic_msg()?  Or maybe $self->annotate(), in the sense that
> > it's not part of the main output, and the runner controls how much
> > annotation you want (e.g., never, only on fail, or always)?  Just some
> > additional brainstorming.
> 
> Hmm... annotate definitely sounds possible.

I like that too.  annotate() it will be then.  Might manage a commit
later tonight ...

[Perlunit-devel] new package?

[Hans D. <co...@po...>]

Hi there,

Maybe an idea to package a new release before any new stuff is being added=
 / refactored? (and with the current speed of changes to do this every=
 month or so?)
Im following the recent changes with great interest, but Im not such a CVS=
 addict to retrieve the latest version being developed (currently I havent=
 got the time; I work with PerlUnit only at home (my job does not include=
 programming)).


_

  Greetings,
  Hans Donner

Re: [Perlunit-devel] new package?

[Piers C. <pdc...@bo...>]

"Hans Donner" <co...@po...> writes:

> Hi there,
>
> Maybe an idea to package a new release before any new stuff is being
> added / refactored? (and with the current speed of changes to do
> this every month or so?) Im following the recent changes with great
> interest, but Im not such a CVS addict to retrieve the latest
> version being developed (currently I havent got the time; I work
> with PerlUnit only at home (my job does not include programming)).

Good idea. I'll see about packaging up what we have and shoving it on
sourceforge and CPAN.

-- 
Piers

   "It is a truth universally acknowledged that a language in
    possession of a rich syntax must be in need of a rewrite."
         -- Jane Austen?

Re: [Perlunit-devel] new package?

[Adam S. <ad...@sp...>]

Piers Cawley (pdc...@bo...) wrote:
> "Hans Donner" <co...@po...> writes:
> > Maybe an idea to package a new release before any new stuff is being
> > added / refactored? (and with the current speed of changes to do
> > this every month or so?) Im following the recent changes with great
> > interest, but Im not such a CVS addict to retrieve the latest
> > version being developed (currently I havent got the time; I work
> > with PerlUnit only at home (my job does not include programming)).
> 
> Good idea. I'll see about packaging up what we have and shoving it on
> sourceforge and CPAN.

That'd be great.  Even if 1.0 doesn't happen this year, much better to
have a 0.15 rather than nothing at all.  I'd certainly like to see
more people trying out my recent changes before we start thinking of a
stable release ...

I notice doc/release-checklist is slightly out of date, incidentally.