perlunit-users

[Perlunit-users] Using @TESTS for changing tests execution order

[Dmitry D. <dd...@ic...>]

Hi All!

I read from TestCase.pm POD that one can populate @TESTS array in order 
to specify the desired order of tests execution:

### quote begin

If you need to specify the test order, you can do one of the
following:

=over 4

=item * Set @TESTS

   our @TESTS = qw(my_test my_test_2);

This is the simplest, and recommended way.

### quote end

But the above does not work for me. I can include only desired tests 
using @TESTS, but they are listed and executed in a random order, 
although I name them as test_001_foo, test_oo2_bar, etc. I think that 
the reason is in the following return statement (line 114 of TestCase.pm):

     return keys %tests;

I decided to override list_tests method like this:

sub list_tests {
     my $self = shift;
     return sort $self->SUPER::list_tests(@_);
}

It works as expected. What do you think? Am I missing something?

Thanks
Dmitry.

Re: [Perlunit-users] Using @TESTS for changing tests execution order

[Matthew A. <mc...@us...>]

On Sat, Oct 04, 2003 at 10:06:19AM +0400, Dmitry Diskin wrote:

> I read from TestCase.pm POD that one can populate @TESTS array in
> order to specify the desired order of tests execution:

Ah, I started at the bottom of my inbox and worked back...  why does
it matter what order the tests run?  The xUnit idea is that each test
(and its associated setup/teardown) should stand alone.


> ### quote begin
> 
> If you need to specify the test order, you can do one of the
> following:
> 
> =over 4
> 
> =item * Set @TESTS
> 
>   our @TESTS = qw(my_test my_test_2);
> 
> This is the simplest, and recommended way.
> 
> ### quote end

> But the above does not work for me. I can include only desired tests
> using @TESTS, but they are listed and executed in a random order,

Hmm, that comment seems to be out of date.  @TESTS is piled into a
hash, then the keys are listed as you said.

> I decided to override list_tests method like this:
> 
> sub list_tests {
>     my $self = shift;
>     return sort $self->SUPER::list_tests(@_);
> }

That would do it, yes.  Did you see in Test/Unit/TestCase.pm,

  # Returns a list of the tests run by this class and its superclasses.
  # DO NOT OVERRIDE THIS UNLESS YOU KNOW WHAT YOU ARE DOING!
  sub list_tests {
      ...

Maybe that comment could be more helpful...  8-/

> It works as expected. What do you think? Am I missing something?

Well it could be that you are missing something.  If you try to avoid
using side effects from each test, then the order should not matter.

I suppose one day the tests may be run in parallel.  That would be
neat, and it would also be incompatible with explicit test ordering.


Matthew  #8-)

Re: [Perlunit-users] Using @TESTS for changing tests execution order

[Dmitry D. <dd...@ic...>]

Matthew Astley wrote:
> On Sat, Oct 04, 2003 at 10:06:19AM +0400, Dmitry Diskin wrote:
> 
> 
>>I read from TestCase.pm POD that one can populate @TESTS array in
>>order to specify the desired order of tests execution:
> 
> 
> Ah, I started at the bottom of my inbox and worked back...  why does
> it matter what order the tests run?  The xUnit idea is that each test
> (and its associated setup/teardown) should stand alone.

Suppose that one test is for database INSERT statement, and the other -- 
for SELECT, and that last -- for DELETE. That way I need to maintain the 
order of tests execution. Of course, I can put all operations into one 
test instead.


> 
> 
>>### quote begin
>>
>>If you need to specify the test order, you can do one of the
>>following:
>>
>>=over 4
>>
>>=item * Set @TESTS
>>
>>  our @TESTS = qw(my_test my_test_2);
>>
>>This is the simplest, and recommended way.
>>
>>### quote end
> 
> 
>>But the above does not work for me. I can include only desired tests
>>using @TESTS, but they are listed and executed in a random order,
> 
> 
> Hmm, that comment seems to be out of date.  @TESTS is piled into a
> hash, then the keys are listed as you said.
> 

Aha.. That is the clue! Thanks. I should study sources myself :)

>>I decided to override list_tests method like this:
>>
>>sub list_tests {
>>    my $self = shift;
>>    return sort $self->SUPER::list_tests(@_);
>>}
> 
> 
> That would do it, yes.  Did you see in Test/Unit/TestCase.pm,
> 
>   # Returns a list of the tests run by this class and its superclasses.
>   # DO NOT OVERRIDE THIS UNLESS YOU KNOW WHAT YOU ARE DOING!
>   sub list_tests {
>       ...
> 
> Maybe that comment could be more helpful...  8-/
> 
> 
>>It works as expected. What do you think? Am I missing something?
> 
> 
> Well it could be that you are missing something.  If you try to avoid
> using side effects from each test, then the order should not matter.
> 
> I suppose one day the tests may be run in parallel.  That would be
> neat, and it would also be incompatible with explicit test ordering.
> 

Hmm.. Never thought about parallel execution.. Indeed it may be a godd idea.

-- 
Dmitry.

Re: [Perlunit-users] Using @TESTS for changing tests execution order

[Phlip <pl...@sy...>]

> Suppose that one test is for database INSERT statement, and the other --
> for SELECT, and that last -- for DELETE. That way I need to maintain the
> order of tests execution. Of course, I can put all operations into one
> test instead.

Uh, one common trick here is to put into setUp() a BEGIN TRANSACTION, then
put into tearDown a ROLLBACK.

But it sounds like you should randomize the order of your tests. Unless you
are testing quite crufty legacy code, where you must rely on a certain
amount of setUp before testing everything, your tests should all be totally
severable.

When you achieve that, come back to me and tell me how you did it. ;-)

--
Phlip
http://www.c2.com/cgi/wiki?TestFirstUserInterfaces

Re: [Perlunit-users] Using @TESTS for changing tests execution order

[Dmitry D. <dd...@ic...>]

Phlip wrote:

>>Suppose that one test is for database INSERT statement, and the other --
>>for SELECT, and that last -- for DELETE. That way I need to maintain the
>>order of tests execution. Of course, I can put all operations into one
>>test instead.
> 
> 
> Uh, one common trick here is to put into setUp() a BEGIN TRANSACTION, then
> put into tearDown a ROLLBACK.
> 
> But it sounds like you should randomize the order of your tests. Unless you
> are testing quite crufty legacy code, where you must rely on a certain
> amount of setUp before testing everything, your tests should all be totally
> severable.
> 
> When you achieve that, come back to me and tell me how you did it. ;-)
> 
> --
> Phlip
> http://www.c2.com/cgi/wiki?TestFirstUserInterfaces
> 

Can you suggest a structure of tests for testing insert, select and 
delete operations? I suppose that I can write something like tis:

sub test_001_insert {
... insert record and do some tets
}

sub test_002_select {
... select record inserted in previous test and do some tets
}

sub test_003_delete {
... delete the above  record and do some tets
}

-- 
Dmitry.

Re: [Perlunit-users] Using @TESTS for changing tests execution order

[Phlip <pl...@sy...>]

> Can you suggest a structure of tests for testing insert, select and 
> delete operations? I suppose that I can write something like tis:
> 
> sub test_001_insert {
> ... insert record and do some tets
> }
> 
> sub test_002_select {
    test_001_insert();
> ... select record inserted in previous test and do some tets
> }
> 
> sub test_003_delete {
    test_002_select();
> ... delete the above  record and do some tets
> }

;-)

-- 
  Phlip

Re: [Perlunit-users] Using @TESTS for changing tests execution order

[Dmitry D. <dd...@ic...>]

Phlip wrote:

>>Can you suggest a structure of tests for testing insert, select and 
>>delete operations? I suppose that I can write something like tis:
>>
>>sub test_001_insert {
>>... insert record and do some tets
>>}
>>
>>sub test_002_select {
> 
>     test_001_insert();
> 
>>... select record inserted in previous test and do some tets
>>}
>>
>>sub test_003_delete {
> 
>     test_002_select();
> 
>>... delete the above  record and do some tets
>>}
> 
> 
> ;-)
> 

Neat :) As far as I can see, the only drawback of this approach is the 
waste of time?

-- 
Dmitry.

Re: [Perlunit-users] Using @TESTS for changing tests execution order

[Phlip <pl...@sy...>]

> >>sub test_003_delete {
> >
> >     test_002_select();
> >
> >>... delete the above  record and do some tets
> >>}
> >
> >
> > ;-)
> >
>
> Neat :) As far as I can see, the only drawback of this approach is the
> waste of time?

As one develops, one hits the test button after changing 1 to 10 lines.

If that gets slow, one spends a few minutes figuring out why and speeding it
up.

The method of last resort is to split the current folder, so its local tests
stay fast. Run the global tests before and after integrating.

If the database can't run three transactions per test run, replace it?

--
  Phlip

Re: [Perlunit-users] Using @TESTS for changing tests execution order

[Bret P. <br...@pe...>]

At 10:27 PM 10/14/2003, Dmitry Diskin wrote:
>Neat :) As far as I can see, the only drawback of this approach is the 
>waste of time?

I'd like to second this approach. If you think it is a waste of time, just 
remove the first two tests. If you want to keep them, then they must be 
worth the time. I can see reasons either way.

Bret


_____________________________________
  Bret Pettichord, Software Tester
    Book - www.testinglessons.com
    Consulting - www.pettichord.com
    Blog - www.io.com/~wazmo/blog
    Hotlist - www.testinghotlist.com