[Junit-trackers] [ junit-Feature Requests-1748683 ] Improve support for grouping/partitioning tests into suites

[SourceForge.net <no...@so...>]

Feature Requests item #1748683, was opened at 2007-07-05 18:55
Message generated for change (Comment added) made by steininger
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=365278&aid=1748683&group_id=15278

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Christian Nelson (xianpants)
Assigned to: Nobody/Anonymous (nobody)
Summary: Improve support for grouping/partitioning tests into suites

Initial Comment:
Often, there are a variety of types of tests on a project, from true unit tests to integration tests which span the spectrum from "almost a unit test" to "end-to-end integration test".  There's value in all of these types of tests and there are many extensions to junit for supporting them (dbunit, spring's test case hierarchy, web tests, etc).  Despite the name, junit is really a harness for running all types of tests (whether or not that was the original intention).

We've run into the situation where we need to partition our tests into multiple suites.  On my current project we have a set of tests which need to be run from a specific machine due to the fact that they reach out to external systems that only accept connections from preset ip addresses.  These tests are integration tests which we run from time to time, not with every build.  It's easy to imagine other examples of suites: unit tests which run uber-fast and integration tests which run more slowly.

I find junit's suite support to be pretty poor and there are a number of real problems with it, including:

* @SuiteClasses(TestClass1.class, ...) is cumbersome for large sets of classes.
* @SuiteClasses(TestClass1.class, ...) is error-prone because it's easy to forget to include a new test case in the suite.
* @RunWith(Suite.class) leads to some tests being run multiple times when you "run all tests" from ant/maven/an ide.  We want tests to run as fast as possible; running some of them twice (or more) doesn't help.

I think all of these problems directly stem from the composite pattern applied to test organization.  The fact that something (a class annotated with @RunWith(Suite.class)) outside of the tests does the grouping doesn't feel natural to me.  I really think of it as every test class (or optionally test method) may optionally be part of one or more named groups, and the assignment happens in the test itself, not some external suite class.  Here's how I'd do it:

1. Test runners (ant, maven, idea, eclipse, etc) could accept an optional parameter which specifies which group(s) are to be run.  If none are specified, then all tests are run.

2. These changes to existing annotations could accommodate this model:

Applied to either test methods or test classes... when applied to a test class, it propagates to all of the test methods in the class.
@Test(group = "online tests")
@Test(groups = {"integration tests", "online tests" })

[Yes, the only difference between the two is the plurality and helper so that an array of one isn't necessary.]

3. Grouping would have to become a first class concept in all of the test runners (or the default/topmost at the very least).

Note that TestNG has support for the kind of grouping I'm describing (see references).

I'm a loyal user of JUnit and prefer it to TestNG because of its simplicity and better (build-in) tool support (everyone ships with junit support).  However, this is one feature that is just poorly implemented in JUnit and I think needs some attention so that is becomes both easier to use and more useful.  I think it can be implemented in such a way that the old functionality could remain for those who use it, users wouldn't /have/ to use the new group features, and users who wanted it could use it.

What do you think?

Christian

References:
http://tech.groups.yahoo.com/group/junit/message/19521
http://testng.org/doc/documentation-main.html#test-groups

----------------------------------------------------------------------

Comment By: Thomas Steininger (steininger)
Date: 2008-04-08 14:50

Message:
Logged In: YES 
user_id=1323884
Originator: NO

i also vote for this feature-request. is there any argument against it?

----------------------------------------------------------------------

Comment By: Alex (toalexsmail)
Date: 2008-03-12 21:48

Message:
Logged In: YES 
user_id=2034775
Originator: NO

I have some related request. See
http://tech.groups.yahoo.com/group/junit/message/20357 for discussion. 

AFAIK this feature is replacement to TestSuite in JUnit 3.8. But it is
less flexible then TestSuite. In JUnit 3.8 you have an option to add
some methods from one class and some methods from the another class.
See also related topic "Filter improvement". I think this
functionality should be available also here, perhaps, with futher
improvement, such as use of patterns.
At least one should be able to run couples of methods from the given
class. When I have some related tests I preferred to run them
altogether and to look what is fails. It helps me both to save the
time and to identify the problem. I feel lack of this in JUnit 4.

----------------------------------------------------------------------

Comment By: David Saff (dsaff)
Date: 2007-07-09 15:50

Message:
Logged In: YES 
user_id=325156
Originator: NO

Let's further discuss this on ju...@ya..., and post a summary
here.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=365278&aid=1748683&group_id=15278