#50 wierd results using group tests
I use 4 consecutive group tests, all of them created at
runtime (to which files are added with addTestFile) and
there seems to be two problems:
1) the current count for "test cases complete" is
wrong. For every test group that is run, it shows the
number of complete tests, then the number of total
tests in the group (supposably), separated with a
slash. But! The number of "total tests in the group"
remains the same for all tests -- it is equal to the
number of tests in the FIRST group, not CURRENT group.
So these numbers look like:
2/2 test cases complete: 5 passes, 0 fails and 0
exceptions.
8/2 test cases complete: 48 passes, 0 fails and 0
exceptions.
10/2 test cases complete: 115 passes, 1 fails and 0
exceptions.
11/2 test cases complete: 117 passes, 1 fails and 0
exceptions.
2) error doesn't get cleared between completely
separate groups. If an error occurs in a group, every
group that comes after it, even though completely
separated (not grouped in a higher group), reports a
failure. In my example (see above), I only have 1 (one)
error in 3rd group, but 4th group also ends up "red"
and reports a failure. This is, I believe, not the
correct behaviour.
screenshot of the uttermost critical situation
Logged In: YES
user_id=695819
Hi.
The miscounting is a known issue and I am deferring the fix
to a later version of SimpleTest. It works if you use
addTestCase() to add group tests, rather the the automatic
addTestFile(). If the problem really is urgent/vital I can
attempt to fix it, but I am hoping that an internal
restructuring will clear down this bug anyway.
The second issue I am tackling now. How come you have
multiple red and green bars on one screen shot? You should
get just one. Can you mail me the runner code you are using
(the top level test script). This is very strange behaviour.
That is, it has already gone terribly wrong long before the
problem you describe is an issue.
yours, Marcus
Logged In: YES
user_id=989152
Hi,
I think maybe it's not really a bug, it was (possibly) a
misuse of the group tests. The reason why I'm getting four
results is because I set up four consecutive groups and run
them separately. Each group takes a reporter as a parameter
and displays its results. The issue occurs if I use the same
instance of the HtmlReporter for all four group tests. It
appears that for each run, the reporter doesn't clear
previous results. This could just as well be wanted
behaviour. If I create a new instance of the reporter, it
seems to work fine.
I've attached the isolated "bug case". Inside the main file
there is a comment that shows how to switch between same/new
instance of the reporter.
Logged In: YES
user_id=695819
Hi.
Regarding the group test counts being wrong, Ithink this is
now fixed in CVS. It was caused by group tests being lazily
instantiated.
yours, Marcus