#1119 JFreechart unit test fails unexpectedly (dependence between unit tests)

1.0.x
open
nobody
None
5
2013-11-21
2013-09-12
Sai
No

Hi there,

I found several JFreechart's (the latest release 1.0.15) unit tests fail unexpected when executed in a different, non-default order. Here is an example:

The test:

org.jfree.chart.axis.junit.SegmentedTimelineTests.testFifteenMinIncludedAndExcludedSegments

passes when executed in its default order. (e.g., executing all tests in SegmentedTimelineTests together)

However, when I re-order the test execution, I found this test fails suspiciously when executed after: org.jfree.chart.axis.junit.SegmentedTimelineTests2.test6

In other words, if you just run:
org.jfree.chart.axis.junit.SegmentedTimelineTests2.test6 and
org.jfree.chart.axis.junit.SegmentedTimelineTests.testFifteenMinIncludedAndExcludedSegments

The later test fails (which I think should pass).

Ideally, each unit test's execution should never affect other tests' results. This is so important to make a unit test's behavior consistently.

Dear JFreechart developers, can you please confirm or refute the above finding, to check whether this behavior is intended or not? or does it reveal a bug in JFreechart, or simply it reveals some smell in the test code?

(I have a couple more tests, if you are interested, I am happy to post them here).

Thanks a lot

-Sai

Discussion

  • Sai
    Sai
    2013-09-12

    Here is the stack trace when the testFifteenMinIncludedAndExcludedSegments test fails:

    testFifteenMinIncludedAndExcludedSegments(org.jfree.chart.axis.junit.SegmentedTimelineTests)junit.framework.AssertionFailedError: null
    at org.jfree.chart.axis.junit.SegmentedTimelineTests.verifyIncludedAndExcludedSegments(SegmentedTimelineTests.java:609)
    at org.jfree.chart.axis.junit.SegmentedTimelineTests.testFifteenMinIncludedAndExcludedSegments(SegmentedTimelineTests.java:587)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

     
  • David Gilbert
    David Gilbert
    2013-11-21

    I agree that the unit tests should be self-contained and the order of running should not impact the results. I wasn't able to reproduce this issue yet, and I'm not going to be able to devote much more time to it unfortunately as I have a lot of other things on my TODO list.