#1119 JFreechart unit test fails unexpectedly (dependence between unit tests)

1.0.x
open
nobody
None
5
2014-09-08
2013-09-12
Sai
No

Hi there,

I found several JFreechart's (the latest release 1.0.15) unit tests fail unexpected when executed in a different, non-default order. Here is an example:

The test:

org.jfree.chart.axis.junit.SegmentedTimelineTests.testFifteenMinIncludedAndExcludedSegments

passes when executed in its default order. (e.g., executing all tests in SegmentedTimelineTests together)

However, when I re-order the test execution, I found this test fails suspiciously when executed after: org.jfree.chart.axis.junit.SegmentedTimelineTests2.test6

In other words, if you just run:
org.jfree.chart.axis.junit.SegmentedTimelineTests2.test6 and
org.jfree.chart.axis.junit.SegmentedTimelineTests.testFifteenMinIncludedAndExcludedSegments

The later test fails (which I think should pass).

Ideally, each unit test's execution should never affect other tests' results. This is so important to make a unit test's behavior consistently.

Dear JFreechart developers, can you please confirm or refute the above finding, to check whether this behavior is intended or not? or does it reveal a bug in JFreechart, or simply it reveals some smell in the test code?

(I have a couple more tests, if you are interested, I am happy to post them here).

Thanks a lot

-Sai

Discussion

1 2 > >> (Page 1 of 2)
  • Sai
    Sai
    2013-09-12

    Here is the stack trace when the testFifteenMinIncludedAndExcludedSegments test fails:

    testFifteenMinIncludedAndExcludedSegments(org.jfree.chart.axis.junit.SegmentedTimelineTests)junit.framework.AssertionFailedError: null
    at org.jfree.chart.axis.junit.SegmentedTimelineTests.verifyIncludedAndExcludedSegments(SegmentedTimelineTests.java:609)
    at org.jfree.chart.axis.junit.SegmentedTimelineTests.testFifteenMinIncludedAndExcludedSegments(SegmentedTimelineTests.java:587)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

     
  • David Gilbert
    David Gilbert
    2013-11-21

    I agree that the unit tests should be self-contained and the order of running should not impact the results. I wasn't able to reproduce this issue yet, and I'm not going to be able to devote much more time to it unfortunately as I have a lot of other things on my TODO list.

     
  • Jan Klass
    Jan Klass
    2014-09-02

    From JFreeChart 1.0.19 source archive, we experience failing tests in some environments.

    We use Ubuntu development VMs and have Jenkins buildserver master and slaves as Ubuntu systems as well.
    Through mvn test, the unit tests seem to always work on all development systems, as well as the buildserver master. However, on the buildserver slave it consistently fails.
    Master and slave are barely different.

    Executing only the SegmentedTimelineTest tests (of which 3 failed) results in no failures.

    Comparing two complete test runs of slave01, the order seems very, but not entirely stable.

    The development system has 4 CPU cores, the master 2, and the slave 6.

    I suspect the 6 cores (with their performance) result in a specific order of execution, that reproducably results in test failure.

     
  • Jan Klass
    Jan Klass
    2014-09-02

    Post awaiting moderation.
  • Jan Klass
    Jan Klass
    2014-09-02

    About the last log; it contains a line
    Concurrency config is parallel='all', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
    To clarify: I did the initial tests with default configuration. Playing around with the surefire settings to limit thread usage etc resulted in this additional line.
    It still fails regardless, in both situations.

     
    Last edit: Jan Klass 2014-09-02
  • Jan Klass
    Jan Klass
    2014-09-03

    Using the surefire runOrder option for alphabetical and reverse alphabetial execution of tests, I was able to identify the culprit.

    Executing org.jfree.chart.axis.SegmentedTimelineAdditionalTest before org.jfree.chart.axis.SegmentedTimelineTest will result in the latter failing. Executing them in reverse order does not fail.

    mvn test -Dtest=org.jfree.chart.axis.SegmentedTimelineAdditionalTest,org.jfree.chart.axis.SegmentedTimelineTest
    

    This was verifyable with the following subset of tests; removing the SegmentedTimelineAdditionalTest made SegmentedTimelineTest not fail.

    mvn test -Dtest=org.jfree.chart.AreaChartTest,org.jfree.chart.axis.AxisTest,org.jfree.chart.axis.CategoryLabelPositionTest,org.jfree.chart.axis.CategoryTickTest,org.jfree.chart.axis.ColorBarTest,org.jfree.chart.axis.CyclicNumberAxisTest,org.jfree.chart.axis.DateTickMarkPositionTest,org.jfree.chart.axis.DateTickTest,org.jfree.chart.axis.ExtendedCategoryAxisTest,org.jfree.chart.axis.ModuloAxisTest,org.jfree.chart.axis.NumberAxis3DTest,org.jfree.chart.axis.PeriodAxisLabelInfoTest,org.jfree.chart.axis.QuarterDateFormatTest,org.jfree.chart.axis.SegmentedTimelineAdditionalTest,org.jfree.chart.axis.SegmentedTimelineTest
    

    However, with the full set of tests, this did not work. It seems another test side effect is in place that neutralizes the issue. Which one still needs to be identified.

     
  • Jan Klass
    Jan Klass
    2014-09-03

    I identified org.jfree.chart.axis.DateAxisTest as the neutralizer.

    Conclusion:

    Tests in org.jfree.chart.axis.SegmentedTimelineTest will fail if org.jfree.chart.axis.SegmentedTimelineAdditionalTest is executed before it but not org.jfree.chart.axis.DateAxisTest before that.

    There has to be a side effect between these.

    Failed tests:
      SegmentedTimelineTest.testFifteenMinIncludedAndExcludedSegments:573->verifyIncludedAndExcludedSegments:595 null
      SegmentedTimelineTest.testFifteenMinSegmentedTimeline:318 expected:<-2208956400000> but was:<-2208960000000>
      SegmentedTimelineTest.testMondayThroughFridaySegmentedTimeline:304 expected:<-2208988800000> but was:<-2208992400000>
    

    With maven runOrder as alphabetical:

    mvn test -Dtest=org.jfree.chart.axis.DateAxisTest,org.jfree.chart.axis.SegmentedTimelineAdditionalTest,org.jfree.chart.axis.SegmentedTimelineTest
    
     
    Last edit: Jan Klass 2014-09-03
  • Jan Klass
    Jan Klass
    2014-09-03

    For minimal error reproduction - test-class specific and test-case specific:

    mvn test -Dtest=org.jfree.chart.axis.SegmentedTimelineAdditionalTest,org.jfree.chart.axis.SegmentedTimelineTest
    
    mvn test -Dtest=org.jfree.chart.axis.SegmentedTimelineAdditionalTest#test1,org.jfree.chart.axis.SegmentedTimelineTest#testFifteenMinIncludedAndExcludedSegments
    

    Result / Error:

    testFifteenMinIncludedAndExcludedSegments(org.jfree.chart.axis.SegmentedTimelineTest)  Time elapsed: 0.006 sec  <<< FAILURE!
    java.lang.AssertionError: null
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertTrue(Assert.java:52)
        at org.jfree.chart.axis.SegmentedTimelineTest.verifyIncludedAndExcludedSegments(SegmentedTimelineTest.java:595)
        at org.jfree.chart.axis.SegmentedTimelineTest.testFifteenMinIncludedAndExcludedSegments(SegmentedTimelineTest.java:573)
    

    For the other two:

    mvn test -Dtest=org.jfree.chart.axis.SegmentedTimelineAdditionalTest#test1,org.jfree.chart.axis.SegmentedTimelineTest#testFifteenMinSegmentedTimeline
    
    testFifteenMinSegmentedTimeline(org.jfree.chart.axis.SegmentedTimelineTest)  Time elapsed: 0.006 sec  <<< FAILURE!
    java.lang.AssertionError: expected:<-2208956400000> but was:<-2208960000000>
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:743)
        at org.junit.Assert.assertEquals(Assert.java:118)
        at org.junit.Assert.assertEquals(Assert.java:555)
        at org.junit.Assert.assertEquals(Assert.java:542)
        at org.jfree.chart.axis.SegmentedTimelineTest.testFifteenMinSegmentedTimeline(SegmentedTimelineTest.java:318)
    
    mvn test -Dtest=org.jfree.chart.axis.SegmentedTimelineAdditionalTest#test1,org.jfree.chart.axis.SegmentedTimelineTest#testMondayThroughFridaySegmentedTimeline
    
    testMondayThroughFridaySegmentedTimeline(org.jfree.chart.axis.SegmentedTimelineTest)  Time elapsed: 0.007 sec  <<< FAILURE!
    java.lang.AssertionError: expected:<-2208988800000> but was:<-2208992400000>
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:743)
        at org.junit.Assert.assertEquals(Assert.java:118)
        at org.junit.Assert.assertEquals(Assert.java:555)
        at org.junit.Assert.assertEquals(Assert.java:542)
        at org.jfree.chart.axis.SegmentedTimelineTest.testMondayThroughFridaySegmentedTimeline(SegmentedTimelineTest.java:304)
    

    @David Gilbert: Now that I have located the issue; is this something you can tackle?

     
    Last edit: Jan Klass 2014-09-03
    • Jan Klass
      Jan Klass
      2014-09-03

      When I remove

              SegmentedTimeline timeline = getTimeline();
              long value = timeline.toTimelineValue(date);
              long ms = timeline.toMillisecond(value);
              Calendar cal2 = Calendar.getInstance(Locale.UK);
              cal2.setTime(new Date(ms));
              Date reverted = cal2.getTime();
              assertTrue("test1", value == (900000 * 34)
                      && date.getTime() == reverted.getTime());
      

      from SegmentedTimelineAdditionalTest#test1, then SegmentedTimelineTest#testMondayThroughFridaySegmentedTimeline does not fails.
      If from this everything except the first line is removed, it still fails.
      Thus, I suspect SegmentedTimelineAdditionalTest.getTimeline to have a (VM-global) side-effect.

       
  • David Gilbert
    David Gilbert
    2014-09-08

    Hi Jan,
    Thanks for your investigation...I'm taking a look.
    Best regards,
    David

     
1 2 > >> (Page 1 of 2)