#1119 JFreechart unit test fails unexpectedly (dependence between unit tests)

[Sai]

Hi there,

I found several JFreechart's （the latest release 1.0.15) unit tests fail unexpected when executed in a different, non-default order. Here is an example:

The test:

org.jfree.chart.axis.junit.SegmentedTimelineTests.testFifteenMinIncludedAndExcludedSegments

passes when executed in its default order. (e.g., executing all tests in SegmentedTimelineTests together)

However, when I re-order the test execution, I found this test fails suspiciously when executed after: org.jfree.chart.axis.junit.SegmentedTimelineTests2.test6

In other words, if you just run:
org.jfree.chart.axis.junit.SegmentedTimelineTests2.test6 and
org.jfree.chart.axis.junit.SegmentedTimelineTests.testFifteenMinIncludedAndExcludedSegments

The later test fails (which I think should pass).

Ideally, each unit test's execution should never affect other tests' results. This is so important to make a unit test's behavior consistently.

Dear JFreechart developers, can you please confirm or refute the above finding, to check whether this behavior is intended or not? or does it reveal a bug in JFreechart, or simply it reveals some smell in the test code?

(I have a couple more tests, if you are interested, I am happy to post them here).

Thanks a lot

-Sai

[Sai]

Here is the stack trace when the testFifteenMinIncludedAndExcludedSegments test fails:

testFifteenMinIncludedAndExcludedSegments(org.jfree.chart.axis.junit.SegmentedTimelineTests)junit.framework.AssertionFailedError: null
at org.jfree.chart.axis.junit.SegmentedTimelineTests.verifyIncludedAndExcludedSegments(SegmentedTimelineTests.java:609)
at org.jfree.chart.axis.junit.SegmentedTimelineTests.testFifteenMinIncludedAndExcludedSegments(SegmentedTimelineTests.java:587)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

[David Gilbert]

I agree that the unit tests should be self-contained and the order of running should not impact the results. I wasn't able to reproduce this issue yet, and I'm not going to be able to devote much more time to it unfortunately as I have a lot of other things on my TODO list.

[Jan Klass]

From JFreeChart 1.0.19 source archive, we experience failing tests in some environments.

We use Ubuntu development VMs and have Jenkins buildserver master and slaves as Ubuntu systems as well.
Through mvn test, the unit tests seem to always work on all development systems, as well as the buildserver master. However, on the buildserver slave it consistently fails.
Master and slave are barely different.

Executing only the SegmentedTimelineTest tests (of which 3 failed) results in no failures.

Comparing two complete test runs of slave01, the order seems very, but not entirely stable.

The development system has 4 CPU cores, the master 2, and the slave 6.

I suspect the 6 cores (with their performance) result in a specific order of execution, that reproducably results in test failure.

[Jan Klass]

[Jan Klass]

About the last log; it contains a line

Concurrency config is parallel='all', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false

To clarify: I did the initial tests with default configuration. Playing around with the surefire settings to limit thread usage etc resulted in this additional line.
It still fails regardless, in both situations.

[Jan Klass]

Using the surefire runOrder option for alphabetical and reverse alphabetial execution of tests, I was able to identify the culprit.

Executing org.jfree.chart.axis.SegmentedTimelineAdditionalTest before org.jfree.chart.axis.SegmentedTimelineTest will result in the latter failing. Executing them in reverse order does not fail.

mvn test -Dtest=org.jfree.chart.axis.SegmentedTimelineAdditionalTest,org.jfree.chart.axis.SegmentedTimelineTest

This was verifyable with the following subset of tests; removing the SegmentedTimelineAdditionalTest made SegmentedTimelineTest not fail.

mvn test -Dtest=org.jfree.chart.AreaChartTest,org.jfree.chart.axis.AxisTest,org.jfree.chart.axis.CategoryLabelPositionTest,org.jfree.chart.axis.CategoryTickTest,org.jfree.chart.axis.ColorBarTest,org.jfree.chart.axis.CyclicNumberAxisTest,org.jfree.chart.axis.DateTickMarkPositionTest,org.jfree.chart.axis.DateTickTest,org.jfree.chart.axis.ExtendedCategoryAxisTest,org.jfree.chart.axis.ModuloAxisTest,org.jfree.chart.axis.NumberAxis3DTest,org.jfree.chart.axis.PeriodAxisLabelInfoTest,org.jfree.chart.axis.QuarterDateFormatTest,org.jfree.chart.axis.SegmentedTimelineAdditionalTest,org.jfree.chart.axis.SegmentedTimelineTest

However, with the full set of tests, this did not work. It seems another test side effect is in place that neutralizes the issue. Which one still needs to be identified.

[Jan Klass]

I identified org.jfree.chart.axis.DateAxisTest as the neutralizer.

Conclusion:

Tests in org.jfree.chart.axis.SegmentedTimelineTest will fail if org.jfree.chart.axis.SegmentedTimelineAdditionalTest is executed before it but not org.jfree.chart.axis.DateAxisTest before that.

There has to be a side effect between these.

Failed tests:
  SegmentedTimelineTest.testFifteenMinIncludedAndExcludedSegments:573->verifyIncludedAndExcludedSegments:595 null
  SegmentedTimelineTest.testFifteenMinSegmentedTimeline:318 expected:<-2208956400000> but was:<-2208960000000>
  SegmentedTimelineTest.testMondayThroughFridaySegmentedTimeline:304 expected:<-2208988800000> but was:<-2208992400000>

With maven runOrder as alphabetical:

mvn test -Dtest=org.jfree.chart.axis.DateAxisTest,org.jfree.chart.axis.SegmentedTimelineAdditionalTest,org.jfree.chart.axis.SegmentedTimelineTest

[Jan Klass]

For minimal error reproduction - test-class specific and test-case specific:

mvn test -Dtest=org.jfree.chart.axis.SegmentedTimelineAdditionalTest,org.jfree.chart.axis.SegmentedTimelineTest

mvn test -Dtest=org.jfree.chart.axis.SegmentedTimelineAdditionalTest#test1,org.jfree.chart.axis.SegmentedTimelineTest#testFifteenMinIncludedAndExcludedSegments

Result / Error:

testFifteenMinIncludedAndExcludedSegments(org.jfree.chart.axis.SegmentedTimelineTest)  Time elapsed: 0.006 sec  <<< FAILURE!
java.lang.AssertionError: null
    at org.junit.Assert.fail(Assert.java:86)
    at org.junit.Assert.assertTrue(Assert.java:41)
    at org.junit.Assert.assertTrue(Assert.java:52)
    at org.jfree.chart.axis.SegmentedTimelineTest.verifyIncludedAndExcludedSegments(SegmentedTimelineTest.java:595)
    at org.jfree.chart.axis.SegmentedTimelineTest.testFifteenMinIncludedAndExcludedSegments(SegmentedTimelineTest.java:573)

For the other two:

mvn test -Dtest=org.jfree.chart.axis.SegmentedTimelineAdditionalTest#test1,org.jfree.chart.axis.SegmentedTimelineTest#testFifteenMinSegmentedTimeline

testFifteenMinSegmentedTimeline(org.jfree.chart.axis.SegmentedTimelineTest)  Time elapsed: 0.006 sec  <<< FAILURE!
java.lang.AssertionError: expected:<-2208956400000> but was:<-2208960000000>
    at org.junit.Assert.fail(Assert.java:88)
    at org.junit.Assert.failNotEquals(Assert.java:743)
    at org.junit.Assert.assertEquals(Assert.java:118)
    at org.junit.Assert.assertEquals(Assert.java:555)
    at org.junit.Assert.assertEquals(Assert.java:542)
    at org.jfree.chart.axis.SegmentedTimelineTest.testFifteenMinSegmentedTimeline(SegmentedTimelineTest.java:318)

mvn test -Dtest=org.jfree.chart.axis.SegmentedTimelineAdditionalTest#test1,org.jfree.chart.axis.SegmentedTimelineTest#testMondayThroughFridaySegmentedTimeline

testMondayThroughFridaySegmentedTimeline(org.jfree.chart.axis.SegmentedTimelineTest)  Time elapsed: 0.007 sec  <<< FAILURE!
java.lang.AssertionError: expected:<-2208988800000> but was:<-2208992400000>
    at org.junit.Assert.fail(Assert.java:88)
    at org.junit.Assert.failNotEquals(Assert.java:743)
    at org.junit.Assert.assertEquals(Assert.java:118)
    at org.junit.Assert.assertEquals(Assert.java:555)
    at org.junit.Assert.assertEquals(Assert.java:542)
    at org.jfree.chart.axis.SegmentedTimelineTest.testMondayThroughFridaySegmentedTimeline(SegmentedTimelineTest.java:304)

@David Gilbert: Now that I have located the issue; is this something you can tackle?

[Jan Klass]

When I remove

~~~~~~
SegmentedTimeline timeline = getTimeline();
long value = timeline.toTimelineValue(date);
long ms = timeline.toMillisecond(value);
Calendar cal2 = Calendar.getInstance(Locale.UK);
cal2.setTime(new Date(ms));
Date reverted = cal2.getTime();
assertTrue("test1", value == (900000 * 34)
&& date.getTime() == reverted.getTime());
~~~~~

from SegmentedTimelineAdditionalTest#test1, then SegmentedTimelineTest#testMondayThroughFridaySegmentedTimeline does not fails.
If from this everything except the first line is removed, it still fails.
Thus, I suspect SegmentedTimelineAdditionalTest.getTimeline to have a (VM-global) side-effect.

[David Gilbert]

Hi Jan,
Thanks for your investigation...I'm taking a look.
Best regards,
David

[David Gilbert]

I think the side effect comes from the initialisation of FIRST_MONDAY_AFTER_1900, a static field which depends on the default timezone in force at the time the class is loaded. I don't know the full intent of this code and why it was made a static field, so I don't know what would be a good fix. I'll spend a little more time on it, but this SegmentedTimeline code is a big time sink (I've removed it from JFreeChart 2).