#1461 [core] RuleSetFactory: Possible threading issue due to PR#75

[Andreas Dangel]

See https://github.com/pmd/pmd/pull/75 for discussion.

[Ryan Gustafson]

I was able to trivially verify that multi-threading support was not working using the -t option. The RuleChain died fairly quickly with ConcurrentModificationExceptions.

It might be worth adding a Test around this, to try to catch regressions via CI. A forced multi-threaded test of the Java rules against the PMD source aught to 1) not fail, 2) produce consistent results if run 2+ times. Not exactly a robust test, but hopefully stands a chance of catching obvious issues before a release happens.

Alternatively, a series of high volume, high concurrency stress tests, could be done pre-release to try to actively shake out such problems. Running all known Rules against a baseline of source from different open source projects and comparing with previous release numbers/reports could be used to spot 1) threading issues, 2) performance problems, 3) false-positive Rule regressions. The -t and -b options are useful here. I had tried doing this once, but I never completed the effort. I'm not sure if something like this now exists.

I have noticed some performance problems while doing analysis on this bug, I'll try to submit some PR around them when I get a chance.

[Alexey Yudichev]

I am using maven-pmd-plugin 3.6 which defaults to PMD 5.3.5. When switching from PMD 5.3.5 to 5.4.1 I start to get a lot of ConcurrentModificationExceptions (first one below) during the build and eventually the build may hang.

Is there a workaround here? I need 5.4.1 as this is what the current version of PMD plugin for IntelliJ IDEA uses.

[08:37:59][Step 2/6] Mar 02, 2016 3:37:59 AM net.sourceforge.pmd.RuleSet apply
[08:37:59][Step 2/6] WARNING: Exception applying rule NPathComplexity on file /data/cmetrading/teamcity_home/agent6/work/d8e9b2b2b3ffbf5f/xasf-container/src/test/java/com/baml/gmt/xasf/container/agent/lifecycle/events/TestAgentLifecycleChangeEvent.java, continuing with next rule
[08:37:59][Step 2/6] java.util.ConcurrentModificationException
[08:37:59][Step 2/6]    at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
[08:37:59][Step 2/6]    at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.lang.rule.stat.StatisticalRuleHelper.applyMinimumValue(StatisticalRuleHelper.java:116)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.lang.rule.stat.StatisticalRuleHelper.apply(StatisticalRuleHelper.java:68)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.lang.java.rule.AbstractStatisticalJavaRule.apply(AbstractStatisticalJavaRule.java:29)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.lang.rule.AbstractDelegateRule.apply(AbstractDelegateRule.java:215)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.RuleSet.apply(RuleSet.java:307)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.RuleSets.apply(RuleSets.java:125)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.SourceCodeProcessor.processSource(SourceCodeProcessor.java:146)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.SourceCodeProcessor.processSourceCode(SourceCodeProcessor.java:76)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.SourceCodeProcessor.processSourceCode(SourceCodeProcessor.java:43)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.processor.PmdRunnable.call(PmdRunnable.java:79)
[08:37:59][Step 2/6]    at net.sourceforge.pmd.processor.PmdRunnable.call(PmdRunnable.java:25)
[08:37:59][Step 2/6]    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[08:37:59][Step 2/6]    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[08:37:59][Step 2/6]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[08:37:59][Step 2/6]    at java.lang.Thread.run(Thread.java:745)

build hangs on this:

"main" prio=10 tid=0x00007f3b4800e800 nid=0x2db1 waiting on condition [0x00007f3b4da42000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000000041e083fe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder.multiThreadedProjectTaskSegmentBuild(MultiThreadedBuilder.java:130)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder.build(MultiThreadedBuilder.java:91)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

    one of PmdThreads (there are several - all doing the same)

    "PmdThread 7" prio=10 tid=0x00007f3a84043800 nid=0x321b runnable [0x00007f3b2ae27000]
   java.lang.Thread.State: RUNNABLE
    at java.util.TreeMap.put(TreeMap.java:560)
    at java.util.TreeSet.add(TreeSet.java:255)
    at net.sourceforge.pmd.lang.rule.stat.StatisticalRuleHelper.addDataPoint(StatisticalRuleHelper.java:47)
    at net.sourceforge.pmd.lang.java.rule.AbstractStatisticalJavaRule.addDataPoint(AbstractStatisticalJavaRule.java:19)
    at net.sourceforge.pmd.lang.java.rule.codesize.AbstractNcssCountRule.visit(AbstractNcssCountRule.java:66)
    at net.sourceforge.pmd.lang.java.rule.AbstractJavaRule.visit(AbstractJavaRule.java:247)
    at net.sourceforge.pmd.lang.java.rule.codesize.NcssMethodCountRule.visit(NcssMethodCountRule.java:26)
    at net.sourceforge.pmd.lang.java.ast.ASTMethodDeclaration.jjtAccept(ASTMethodDeclaration.java:25)
    at net.sourceforge.pmd.lang.java.rule.codesize.AbstractNcssCountRule.visit(AbstractNcssCountRule.java:55)
    at net.sourceforge.pmd.lang.java.rule.AbstractJavaRule.visit(AbstractJavaRule.java:111)
    at net.sourceforge.pmd.lang.java.ast.ASTClassOrInterfaceBodyDeclaration.jjtAccept(ASTClassOrInterfaceBodyDeclaration.java:41)
    at net.sourceforge.pmd.lang.java.rule.codesize.AbstractNcssCountRule.visit(AbstractNcssCountRule.java:55)
    at net.sourceforge.pmd.lang.java.rule.AbstractJavaRule.visit(AbstractJavaRule.java:107)
    at net.sourceforge.pmd.lang.java.ast.ASTClassOrInterfaceBody.jjtAccept(ASTClassOrInterfaceBody.java:22)
    at net.sourceforge.pmd.lang.java.rule.codesize.AbstractNcssCountRule.visit(AbstractNcssCountRule.java:55)
    at net.sourceforge.pmd.lang.java.rule.AbstractJavaRule.visit(AbstractJavaRule.java:83)
    at net.sourceforge.pmd.lang.java.ast.ASTClassOrInterfaceDeclaration.jjtAccept(ASTClassOrInterfaceDeclaration.java:26)
    at net.sourceforge.pmd.lang.java.rule.codesize.AbstractNcssCountRule.visit(AbstractNcssCountRule.java:55)
    at net.sourceforge.pmd.lang.java.rule.AbstractJavaRule.visit(AbstractJavaRule.java:223)
    at net.sourceforge.pmd.lang.java.ast.ASTTypeDeclaration.jjtAccept(ASTTypeDeclaration.java:36)
    at net.sourceforge.pmd.lang.java.rule.codesize.AbstractNcssCountRule.visit(AbstractNcssCountRule.java:55)
    at net.sourceforge.pmd.lang.java.rule.AbstractJavaRule.visit(AbstractJavaRule.java:203)
    at net.sourceforge.pmd.lang.java.rule.AbstractJavaRule.visitAll(AbstractJavaRule.java:31)
    at net.sourceforge.pmd.lang.java.rule.AbstractJavaRule.apply(AbstractJavaRule.java:25)
    at net.sourceforge.pmd.lang.java.rule.AbstractStatisticalJavaRule.apply(AbstractStatisticalJavaRule.java:28)
    at net.sourceforge.pmd.lang.rule.AbstractDelegateRule.apply(AbstractDelegateRule.java:215)
    at net.sourceforge.pmd.RuleSet.apply(RuleSet.java:307)
    at net.sourceforge.pmd.RuleSets.apply(RuleSets.java:125)
    at net.sourceforge.pmd.SourceCodeProcessor.processSource(SourceCodeProcessor.java:146)
    at net.sourceforge.pmd.SourceCodeProcessor.processSourceCode(SourceCodeProcessor.java:76)
    at net.sourceforge.pmd.SourceCodeProcessor.processSourceCode(SourceCodeProcessor.java:43)
    at net.sourceforge.pmd.processor.PmdRunnable.call(PmdRunnable.java:79)
    at net.sourceforge.pmd.processor.PmdRunnable.call(PmdRunnable.java:25)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

[Andreas Dangel]

You're right, these exceptions are occurring probably due to this bug.

There seems no workaround here with maven-pmd-plugin, as the threads-option can't be changed (it defaults to Runtime.getRuntime().availableProcessors().

If you really need PMD 5.4.1 right now, the only option seems to be, to run it from commandline or via ant - the PMDTask for ant also supports the "threads" option. To workaround this threading issue, you'll need to run PMD with the option -threads 0.

[Andreas Dangel]

I've added a unit test (see 54ce3036) to hopefully prevent the problem in the future. I basically reverted the commit from PR #75.
Under documentation, I added one more paragraph -> see How to write a rule

The fix will be included in the next PMD versions (5.3.7, 5.4.2 and 5.5.0).

@rgustav - Please create a separate bug for the performance issues you've seen. Thanks!