#1530 [java] Parser exception on Java code

[Juan Sotuyo]

I'm running PMD over PMD-Core's sourcecode, and I'm coming accross a parse exception on DAAPathFinder

pmd-core/src/main/java/net/sourceforge/pmd/lang/dfa/pathfinder/DAAPathFinder.java - Error while parsing src/main/java/net/sourceforge/pmd/lang/dfa/pathfinder/DAAPathFinder.java

The offending line is line 336:

((PathElement) stack.getLastLeaf().getUserObject()).currentChild++;

The cause, is that "++" is unexpected.

I've been debugging the JavaParser class, and the problem seems to arise from it not being able to detect the precense of the post-increment.

The syntax file seems about right:

void StatementExpression() :
{}
{
  PreIncrementExpression()
|
  PreDecrementExpression()
|
  LOOKAHEAD( PrimaryExpression() ("++" | "--") ) PostfixExpression()
|
  PrimaryExpression()
  [
  AssignmentOperator() Expression()
  ]
}

void PostfixExpression() #PostfixExpression((jjtn000.getImage() != null)):
{}
{
  PrimaryExpression() [ "++" {jjtThis.setImage("++");} | "--" {jjtThis.setImage("--");} ]
}

However, on the generated code:

    try {
      switch (jj_nt.kind) {
      case INCR:
        PreIncrementExpression();
        break;
      case DECR:
        PreDecrementExpression();
        break;
      default:
        jj_la1[121] = jj_gen;
        if (jj_2_73(2147483647)) {
          PostfixExpression();
        } else {
          ...
        }

jj_2_73(2147483647) always evaluates to false. Something seems wrong in the lookahead, but I can't figure it out, and my Antlr2 knowledge is somewhat limited.

I'm still tinkering with it, but at this point, it's probably better off in someone else's hands.

[Juan Sotuyo]

I've been looking around javacc's OS grammar repository, and all 4 Java grammas define StatementExpression with post operators inline, not using PostfixExpression.

In all cases (for instance this one), the following comment can be found:

/*

 * The last expansion of this production accepts more than the legal
 * Java expansions for StatementExpression.  This expansion does not
 * use PostfixExpression for performance reasons.
 */

Defining it as follows seems to work as expected, but doesn't feel right:

 void StatementExpression() :
{}
{
  PreIncrementExpression()
|
  PreDecrementExpression()
|
  PrimaryExpression()
  [
    "++" {jjtThis.setImage("++");}
  |
    "--" {jjtThis.setImage("--");}
  |
    AssignmentOperator() Expression()
  ]
}

I'm still not sure what the problem is with the LOOKAHEAD though...

[Juan Sotuyo]

This change to StatementExpression does not only feel wrong, but makes some tests and rules fail, since nodes will be different....

It seems we really need to get the lookahead to work somehow.

[Juan Sotuyo]

After lots more of tinkering, the offending changeset seems to be this one:

https://github.com/pmd/pmd/commit/0d20fe7e1c66568b4aa1a5c40b48621ac5a911bd

Reverting that single rule from the grammar, the code is parsed properly for DAAPathFinder but obviously the issue #1484 is regressed.

[Juan Sotuyo]

I've no idea why, but if we edit UnaryExpressionNotPlusMinus to rewrite:

| LOOKAHEAD("(" <IDENTIFIER> ")" "+") PostfixExpression()

as a semantic lookahead, insted of a syntactic one (yeah, I've been reading a lot on JavaCC):

| LOOKAHEAD( { getToken(1).kind == LPAREN && getToken(2).kind == IDENTIFIER && getToken(3).kind == RPAREN && getToken(4).kind == PLUS } ) PostfixExpression()

It will work on all test scenarios, including #1484 and this one. I've submited a PR, although I'd appreaciate if someone was able to explain the root of the issue...

Sorry for the super spammy thread, thanks for bearing with me :)

[Andreas Dangel]

Hi - many thanks for your effort!!! I'll integrate your PR this weekend. Thanks again!

[Andreas Dangel]

This bug will be fixed with PMD 5.4.3, 5.5.2 and later.

Commit: https://github.com/pmd/pmd/commit/a10f98e0c4093d5b8c4fd1600447a75b2b87d19f and cherry-picked to https://github.com/pmd/commit/283dc00a43741cd4b1e48324f63b22b12c30dcd4