[Ctags-devel] Code suggestions - optional command terminators

[David F. <fis...@ia...>]

I am hoping one of the other languages has a similar syntax where lines can
conditionally have a terminator.

I am having a very hard time correctly parsing this code and hope someone
can give me some suggestions.

Here is a typical code block for a function in javascript (assume this is
the only thing in test.js):
function checkForUpdate() {
    if( 1==1 ) {
        document.write("hello from checkForUpdate<br>");
    }
    return 1;
}

So you have an idea how the new javascript parser works I will show you what
functions are called:
We read "function" and call parseFunction.
parseFunction checks for a function name "checkForUpdate", skips over
arguments "()" and looks for a starting block "{".
It then creates the "checkForUpdate" function tag.
Then it calls parseBlock (between the curly braces).

parseBlock finds the "if", calls parseIf.
parseIf skips over the arguments "(1==1)" and finds the "{".
It then calls parseBlock to deal with the curly braces.

parseBlock finds the "document" and parses the line looking for a line
terminator.
A line terminator in javascript is ";".
parseBlock returns to parseIf.
parseIf returns to parseBlock.
parseBlock returns to parseFunction.

parseFunction finds "return" and processes the line until the terminator.
The "}" indicates the completion of parseFunction.


Hopefully, that is clear enough for everyone.

Here is where the problem creeps in.

This is also valid javascript (though not for C):
function checkForUpdate() {
    if( 1==1 ) {
        document.write("hello from checkForUpdate<br>")
    }
    return 1;
}

Notice the "document" line does not have an ending ";".
So when I am processing the "document line" I am expecting to read until a
semi-colon (or a curly brace "}").  So in fact I find the closing brace for
the IF block and now I have moved one token beyond where I need to be.  So
parsing of the rest of the file is off.

The document line could have been replaced with an inner function like this:
function checkForUpdate() {
    if( 1==1 ) {
        var f2 = function() {}
    }
    return 1;
}

So basically, when I process the "document" line I realize it is just a
statement and not one I specifically parse, so just skip ahead to it's
terminator.  These statements could get long and complex with string
concatentations and the likes.  Skipping ahead is about the only thing I can
think of, since I couldn't possible parse it.

Any suggestions on how to deal with optional command terminators?

If I could "unreadToken()", then I could probably detect this case.  But I
would need to be able to push the token back onto the stack.  I am not sure
if the ctags infrastructure provides this feature.  So far everything I ask
for something to help me in these cases, Elliott points me at something that
does exactly what I need.  Come on Elliot!  Give it to me again!

Thanks for any help.
Dave