Incomplete parse. Last successful match: programPrototype < sourceUnit < compilationGroup

[Simon Sobisch]

This error occurs for a self defined extension (via -Dkoopa.cobolFileExtensions=MAK). The file looks like

      *-----------------------------------------------------------------
      * *MACRO STARTSFILE
      * Function: Routine for SQL handling
      *-------------------------------------------------------------
       START-KEY-03-filename SECTION.
      *00.
           PERFORM sql-something
           ;START-SQL filename KEY 03
      *
           continue.
      *-----------------------------------------------------------------

The ;something is a macro for a non-standard preparser (the ; are placed in column 7 [!] to 60), and "of course" especially the macros are what we want to parse.

[Simon Sobisch]

Same happens with -Dkoopa.copybooks=CPY,MAK.

[KrisDS]

This gets accepted as a copybook if you comment out the line with the semicolon.

I'm wondering how I could help here. Given that you're talking about a non-standard preparser it should not impact the default Cobol chain. I could give you an option to include extra tokenizers at some custom point in the chain (like extension points) ? Then you could handle them there ?

If you also want to parse these statements you will have to extend the grammar as well. Probably a custom subclass of the Cobol grammar which tweaks the statement definition.

[Simon Sobisch]

Sounds good - despite the part that I have no idea how to do so ;-)

[KrisDS]

What is the overall goal ? Are you interested in grabbing only these lines ? Or do you really need to parse them and the Cobol code together ?

[Simon Sobisch]

Overal goal: documentation of the complete sources (parse the macros and COBOL code together) including: which files and screens are used where and how (95% of the files and all screens are "hidden" in the macros), what source files exists and where are they used (COPY SOMECOPY, **MACRO MACRONAME, CALL 'SOMEPROG').
If possible: cross reference for the variables in the used files (where are they read/updated [luckily they always have the same name because the data definition is included via a copybook]).
Marvelous if the business logic can be parsed (create a list/csv/whatever similar to a decission model notation), even if the rules are just the COBOL sources like IF someflag and PERFORM stuff.

[KrisDS]

All of that seems doable, except for the custom macros. You will really need to extend Koopa to handle that. Which will mean tweaking the tokenizer sequence to recognize the start of macros. And extend the grammar to parse them for you.

Now I know people have extended the grammar without much problem. But I don't think anyone has really tried tweaking the tokenizers to extend those. So that might be a bit painful still. I'll have to consider a bit how I can make that easier.

An alternative workaround could be to scan for the macros first, and turn them into comments before parsing. After the parse you could then scan the syntax tree for those comments, and parse the macros as you please.

[Simon Sobisch]

An alternative workaround could be to scan for the macros first, and turn them into comments before parsing. After the parse you could then scan the syntax tree for those comments, and parse the macros as you please.

Interesting approach and looks like a minor change. Can I do this with an own preparser? If yes: how to setup it (ideally with adding the copybook for parsing in WORKING-STORAGE SECTION that is normally included when resolving the specific Macro)?

[KrisDS]

Yes. If you choose to do this in Java you could just read the file line by line, and look for lines with a macro. Then replace those with a commented out version, or an equivalent COPY statement in the case of those specific macros. String it all together and feed that to Koopa. After which you can grab the syntax tree, and query it for the macros you commented out.

Check out examples/basic and examples/jaxen to get a rough starting point.

Like I said, I'd love to be able to offer you something better than this, but I need some time to figure that out and implement it. I have seen another codebase with custom macros, so maybe I should just tailor it to that specific use case.