#39 Improvement: handle compiler directives and source formatting directives in the tokenizer stages

[KrisDS]

Right now the grammar contains several grammar rules related to compiler directives and source formatting. Ideally these would already be handled in the tokenizer stages, not the parser.

Reasons for doing so:

• These directives can appear pretty much anywhere, and handling all that in the grammar could/would add a lot of complexity.

• Extending the functionality of the preprocessor is easier if we can intepret these directives already in the tokenizer stages.

Ideally, describing the syntax of these directives should be as easy as describing the syntax of Cobol; if it can use the same mechanism that would be even better.

[Luis R.]

Would it be possible to move the CICS and SQL parsers to a different stage too? There's a lot more to SQL than the stuff covered by the current rules, and not being COBOL, it really doesn't belong in the same grammar definition file.

(Perhaps we should address this in a separate issue?)

[KrisDS]

My idea for the embedded languages was to set up separate grammars/parsers for each of them (using whichever technology is most appropriate), and then link to those from within the Cobol grammar. Syntactically it could look something like this:

EXEC SQL (--SQLGrammar.sqlStatement--> END-EXEC) END-EXEC

So, skip up to the END-EXEC token, and then try parsing the skipped tokens as an sqlStatement defined by an SQLGrammar.

All of this is just a rough idea, and should probably become a separate issue. I'll create one for this now.

[KrisDS]

Done. See ticket [#40] for the CICS/SQL stuff.

[KrisDS]

Closing this. Part of this ticket is already covered (grammar composition, preprocessing directives in a base grammar). Other parts (actually handling compiler directives in the tokenizers) have(/can) become separate tickets.