#50 Preprocessing option from command-line

[Inaki GE]

Hi,
I wonder if there is any option to invoke the preprocessing option with the copybook path directly from the command line.
When I try to execute the option in the application, it freezes. The problem could be that the number of file in my Copybook path is huge (more than 40,000 files). Maybe from the command line, this issue would be solved.
Thank you very much for your support.
Regards.

[KrisDS]

Could you elaborate a bit more on what is going wrong ? What are the exact steps you're doing ? At what point in the process does it freeze.

Please also mention the platform you're on. I have noticed that on Mac OS X Yosemite, for instance, the Java File dialog hangs sometimes for unknown reasons...

[Inaki GE]

Hi,
The steps I follow are the following:
1) Execute Koopa.
2) Select the copybooks path.
3) Enable the preprocessing (alpha) option.
4) Parse the .cbl.

At the last point I get no response from the application.
Anyway with the preprocessing option disabled, sometimes when I try to parse a huge file (hundreds of thousand lines) directly in the application I also get no response. But if I execute Koopa from the command line "java -cp koopa.jar..." (extending the memory) there is no problem in parsing the file.

That´s why I want to know how to invoke to Koopa from the command line enabling the preprocessing option and defining a copybook path. My final goal is to get the Copy values in the xml generated.

If you need further information or some example, don't hesitate to contact me.

Thanks.
Regards.

PD: My platform is Windows 7.

[KrisDS]

When you say no response: is it hanging ? Or just slow ? You could try tweaking the log4j.properties file to see it is doing anything at all. Or maybe attach a VisualVM of jconsole to the process.

Anyway, as the preprocessing is still very experimental there is not much support for it yet. I exposed it a bit in the GUI so people could more easily play with it, but that's as far as it goes for now.

Having said that, if you're comfortable with Java I would suggest exploring the Cobol85PreprocessingTest class a bit. It will show you the few calls it takes to set up the preprocessing options. And then you can enable it and pass your copybook paths programmatically. Not as nice as a command line option, but this could get you unstuck in the shortest time.

[Inaki GE]

Hi again,
I have renamed the copy files as .cpy and Koopa, at least, parses the file.
The thing is that copy values are not displayed in the tree and are omitted by Koopa. If you want to check it, I attach the .cbl and the .cpy files.
Thanks.
Regards

[KrisDS]

Hey,

I tried your example, and it works fine when I process it in the GUI. Here is an excerpt of the XML output:

        <execStatement>
          <execSQLStatement>
            <!-- =======================                                           -->
            <t><![CDATA[EXEC]]></t>
            <t><![CDATA[SQL]]></t>
            <sqlStatement>
              <includeStatement>
                <t><![CDATA[INCLUDE]]></t>
                <textName>
                  <identifier>
                    <t><![CDATA[SQLCA]]></t>
                  </identifier>
                </textName>
              </includeStatement>
            </sqlStatement>
            <t><![CDATA[END-EXEC]]></t>
          </execSQLStatement>
        </execStatement>
        <t><![CDATA[.]]></t>
        <recordDescriptionEntry>
          <dataDescriptionEntry>
            <dataDescriptionEntry_format1>
              <levelNumber>
                <!--   COPY DE LA RUTINA DE ERROR-DB2-BATCH                            -->
                <t><![CDATA[01]]></t>
              </levelNumber>
              <dataDescName>
                <dataName>
                  <cobolWord>
                    <t><![CDATA[RRUT1]]></t>
                  </cobolWord>
                </dataName>
              </dataDescName>
              <t><![CDATA[.]]></t>
            </dataDescriptionEntry_format1>
          </dataDescriptionEntry>
        </recordDescriptionEntry>

The execStatement comes from the main CBL file. The dataDescriptionEntry comes from the CPY file.

Are you not seeing the same ?

[Inaki GE]

Hi again,

I was reviewing only the tree structure in the GUI application and as no xpath value was displayed for the COPY sentence, I thought that Koopa did not perform it well. I have checked the output in .xml and it seems to be OK.

Referring to the command line option, are you planning to add in the short term the preprocessing option and copybook path parameters when invoking Koopa from the command line?

Thank for your support.
Thanks.

[KrisDS]

The GUI always shows the original source file. I haven't figured out how to nicely show the expanded version.

Anyway, I added the preprocessing CLI options in [r199]. Here's the usage now:

Usage: ToXml [--free-format] 
             [--preprocess -I <copyboopath>] 
             <source-path> <target-path>

Multiple "-I" are allowed if you have more than one lookup path.

Please let me know if this resolves your problem.

Cheers,

Kris

[Inaki GE]

Hi,

I have tested the preprocessing option from the command line for different programs and it works nice. The generated XML seems to be OK and if the COPY file invoked does not exist in the copybook path a null pointer exception is displayed in the console.

I expect to gather all the copy files and execute the Koopa parser with a significant number of .cbl files. If I detect any error, I will let you know.

Thank you for your support.
Regards.

[KrisDS]

Could you share the stacktrace for the NullPointerException ? I'll try to make that a bit friendlier. Thanks.

[Inaki GE]

Hi,
Here you have the stacktrace that is displayed in the console when a COPY file is not found. After the issue Koopa no longer parses the rest of the files contained in the same directory.
Thanks.

Exception in thread "main" java.lang.NullPointerException
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileReader.<init>(Unknown Source)
at koopa.cobol.parser.cobol.preprocessing.PreprocessingTokenizer.nxt1(Pr
eprocessingTokenizer.java:143)
at koopa.cobol.parser.cobol.preprocessing.PreprocessingTokenizer.nxt1(Pr
eprocessingTokenizer.java:27)
at koopa.core.sources.BasicSource.next(BasicSource.java:26)
at koopa.core.parsers.BasicParseStream.forward(BasicParseStream.java:58)
at koopa.core.grammars.KoopaGrammar.skipSeparators(KoopaGrammar.java:657)
at koopa.core.grammars.KoopaGrammar$13.accepts(KoopaGrammar.java:437)
at koopa.core.grammars.KoopaGrammar$9.accepts(KoopaGrammar.java:326)
at koopa.core.grammars.KoopaGrammar$10.accepts(KoopaGrammar.java:345)
at koopa.core.grammars.KoopaGrammar$9.accepts(KoopaGrammar.java:326)
at koopa.core.grammars.KoopaGrammar$1.accepts(KoopaGrammar.java:127)
at koopa.core.grammars.KoopaGrammar$10.accepts(KoopaGrammar.java:345)
at koopa.core.grammars.KoopaGrammar$1.accepts(KoopaGrammar.java:127)
at koopa.core.grammars.KoopaGrammar$10.accepts(KoopaGrammar.java:345)
at koopa.core.grammars.KoopaGrammar$1.accepts(KoopaGrammar.java:127)
at koopa.core.grammars.KoopaGrammar$10.accepts(KoopaGrammar.java:345)
at koopa.core.grammars.KoopaGrammar$7.accepts(KoopaGrammar.java:268)
at koopa.core.grammars.KoopaGrammar$9.accepts(KoopaGrammar.java:326)
at koopa.core.grammars.KoopaGrammar$1.accepts(KoopaGrammar.java:127)
at koopa.core.grammars.KoopaGrammar$10.accepts(KoopaGrammar.java:345)
at koopa.core.grammars.KoopaGrammar$8.accepts(KoopaGrammar.java:292)
at koopa.core.grammars.KoopaGrammar$1.accepts(KoopaGrammar.java:127)
at koopa.core.grammars.KoopaGrammar$12.accepts(KoopaGrammar.java:410)
at koopa.core.grammars.KoopaGrammar$9.accepts(KoopaGrammar.java:326)
at koopa.core.grammars.KoopaGrammar$10.accepts(KoopaGrammar.java:345)
at koopa.core.grammars.KoopaGrammar$1.accepts(KoopaGrammar.java:127)
at koopa.core.grammars.KoopaGrammar$12.accepts(KoopaGrammar.java:410)
at koopa.core.grammars.KoopaGrammar$9.accepts(KoopaGrammar.java:326)
at koopa.core.grammars.KoopaGrammar$10.accepts(KoopaGrammar.java:345)
at koopa.core.grammars.KoopaGrammar$9.accepts(KoopaGrammar.java:326)
at koopa.core.grammars.KoopaGrammar$1.accepts(KoopaGrammar.java:127)
at koopa.core.grammars.KoopaGrammar$10.accepts(KoopaGrammar.java:345)
at koopa.core.grammars.KoopaGrammar$7.accepts(KoopaGrammar.java:268)
at koopa.core.grammars.KoopaGrammar$1.accepts(KoopaGrammar.java:127)
at koopa.core.parsers.Parser.accepts(Parser.java:25)
at koopa.cobol.parser.cobol.CobolParser.parse(CobolParser.java:136)
at koopa.cobol.parser.cobol.CobolParser.parse(CobolParser.java:67)
at koopa.cobol.parser.cobol.ParsingCoordinator.parse(ParsingCoordinator.
java:72)
at koopa.app.cli.ToXml.toXml(ToXml.java:134)
at koopa.app.cli.ToXml.process(ToXml.java:101)
at koopa.app.cli.ToXml.process(ToXml.java:109)
at koopa.app.cli.ToXml.main(ToXml.java:84)

[KrisDS]

Thanks. That should be fixed in [r206] now.

[Inaki GE]

It works nice now and the output generated if a copy file is not found is friendlier.
Thanks.