LLM usage to create COBOL code... still fails

[Ralph Linkletter]

I presented Claude with an assignment to create a sales order entry application. Not just a program but an application.
Claude dutifully found code fragments on the internet and stitched together those fragments to create a single program - 439 lines long. - SOECLAUDE.COB.

I compiled the program with GnuCOBOL - it produced 39 errors. If I were to fix those errors and keep resubmitting it for compile I would be at it for quite a while.

I compiled the same code with Micro Focus (MF).
MF recovers from syntax errors a wee bit better than GnuCOBOL,
The MF version produced:

COBCH0498S Procedure name INITIALIZE-PROGRAM undeclared, line 129 : C:\GEDIT\COBOL\SOECLAUDE.COB(129,19)
Compilation complete with 292 errors, 0 warnings, 0 notices and an exit code of 12

Build finished with 292 errors, 0 warnings, and 0 notices.
Completed in 1 seconds

Beyond the complexity of "Hello World" or a simple math problem Claude ain't ready for prime time.

Just a snippet of the code.

          SELECT INVENTORY-FILE ASSIGN TO 'INVFILE'
              ORGANIZATION IS INDEXED
              ACCESS MODE IS DYNAMIC
              RECORD KEY IS ITEM-CODE.

          SELECT INVOICE-FILE ASSIGN TO 'INVFILE'
              ORGANIZATION IS SEQUENTIAL
              ACCESS MODE IS SEQUENTIAL.

Can you see the confusion / naivete Claude is regurgitating as gospel fact.

Ralph

[Simon Sobisch]

I've seen similar "confusion", including ... fantasies about statements (you know, it looks so English, so sometimes LLMs just use sentences/phrases that sound nice).

Note: This is highly dependent on the LLM used, LLMs that were exposed to much more COBOL code or are especially trained for COBOL do indeed perform much better.
I'd expect IBMs to be much better, but also others like mAInframer berform better.
For a list of LLMs you may run locally, see https://huggingface.co/search/full-text?q=cobol&type=model .

Also: LLMs that may run local processes and know about cobc -fsyntax-only or similar (depending on the setup you can white-list commands like these to be executed without interaction) and can iterate over errors themselves can often handle compile-errors on their own, while possibly taking more time and GPU/CPU (not that big issue as long as you run everything locally and can work on something else in the meantime).

To be used "for production code" you'd normally want to (let someone else) fine-tune/train the model on a given house-style and the knowledge of copybooks to use.

[Mickey White]

Perhaps, since it appears that the link's LLM's mostly point to Mainframe code; wouldn't it be beneficial to have it also include GnuCOBOL SVN ( Source and Contributions ) so it would have a variety of examples?

[Simon Sobisch]

MF recovers from syntax errors a wee bit better than GnuCOBOL,
The MF version produced:

COBCH0498S Procedure name INITIALIZE-PROGRAM undeclared, line 129 : C:\GEDIT\COBOL\SOECLAUDE.COB(129,19)
Compilation complete with 292 errors, 0 warnings, 0 notices and an exit code of 12

please try to minimize the code to catch where cobc seems to "take the wrong way" - improving error-recovery and diagnostics is something we aim to do - but need to be presented with code samples.