Unallocated Memory issue when calling COBOL program from C or Python

[Anonymous]

Hi, I am very new to COBOL and recently for a project I was trying to call a COBOL program from within Python. I could not find any existing guide/blog on this (plan to write one myself) but I did find ctypes python library and this part of the GNUCOBOL documentation.
So I combined the two and was successfully able to call the COBOL program from within Python but the issue occurred when I tried to do it in a loop and it's a rather interesting issue. The program was crashing at the second iteration, always, with this message: attempt to reference unallocated memory (signal SIGSEGV)

After some debugging, I found that the crash was happening at the DIVIDE statement. Here is a sample to try it out:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. called.

       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01 INTEREST PIC SV9(8).
       01 TESST PIC 9(5) VALUE '50001'.
          LINKAGE SECTION.
          01 SID PIC 9(4).
          01 NAME PIC A(15).

       PROCEDURE DIVISION USING SID, NAME.
           DISPLAY 'In Called Program'.
           DISPLAY SID.
           DIVIDE 100 INTO TESST GIVING INTEREST.
           DISPLAY INTEREST.
           MOVE 'Tom' TO NAME.
           DISPLAY NAME.
           EXIT PROGRAM.

The above is the program being called, we compile it as: cobc called.cbl which gives us the called.so file.
Next is the python code:

from ctypes import * 

NAME = create_string_buffer(b"Hellooooo")
SID = create_string_buffer(b"100") 

for i in range(1, 4):
    cbl = cdll.LoadLibrary("called.so")
    cbl.cob_init(0, None)
    ret = cbl.called(byref(SID), byref(NAME))
    print(ret, NAME.value, i)
    cbl.cob_tidy()

And this is the result:

[pc@pc COBOL]$ python test_ctype_cbl2.py 
In Called Program
100
+.01000000
Tom      
0 b'Tom       ' 1
In Called Program
100

attempt to reference unallocated memory (signal SIGSEGV)

There are some more interesting observations:

1. This is not a Python issue as I have tried the same thing with C and it behaves similarly, I can share the code if required.
2. When called from another COBOL program, everything works fine. I can share that code too.
3. This works with Micro Focus COBOL. The python code requires some changes (like removing cob_init() and stuff) but there is no crash there.
4. If I change DIVIDE 100 INTO TESST GIVING INTEREST. to DIVIDE 100 INTO TESST. or DIVIDE 100 INTO 1000 GIVING INTEREST. then it is working fine.
5. I tried to replace the DIVIDE with COMPUTE statement but even that is crashing.

I wish to find a solution to this problem with everyone's help.
Thanks.

[Brian Tiffin]

I just noticed the pending messages in the queue, so this may be a rushed response.

GnuCOBOL modules, by default, like to be called from the command line or from other GnuCOBOL modules. There is some COBOL run time assumptions in COBOL binaries.

Edward added the CALL CONVENTION clause to SPECIAL-NAMES, that is paired with a special mode to the PROCEDURE DIVISION clause to tell GnuCOBOL to not assume it was called from another COBOL module.

I bumped into this a long time ago, and hacked the compiler to accept EXTERN, but Edward did it the right way, more to Standard (or at least standard expectation). I still use extern when I use the call-convention though, just because. I'm pretty sure the is extern can be any user defined symbol, though I never try it, because I use extern. ;-)

::cobolfree
       environment division.
       configuration section.
       special-names.
           call-convention 0 is extern.

and

::cobolfree
       procedure division extern using
                          by value conn ev by reference vdata
                          returning omitted.

Use whatever the linkage params and return needs actually are (I just took the first sample I found in my file pile for the code fragment).

That should help, Anon.

Oh, one other important item. Something in the executable chain needs to ensure the COBOL run time is initialized. That can be a direct C level call to cob_init()or using
cobc -fimplicit-init ... when the COBOL module is compiled. (I always use the latter), and let the compiler insert the initialization where it deems best.

Main programs (cobc -x) always do the run time initialization, and for most modules called from COBOL, the init has already happened before the CALL. This is only needed when a non COBOL program calls a COBOL module.

For completeness, void cob_init(int argc, char **argv) which is usually called with a simple cob_init(0, NULL);. You can tidy up libcob resources with int cob_tidy(void);.

Cheers,
Blue

[Sudhanshu Dubey]

Thanks for your response @btiffin.
I am sorry but I don't quite understand how to use CALL-CONVENTION and extern.
I tried to modify my sample code:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. called.

       environment division.
       configuration section.
       special-names.
           call-convention 0 is extern.

       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01 INTEREST PIC SV9(8).
       01 TESST PIC 9(5) VALUE 50001.
          LINKAGE SECTION.
          01 SID PIC 9(4).
          01 NAME PIC A(15).

       PROCEDURE DIVISION extern USING by reference SID, NAME.
           DISPLAY 'In Called Program'.
           DISPLAY SID.
           DIVIDE 100 INTO TESST GIVING INTEREST.
           DISPLAY INTEREST.
           MOVE 'Tom' TO NAME.
           DISPLAY NAME.
           EXIT PROGRAM.

But it didn't work. I may be asking too much but can you modify this code to demonstrate what you were trying to tell?

Thanks

[Brian Tiffin]

:-)

You had it mostly right, Sudhanshu.

Safe to ignore a lot of what I said in the first response for this one. I glossed over critical details in my rush to give a hint. The hint might come in handy as you progress, for other extern calls, but not for this.

You were already well on your way to having this work. I just shuffled around the load, the cob_init and the cob_tidy to keep the link loader cache and libcob sane.

Your original COBOL:

::cobolfree
       IDENTIFICATION DIVISION.
       PROGRAM-ID. called.

       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01 INTEREST PIC SV9(8).
       01 TESST PIC 9(5) VALUE '50001'.
          LINKAGE SECTION.
          01 SID PIC 9(4).
          01 NAME PIC A(15).

       PROCEDURE DIVISION USING SID, NAME.
           DISPLAY 'In Called Program'.
           DISPLAY SID.
           DIVIDE 100 INTO TESST GIVING INTEREST.
           DISPLAY INTEREST.
           MOVE 'Tom' TO NAME.
           DISPLAY NAME.
           EXIT PROGRAM.

Slightly rejigged Python (I added the ./ relative path spec for called.so, as Python didn't seem to want to look in $PWD in LoadLibrary...)

::python
from ctypes import *

NAME = create_string_buffer(b"Hellooooo")
SID = create_string_buffer(b"100")

cbl = cdll.LoadLibrary("./called.so")
cbl.cob_init(0, None)

for i in range(1, 4):
    ret = cbl.called(byref(SID), byref(NAME))
    print(ret, NAME.value, i)

cbl.cob_tidy()

That moves the load and init/tidy outside the loop. And a run of

::text
prompt$ cobc called.cob
prompt$ python3 calling.py
In Called Program
100
+.01000000
Tom
0 b'Tom       ' 1
In Called Program
100
+.01000000
Tom
0 b'Tom       ' 2
In Called Program
100
+.01000000
Tom
0 b'Tom       ' 3

And the second try of COBOL you used is bang on too, in terms of syntax. Either will work in this case (with the Python change above). Except unnecessary due to the proper init call for libcob setup and not accessing any magic special registers (explained below).

I think, in this short trial anyway. The extern thing was added to GnuCOBOL to get round a COBOL to COBOL calling sequence that allows for a little bit of reflection. GnuCOBOL actually builds a shadow array of parameters passed when it calls a module, so routines can get at the cob_field structure addresses, not just the data reference. cob_field holds things like size, usage flags along with the data address of a COBOL field. Along with setting a more commonly used special register, NUMBER-OF-CALL-PARAMETERS that can be used in called modules. If you don't touch that space, or the register, the extern call convention isn't needed. Doesn't hurt really, in this case, but the complication isn't needed. If your routine did try and access NUMBER-OF-CALL-PARAMETERS, it'd try to dereference a null, and back to kablooey.

The extern convention flag tells the run-time to not assume that shadow array or register is set when it enters a module code space, which you don't need for this example.

The CALL-CONVENTION thing might come in handy as you get more complicated callables though, Sudhanshu, so knowing about it does not hurt, and there is support for a few different conventions now in GnuCOBOL.

For the most part, you want to load and unload dynamic libraries once, and only once per process run. The change I did to the Python does that. If it becomes absolutely necessary
to load/unload over and over, we can talk more and dig deeper to ensure low level caches don't get in the way.

Hope I haven't confused more than helped.

Have good,
Blue

[Sudhanshu Dubey]

Thanks a lot for all the help @btiffin !!!

For the most part, you want to load and unload dynamic libraries once, and only once per process run.

This is the line that clicked for me and made me realise what I was doing wrong the whole time.
Quite ironic how me trying to clean the resources and actually overdoing it was the issue and that the compiler was telling me this all along:
attempt to reference unallocated memory

The actual project that I am working on is this: https://github.com/openmainframeproject-internship/COBOL-Modernization/tree/master/src/modern_UI/LoanCalculatorUI
I am trying to call COBOL from a Flask web app and being a web app we are not supposed to close it after a single run. But loading the library in a global scope and calling cob_tidy() when the application is about to close did the trick. (We are waiting for the COBOL code to be made open source to add it in the repo.)

It was quite naive of me to not do the feasibility study before deciding to use Flask for this purpose but I am still glad that this endeavour helped me ask questions that I (or maybe anyone) would never ask.

Thanks again for all the help!

[Brian Tiffin]

Cool. Nice to see this project and the others.

And...

If you do any serious thing -> GnuCOBOL, or GnuCOBOL -> thing, prepare to see attempt to reference unallocated memory more than once during development. :-)

GnuCOBOL is quite robust, but it has fairly complex areas of the run time that are extremely fragile until bolted into the right slots. The LINKAGE SECTION of COBOL has pretty much been a here be dragons, you are on your own area since the 1960s. Mixing run times means even more dragons.

Don't feel naive though. Learning is learning. There are areas of computing that you cannot always derive from first principles or prior knowledge, but need to experience. Flask is well designed tech, and adding GnuCOBOL to that mix could be a very worthwhile thing, Sunhanshu.

I'm all for exploring the edges of COBOL, and expanding those horizons thanks to GnuCOBOL.

Have good, make well,
Blue