#14 Occurs-Depending bug on last field

[Anonymous]

I'm get an 'Error calculation Occurs Depending On' error when trying to use the following copybook (slightly modified from the tests):

        01 Location-Details.
           03 Record-Type               pic x.
           03 Level-Count               pic s99.
           03 Attr-count                pic s99.
           03 Location-Levels.
              05 occurs 1 to 5 depending on Level-Count.
                 10 Level               pic 999.
                 10 occurs 1 to 5 depending on Attr-count.
                    15 Attr             pic 99.

Adding the following two lines to the end of the file fixes the issue so it seems to be a problem with the 'occurs-depending' field being the only one and also being in the final group in the copybook.

              05 occurs 1 to 5 depending on Attr-count.
                 10 Attribute           pic 99.

Unfortunately I don't have control over the copybooks I'm using so can't use the above workaround, but hopefully this will help to resolve the issue. I understand the 'occurs-depending' feature is not production-ready but is there another way to fix this issue?

Thanks

[Bruce Martin]

Are you using the copybook for reading or writing ???,

also the full stack trace would be usefull

[Bruce Martin]

I have just tested the copybook with my Source code and can not see a problem with processing it.
I have just updated subversion with latest code.

Can you download the latest source and get a Stack Trace. The error could be caused by a array size being invalid / not set.

[Michael Zoghby]

Hey Bruce -

I saw an occurs-depending error and thought this was a good place to chime in. This could still be something I'm doing wrong but I wanted to see if you could have a look at it.

This is my copybook:

01  DTLNK-RCRD.
  05  DTLNK-FULL-ACCT-NMBR-ID.
    10  DTLNK-SYS-BANK-ID         PIC X(04).
    10  DTLNK-PRIN-BANK-ID        PIC X(04).
    10  DTLNK-AGNT-BANK-ID        PIC X(04).
    10  DTLNK-CARD-ACCT-16-ID-1   PIC X(1).
    10  DTLNK-CARD-ACCT-16-ID-2   PIC X(15).
  05  DTLNK-CLNT-BANK-ID          PIC X(04).
  05  DTLNK-ELMN-TABL.
    10  DTLNK-NMBR-ELMN-NR    PIC S9(04) COMP.
    10  DTLNK-DCSN-ELMN       OCCURS 0 TO 1000 TIMES
                                                             DEPENDING ON DTLNK-NMBR-ELMN-NR.
      15  DTLNK-ACS-ELMN-ID   PIC X(04).
      15  DTLNK-DMM-ELMN-ID   PIC S9(09) COMP.
      15  DTLNK-ELMN-TX       PIC X(12).

The error I'm seeing appears to be related to the "depends on" count being a COMP field - when I remove the COMP
(which obviously doesn't work, but I wanted to see) the process fail normally. When I run it as-is, I get a NegativeArraySizeException
from the ContinuousLineReader. Can you recommend a change I can make that will allow this to process? We have other occurs-depending files
that are working right now with no problem. That COMP seems to be the 'X' factor...

784f43a2f718:hydra rec320$ java -jar target/hydra-1.3.3.jar ebcdic datalink_cpy.txt datalinkconfig2.json datalink.dat ascii output/
2019-05-10 13:33:04 DEBUG CopybookReader:325 - ------------- SINGLE RECORD CONFIGURATION --------------
2019-05-10 13:33:04 DEBUG CopybookReader:326 - RECORD: DTLNK-RCRD
2019-05-10 13:33:04 DEBUG CopybookReader:327 - FIND DTLNK-SYS-BANK-ID USING VARIABLE
2019-05-10 13:33:04 DEBUG CopybookReader:328 - ----------- END SINGLE RECORD CONFIGURATION --------------
2019-05-10 13:33:04 DEBUG CopybookReader:682 - Record w/ ID: VARIABLE has a length of: 34
java.lang.NegativeArraySizeException
    at net.sf.JRecord.IO.ContinuousLineReader.readImplementation(ContinuousLineReader.java:155)
    at net.sf.JRecord.IO.AbstractLineReader.read(AbstractLineReader.java:176)
    at com.place.librarytools.copybookreader.CopybookReader.queueEbcdicLines(CopybookReader.java:852)
    at com.place.librarytools.copybookreader.CopybookReader.queueRecordLine(CopybookReader.java:744)
    at com.place.librarytools.copybookreader.CopybookReader.beginFileProcessing(CopybookReader.java:699)
    at com.place.librarytools.copybookreader.CopybookReader.processEbcdicLines(CopybookReader.java:654)
    at com.place.librarytools.copybookreader.CopybookReader.readCobolFile(CopybookReader.java:587)
    at com.place.librarytools.copybookreader.CopybookReader.readCobolFile(CopybookReader.java:572)
    at com.place.App.readFileData(App.java:260)
    at com.place.App.main(App.java:104)
Exception in thread "main" java.lang.NullPointerException
    at com.place.librarytools.copybookreader.CopybookReader.queueEbcdicLines(CopybookReader.java:869)
    at com.place.librarytools.copybookreader.CopybookReader.queueRecordLine(CopybookReader.java:744)
    at com.place.librarytools.copybookreader.CopybookReader.beginFileProcessing(CopybookReader.java:699)
    at com.place.librarytools.copybookreader.CopybookReader.processEbcdicLines(CopybookReader.java:654)
    at com.place.librarytools.copybookreader.CopybookReader.readCobolFile(CopybookReader.java:587)
    at com.place.librarytools.copybookreader.CopybookReader.readCobolFile(CopybookReader.java:572)
    at com.place.App.readFileData(App.java:260)
    at com.place.App.main(App.java:104)
784f43a2f718:hydra rec320$ 

Anything you can tell me will be a huge help. As always - JRecord is hugely important to my operations here and I bless you every day for writing it so I didn't have to. Thank you for your hard work!

[Bruce Martin]

I will look at it

[Bruce Martin]

I have had look, at my version of JRecord; everything looks to be working.
I would suggest double checking everything (if you have not already). In particular

• Check that you have the latest copybook and not an older version
• Check Cobol Dialect (unlikely in this case) but there differences in field sizes
• Check the file structure is correct this could cause this problem

I am also happy to look at the file / copybook to see the problem. - Do you have my e-mail address ???

[Michael Zoghby]

Thank you for responding so quickly! I realized I didn't even post the version I'm using - 0.81.4:

<dependency>
    <groupId>net.sf.JRecord</groupId>
    <artifactId>JRecord</artifactId>
    <version>0.81.4</version>
</dependency>

I have attached the copybook and very small (9 kb) sample file I have in my possession. It's possible I simply need to upgrade my JRecord version but I read that 0.90 was a pretty huge departure from previous versions and our application uses a good number of the detailed tools present in JRecord - I was concerned about the disruption of a big upgrade.

I don't have your email address but I'm happy to switch to email if that's easier than working here.

[Bruce Martin]

In your code you have

            .setFileOrganization(Constants.IO_CONTINOUS_NO_LINE_MARKER)

        // Try changing to: 

            .setFileOrganization(Constants.IO_VB_DUMP)

Continuous vs VB Dump

With IO_CONTINOUS_NO_LINE_MARKER you are expecting

 <line 1><line 2><line 3> ... <line n>

what you have is

<block length=""><record length=""><line 1=""><record length=""><line 2=""> ... <block length=""><record length=""><line n=""></line></record></block></line></record></line></record></block>

So when JRecord looks for the array count, it looks in the wrong place (it is looking in
DTLNK-CARD-ACCT-16-ID a character field).

It is actually a safer way to recieve files because you have the Mainframe Record-Lengths

RecordEditor

If you supply both a Cobol-Copybook and a Sample file to either the:

• Layout Import (Top level menu Record Layout >>> Load Cobol Copybook)
• Cobol~JRecord Generate (Top level menu REcord Layout >>> Load Cobol Copybook)

The RecordEditor will check for the various VB formats (it does not check for
IO_CONTINOUS_NO_LINE_MARKER though.

Layout Import:

[Michael Zoghby]

So you're 100% correct on using the continuous line bit...
The issue is I'm also standing up a continuous line reader to read the file because it was the only thing that worked with my complex records.
I made the change you noted to VB_IO_DUMP from IO_CONTINOUS_NO_LINE_MARKERand am still seeing the same error as above (NegativeArraySize)

JRecord version 0.81.4:
This is where I stand up my builders and copybook:

private void setCopyBook(String copyBookName) throws IOException {
    // create stream of incoming copybook for use for layout access
    try (InputStream copyBookInputStream = FileStreamFactory.createInputStream(copyBookName)) {
        // the loader is used for faster access to our layout
        CobolCopybookLoader copyBookLoader = new CobolCopybookLoader();
        this.copyBook = copyBookLoader.loadCopyBook(copyBookInputStream, COPYBOOK_NAME,
                CopybookLoader.SPLIT_01_LEVEL, 0, CHARSET_EBCDIC, Cb2xmlConstants.USE_STANDARD_COLUMNS,
                Convert.FMT_MAINFRAME, 0, null);

        // the builder is used to read our file - it gives us access to more granular record identification tools
        this.builder = JRecordInterface1.COBOL.newIOBuilder(copyBookName)
                .setDialect(ICopybookDialects.FMT_MAINFRAME)
                .setSplitCopybook(CopybookLoader.SPLIT_01_LEVEL)
                .setFont(CHARSET_EBCDIC)
                .setFileOrganization(Constants.IO_VB_DUMP);
    }
}

and later:

// continuous line reader - we specified a no-break file when we setup our copybook reader
ContinuousLineReader r = new ContinuousLineReader();

// depending on whether our record is fixed length or variable, we need to use different reader configurations
// for fixed length, we get a reader with a fixed length and specify a reference record
// for variable, we need our builder to assemble the reader with its pre-configured records
// and IDs for those field values
if (getConfiguration().getReferenceRecord() != null) {
    // defaulting to our zero index
    EbcdicCobolRecordDecider decider = new EbcdicCobolRecordDecider(ZERO);
    this.copyBook.setRecordDecider(decider);
    r.open(getInputStream(), this.copyBook.getRecord(getConfiguration().getReferenceRecord()).asLayoutDetail());
} else {
    r.open(getInputStream(), this.builder.getLayout());
}

this.ebcdicReader = r;

and I'm using the ExternalRecordSelector objects to allow my reads to work when I don't use the record decider:

// set the external records for the builder and the config for mapping
for (RecordMappings recordMap : this.recordMappings) {
    // builder config
    // the RecordMap takes an array in the case where a copybook entry might be mapped to multiple IDs
    ExternalGroupSelection groupSelection = ExternalGroupSelection.newOr();

    for (String recordId : recordMap.getFieldIds()) {
        if (recordId.contains(COMP_FIELD_SPLIT)) {
            readerLogger.debug("------------- MULTI RECORD CONFIGURATION --------------");
            readerLogger.debug("RECORD: " + recordMap.getRecordName());

            String[] cbFieldNames = recordMap.getFieldName().split(ESCAPED_COMP_FIELD_SPLIT, -1);
            String[] cblRecordIds = recordId.split(ESCAPED_COMP_FIELD_SPLIT, -1);

            ExternalGroupSelection andSelection = ExternalGroupSelection.newAnd();
            for (int i = 0; i < cbFieldNames.length; i++) {
                readerLogger.debug("FIND " + cbFieldNames[i] + " USING " + cblRecordIds[i]);
                andSelection.add(
                        new ExternalFieldSelection(cbFieldNames[i], cblRecordIds[i]));
            }
            readerLogger.debug("----------- END MULTI RECORD CONFIGURATION --------------");

            // add AND selector to record selection
            groupSelection.add(andSelection);

        } else {
            groupSelection.add(
                    new ExternalFieldSelection(recordMap.getFieldName(), recordId));

            // if logging is enabled, write the mappable value
            readerLogger.debug("------------- SINGLE RECORD CONFIGURATION --------------");
            readerLogger.debug("RECORD: " + recordMap.getRecordName());
            readerLogger.debug("FIND " + recordMap.getFieldName() + " USING " + recordId);
            readerLogger.debug("----------- END SINGLE RECORD CONFIGURATION --------------");
        }
    }

    this.builder.setRecordSelection(recordMap.getRecordName(), groupSelection);
}

and an example of the JSON I feed it to help it find the records:

{
    "recordName": "DTLNK-RCRD",
    "fieldName": "DTLNK-CARD-ACCT-16-ID-1",
    "id": ["1", "2", "4", "5"]
}

This code works for every copybook I've every thrown at JRecord except this one with the COMP occurs-depending.
I know almost nothing about Mainframes - JRecord has literally saved my life the last year or so. I think I follow what you're saying about the record block length being
captured by the VB dump but I don't know what that means for the rest of my working code if I make more changes.

[Bruce Martin]

Problem:

Problem is you are hard coding the ContinuousLineReader

You will need some flag to indicate the type of Reader.

Possible Solution

This is one (of many) possible solution. Hopefully it will give you idea's

JSon

add a FileStructure Field i.e. something like:

{
    "recordName": "DTLNK-RCRD",
    "fieldName": "DTLNK-CARD-ACCT-16-ID-1",
    "fileStructure":  "VB_DUMP",
    "id": ["1", "2", "4", "5"]
}

setCopyBook

You do not need:

        CobolCopybookLoader copyBookLoader = new CobolCopybookLoader();
        this.copyBook = copyBookLoader.loadCopyBook(copyBookInputStream, COPYBOOK_NAME,
                CopybookLoader.SPLIT_01_LEVEL, 0, CHARSET_EBCDIC, Cb2xmlConstants.USE_STANDARD_COLUMNS,
                Convert.FMT_MAINFRAME, 0, null);

if you need the copyBook field you can do

        copybook = builder.getLayout();

Creating Reader

        int fileStructure = Constants.IO_CONTINOUS_NO_LINE_MARKER;

        if (getConfiguration().getReferenceRecord() != null) {
            if ("VB_DUMP".equals(getConfiguration().getFileStructure()) {
                fileStructure = Constants.
            }

            // You should not need this:
            //EbcdicCobolRecordDecider decider = new EbcdicCobolRecordDecider(ZERO);
            //this.copyBook.setRecordDecider(decider);

            //You only need to set one of RecordDecider and RecordSelection
            builder.setRecordSelection(...);
        } else {
            // You can probably do the following ???
            fileStructure = Constants.IO_FIXED_LENGTH_RECORDS;
        }
        builder.setFileOrganization(fileStructure)
        AbstractLineReader r = builder.newReader(getInputStream());
        this.ebcdicReader = newReader;

---

With regards JRecord 0.90 vs 0.84 I have tried to keep it backward
compatible as possible

• Some classes may of moved, these will be largely 'internal' classes.
• Some classes will be deprecated.

Only change to 0.90 if you need something from version 0.90. But make sure you have
good test coverage in case you need to move.

[Michael Zoghby]

So I think I understand - my problem was I was smashing all these different things together striving for a "read all comers" single solution. You're telling me this file is unique because it's a single record "dump" that is differently organized than the formats I'm use to - multiple 01 records in a copybook each with a "record type" and the layouts can be different lengths.

@Test
public void vbDump() throws Exception {
    ICobolIOBuilder builder =  JRecordInterface1.COBOL.newIOBuilder("datalink_cpy.txt")
            .setDialect(ICopybookDialects.FMT_MAINFRAME)
            .setSplitCopybook(CopybookLoader.SPLIT_01_LEVEL)
            .setFont(CHARSET_EBCDIC)
            .setFileOrganization(Constants.IO_VB_DUMP);

    AbstractLineReader reader = builder.newReader(new FileInputStream("datalink.dat"));
    AbstractLine recordLine;
    while(true) {
        if ((recordLine = reader.read()) != null) {
            System.out.println(recordLine.getFullLine());
        } else {
            break;
        }
    }
}

So - again, total COBOL/mainframe ignorance - is there a good way to tell just by looking at a copybook if it's a "dump" that the builder can just hand me a reader for versus a layout I need to configure selectors to read? I've attached a copybook example I would describe as my "usual" work.

[Bruce Martin]

So I think I understand - my problem was I was smashing all these different things together striving for a "read all comers" single solution. You're telling me this file is unique because it's a single record "dump" that is differently organized than the formats I'm use to

• YES.

So - again, total COBOL/mainframe ignorance - is there a good way to tell just by looking at a copybook if it's a "dump"

• NO, It is probably down to how the file is transmitted from the mainframe. It is possible to test wether you have a VB_DUMP (or VB file) - just try reading a couple of records. If it is not a VB_DUMP you will gen an exception (basically with in 3 records). The only problem is if there is only one record there is a small chance of a false positive (very unlikely).

---

I have added a new a new wiki entry Mainframe File System

Other wikis

• JRecord File Strucutures describes FileStrucure / FileOrganisation in JRecord
• MainFrame VB gives a bit more detail of the Mainframe VB format - probably not worth reading