We are attempting to use JRecord to help us with converting IBM mainframe files. We're working from the examples that are listed on the main JRecord site to pass in a Cobol copybook, a binary data file, and a desired output file to write the converted information.
So far we have realized that we need to access any packed decimals we're converting through the getFieldValue(String fieldName) method, rather than the getFullLine() (both in AbstractLine) method because the latter does not access the FieldValue object underneath, which is where the EBCDIC to ASCII conversion is actually called (see below). The problem is that we're not sure how to access a container that has a list of all the field names (or, alternatively, the FieldDetail objects). The code I've downloaded from sourceforge is missing some of the sources that are included in the Jars. I've just found out that there is a SVN repository that hosts the code as well, so I'm going to look into that to see if I can solve the problem.
For the time being, I was hoping you could help us figure out how to access one of those containers so we can iterate across it for each line to call the getFieldValue() method on each field and whether that is indeed the best practice for building a class that uses the JRecord library to dynamically parse and convert copybooks.
Thanks,
John
Example of copybook and output with getFullLine() vs getFieldValue()
As you found out, getFullLine() retrieves the Record as Text.
The LayoutDetail Object is the Schema (Description of the file). Many objects
(AbstractLine's, LineReader's) have a getLayout() method to retrieve the LayoutDetail.
The LayoutDetail basically has the following format:
LayoutDetail
+ -------- RecordDetail (one or more records)
+ ------------ FieldDetails (one or more fields).
I presume you are using single Record files (i.e. No header / Footer Records
in the file). Here is an example of printing each record:
~~~~~~
int fileStructure = Constants.IO_FIXED_LENGTH;
CobolIoProvider ioProvider = CobolIoProvider.getInstance();
AbstractLineReader reader = ioProvider.getLineReader(
fileStructure, Convert.FMT_MAINFRAME,
CopybookLoader.SPLIT_NONE, copybookName, salesFile
);
LayoutDetail schema = reader.getLayout();
int recordId = 0; // Assuming only one record type In the file
while ((saleRecord = reader.read()) != null) {
lineNum += 1;
System.out.println("Line " + lineNum + " Record : " + schema.getRecord(recordId).getRecordName());
for (int i = 0; i < schema.getRecord(recordId).getFieldCount(); i++) {
FieldDetail field = schema.getRecord(recordId).getField(i);
System.out.println(
"\t" + field.getName()
+ "\t\t" + saleRecord.getFieldValue(field).asString());
}
System.out.println();
System.out.println();
}
reader.close();
Here is a multi-record file where the "Record Type" determines the type of Record (possible values for record Type "H1", "S1", "D1". It uses a Xml-Record-Layout instead
of Cobol but the principle is just the same:
StringinstallDir=TstConstants.SAMPLE_DIRECTORY;StringamsPoFile=installDir+"Ams_PODownload_20041231.txt";StringcopybookName=TstConstants.RECORD_EDITOR_XML_DIRECTORY+"ams PO Download.Xml";intlineNum=0;AbstractLineamsPoRecord;try{LayoutDetailschema=CopybookLoaderFactory.getInstance().getLayoutRecordEditXml(copybookName,null);/* with XML copybooks, get the file structure from layout */intfileStructure=schema.getFileStructure();HashMap<String,RecordDetail>recordIdxMap=newHashMap<String,RecordDetail>();recordIdxMap.put("H1",schema.getRecord(schema.getRecordIndex("amsPODownload:Detail"))); recordIdxMap.put("D1", schema.getRecord(schema.getRecordIndex("amsPODownload:Header"))); recordIdxMap.put("S1", schema.getRecord(schema.getRecordIndex("amsPODownload:Allocation"))); AbstractLineReader reader = LineIOProvider.getInstance().getLineReader(fileStructure); reader.open(amsPoFile, schema); while ((amsPoRecord = reader.read()) != null) { String recordType = amsPoRecord.getFieldValue("RecordType").asString(); lineNum += 1; if (recordIdxMap.containsKey(recordType)) { RecordDetail recordDetail = recordIdxMap.get(recordType); System.out.println("Line" + lineNum + "Record:" + recordDetail.getRecordName()); for (int i = 0; i < recordDetail.getFieldCount(); i++) { FieldDetail field = recordDetail.getField(i); System.out.println( "" + field.getName() + "" + amsPoRecord.getFieldValue(field).asString()); } } else { System.out.println("InvalidRecordType:" + recordType + "atLineNumber:" + lineNum + amsPoRecord.getFullLine()); } System.out.println(); System.out.println(); } reader.close(); } catch (Exception e) { System.out.println("~~>" + lineNum + ""+e.getMessage());System.out.println();e.printStackTrace();}
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the response. We had found the getLayout() and getFieldMap() methods in the hours between the question and your response, but we've run into a different issue. We're dealing with multi-line, single record files (i.e. no header/footer, each line is a row of data) and the library is correctly converting the first line of data, but doing it incorrectly for the remainder of the lines (see below for our first 3 lines).
I've changed my code to use what you posted above and as I step through the execution, I'm seeing the correct field names and types (or at least the ints that stand for the types are coming across the same, as 35 for comp-3 and 31 for comp) throughout the lines and the packed decimal conversions are all getting called. This leads me to believe that the issue is either within the data on our end (which I'm looking into) or in one of the steps to read in the data (which I'd assume would be the Line and/or LineReader classes).
Is there any additional set up we need to be doing for the library to work with our type of files? Have you seen this issue pop up before and recognize it as a common data (offset, corruption, conversion, etc.) problem? Also, do you have a sample data & copybook file that we can run against this to make sure we have everything set up properly? We're doing most of our testing against one file, but we've tried a bunch of others just to see if any work and we're getting the same issues.
Thanks,
John
p.s. there is a bunch of garbage hex in the output in eclipse that isn't showing up here
Line 2 Record : AFS_200_mod
C200-BK-NO 16448
C200-SERV-UNIT
C200-OBG-NO 404040404040
C200-OBL-NO 4040400001f0
C200-CHG-CODE 004
C200-ADD-DATE 0
C200-NUM-DAYS ff0
C200-ITEM 00200
C200-FIELD-NAME 6-07-20
C200-FIELD-DATA DAYS
C200-EXCEP-MSG 999DAYS SINCE LAST AUDIT EXCEEDS MAXI
C200-SEQ d4e4d440
Line 3 Record : AFS_200_mod
C200-BK-NO 16448
C200-SERV-UNIT
C200-OBG-NO 404040404000
C200-OBL-NO 2f404040
C200-CHG-CODE
C200-ADD-DATE
C200-NUM-DAYS 1f0f0
C200-ITEM 040
C200-FIELD-NAME
C200-FIELD-DATA 7-20
C200-EXCEP-MSG S 999DAYS SINCE LAST AU
C200-SEQ c4c9e340
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As an update, it seems the issue was on our end. The copybooks that were provided to us did not include an abitrary amount of space between the final pieces of data and the end of record delimiter in our systems. By adding a line (below) to our copybooks, we were able to get JRecord to function as intended.
10 FILLER PIC X(16) VALUE SPACE.
Thanks for the help,
John
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We're looking into extending the JRecord code to allow the project to function with our specific needs but we've stumbled upon some missing source code files (e.g. TreeDetails.java). Is there somewhere you have the full source code avialable, or is there any way that you can make it available to us?
Thanks,
John
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Where did you get the source code ??? and which class references TreeDetails.java ???. TreeDetails should not be part of standard JRecord. So I would like to find the reference.
TreeDetails.java is part of RecordEditor's version of JRecord. It is used in AvroEditor / Protocol_Buffers_Editor for representing the Tree-Structure in Avro / Protocol_Buffers formats. It should not be needed in standard JRecord.
Anyway the RecordEditor Repositry has a different version of JRecord:
cb2xml and JRecord_Common will be similar (but updated from the last JRecord version) The JRecord code here will be different from what you already have. It does include TreeDetails which implements Interface AbstractTreeDetails.
Many of the utilitiesuse the RecordEditor version
Last edit: Bruce Martin 2013-06-05
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
JRecord_0.68.3_Updated_Jars.zip/lib/JRecord.jar/net/sf/JRecord/Details/ contains classes which you indicate should not be part of the standard JRecord distribution.
The .class files that do not have corresponding .java files include
TreeDetails.class
AbstractChildDetails.class
AbstractLayoutDetails.class
...
No need to do anything about it at this point.
I am posting this so that anyone else who may be trying to use JRecord-0.68.3 does not stumble and get confused over this issue.
Michael
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
We are attempting to use JRecord to help us with converting IBM mainframe files. We're working from the examples that are listed on the main JRecord site to pass in a Cobol copybook, a binary data file, and a desired output file to write the converted information.
So far we have realized that we need to access any packed decimals we're converting through the getFieldValue(String fieldName) method, rather than the getFullLine() (both in AbstractLine) method because the latter does not access the FieldValue object underneath, which is where the EBCDIC to ASCII conversion is actually called (see below). The problem is that we're not sure how to access a container that has a list of all the field names (or, alternatively, the FieldDetail objects). The code I've downloaded from sourceforge is missing some of the sources that are included in the Jars. I've just found out that there is a SVN repository that hosts the code as well, so I'm going to look into that to see if I can solve the problem.
For the time being, I was hoping you could help us figure out how to access one of those containers so we can iterate across it for each line to call the getFieldValue() method on each field and whether that is indeed the best practice for building a class that uses the JRecord library to dynamically parse and convert copybooks.
Thanks,
John
Example of copybook and output with getFullLine() vs getFieldValue()
01 DCLC200. 10 C200-BK-NO PIC S9(4) USAGE COMP. 10 C200-SERV-UNIT PIC X(5). 10 C200-OBG-NO PIC S9(11)V USAGE COMP-3. 10 C200-OBL-NO PIC S9(11)V USAGE COMP-3. 10 C200-CHG-CODE PIC X(3). 10 C200-ADD-DATE PIC X(10). 10 C200-NUM-DAYS PIC S9(6)V USAGE COMP-3.getFieldValue()
1 000092039 2217941 5802012-12-28146
getFullLine()
00009 (some garbage hex that cant be displayed) m 5802012-12-28 (more garbage hex)
As you found out, getFullLine() retrieves the Record as Text.
The LayoutDetail Object is the Schema (Description of the file). Many objects
(AbstractLine's, LineReader's) have a getLayout() method to retrieve the LayoutDetail.
The LayoutDetail basically has the following format:
LayoutDetail + -------- RecordDetail (one or more records) + ------------ FieldDetails (one or more fields).I presume you are using single Record files (i.e. No header / Footer Records
in the file). Here is an example of printing each record:
~~~~~~
int fileStructure = Constants.IO_FIXED_LENGTH;
CobolIoProvider ioProvider = CobolIoProvider.getInstance();
AbstractLineReader reader = ioProvider.getLineReader(
fileStructure, Convert.FMT_MAINFRAME,
CopybookLoader.SPLIT_NONE, copybookName, salesFile
);
LayoutDetail schema = reader.getLayout();
int recordId = 0; // Assuming only one record type In the file
while ((saleRecord = reader.read()) != null) { lineNum += 1; System.out.println("Line " + lineNum + " Record : " + schema.getRecord(recordId).getRecordName()); for (int i = 0; i < schema.getRecord(recordId).getFieldCount(); i++) { FieldDetail field = schema.getRecord(recordId).getField(i); System.out.println( "\t" + field.getName() + "\t\t" + saleRecord.getFieldValue(field).asString()); } System.out.println(); System.out.println(); } reader.close();~~~~~
I will do a multi-record Example and post that latter.
Bruce
Here is a multi-record file where the "Record Type" determines the type of Record (possible values for record Type "H1", "S1", "D1". It uses a Xml-Record-Layout instead
of Cobol but the principle is just the same:
Bruce,
Thanks for the response. We had found the getLayout() and getFieldMap() methods in the hours between the question and your response, but we've run into a different issue. We're dealing with multi-line, single record files (i.e. no header/footer, each line is a row of data) and the library is correctly converting the first line of data, but doing it incorrectly for the remainder of the lines (see below for our first 3 lines).
I've changed my code to use what you posted above and as I step through the execution, I'm seeing the correct field names and types (or at least the ints that stand for the types are coming across the same, as 35 for comp-3 and 31 for comp) throughout the lines and the packed decimal conversions are all getting called. This leads me to believe that the issue is either within the data on our end (which I'm looking into) or in one of the steps to read in the data (which I'd assume would be the Line and/or LineReader classes).
Is there any additional set up we need to be doing for the library to work with our type of files? Have you seen this issue pop up before and recognize it as a common data (offset, corruption, conversion, etc.) problem? Also, do you have a sample data & copybook file that we can run against this to make sure we have everything set up properly? We're doing most of our testing against one file, but we've tried a bunch of others just to see if any work and we're getting the same issues.
Thanks,
John
p.s. there is a bunch of garbage hex in the output in eclipse that isn't showing up here
Line 1 Record : AFS_200_mod
C200-BK-NO 1
C200-SERV-UNIT 00009
C200-OBG-NO 2039
C200-OBL-NO 2217941
C200-CHG-CODE 580
C200-ADD-DATE 2012-12-28
C200-NUM-DAYS 147
C200-ITEM
C200-FIELD-NAME CHARGE CODE
C200-FIELD-DATA 580
C200-EXCEP-MSG ACCRUAL/BILLING SCHEDULE MISMATCH
C200-SEQ 1
Line 2 Record : AFS_200_mod
C200-BK-NO 16448
C200-SERV-UNIT
C200-OBG-NO 404040404040
C200-OBL-NO 4040400001f0
C200-CHG-CODE 004
C200-ADD-DATE 0
C200-NUM-DAYS ff0
C200-ITEM 00200
C200-FIELD-NAME 6-07-20
C200-FIELD-DATA DAYS
C200-EXCEP-MSG 999DAYS SINCE LAST AUDIT EXCEEDS MAXI
C200-SEQ d4e4d440
Line 3 Record : AFS_200_mod
C200-BK-NO 16448
C200-SERV-UNIT
C200-OBG-NO 404040404000
C200-OBL-NO 2f404040
C200-CHG-CODE
C200-ADD-DATE
C200-NUM-DAYS 1f0f0
C200-ITEM 040
C200-FIELD-NAME
C200-FIELD-DATA 7-20
C200-EXCEP-MSG S 999DAYS SINCE LAST AU
C200-SEQ c4c9e340
As an update, it seems the issue was on our end. The copybooks that were provided to us did not include an abitrary amount of space between the final pieces of data and the end of record delimiter in our systems. By adding a line (below) to our copybooks, we were able to get JRecord to function as intended.
10 FILLER PIC X(16) VALUE SPACE.
Thanks for the help,
John
Bruce,
We're looking into extending the JRecord code to allow the project to function with our specific needs but we've stumbled upon some missing source code files (e.g. TreeDetails.java). Is there somewhere you have the full source code avialable, or is there any way that you can make it available to us?
Thanks,
John
Where did you get the source code ??? and which class references TreeDetails.java ???. TreeDetails should not be part of standard JRecord. So I would like to find the reference.
If you are using version 0.68.3 already, try downloading it
at https://sourceforge.net/projects/jrecord/files/jrecord/Version_0.68.3/
TreeDetails.java is part of RecordEditor's version of JRecord. It is used in AvroEditor / Protocol_Buffers_Editor for representing the Tree-Structure in Avro / Protocol_Buffers formats. It should not be needed in standard JRecord.
Anyway the RecordEditor Repositry has a different version of JRecord:
https://sourceforge.net/p/record-editor/code/HEAD/tree/Source/
cb2xml and JRecord_Common will be similar (but updated from the last JRecord version) The JRecord code here will be different from what you already have. It does include TreeDetails which implements Interface AbstractTreeDetails.
Many of the utilitiesuse the RecordEditor version
Last edit: Bruce Martin 2013-06-05
Bruce (& potential users of JRecord-0.68.3),
Be advised that the release of JRecord 0.68.3 posted at sourceforge is inconsistent.
https://sourceforge.net/projects/jrecord/files/jrecord/Version_0.68.3/JRecord_0.68.3_Updated_Jars.zip
JRecord_0.68.3_Updated_Jars.zip/JRecord/src/net/sf/JRecord/Details/ looks correct.
JRecord_0.68.3_Updated_Jars.zip/lib/JRecord.jar/net/sf/JRecord/Details/ contains classes which you indicate should not be part of the standard JRecord distribution.
The .class files that do not have corresponding .java files include
TreeDetails.class
AbstractChildDetails.class
AbstractLayoutDetails.class
...
No need to do anything about it at this point.
I am posting this so that anyone else who may be trying to use JRecord-0.68.3 does not stumble and get confused over this issue.
Michael