Jrecord EBCDIC to ASCII conversion issue in Linux

[ShreyasV]

Hello,

I am currently facing a weird issue.
I have a working program which uses Jrecord to read an ebcdic file and displays the contents in ascii.
This is working correctly as expected in Eclipse.
But when the program is packaged as a Runnable jar and executed on Linux server. the output is having special chars, and comp-3 fields are not correctly converted.
But the same jar when run in windows command prompt will produce proper output.

Has anyone faced this issue earlier?
Appreciate your guidance.

Thanks

[Bruce Martin]

output is having special chars, and comp-3 fields are not correctly converted - this suggest to me there is some sort of conversion occuring on the file. Either a Ebcdic-Unicode or ebcdic to ascii.
I really need to see the code (and probably a data file) to understand what is going on. I will try and send you my direct e-mail address incase you do not want to put the program up on the web-site

One possible cause is on Windows the default encoding is ascii while most linux's it is utf-8.

Two things I would suggest (if you have not done so):

1. Make sure you use the same file on linux and windows
2. Check for .setFont("") this will perform differetly on linux as apossed windows i.e

        ioBuilder
               .setFont("")

[Thomas George]

Hi Bruce,
Can you provide the code link which is used for converting ebcdic to ascii using java.

[Bruce Martin]

For JRecord use the setFont option, for US EBCDIC (cp037 or IBM037)

      ioBuilder
               .setFont("cp037")

There are lot of ebcdic's though. German EBCDIC is cp273/IBM273

I java in general you can do

     String s = new String(byteArray, "cp037);

Generate JRecord code

If you use the code-generator in the recordeditor you can specify a

• Copybook
• Data file

It should work out you have an ebcdic file and generate the appropriate code

see StackOverflow answer

[ShreyasV]

Hi bruce

Thanks for your input!
Below is the partial code we are using, to create a IOBuilder and reader :

ICobolIOBuilder iob = JRecordInterface1.COBOL.newIOBuilder(copyBook)
                        .setFont("cp037").setFileOrganization(Constants.IO_FIXED_LENGTH)
                        .setSplitCopybook(CopybookLoader.SPLIT_NONE);
AbstractLineReader reader = iob.newReader(ebcdicFile);
                AbstractLine line;
 while(fieldCounter<totalFieldCount){
                            printStmnt=printStmnt + line.getFieldValue(fieldsArray[fieldCounter]).asString().trim() +"\t\t  |"  ;
                            fieldCounter++;

I have tried the .setFont option with both cp037 and IBM037 values, but the same behaviour is observed.

I will not be able to share the actual data as it is not permitted.

Appreciate your guidance.

Thank you!!

[Bruce Martin]

This code should work the same on linux as on Windows. Its clutching at straws time.

I think it is time to investigate java:
which version of java is running on linux
The encodings CP037 / IBM037 are in an Optional jar. If this is jar is missing / corrupted, java will still work (provided you dot use thing like cp037). Please run the following code on the linux system (it will thow an exception if cp037 is missing):

public static void main(String[] args) throws UnsupportedEncodingException {
        System.out.println(
                new String(
                        "abcAB123".getBytes("cp037"),
                        "cp037"
                        )
                );
    }

----

It may be worth trying on a different Linux instalation -

• indicates wether it is a generic linux problem or specific to your linux box.

You could install Virtualbox and install a recent Linux distribution (I would suggest a light weight version e.g. LUBUNTU)

[ShreyasV]

Hi Bruce

Thanks for the inputs.
I have tried out the mentioned code above and it runs fine without throwing any errors oon linux box.
However, when I was looking at the data, i could see that the first line is converted perfectly, but few fields from the 2nd line are being concated at the end of 1st line and the fields starting from 2nd line are not correctly extracted.

As I cant give the actual data, PFB an example of how the data appears in Linux output.
Eg:
row1: line1field1 |line1field2 |line1field3 |line1field4 |line1field5 |line1field6 |line2field1 |line2field2_split1|line2field2_split2
row2: |line2field3_split1|line2field3_split2|line2field4_split1|line2field4_split2 etc....

This is only happening on Linux, but windows cmd and eclipse output are perfect.

Hope to resolve this soon!
Thanks in advance for the help!

[Bruce Martin]

That looks like a mismatch between the copybook and the Data. I would be checking the file and copybook are the same in Windows & linux.

I suggest Stting up a directory containing:
All jars
The Cobol Copybook
* File being read.

Run the program on Wandows then move directory to linux and run it again (or vice versa).

It is probably be a small difference between the way things are setup in windows and linux

[ShreyasV]

Hi Bruce An update on the issue.
We moved the .java file to linux and compiled it on the server and ran the class files. it runs without any issue :)
But if we package the jar in eclipse and use it on Linux the issue turns up.
Looks like we'll have to proceed with the .class files we have generated, instead of using the jar.

Thanks for all your support!

[Bruce Martin]

At least you got it working. I presume there is an issue with your eclipse setup / build job

[ShreyasV]

Hi Bruce,
We went throu the code and the copybook files again, and you were right in pointing out that this issue might be due to mismatch between data fie and copybook.
It turns out there are a few fields added to the file by upstream teams, but they failed to update the changes in copybook.
After the updation we were able to read the files correctly.

One more thing we are considering is reading the EBCDIC files that are present on HDFS.
Our use case would be to Read EBCDIC file present on HDFS and print out converted ascii data.
Are there any other libraries tat needs to be used for this use case?
Any sample code available would be helpful as well!

Request you to point me in right direction.

Thanks for all the help!

[Bruce Martin]

Have a look at this discaussion:

• https://sourceforge.net/p/jrecord/discussion/678633/thread/27d23c5b/

it lists several packages that use JRecord with HADOOP, not sure how relavent they are. Packages include:

• https://github.com/tmalaska/CopybookInputFormat
• https://github.com/ianbuss/CopybookHadoop (clone of the above ??)
• https://github.com/gss2002/copybook_formatter
• https://github.com/gss2002/ftp2hdfs

I you can gett record as an array of byes, another option could be to use a IByteRecordReader . See:

• https://sourceforge.net/p/jrecord/wiki/IByteRecordReader%20and%20IByteRecordWriter%20interfaces/
• https://sourceforge.net/p/jrecord/discussion/678634/thread/8a04f1a1/?limit=25#cefd/8fd5

Finally creating a CodeGen Template for Hadoop might be possible

[Jasminder pal s sehgal]

Hello Bruce, I;m getting the flollowing error

Exception in thread "main" java.lang.NoSuchMethodError: net.sf.cb2xml.CobolPreprocessor.preProcess(Ljava/io/Reader;II)Ljava/lang/String;
    at net.sf.cb2xml.DoCobolAnalyse.doAnalysis(Unknown Source)
    at net.sf.cb2xml.Cb2Xml3$BldrImp.getCopybook(Unknown Source)
    at net.sf.cb2xml.Cb2Xml3$BldrImp.asCobolItemTree(Unknown Source)
    at net.sf.JRecord.External.Def.Cb2Xml.getCopybook(Cb2Xml.java:194)
    at net.sf.JRecord.External.base.BaseCobolItemLoader.loadCopyBook(BaseCobolItemLoader.java:142)
    at net.sf.JRecord.External.base.BaseCobolItemLoader.loadCopyBook(BaseCobolItemLoader.java:125)
    at net.sf.JRecord.External.CobolCopybookLoader.loadCopyBook(CobolCopybookLoader.java:18)
    at net.sf.JRecord.IO.builders.CreateExternalFromFile.createExternalRecordImp(CreateExternalFromFile.java:48)
    at net.sf.JRecord.IO.builders.CreateExternalBase.createExternalRecord(CreateExternalBase.java:96)
    at net.sf.JRecord.IO.builders.CblIOBuilderMultiSchemaBase.getExternalRecordImpl(CblIOBuilderMultiSchemaBase.java:124)
    at net.sf.JRecord.IO.builders.CblIOBuilderBase.getExternalRecord(CblIOBuilderBase.java:407)
    at net.sf.JRecord.IO.builders.CblIOBuilderBase.getLayout(CblIOBuilderBase.java:481)
    at net.sf.JRecord.IO.builders.CblIOBuilderBase.newReader(CblIOBuilderBase.java:253)
    at net.sf.JRecord.IO.builders.CblIOBuilderBase.newReader(CblIOBuilderBase.java:245)
    at EBCDIC_Reader.main(EBCDIC_Reader.java:80)

My code is below::

CobolIoProvider ioProvider = CobolIoProvider.getInstance();
        String copyBook = "C:\\Users\\S5775301\\Desktop\\input\\copybook.cbl";
        String DatFile ="C:\\Users\\S5775301\\Desktop\\input\\copybook-data.txt";

        int lineNumber = 0;
        AbstractLine line;
        try{

            ICobolIOBuilder ioBuilder = JRecordInterface1.COBOL
                    .newIOBuilder(copyBook)
                    .setFont("cp037")
                    .setFileOrganization(Constants.IO_FIXED_LENGTH_RECORDS);

            AbstractLineReader reader  = ioBuilder.newReader(DatFile);
            while ((line = reader.read()) != null) {
                //System.out.println(line.getFullLine());

                lineNumber += 1;
                System.out.println(
                        "  "    + line.getFieldValue("CONTACT-NAME").asString()
                                + "  \t " + line.getFieldValue("CONTAT-NUM1").asString()
                                + "  \t " + line.getFieldValue("CONTAT-NUM2").asString()
                                + "  \t " + line.getFieldValue("CONTAT-NUM3").asString()
                                + "  \t " + line.getFieldValue("CONTAT-NUM4").asString()
                                + "  \t " + line.getFieldValue("CONTAT-NUM5").asString()
                                + "  \t " + line.getFieldValue("CONTAT-NUM6").asString());

            }

        }catch(Exception e){
            e.printStackTrace();
        }

Please help!! i'm using latest JRecorder version to convert EPCDIC data.

[Bruce Martin]

Which version of Java ???

• is it 11 ??
• or an old version of java (it needs to be atleast Java 7) for the latest JRecord (Strictly the cb2xml cobol preprocessor has a Java 7 limit).

Also I will attach cb2xml.jar just in case your jar has been corrupted

[Jasminder pal s sehgal]

Thanks Bruce so much. I was using older version of Java. I have one query.

I'm able to convert EBCDIC data using copybook (.cbl format)
Now, If we use .cbl format ,then we have to manuall enter the field names in our code to get data.

I was thinking, whether you have any utility which take copybook data in XML format and EBCDIC data and automatically extract the field name and process the data.

I have tried cbl2xml and it works well, I'm able to get Copybook data in XML format. But when i try to parse this XML file. It shows error.

IIOBuilder ioBuilder = JRecordInterface1.SCHEMA_XML
                                                         .newIOBuilder(copyBookFile)
                                                             .setFont("cp037")
                                                             .setFileOrganization(Constants.IO_FIXED_LENGTH_RECORDS);

            AbstractLineReader reader = ioBuilder.newReader(EBCFilePath);

I'm getting following error

Error loading copybook; Expected RECORD Tag, but got copybook
net.sf.JRecord.Common.RecordException: Error loading copybook; Expected RECORD Tag, but got copybook
    at net.sf.JRecord.IO.builders.CblIOBuilderMultiSchemaBase.getExternalRecordImpl(CblIOBuilderMultiSchemaBase.java:161)
    at net.sf.JRecord.IO.builders.CblIOBuilderBase.getExternalRecord(CblIOBuilderBase.java:419)
    at net.sf.JRecord.IO.builders.CblIOBuilderBase.getLayout(CblIOBuilderBase.java:493)
    at net.sf.JRecord.IO.builders.CblIOBuilderBase.newReader(CblIOBuilderBase.java:253)
    at net.sf.JRecord.IO.builders.CblIOBuilderBase.newReader(CblIOBuilderBase.java:245)
    at EBCDIC_Reader.convertEBCtoFinal(EBCDIC_Reader.java:102)
    at EBCDIC_Reader.main(EBCDIC_Reader.java:190)

Thanks alot, You are god of COBOL :))

[Jasminder pal s sehgal]

Ok Bruce, I got the error. Cb2XMl generates XML with follwing header

<?xml version="1.0" encoding="UTF-8"?>

-<copybook cb2xml-format="2017" dialect="Mainframe" filename="oip_ioice_ci_f_cioicci_181102.cpy">

-<item display-length="1000" storage-length="1000" position="1" name="CI-CUSTOMER-PLAN-ACCOUNT-INFO" level="01">

<item display-length="15" storage-length="15" position="1" name="CI-PLAN-ACCOUNT-NO" level="05" picture="X(15)"/>

But, Your code wants to parse XML in following format :

<?xml version="1.0" ?>
   <RECORD RECORDNAME="DTAR020" COPYBOOK="DTAR020" DELIMITER="&lt;Tab&gt;" 
           FONTNAME="CP037" FILESTRUCTURE="Default" STYLE="0" RECORDTYPE="RecordLayout"
           LIST="Y" QUOTE="" RecSep="default">
           <FIELDS>
                   <FIELD NAME="KEYCODE-NO" POSITION="1"  LENGTH="8" TYPE="Char" />
                   <FIELD NAME="STORE-NO"   POSITION="9"  LENGTH="2" TYPE="Mainframe Packed Decimal (comp-3)" />
                   <FIELD NAME="DATE"       POSITION="11" LENGTH="4" TYPE="Mainframe Packed Decimal (comp-3)" />
                   <FIELD NAME="DEPT-NO"    POSITION="15" LENGTH="2" TYPE="Mainframe Packed Decimal (comp-3)" />
                  <FIELD NAME="QTY-SOLD"   POSITION="17" LENGTH="5" TYPE="Mainframe Packed Decimal (comp-3)" />
                  <FIELD NAME="SALE-PRICE" POSITION="22" LENGTH="6" DECIMAL="2" TYPE="Mainframe Packed Decimal (comp-3)" />
          </FIELDS>
  </RECORD>

So, what should i do now?

[Bruce Martin]

To process cb2xml xml files:

 JRecordInterface1.CB2XML.newIOBuilder(copybookFileame)

but there is a much better option - CodeGen utility, this can be run from

• Command line
• From java
• From the RecordEditor. The RecordEditor can use a sample Data File to help select the correct options.
  Have a look at Generating Java JRecord Code from a Cobol Copybook

Which-ever way you run CodeGen, there are three basic Templates:

• standard This template generates

  • a Cobol Field Nameclass
  • sample read / write programs. These are not intended to be fully working programs but sample programs to get you started

• lineWrapper this template generates

  • a Cobol Field Name class
  • a Wrapper class (for JRecord lines) where access the fields by java getter/setter's for each field.
  • Sample read / write programs

• pojo - This template generates a pojo class, cobol to java-pojo class, IO Builder for the pojo and pojo readers/writers. It works well for simple Cobol files, for
  more complicated files (e.g. with complicated redefines) you are probably better using one of the above

The standard template is probably what you are currently reting to achieve. Although the other templates ma be useful.

The Code for CodeGen is in Source/Other_Source/JRecord_CodeGen directory of the JRecord zip files. It uses Velocity 1.7 templates to generate the Code.

---

If you want to write generic programs I can tell you how to get at the schema information.

[Bruce Martin]

On a side note RecordEditor and ReCsvEditor can generate Java~JRecord code to read/write the CSV for you.

[Jasminder pal s sehgal]

Thanks, It helped.
May the force be with you :))