#52 MF output format

[Sjaak Westdijk]

Hi,

With the migration from MF NetExpress to GnuCOBOL we could get the output format right. MF print <cr> and <cr><lf> in a not so consistent way. </lf></cr></cr>

To get our files the same as MF I created a patch set containing a modification of the cob_file struct to keep track of the write mode (AFTER/BEFORE) and modifications to fileio.c to print <cr> or <lf> on the correct places. With this modifications files are for 99% equal to MF. The only thing left is the issue with OPEN EXTEND, MF clears the last <cr> in the file.</cr></lf></cr>

this modifications are only for MF compatibility. So if they are incorporated, they should only be active when compiling with --std=mf

Sjaak

[Simon Sobisch]

Assign to Ron because of mf-compat.

[Simon Sobisch]

Ticket moved from /p/gnucobol/bugs/780/

[Sjaak Westdijk]

We detected the same issues on the sequential organization files. So the patch is extended to handle this as well. Also a modified test program.

[Ron Norman]

I've done some testing of your additional test case using FILE6, FILE7, FILE 8 and using Microfocus COBOL. I added code to read back and DISPLAY the FILE8 records. Using the BEFORE/AFTER on WRITE for a SEQUENTIAL file results in a file which is NOT read back correctly by Microfocus COBOL.
If I add ORGANIZATION LINE SEQUENTIAL then it is read back correctly.
So I am not sure if using BEFORE/AFTER on the WRITE for is a valid thing to do here.

[Ron Norman]

If you do not use BEFORE/AFTER with WRITE and undefined ORGANIZATION then the file ends up looking like a RECORD SEQUENTIAL file using Microfocus and GnuCOBOL.
If there are WRITE BEFORE/AFTER and undefiend ORGANIZATION then records written using no BEFORE/AFTER look like RECORD SEQUENTIAL format but then the records written using BEFORE/AFTER look like LINE SEQUENTIAL format.
So the data file gets very confused.

[Ron Norman]

A possible option here maybe to track is a file ORGANIZATION was in the code or not.
If defined as LINE SEQUENTIAL then WRITE BEFORE/AFTER is accepted.
If not LINE SEQUENTIAL then using WRITE BEFORE/AFTER causes a compile error.
If the ORGANIZATION was not defined and there are WRITE BEFORE/AFTER in the program then default the file type to LINE SEQUENTIAL.
If ORGANIZASTION was not defined and there are no WRITE BEFORE/AFTER in the program then default the file type to RECORD SEQUENTIAL.

[Sjaak Westdijk]

You can skip FILE8. It was more or less a monkey proof test on our side which will not used in normal situations (we think).

Modifying the ORGANIZATION to LINE SEQUENTIAL is probably not a good idea as the file layout will be different (missing ending spaces for example) and we are expecting issues when third parties read to files which are expecting to be SEQUENTIAL. So I think BEFORE/AFTER must be acceptable for SEQUENTIAL (as MF does) but I agree switching between BEFORE and AFTER is causing issues.

[Ron Norman]

My main point is that although Microfocus allows the BEFORE/AFTER it creates a file which Microfocus is not able to correctly read. So I question the validity of doing this.

A SEQUENTIAL file with 30 byte records consists of a sequence of 30 byte records in the file with nothing between records.
A LINE SEQUENTIAL file is variable length and records are separated by CR LF on Windows and LF on Unix with the CR being optional.

But when you use WRITE BEFORE/AFTER with a [RECORD] SEQUENTIAL file the data on disk ends up as part fixed size SEQUENTIAL records with no separator and part LINE SEQUENTIAL records with CR/LF. Its a mess.

[Sjaak Westdijk]

I fully understand your point and share your thoughts about this. Its strange that MF can write a file, but can't read it.

Questions is: what is MF compatibility.

In my opinion:
1. If MF can write such file, GnuCobol should also have the possibilty in mf compat mode
2. if MF can read such file, GnuCobol should also.
3. If MF can't read the file, GnuCobol may have the possibilty if it can't its also ok.

In our application we use such "feature" and can't change the file "layout" as third parties are using it a lot of years.

So hopefully you end up to implement such "feature" else I have to patch the fileio.c at every release for now.

[Ron Norman]

The third party must have special software to read the files...
If the file is SEQUENTIAL the format on disk is fixed size records with no separator between the records.
For GnuCOBOL, to be able to reread the file, when it is written using WRITE BEFORE/AFTER, then if GnuCOBOL just wrote empty records to the file, then it would be possible to reread the SEQUENTIAL file using COBOL.
But mixing up fixed size records with CR & LF is not what the COBOL I/O routine of either Microfocus or GnuCOBOL is expecting.
Can you verify that your data files actually do end up in this strange collection of data?

[Simon Sobisch]

Hm, WRITE BEFORE/AFTER with SEQUENTIAL files is always special and if I remember correctly GnuCOBOL never matched MF (or ACUCOBOL-GT) here.

@jamesbwhite: Do you have an option to compile and run https://sourceforge.net/p/gnucobol/patches/_discuss/thread/74d7e91f5c/8c96/attachment/testcrlf.pco (just rename as needed) on big iron and check what IBM does here?

[Mickey White]

I will look at it tomorrow.

[Mickey White]

@sf-mensch
The program had to be changed for the FD section.
Line-sequential support is limited to z/OS UNIX files. ( I am not running on the Linux OS side).
Below is the only changes I made. Then I listed the output of each file. Remember EBCDIC here.

FILE-CONTROL.                        
    SELECT FILE1                     
        ASSIGN TO FILE1.             
    SELECT FILE2                     
        ASSIGN TO  FILE2.            
    SELECT FILE3                     
        ASSIGN TO FILE3.             
    SELECT FILE4                     
        ASSIGN TO FILE4.             
    SELECT FILE5                     
        ASSIGN TO  FILE5.            
DATA DIVISION.

File1
=COLS> ----+----1----+----2----+----3
** ***** Top of Data ****
==MSG> -CAUTION- Data contains invalid (non-display) characters. Use command
==MSG> ===> FIND P'.' to position cursor to these
000001 A
C44444444444444444444444444444
100000000000000000000000000000
------------------------------------------------------------------------------
000002 µ
005A00000000000000000000000000
05F000000000000000000000000000
------------------------------------------------------------------------------
** **** Bottom of Data ****

File2
=COLS> ----+----1----+----2----+----3
** ***** Top of Data ****
==MSG> -CAUTION- Data contains invalid (non-display) characters. Use command
==MSG> ===> FIND P'.' to position cursor to these
000001 » ;Q
000002 P
000003
000004 1
000005
000006 2
000007 1
000008 2
000009
000010 1
000011
000012 2
000013
000014 0
000015
000016 0
** **** Bottom of Data ****

File3
=COLS> ----+----1----+----2----+----3
** ***** Top of Data ****
==MSG> -CAUTION- Data contains invalid (non-display) characters. Use command
==MSG> ===> FIND P'.' to position cursor to these
000001 A
000002 ;&
** **** Bottom of Data ****

File4
=COLS> ----+----1----+----2----+----3
** ***** Top of Data ****
000001 1P
000002 1
000003 02
** **** Bottom of Data ****

File5
=COLS> ----+----1----+----2----+----3
** ***** Top of Data ****
000001 1P
000002 1
000003 02
** **** Bottom of Data ****

[Mickey White]

Oh, you can download the latest COBOL IBM manuals (language reference and programmer guide and more, ) here:
https://public.dhe.ibm.com/software/zseries/cobol/docs/zOSCOBOLV6R2/en/cobolv6r2_en.zip

[Ron Norman]

The data file format on z/OS will be very different and is not a valid comparison to what Microfocus (or GnuCOBOL) implement on Windows & Linux/Unix.
On the IBM mainframe the data file structure is manage by the operating system and not the COBOL compiler.

[Ron Norman]

Since using WRITE BEFORE/AFTER with 'RECORD' SEQUENTIAL format files using Microfocus COBOL creates a data file which is not possible for even Microfocus to read back correctly, we should consider this to be a bug.
I see only two possible solutions.
1) if WRITE BEFORE/AFTER is used with a file of unspecified format then process the file as LINE SEQUENTAIL for writting. However you would not be able to read file correctly as RECORD SEQUENTIAL.
2) If WRITE BEFORE/AFTER is used then insert the number of empty (spaces or low-values) fixed size records so that the file format remains correct as a RECORD SEQUENTIAL data file.

I think option 2 is a better solution to this problem.

[Simon Sobisch]

We've missed the main point of this issue so far 8-|
There are multiple sequential file types available:

• plain "record sequential", in other places named "fixed format record sequential"
• line sequential
• record sequential, with line advancing "suitable for a printer", in other places named "printer sequential file", in ACUCOBOL (which does not have LINE ADVANCING you only get there if the device-name is PRINTER orPRINTER-1, which is also documented with MF, but there at the WRITE statement as "listing devices")

The rules for 2 are clear and now correctly handled with a changed default in 4.x to split the lines (per COBOL 202x and MF). A file type 1 should be possible to read and write within the COBOL system - also per MF docs, so I'd say if it provides a file different then either it has an undocumented feature to "imply" LINE ADVANCING as soon as a WRITE BEFORE/AFTER is seen (can someone test this with CALLFH, please and check for the line-advancing attribute of the file?) or has a bug.
Type 3 is a special version of type 2.

TODOs:

1. check at MF with CALLFH if a RECORD SEQUENTIAL file without LINE ADVANCINGbut with WRITE AFTER 2 has the line advancing bit set (mostly to know).
2. In GnuCOBOL:
   a. set LINE ADVANCING bit also if the device-name PRINTER/PRINTER-1 is used.
   b. set LINE ADVANCING bit also if an ADVANCING phrase is used at the WRITE
3. check GnuCOBOL's use of the line advancing flag, it seems to be suspiciously... unused; for example verify that this flag is used to create a file starting with 0x0d ( OPEN OUTPUT, likely also on OPEN EXTEND) and that each write (maybe not on ADVANCING 0? ) ends with an 0x0d and neither 0x0d nor 0x0a are depending on the operating system here.
4. remove before/after handling in record sequential - because those only apply with line-advancing - a special version of line-sequential
5. check if/where the testsuite needs adjustments

Because of "line advancing" being internally "special line sequential" the patch part for this in "record sequential" will be obsolete.

[Ron Norman]

Only 'LINE SEQUENTIAL' with an optional ADVANCING option is suitable for a PRINTER.
A RECORD SEQUENTIAL file is never intended to be sent to a PRINTER.

[Simon Sobisch]

MF docs say that a RECORD SEQUENTIAL with LINE ADVANCING clause (which is what the test uses) is a printer record sequential file - that was the main point of my long post (and that this isn't the case for GnuCOBOL yet).

[Ron Norman]

Agreed. LINE ADVANCING implies LINE SEQUENTIAL.
I will check and make sure that is the case for GnuCOBOL.

But the additional test case which has no mention of the file organization at all but is supposed to default to SEQUENTIAL (and I believe RECORD SEQUENTIAL) is the issue.
Usnig WRITE BEFORE/AFTER with a RECORD SEQUENTIAL file does not appear to be correctly implemented in either Microfocus or GnuCOBOL as currently such a file is NOT able to be read back and retrieve the same data records as had been written.

[Simon Sobisch]

For ACUCOBOL the "listing devices" are the option to specify LINE ADVANCING (please add this for GnuCOBOL's PRINTER and PRINTER-1 handling, too [adjustment on cobc side]).

ACUCOBOL docs makes it explicit, MF implicit that a file which uses WRITE BEFORE/AFTER automatically is LINE ADVANCING, too - in this case there is no "bug" in MF, because those clauses always say "that's not 'plain record sequential'" (not yet for GnuCOBOL, which I think it should be).

[Mickey White]

At my work, we are using GnuCobol 1.1 and this is what we have based programs on, (from the 1.1 programmers guide):
3. Files declared as ORGANIZATION LINE SEQUENTIAL will consist of records terminated by an ASCII line-feed character (X”10”). When reading a LINE SEQUENTIAL file, records in excess of the size implied by the file’s FD will be truncated while records shorter than that size will be padded to the right with the PADDING CHARACTER value.

[Sjaak Westdijk]

We extended the test somewhat more. And find out that MF makes a difference between WRITE (which will be hard set WRITE BEFORE in GnuCOBOL) and WRITE BEFORE in the way the handle the <cr><lf>. So I disabled the hard set of WRITE_BEFORE and add some extra tests. So with the attached patches all files from MG and GnuCOBOL are equal. </lf></cr>