#1277 National (PIC N) data: MOVE, SPACE fill, INITIALIZE, item size and comparisons work on bytes

[Brent Rector]

National (PIC N) data is wrong in six places. A national item is stored as UTF-16 big-endian (N"CD" is 00 43 00 44), but MOVE, the space fill, INITIALIZE, the item size and the comparison routines treat its storage as alphanumeric bytes. The six cases:

1. MOVE of an alphanumeric item to a national item copies the bytes without conversion.
2. The SPACE fill of a national item stores 0x20 bytes, not national spaces.
3. INITIALIZE of a national item generates no code.
4. PIC N(n) USAGE NATIONAL is twice the size of PIC N(n).
5. Comparing national operands of unequal length pads with 0x20 bytes.
6. Comparing an alphanumeric or integer operand with a national operand compares raw bytes.

Each case gives a different wrong result and needs its own fix, so each has its own section. Only case 4 has a patch.

How the programs were run

Every program below compiles with exit code 0. With default warnings, cobc prints only -Wunfinished warnings ("handling of USAGE NATIONAL is unfinished; implementation is likely to be changed" and "handling of national literal is unfinished; implementation is likely to be changed"), with source context. To keep this report short, the commands add -Wno-unfinished. The run output is the same with or without that option. With it, the 3.2.0 compile prints only these two lines, which come from the Ubuntu package's C compiler flags:

<command-line>: warning: ‘_FORTIFY_SOURCE’ redefined
<command-line>: note: this is the location of the previous definition

and the 3.3-dev compile prints nothing. For 3.3-dev the commands are the same, using that build's cobc, with LD_LIBRARY_PATH pointing at its libcob.

1. MOVE alphanumeric to national copies the bytes without conversion

mvnat.cob (attached):

IDENTIFICATION DIVISION.
PROGRAM-ID. MVNAT.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 NAT PIC N(2).
PROCEDURE DIVISION.
MOVE "CD" TO NAT
IF NAT = N"CD" DISPLAY "EQUAL" ELSE DISPLAY "NOT EQUAL" END-IF
STOP RUN.

What happens

On 3.2.0, cobc -std=cobol2014 -fformat=free -Wno-unfinished -x mvnat.cob && ./mvnat. The compile exits 0 and the run exits 0:

NOT EQUAL

3.3-dev r5751 gives the same output, exit 0.

natbytes.cob (attached) displays four national items through PIC X(4) REDEFINES: MOVED after MOVE "CD", then VALUE N"CD", VALUE SPACE and VALUE N" ". ./natbytes | od -An -tx1 prints the following, byte-identical on both versions:

 43 44 20 20 0a 00 43 00 44 0a 20 20 20 20 0a 00
 20 00 20 0a

After the MOVE the item holds 43 44 20 20, while GnuCOBOL's own N"CD" is 00 43 00 44. Moving a PIC X(2) VALUE "CD" data item gives the same result.

What should happen

The program prints EQUAL. ISO/IEC 1989:2023:

• §14.9.25.3 syntax rule 10: "table 16, Validity of types of MOVE statements, specifies the validity of the move." Table 16 lists alphanumeric to national as valid.
• §14.9.25.4 general rule 6: "Any necessary conversion from alphanumeric character to national character representation shall be performed, before any alignment, in accordance with a correspondence defined by the implementor."
• §14.6.8.5: "The data in the sending operand is transferred, after any necessary conversion, to the receiving character positions and aligned at the leftmost character position in the data item with space fill or truncation to the right, as required."

The correspondence is GnuCOBOL's to define. But it maps characters to characters, and two alphanumeric characters packed into one national character position is not such a mapping.

Where the fix belongs

The generated C for line 7 is cob_move ((cob_field *)&c_1, &f_1);. cob_move in libcob/move.c has no case for a national destination. An alphanumeric source falls to the default arm and calls cob_move_alphanum_to_alphanum, which copies the bytes and pads with alphanumeric spaces.

2. SPACE fill of a national item stores 0x20 bytes, not national spaces

nspace.cob (attached):

IDENTIFICATION DIVISION.
PROGRAM-ID. NSPACE.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 NV PIC N(1) VALUE SPACE.
01 NM PIC N(2).
01 NP PIC N(2).
PROCEDURE DIVISION.
MOVE SPACES TO NM
MOVE N"A" TO NP
IF NV = N" "  DISPLAY "1 EQUAL" ELSE DISPLAY "1 NOT EQUAL" END-IF
IF NM = N"  " DISPLAY "2 EQUAL" ELSE DISPLAY "2 NOT EQUAL" END-IF
IF NP = N"A " DISPLAY "3 EQUAL" ELSE DISPLAY "3 NOT EQUAL" END-IF
STOP RUN.

All three comparisons are between operands of equal length, so the padding in case 5 does not come into it.

What happens

On 3.2.0, cobc -std=cobol2014 -fformat=free -Wno-unfinished -x nspace.cob && ./nspace. The compile exits 0 and the run exits 0:

1 NOT EQUAL
2 NOT EQUAL
3 NOT EQUAL

3.3-dev r5751 gives the same output, exit 0.

The natbytes.cob output in case 1 shows VALUE SPACE as 20 20 20 20 and VALUE N" " as 00 20 00 20.

What should happen

The program prints 1 EQUAL, 2 EQUAL, 3 EQUAL. ISO/IEC 1989:2023:

• §8.3.3.6.4 general rule 1: "When a figurative constant is used in a context requiring national characters, the figurative constant represents a national character value."
• §14.6.8.5 (quoted in case 1): the receiver is aligned "with space fill ... as required". The receiver's character positions are national, so the fill character is the national space.
• §8.8.4.2.10 general rule 1: "The operands are determined to be equal if all pairs of corresponding national characters are equal."

Where the fix belongs

In cobc/codegen.c, output_initialize_to_value handles a cb_space VALUE with output_figurative (..., ' ', ...), a byte fill, whatever the field's category. In libcob/move.c, cob_move_alphanum_to_alphanum pads with ' ', and cob_move sends a national receiver there for cob_all_space and for a shorter sender. A national-to-national move that pads with the national space is also needed. This fix has to land with the comparison changes in cases 5 and 6; see "Cases 1, 2, 3, 5 and 6 have no patch" below.

3. INITIALIZE of a national item generates no code

ninit.cob (attached):

IDENTIFICATION DIVISION.
PROGRAM-ID. NINIT.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 NAT PIC N(1) VALUE N"A".
01 GRP.
   05 GN PIC N(1) VALUE N"A".
   05 GX PIC X(1) VALUE "A".
PROCEDURE DIVISION.
INITIALIZE NAT GRP
IF NAT = N"A" DISPLAY "NAT UNCHANGED" ELSE DISPLAY "NAT CHANGED" END-IF
IF GN = N"A" DISPLAY "GN UNCHANGED" ELSE DISPLAY "GN CHANGED" END-IF
IF GX = "A" DISPLAY "GX UNCHANGED" ELSE DISPLAY "GX CHANGED" END-IF
STOP RUN.

What happens

On 3.2.0, cobc -std=cobol2014 -fformat=free -Wno-unfinished -x ninit.cob && ./ninit. The compile exits 0 and the run exits 0:

NAT UNCHANGED
GN UNCHANGED
GX CHANGED

3.3-dev r5751 gives the same output, exit 0.

The alphanumeric GX is space-filled. The national items, both the elementary NAT and GN inside the group, are left as they were. In the C that cobc -C generates (the same on both versions), the whole statement is one store, for GX:

  /* Line: 10        : INITIALIZE         : ninit.cob */
  *(cob_u8_ptr)(b_2 + 2) = (unsigned char)' ';

The output is the same with -std=default, cobol2002, ibm and mf, on both versions.

What should happen

The program prints NAT CHANGED, GN CHANGED, GX CHANGED. ISO/IEC 1989:2023:

• §14.9.20.4 general rule 4: "the effect of the execution of the INITIALIZE statement is as though a series of implicit MOVE or SET statements" were executed, each with an elementary receiving item.
• §14.9.20.4 general rule 5 c) 4.: an elementary item referenced by identifier-1, or contained in the group identifier-1 references (rule 5 b)), is a receiving operand when "Neither the REPLACING phrase nor the VALUE phrase is specified."
• §14.9.20.4 general rule 6, the table of sending operands: National, "Figurative constant national SPACES".

Where the fix belongs

In cobc/codegen.c, initialize_uniform_char returns -1 for a national field. For INITIALIZE_DEFAULT, output_initialize then falls through to output_initialize_compound, which only visits children. An elementary field has no children, so nothing is emitted, and a national child of a group takes the same path. The fill that a fix produces must be the national space (case 2): output_initialize_to_default moves cb_space.

4. PIC N(n) with an explicit USAGE NATIONAL is twice the size of PIC N(n)

natuse.cob (attached):

IDENTIFICATION DIVISION.
PROGRAM-ID. NATUSE.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 IMPLICIT-N PIC N(4).
01 EXPLICIT-N PIC N(4) USAGE NATIONAL.
PROCEDURE DIVISION.
MOVE N"ABCDEFGH" TO IMPLICIT-N EXPLICIT-N
DISPLAY FUNCTION LENGTH(IMPLICIT-N) " " FUNCTION LENGTH(EXPLICIT-N)
DISPLAY FUNCTION BYTE-LENGTH(IMPLICIT-N) " " FUNCTION BYTE-LENGTH(EXPLICIT-N)
IF IMPLICIT-N = N"ABCD" DISPLAY "IMPLICIT EQUAL" ELSE DISPLAY "IMPLICIT NOT EQUAL" END-IF
IF EXPLICIT-N = N"ABCD" DISPLAY "EXPLICIT EQUAL" ELSE DISPLAY "EXPLICIT NOT EQUAL" END-IF
STOP RUN.

The 8-character sender fills both receivers after truncation, so the space fill in case 2 does not come into it.

What happens

On 3.2.0, cobc -std=cobol2014 -fformat=free -Wno-unfinished -x natuse.cob && ./natuse. The compile exits 0 and the run exits 0:

4 8
8 16
IMPLICIT EQUAL
EXPLICIT NOT EQUAL

3.3-dev r5751 gives the same output, exit 0.

What should happen

The program prints 4 4, then two equal byte lengths (their value is the implementor's), IMPLICIT EQUAL, EXPLICIT EQUAL. ISO/IEC 1989:2023:

• §13.18.60.3 syntax rule 13 a): "if the explicit or implicit picture character-string contains the symbol 'N', a USAGE NATIONAL clause is implied". So PIC N(4) and PIC N(4) USAGE NATIONAL describe the same item.
• §13.18.40.4 general rule 14: "Each symbol 'N' represents a national character position ... Each symbol 'N' is counted in the size of the item."
• §15.50.4 returned value rule 2: for an elementary item of usage national, LENGTH is "an integer equal to the length of argument-1 in national character positions".
• §14.6.8.5 (quoted in case 1): the move truncates on the right, leaving N"ABCD".

Why it happens

cb_build_picture in cobc/tree.c already counts each 'N' as COB_NATIONAL_SIZE bytes (size += n * (COB_NATIONAL_SIZE - 1)). Then, in the CB_USAGE_NATIONAL case of compute_size in cobc/field.c, f->size = f->pic->size * COB_NATIONAL_SIZE multiplies by it again. An item with no USAGE clause keeps usage DISPLAY and is sized once. The ChangeLog entry of 2023-01-30 ("actually pass NATIONAL usage if specified") appears to be the change that sends the explicit form down the doubling path.

Proposed fix

In the CB_USAGE_NATIONAL case of compute_size, a PICTURE of category national takes the picture size as it is, which is what an item without the USAGE clause gets. Other pictures with USAGE NATIONAL, such as PIC 9(2) USAGE NATIONAL, are still multiplied by COB_NATIONAL_SIZE, because their picture size counts each symbol once.

The patch is national-storage-move-compare.patch (attached). It is against 3.3-dev r5751 and applies with patch -p1 at the top of the source tree. With the patch, the same commands give, for natuse:

4 4
8 8
IMPLICIT EQUAL
EXPLICIT EQUAL

The compile and the run exit 0. The other six programs give the same output as without the patch.

make check on 3.3-dev r5751 with the patch: 1,373 tests behaved as expected, 9 were skipped, 0 failed. Unpatched, 1,372 behaved as expected, 9 were skipped and 0 failed; the difference is the new test, which passes. On the unpatched build its program reports a LENGTH of 08, a BYTE-LENGTH of 16 against 08, and a wrong result of the MOVE. The patch adds the testsuite case "FUNCTION LENGTH and BYTE-LENGTH of PIC N USAGE NATIONAL" in run_functions.at and ChangeLog entries in cobc and tests.

5. Comparing national operands of unequal length pads with 0x20 bytes

ncmppad.cob (attached):

IDENTIFICATION DIVISION.
PROGRAM-ID. NCMPPAD.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 SHORT-N PIC N(1) VALUE N"A".
01 LONG-N  PIC N(3) VALUE N"A  ".
PROCEDURE DIVISION.
IF LONG-N = N"A  " DISPLAY "1 EQUAL" ELSE DISPLAY "1 NOT EQUAL" END-IF
IF SHORT-N = LONG-N DISPLAY "2 EQUAL" ELSE DISPLAY "2 NOT EQUAL" END-IF
IF N"A" = LONG-N DISPLAY "3 EQUAL" ELSE DISPLAY "3 NOT EQUAL" END-IF
STOP RUN.

Every item holds the characters of its own national VALUE literal. Line 1 is the equal-length control.

What happens

On 3.2.0, cobc -std=cobol2014 -fformat=free -Wno-unfinished -x ncmppad.cob && ./ncmppad. The compile exits 0 and the run exits 0:

1 EQUAL
2 NOT EQUAL
3 NOT EQUAL

3.3-dev r5751 gives the same output, exit 0.

What should happen

The program prints 1 EQUAL, 2 EQUAL, 3 EQUAL. ISO/IEC 1989:2023:

• §8.8.4.2.9: "An operand of class national may be compared with another operand of class national."
• §8.8.4.2.10 general rule 2: "comparison proceeds as though the shorter operand were extended on the right by sufficient national spaces to make the operands of equal length."
• §8.8.4.2.10 general rule 1: "The operands are determined to be equal if all pairs of corresponding national characters are equal."

Only equality is tested, so the result does not depend on the national collating sequence.

Where the fix belongs

cob_cmp in libcob/common.c has its national branch under #if 0 ("FIXME later: must cater for national fields, too"). Two national operands therefore reach cob_cmp_alnum, which compares the excess of the longer operand with COB_SPACES_ALPHABETIC through compare_spaces, or with ' ' on the collating path. The excess of a national operand holds 00 20, which never matches 20 20. Padding with national spaces here has to land with the space-fill fix in case 2; see "Cases 1, 2, 3, 5 and 6 have no patch" below.

6. Comparing an alphanumeric or integer operand with a national operand compares raw bytes

ncmpcnv.cob (attached):

IDENTIFICATION DIVISION.
PROGRAM-ID. NCMPCNV.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 ALPHA-1 PIC X(2) VALUE "AB".
01 NAT-1   PIC N(2) VALUE N"AB".
01 INT-1   PIC 9(4) VALUE 1234.
PROCEDURE DIVISION.
IF NAT-1 = N"AB" DISPLAY "1 EQUAL" ELSE DISPLAY "1 NOT EQUAL" END-IF
IF ALPHA-1 = NAT-1 DISPLAY "2 EQUAL" ELSE DISPLAY "2 NOT EQUAL" END-IF
IF "AB" = NAT-1 DISPLAY "3 EQUAL" ELSE DISPLAY "3 NOT EQUAL" END-IF
IF INT-1 = N"1234" DISPLAY "4 EQUAL" ELSE DISPLAY "4 NOT EQUAL" END-IF
STOP RUN.

No MOVE is executed. Line 1 is the control.

What happens

On 3.2.0, cobc -std=cobol2014 -fformat=free -Wno-unfinished -x ncmpcnv.cob && ./ncmpcnv. The compile exits 0 and the run exits 0:

1 EQUAL
2 NOT EQUAL
3 NOT EQUAL
4 NOT EQUAL

3.3-dev r5751 gives the same output, exit 0.

What should happen

All four lines print EQUAL. ISO/IEC 1989:2023:

• §8.8.4.2.6: "The alphanumeric operand is treated as though it were converted and moved in accordance with the rules of the MOVE statement from an alphanumeric elementary data item to a temporary elementary data item of class national with the same length in terms of character positions as the alphanumeric operand." (lines 2 and 3)
• §8.8.4.2.5: "The numeric integer operand shall be an integer literal or an integer numeric data item of usage display or national." and "The integer operand is treated as though it were moved, according to the rules of the MOVE statement, to an elementary data item of the same length in terms of character positions as the number of digits in the integer, and of the same class and usage as the alphanumeric or national operand". 1234 compares as N"1234" (line 4).
• §8.8.4.2.10 general rule 1 (quoted in case 2).

Where the fix belongs

cob_cmp in libcob/common.c, the same #if 0 branch as in case 5. The code after it ("all else -> alphanumeric comparision") sends a mixed pair to cob_cmp_alnum, a byte comparison. The non-national operand has to be converted to national first, using the same conversion that case 1 needs for MOVE. Fixing MOVE alone does not fix this path.

Cases 1, 2, 3, 5 and 6 have no patch

These five defects depend on each other, so a fix for one of them alone changes results that are right today only because two wrong byte representations agree. With 3.3-dev r5751 and the same command:

IDENTIFICATION DIVISION.
PROGRAM-ID. NACC.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 SHORT-N PIC N(1) VALUE N"A".
01 LONG-N  PIC N(3).
PROCEDURE DIVISION.
MOVE N"A" TO LONG-N
IF SHORT-N = LONG-N DISPLAY "EQUAL" ELSE DISPLAY "NOT EQUAL" END-IF
IF LONG-N = SPACES DISPLAY "SP" ELSE DISPLAY "NOT SP" END-IF
MOVE SPACES TO LONG-N
IF LONG-N = SPACES DISPLAY "SP" ELSE DISPLAY "NOT SP" END-IF
STOP RUN.

prints

EQUAL
NOT SP
SP

All three results are correct, but only because the MOVE pads with 0x20 bytes and the comparisons pad and compare with 0x20 bytes. Fixing the comparison padding (case 5) alone turns the first line into NOT EQUAL, and fixing the space fill (case 2) alone turns the last line into NOT SP.

A consistent fix changes these together:

• the padding and figurative SPACE fill in libcob/move.c (cob_move_alphanum_to_alphanum, cob_move_all);
• the VALUE and INITIALIZE code in cobc/codegen.c (output_initialize_to_value, initialize_uniform_char);
• the national branch of cob_cmp together with cob_cmp_all in libcob/common.c, which compares a figurative constant byte by byte.

Cases 1 and 6 also need a correspondence from alphanumeric to national characters. The scanner's "poor-man's conversion iso-8859 -> utf-16" for national literals is one GnuCOBOL already uses, but which one to use at run time is for the maintainers to choose. We therefore name the sites and attach no patch for these cases.

Related

No existing ticket reports any of these. cobc's own -Wunfinished warning ("handling of USAGE NATIONAL is unfinished") and the FIXME in cob_cmp show that national support is known to be incomplete in general.

Details

Versions: GnuCOBOL 3.2.0 (Ubuntu gnucobol3 3.2-4build1) and 3.3-dev r5751 (CI snapshot 3.3-dev.138 of 2026-10-29), x86_64 Linux. All runs were repeated on 2026-10-30.

Attached: mvnat.cob, natbytes.cob, nspace.cob, ninit.cob, natuse.cob, ncmppad.cob, ncmpcnv.cob, one per case, and national-storage-move-compare.patch, a proposed patch for case 4.

Editions: the rules quoted above are from ISO/IEC 1989:2023. Each is also in the committee draft of the 2014 edition (CD 1.2, 2009) and in the 2002 edition's FCD, so all seven programs are valid under -std=cobol2014. National data is a 2002 feature.

How the fix sites were found: we read the r5751 source and the C that cobc -C generates. The patch confirms the site for case 4. For the other cases we did not trace them in a debugger, so they are inferred.