#4226 Valid UTF-16 numeric string escape loses its high byte

[Christopher Bazley]

1. Sample code that reproduces the problem.

unsigned int object[] = u"\x100";

1. Exact command used to run SDCC on this sample code.

sdcc --c1mode -mz80 --std=c2y -o bug.asm < bug.c

The source is supplied directly as preprocessed input. No preprocessor is run.

1. SDCC version tested.

SDCC 4.6.4 #17081 (Linux)

1. Observed and expected behaviour.

The compiler exits successfully without diagnostics. Its Z80 assembly contains:

__xinit__object:
    .byte 0,0
    .byte 0,0

The first UTF-16 code unit should have value 256, followed by the zero terminator.
The expected little-endian bytes are 0,1,0,0. The escape is representable as
char16_t; this is valid source, not a required-diagnostic case.

This is separate from the missing range diagnostic in [bugs:#4179]; this sample is within range and must preserve its value.

1. Additional checks.

Using u"\u0100" emits 0,1,0,0 as expected. Using u"A" emits 65,0,0,0.
All three variants were compared on unmodified r17081 and a standalone candidate bugfix for [bugs:#4179]. The numeric-escape defect remains on both. Sources, commands, return codes, logs and generated assembly are in evidence/direct/wide-numeric-escape.json and adjacent files. This observation is based on emitted data; no runtime execution is claimed for this reproducer.

Implementation observation: the existing string decoding path appears to narrow numeric escapes before constructing the wide elements.

[Christopher Bazley]

Please consider the attached candidate patch to fix this bug. The patch looks large, but that is mostly because an existing function, utf_32_from_utf_8 was moved from src/SDCCval.c to src/SDCCutil.c.
An existing comment already said:

// TODO: Move this function to SDCCutil?

Regression tests passed on ucz80. valdiag tests passed on all targets.

The candidate shares escape parsing between byte and wide strings. The wide path produces 32-bit elements directly, preserving numeric code units without a UTF-8 round trip. Numeric UTF-16 escapes retain one 16-bit code unit, including a surrogate value; UTF-32 and SDCC's wchar_t literals retain all 32 bits. Ordinary/u8 strings and inline assembly keep the existing byte path.

The patch includes ChangeLog and a six-case regression test. It changes no diagnostic messages or valdiag expectations and needs no part of the fix for [bugs:#4179]. Both candidates touch string decoding, so the pending [bugs:#4179] patch will need reconciliation if this one is merged first.

[Philipp Klaus Krause]

How about moving the function to SDCCutil in a first step (along with utf_16_from_utf_32), then look at the smaller patch for the bugfix?

[Christopher Bazley]

How about moving the function to SDCCutil in a first step (along with utf_16_from_utf_32), then look at the smaller patch for the bugfix?

Good idea. The split patches are attached.
The first moves utf_32_from_utf_8 and utf_16_from_utf_32 to SDCCutil.c. Function bodies are unchanged; only static linkage and the obsolete TODO comments are removed. Both declarations are added to SDCCutil.h.

The second uses parseEscape and escapeValueInRange from the patch for [bugs:#4179] before storing wide numeric code units. Ordinary/u8 strings keep copyStrWithEscapeLimit; inline assembly keeps unrestricted copyStr. A shared internal decoder supports byte and wide output. Wide numeric escapes no longer pass through a byte buffer or get interpreted as UTF-8. All 16-bit numeric units, including surrogate values, are preserved. Existing Unicode-range warnings for universal escapes are also retained. Numeric UTF-32 values above the Unicode range remain permitted; universal escapes and numeric escapes have different rules.

[Philipp Klaus Krause]

First patch is ok; I'll look at the second later this week.

[Christopher Bazley]

First patch is ok; I'll look at the second later this week.

Thanks for the review. The first patch is now committed as [r17136].