xstrings Files

This release fixes invalid UTF-8 handling in ExpandTabs, Squeeze and the justification helpers, and makes FirstRuneToUpper / FirstRuneToLower honor all Unicode case mappings.

Most fixes preserve bytes which were not supposed to be changed. As a result, invalid UTF-8 bytes are no longer rewritten as U+FFFD in these functions; valid UTF-8 output is unchanged.

Bug fixes

ExpandTabs: invalid bytes after the first tab

ExpandTabs copied the input prefix before the first tab, but every later rune was written back with WriteRune. Invalid bytes decode to utf8.RuneError and were rewritten as U+FFFD, so the same byte was preserved before a tab and corrupted after it.

• ExpandTabs("\t\xffx", 4) returned " \uFFFDx" instead of " \xffx".
• ExpandTabs("\xff\t\xfe", 4) returned "\xff \uFFFD" instead of "\xff \xfe".

Non-tab bytes are now copied unchanged.

Squeeze: retained bytes and duplicate detection

Two separate issues affected invalid input:

• After the first squeeze, retained runes were written with WriteRune, rewriting invalid bytes as U+FFFD: Squeeze("aa\xffb", "a") returned "a\uFFFDb" instead of "a\xffb", and even a non-matching pattern such as "z" changed the bytes (#73).
• Duplicate detection compared decoded runes. Every invalid byte decodes to RuneError, so distinct invalid bytes, and an invalid byte next to a valid U+FFFD, were treated as repeated runes and collapsed: Squeeze("\xff\xfe", "") returned "\xff" and Squeeze("\xff\uFFFD", "") returned "\xff" (#76).

Retained bytes are now copied unchanged, and adjacent runes are compared by their original bytes. Identical bytes still squeeze normally when the rune is eligible for squeezing, while distinct sequences are kept.

LeftJustify, RightJustify, Center: truncated pad strings

When the pad string was cut in the middle of its runes, writePadString decoded each pad rune and wrote it back with WriteRune. Invalid bytes in that partial pad became U+FFFD, although complete pad repetitions were already copied unchanged.

• LeftJustify("a", 3, "x\xffy") returned "ax\uFFFD" instead of "ax\xff".
• RightJustify("a", 2, "\xffx") returned "\uFFFDa" instead of "\xffa".
• Center("a", 4, "x\xffy") returned "xax\uFFFD" instead of "xax\xff".

Partial pad bytes are now copied unchanged (#75).

FirstRuneToUpper / FirstRuneToLower: Unicode case mappings outside Ll/Lu

The functions only changed the first rune when unicode.IsLower / unicode.IsUpper was true. Characters with case mappings outside those categories were skipped, including titlecase digraphs, Roman numerals and circled letters.

• FirstRuneToUpper("Džuro") returned "Džuro" instead of "DŽuro".
• FirstRuneToLower("Džuro") returned "Džuro" instead of "džuro".
• FirstRuneToUpper("ⅳ chapter") returned "ⅳ chapter" instead of "Ⅳ chapter".
• FirstRuneToUpper("ⓐ label") returned "ⓐ label" instead of "Ⓐ label".

The functions now apply unicode.ToUpper / unicode.ToLower and return the original string when the first rune has no mapping. Suffixes, uncased runes and invalid UTF-8 are handled as before (#72).

Behaviour changes

• FirstRuneToUpper and FirstRuneToLower now change every rune with a standard Unicode simple case mapping, not only runes in Ll/Lu. This includes titlecase, letterlike and circled characters.
• ExpandTabs, Squeeze, LeftJustify, RightJustify and Center now preserve invalid UTF-8 bytes instead of rewriting them as U+FFFD in the affected paths. The output is byte-preserving, but remains invalid UTF-8 if the input is invalid. Valid UTF-8 results are unchanged.
• Squeeze no longer treats different invalid byte sequences as repeated runes. Repeated identical bytes are still squeezed under the same pattern rules.
• No public API signatures changed.

Tests

• Regression cases for invalid bytes before and after tabs, squeeze points, and partial pad strings.
• Cases for distinct invalid bytes, invalid bytes next to valid U+FFFD, and repeated identical invalid bytes under empty, matching and non-matching patterns.
• Titlecase digraph, Roman numeral and circled-letter cases for both first-rune case functions.
• go test ./..., go test -race ./..., go vet ./..., gofmt and git diff --check pass. Statement coverage remains at 99.9%.

Thanks

Thank you to the contributors of this release:

• [#72] Honor Unicode mappings when changing the first rune case by @ac1982
• [#73] Preserve untouched bytes when squeezing strings by @yuefdev
• [#74] Preserve untouched bytes when expanding tabs by @yuefdev
• [#75] Preserve pad bytes when justifying strings by @huandu
• [#76] Do not treat distinct invalid bytes as repeated runes by @huandu

Full Changelog: https://github.com/huandu/xstrings/compare/v1.6.2...v1.6.3