xstrings Files

Bug fixes for Scrub, Slice, Successor, ToCamelCase and Translate, plus a much bigger test suite.

Bug fixes

Scrub

• When a run of invalid bytes was at the end of the input, Scrub appended the already written prefix a second time and kept the invalid bytes as they were: Scrub("abc\xFF", "*") returned "abcabc\xFF" instead of "abc*", and Scrub("ab\xFFcd\xFE", "*") returned "ab*cdcd\xFE" instead of "ab*cd*" (#60, thanks @vitalivo).

Slice

• Slice with a negative end (i.e. "slice to the end of the string") did not validate start against the rune count, so an out-of-range start returned an empty string instead of panicking: Slice("abc", 4, -1), Slice("中文", 3, -1) and Slice("", 1, -1) all returned "". They now panic with "out of range", as the document promises (#61, thanks @vitalivo).

Successor

• The carry rune generated by '9' was inserted as '0' instead of '1', so Successor("9") returned "00" instead of "10" (and "99" → "000", "9z" → "00a") (#62).
• The prefix in front of an inserted carry rune was sliced with a rune index used as a byte offset, which cut multibyte runes and produced invalid UTF-8: Successor("中z") returned "\xe4aa" instead of "中aa" (#63).
• A carry was allowed to cross a letter/digit boundary after a non-alphanumeric rune, which differs from Ruby's String#succ: Successor("8a_9") returned "8b_0" instead of "8a_10", Successor("1*z") returned "2*a" instead of "1*aa" (#67). A differential test against Ruby over 20000 random ASCII strings now reports zero mismatches.

ToCamelCase / ToPascalCase

• A string which contains connectors only got its last rune duplicated: ToCamelCase("_") returned "__", ToCamelCase(" ") returned " ", ToCamelCase("-_-") returned "-_--" (#64).

Translate / Delete

• Once one rune had been translated, every unmatched rune which decodes to utf8.RuneError was dropped. A valid U+FFFD rune which did not match the pattern disappeared, and invalid bytes were removed as well: Translate("a\uFFFDb", "a", "x") returned "xb" instead of "x\uFFFDb", Delete("a\uFFFDb", "a") returned "b" instead of "\uFFFDb" (#65).
• A mapping whose target rune is U+0000 was silently ignored, because 0 was used as the "no mapping" marker: Translate("hello", "h", "\x00") returned "hello" instead of "\x00ello" (#66).

Behaviour changes

Two results change on purpose. Please check them if you depend on the exact output:

• Translate/Delete now keep every rune which does not match the pattern, including a valid U+FFFD, and invalid bytes are passed through as is instead of being rewritten as U+FFFD. As a consequence, Translate("hel\uFFFDlo", "^\uFFFD", "H") returns "HHH\uFFFDHH" (it returned "HHHHH" before), and Len(Delete(s, pattern)) + Count(s, pattern) == Len(s) now holds for every input.
• Successor stops a carry in front of a rune of another kind (letter vs. number), which is what Ruby's String#succ does. As before, only ASCII runes are treated as alphanumeric (documented in Successor), so for input which mixes CJK and ASCII the rule applies to the ASCII runes: Successor("中cZ英ZZ文zZ混9zZ9杂99进z位") returns "中cZ英ZZ文zZ混9zZ9杂99进aa位" (it returned "中dA英AA文aA混0aA0杂00进a位" before).

Tests

• Statement coverage grows from 97.4% to 99.9%.
• Boundary and regression tests for every exported function.
• The Translator API (HasPattern, TranslateRune, reusing a Translator) is covered by tests now; it had none before.
• Regression tests for all of the fixes listed above.

Thanks

Thank you @vitalivo for the two pull requests in this release:

• [#60] Replace trailing invalid byte runs in Scrub
• [#61] Check the start bound of open-ended rune slices

Full Changelog: https://github.com/huandu/xstrings/compare/v1.5.0...v1.6.0