#4711 regsub example doesn't perform as advertized

[Jasper]

It's this one:

Convert all non-ASCII and Tcl-significant characters into \u escape sequences by using regsub and subst in combination:

# This RE is just a character class for everything "bad"
set RE {[][{};#\\\$\s\u0080-\uffff]}

# We will substitute with a fragment of Tcl script in brackets
set substitution {[format \\\\u%04x [scan "\\&" %c]]}

# Now we apply the substitution to get a subst-string that
# will perform the computational parts of the conversion.
set quoted [subst [regsub -all $RE $string $substitution]]

===========================================
The RE for "bad" characters includes \s which matches all whitespace characters including newline. However, inserting \ before newline before calling subst on a string containing it does not preserve the newline, it causes it to be replaced by a space, so the whole procedure replaces newlines with \u0020

Not sure how to get this to work properly! Little help?

[Donal K. Fellows]

Good catch. Newlines need to be handled specially. :-(

[Donal K. Fellows]

Updated example text is as below.

# This RE is just a character class for almost everything "bad"
set RE {[][{};#\\\$ \r\t\u0080‐\uffff]}

# We will substitute with a fragment of Tcl script in brackets
set substitution {[format \\\\u%04x [scan "\\&" %c]]}

# Now we apply the substitution to get a subst‐string that
# will perform the computational parts of the conversion. Note
# that newline is handled specially through string map since
# backslash‐newline is a special sequence.
set quoted [subst [string map {\n {\\u000a}} \ [regsub -all $RE $string $substitution]]]

[Donal K. Fellows]

Fixed in HEAD and 8.5 branch (alas, just missed the train for 8.5.9...)