Example:

   encoding convertfrom utf-8 \x3c\xc0\xbc
   <<

Most UTF-8 decoders handle this the same as other invalid UTF-8 sequences: Just output valid UTF-8 corresponding with the individual bytes. The original example then becomes:

   encoding convertfrom utf-8 \x3c\xc0\xbc
   <À¼

The characters "À¼" are 'safe' in HTML, since no characters > \x7f have a special meaning.

See also

https://core.tcl.tk/tcl/tktview?name=2564363

From TCLCORE:

"Tcl_UtfToUniChar() is documented as:
    <http://core.tcl.tk/tcl/artifact?ln=142-145&name=eaf0d058d9f17309>
Text:
    If the input is not in proper UTF-8 format, \fBTcl_UtfToUniChar\fR will
    store the first byte of \fIsrc\fR in \fI*chPtr\fR as a Tcl_UniChar
    between 0x0000 and 0x00ff and return 1.

This turns out to be a lie: Some byte-sequences, which are clearly
non-proper according to the UTF-8 standard, this function returns
something else than the first byte. That's what I'm proposing to fix."

Thank you, that's the clear explanation I needed to see.  This is indeed
a good change to make.  Merged to core-8-6-branch. Will further merge
forward.

Seems this has been contrary to documentation since it arrived
in Tcl 8.1.  Possible explanation is a changing standard of
what is considered "proper UTF-8 format" and whether multiple
encodings are permissible for the same codepoint?  Clear standard
now is that they are not.

One thing that might still be improved (?) is to mention the
exception \xc0 \x80 encoding for U+0000 that the routine
explicitly makes.

The security people (I hope I may say belonging to those to some extent) say - it is rather undefined behavior, but not directly a kind of vulnerability in sense of "tcl has possible security-leak with grave impact".

The fact is: if you read something from foreign input and want to escape/validate it somehow, you should act in the target encoding (and never with the raw input). This is one of the first rules, that will be applied by security-audits.

I mean that it should be converted previously... Or twice (before and hereafter).

Just as example:

% puts [set buf "\xc0\xbcscript ..."]
À¼script ...
% set buf [encoding convertfrom utf-8 $buf]; #** important before validate **
<script ... 
% if {[regexp -nocase {<script} $buf]} {error NOT-PERMITTED} 
NOT-PERMITTED 
</code>

Just as some thoughts for your info. I'm doing already too much sec-audits (including the source code).

Don't misunderstand my amount, I'm not with Don here (in contrary the changes looks good to me), but... But I do not see this necessarily as argument for theoretical vulnerability to fix ASAP. But also I don't see still any reasons to revert it ASAP.

Excerpt from the wiki:

More recent converters translate the first byte of an invalid sequence to a replacement character and continue parsing with the next byte. These error bytes will always have the high bit set. This avoids denial-of-service bugs, and it is very common in text rendering such as browser display, since mangled text is probably more useful than nothing for helping the user figure out what the string was supposed to contain. Popular replacements include:

- The replacement character "�" (U+FFFD) (or EF BF BD in UTF-8)

- The invalid Unicode code points U+DC80–U+DCFF where the low eight bits are the byte's value.[16] Sometimes it is called UTF-8B[17]

- The Unicode code points U+0080–U+00FF with the same value as the byte, thus interpreting the bytes according to ISO-8859-1[citation needed]

- The Unicode code point for the character represented by the byte in CP1252,[citation needed] which is similar to using ISO-8859-1, except that some characters in the range 0x80–0x9F are mapped into different Unicode code points. For example, 0x80 becomes the Euro sign, U+20AC.

Tcl uses the 3th possibility for all other invalid sequences it checks for, so it makes most sense to do the same here.

I think the UTF-8 standard is absolutely clear about what are invalid byte sequences and what not! Tcl already handles many invalid byte sequences, just these ones have always been forgotten.

See: https://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences

Currently we only handle 1-3.

Actually, I thought that Tcl already handled this long ago. Fossil has a function invalid_utf8() which does the same checks.

Please talk to security people, it's embarrassing not to handle this nowadays, really!

This is far too big a change to make without talking about it.

Please revert.

Fixed in core-8-6-branch and trunk