Starting with checkin

https://core.tcl-lang.org/tcl/info/b5633ba

there is a regression in the function of Tcl_NumUtfChars() in the default build.
Before that checkin:

% lmap i {0 1 2 3 4} {testnumutfchars [testbytestring \xf0\x9f\x92\xa9] $i}
0 1 2 3 4

With that checkin:

% lmap i {0 1 2 3 4} {testnumutfchars [testbytestring \xf0\x9f\x92\xa9] $i}
0 1 2 2 2

Besides the change in results, the new results also suggest that examination
of the buffer is running past the end and reading bytes it should not.

Remainder of the fix committed here: [2a7beb9d187eff66], together with the fix for [ed29806baf]

As a base for further discussion, I now merged the bug-31aa44375d branch to core-8-6-branch. Testcase encoding-15.5 is now marked as "knownBug" because this is IMHO an important feature: The UtfToUtf encoding is meant to fix all kinds or problems when extenal byte sources find their way into Tcl. A 4-byte UTF-8 sequence is currently illegal in Tcl, but it could be legal to the outside world. Therefore it should be translated to a surrogate pair, so Tcl can handle it as intended. It is possible to fix this using special code in the Utf0toUtf encoder/decoder. This test-case checks for that, and it's broken now. That should be fixed. I'll prepare a solution for that.

So, discussion not over yet, but at least we have a base to talk about.

Fresh start on a bugfix branch building on other reforms and fixes.

See also: a179564826

So, concluding: the main change you don't agree with is this one:

https://core.tcl-lang.org/tcl/info/aed6634d2ccf2107

(there were later commits, refining this, but this was the main one)

Well, I created a "bug-31aa44375d" branch, which basically reverts this commit. It means that internal 4-byte UTF-8 sequences (however hard to generate those, they are still officially "invalid") are handled as 4 separate bytes when stepping over it.

Since this change only involved invalid 4-byte sequences (apart from the bug in Tcl_UtfCharComplete() you found, which is now fixed) I didn't think it needed a TIP.

So far, let's see what Travis thinks of it.

Please keep in mind my bug is filed against a default build of the 8.6 branch.

In this ticket I am not complaining about changed behavior in 8.7 and 9.0, or
in custom builds. Only that the default 8.6 should keep doing what it has been doing.

Default 8.6 builds do not know surrogates from a hole in the ground.

Well, first remark:  Tcl 9.0 gives the right answere here:

% lmap i {0 1 2 3 4} {testnumutfchars [testbytestring \xf0\x9f\x92\xa9] $i}
0 1 2 3 1

But ... for Tcl 8.x, we consider 4-byte sequences as consisting on 2 surrogates. So, there the correct answer should be:

% lmap i {0 1 2 3 4} {testnumutfchars [testbytestring \xf0\x9f\x92\xa9] $i}
0 1 2 3 2

You are correctly noticing that Tcl_UtfCharComplete() does not always match with Tcl_UtfToUniChar: Sometimes only 1 byte is demanded, but Tcl_UtfToUniChar() still reads 3 bytes. That's corrected in [166c0270e7812ba4 now. That change changes your test-case into the right answer.

Still, the documentation needs to be adapted explaining why this is the right answer. So, I'm not closing this ticket yet.

Note there are also situations (byte two of 4 bytes sequence) where
Tcl_UtfComplete() demands only one byte, but Tcl_UtfToUniChar will
then examine three.

Raising priority due to memory access outside of bounds.

That change appears to start here:

https://core.tcl-lang.org/tcl/info/a7a44cf3cab2e70f

Note that in the default TCL_UTF_MAX = 3 build, Tcl_UtfCharComplete
is only demanding 3 bytes when it sees what should be a lead byte for
a 4-byte sequence.

Looks like the consequence is that it is happy knowing 3 bytes are available,
then a call to Tcl_UtfToUniChar actually reads 4 bytes, overrunning the buffer.

At a minimum these two routines have to be in agreement with each other.

This part of the problem may go back farther than the regression.