# Design of Tcl string values for Tcl 9

## DRAFT WORK IN PROGRESS. NOT (YET) NORMATIVE

All Tcl values are strings, but what is a string?

The aim of this document is to spell out what the answer to that question
has been, what it should be, and how we get from one to the other.

## Fundamentals

A ***string*** is a sequence of zero or more symbols, each ***symbol***
a member of a symbol set known as an ***alphabet***.  For example, the
Tcl string **cat** is the sequence of three symbols **c**, **a**, **t**.

Each symbol in the Tcl alphabet is associated with a non-negative integer,
known as the ***code*** of that symbol.  Distinct symbols have distinct
codes.  Said another way, two symbols associated with the same code are
equivalent.  Schemes that associate symbols with code values are known
as ***character sets***.  For example, in Tcl's character set, the
symbols **c**, **a**, and **t** are associated with code
values 99, 97, and 116 respectively.

There is an ordering imposed on the alphabet by the numeric order of
the associated code values.  This ordering allows symbols to be compared
and sorted.  The natural extension of that ordering to sequences
establishes one well-defined way to compare, order and sort string values.

The number of symbols in a string's symbol sequence is the ***length***
of that string.  

A symbol within a string can be identified by its place in the sequence,
counting with an integer ***index*** that starts at 0.  A string of length *N*
has a symbol at each of the index values 0 through *N*-1.  We can
represent the symbol found at index *i* within string *s* with the
notation *s*[*i*].

Given a string *s* of length *N*, every pair of an index
value *i*, 0 <= *i* <= *N*, and a length *L*, 0 <= *L* <= *N* - *i*,
defines a substring of *s* of length *L* built from the sequence
of symbols *s*[*i*], ..., *s*[*i* + *L* - 1]. These substrings are also
rooted to a particular location in the original string.

Given a string *s* of length *M* and a string *t* of length *N*, the
concatenation of *s* and *t* is the string *u* of length *L* = *M* + *N*,
with *u*[0] = *s*[0], ... *u*[*M* - 1] = *s*[*M* - 1],
*u*[*M*] = *t*[0], ... *u*[*L* - 1] = *t*[*N* - 1].

In all of these descriptions, the string values act in ways with a
direct analog to C arrays of integer values.  This is a familiar collection
of behaviors that a broad collection of programmers can successfully reason
about without extensive training in the complexities of character
sets and encodings.  The basics are accessible to even novice programmers.
*Make the easy things easy; make the harder things possible.*

To be able to create and store an arbitrary string, a Tcl interpreter has
the minimum need for a set of commands or substitutions with the primitive
abilities to create an empty string, to create each string of length 1
(one for each symbol in the alphabet), and the ability to concatenate
arbitrary strings.  Other important primitives are the ability to report
the length of a string, index into a string, take a substring from a
string, and compare symbols and strings.  

## Tcl strings as the universal value set

We most frequently think of strings as a data type with the purpose of
entering, creating, storing, manipulating, processing and producing
text.  In Tcl, though, string values also serve as the universal value
set.  If a value cannot be expressed as a Tcl string, it is not a Tcl value.
The representations of other value sets in Tcl is in large part an exercise
in creating schemes to encode the values of those other value sets in
the form of Tcl strings.  Because of that, we want the set of Tcl strings
to permit encodings of other value sets that are clear, simple, efficient
and convenient.  The history of growing Tcl's alphabet has been driven by
the desire to better provide for the encoding of another value set that
has not easily been accommodated by the legacy alphabet.

## Implementations

The fundamentals above describe Tcl's value strings at an abstract level,
but to make Tcl interpreters and Tcl libraries we have to create programs 
that exhibit the described abstract behviors.  We will have much more
detailed things to say about string representations later, but for now
suffice it to say that a proper implementation of the string abstraction
needs to reproduce all the fundamentals faithfully.  If a representation
is used that allows multiple representations of a single symbol, or
multiple representations of a single string, this can be a matter of
some difficulty.  Representations that include the possibility of states
that are not valid string values at all are also cases that need
careful consideration.

## Tcl alphabet versions

This section looks into Tcl's history, which can be tricky.  History gets
messy.  It is full of steps and mis-steps and it can be difficult to get
agreement looking back about which were which.

In Tcl 7, the alphabet for Tcl strings was a set of symbols associated
with code values 1 through 255.  This is exactly the set
of **NUL**-terminated C strings.  The symbols with code values 1 through 127
are defined to follow the ASCII character set.  This is important to the
definition of Tcl because all symbols with syntactic meaning in Tcl scripts
are in the ASCII set.  The symbols with code values 128 through 255 were
less stringently specified, but such symbols could nevertheless be reliably
created, stored, processed and produced by Tcl programs.  This set of
string values continues to have relevance in Tcl today, because it is
exactly this set that can be passed as arguments to Tcl commands defined
via **Tcl_CreateCommand**.

Even though no Tcl 7 value could contain a symbol with code 0, Tcl still
offered a substitution suggesting it was possible.
```
	% string length <\x01>
	3
	% string length <\x00>
	1
```
This seems to be a mis-step, where the \\x00 substitution should have
raised an error.  The fact that everything created by it broke expectations
in some way supports that judgment.  But it's not beyond imagination for
someone to take another view.

The value set of arbitrary binary data, in the form of byte sequences,
is not well served by the Tcl 7 alphabet.  No encoding into Tcl 7 strings
can be both simple and efficient.  Either it must be variable-width, or
it must use a fixed width of at least two symbols per byte.  There are
certainly ways to encode arbitrary binary data using only the Tcl 7 alphabet,
but the commands of Tcl 7 never chose one for the core commands of the
language to use.  The problem was left unsolved so that a [**read**] from a
binary channel returned a value that the rest of Tcl simply truncated at
the first **NUL**.

In Tcl 8.0, the Tcl alphabet added a symbol with code value 0.  This allowed
the direct encoding of a byte sequences of length *N* by a string of
length *N* with each symbol determined by the code given by the byte value.
Aribtrary binary data could be stored in Tcl variables, and processed by any
commands created by the new **Tcl_CreateObjCommand**.  Legacy commands
still created by **Tcl_CreateCommand** remained "binary unsafe".

Note that the alphabet strictly grew between Tcl 7 and Tcl 8.0.  All string
values representable in Tcl 7 remained representable in Tcl 8.0.  Internally
there was reform in representation (counted strings replaced terminated
strings), but the concept of the Tcl string value accessible to scripts
changed along an upward compatible path.

Tcl 8.0 string values suffered from two deficits.  First, The internals
still used two representations that were not reconciled to provide the
same functionality.  Second, the international character sets of 
increasing importance could not be encoded into Tcl string values in
ways that were both simple and efficient.  International character set text
became the next value set prompting an expansion of Tcl's alphabet.

In Tcl 8.1 (released April, 1999), the Tcl alphabet expanded to a set of
symbols with code values ranging from 0 to 65,535 (0x0000 to 0xFFFF).
Where a code value had an assigned symbol in Unicode 2.0, the symbol of
Tcl's alphabet agreed with the Unicode character set.  This implied
continued symbol agreement with ASCII.  All assigned codepoints of
Unicode 2.0 fit in this alphabet.  Tcl could store all Unicode text values
in a simple and efficient way in its string values.  It could perhaps be
said that Tcl 8.1 strings *were* Unicode 2.0 text values. The Tcl
documentation is certainly phrased in those terms.  In hindsight, given
the later divergence in the development of Tcl and Unicode, that's
not the most useful perspective.  It is more useful to think of the set 
of Tcl 8.1 strings as a superset of the set of Unicode 2.0 text values.
This set of string values came to be known as UCS-2 to distinguish it
from later versions of Unicode.

Tcl 8.1 added a new backslash substitution syntax, **\\u**___HHHH___,
capable of producing every symbol in the Tcl alphabet.  Built-in Tcl
commands such as **format** and **scan** were extended to produce and
accept symbols and codes in the extended alphabet.  Once again the alphabet
strictly grew, preserving upward compatibility in the set of string values
available to Tcl programs.

The representations of strings inside the Tcl 8.1 library were 
substantially reformed to support the extended alphabet.  These
representations appeared in parts of the C programming interface
of the Tcl library, so extensions written for Tcl 7 or Tcl 8 had
to be adapted to the reforms.

Tcl 8.1 string values did bring with them a new mis-step.  A new
command **encoding** provided scripts with the ability to transform
Unicode text values into a variety of encoded forms.  This **encoding**
command included support for an encoding called **identity** that
empowers scripts to store an arbitrary byte sequence inside an internal
representation for Tcl strings.  The consequence is that scripts
can create multiple representations for the same string that are not
consistently treated as the same string, contrary to our fundamental
expectations.
```
	% info patch
	8.1.1
	% set s \u0080
	€
	% set t [encoding convertfrom identity \x80]
	??
	% string length $s
	1
	% string length $t
	1
	% scan [string index $s 0] %c code; set code
	128
	% scan [string index $t 0] %c code; set code
	128
	% string equal $s $t
	0
```

While scripts can avoid use of the **identity** encoding (perhaps treating
any use of it as introducing *undefined* behavior), the underlying
implementation flaws that allow for its mischief are available to all
Tcl extensions, so the failure of Tcl 8.1 strings to fully conform
to fundamental expectations still lurks.  More on this when we 
examine representations below.

Tcl 8.2 (released August, 1999) kept the same string alphabet, but
revised all relevant commands to process symbols according to the
Unicode 2.1 standard.  Notably this included the assignment of
code point **U+20AC** to the symbol EURO SIGN.  The Tcl 8.3.* series
of releases (February 2000 - October 2002) continued that same standard
of Unicode support.

From Chapter 1 of *The Unicode Standard, Version 2.0*,
"**The Unicode Standard is a fixed-width, uniform encoding scheme for written characters and text.**"
Chapter 2 presents a set of desgin principles.  The first principle states,
in part,
"**Plain Unicode text consists of pure 16-bit Unicode character sequences.**"
The second principle states 
"**The full 16-bit codespace... is available to represent characters.**"
A plain reading of those designs and assurances suggests that sequences
of symbols from a 16-bit alphabet will be a suitable fixed-width
representation for Unicode text.  The second design principle, though, goes
on to describe a "*surrogate extension mechanism*" which uses a pair of
16-bit code values to represent a single character.  This mechanism, of
course, makes the specified Unicode definition a variable-width encoding
of abstract characters, not a fixed-width encoding at all.

That said, Unicode 2.0, does specify surrogate pairs as the way to
represent a full collection of 1,114,112 distinguishable symbols
in the potential Unicode collection.  It also reserves the surrogate
code points to be used properly in Unicode text only in the formation
of such pairs.  It did not assign any characters to those pairs, instead
describing the mechanism as one "for encoding extremely rare characters".
Tcl string values processed symbols corresponding to Unicode surrogates
no differently from any other symbols in the Tcl alphabet.  In this state
of things, it is clear that the set of Tcl strings is strictly a superset
of the set of well-formed Unicode text.  This is not only because of
unassigned codepoints awaiting their symbols, but includes an ability
to store symbol sequences in Tcl string values that can never become
well-formed Unicode text at any point in the future.

Unicode 3.1.0 was released March 2001.  This was the first version
of the Unicode Standard that assigned characters to surrogate code
pairs in the 16-bit encoding, which in Unicode 3 came to have the 
name UTF-16.  In Unicode 3 it was acknowledged that UTF-16 is a
variable-length encoding.  Tcl's source code was updated to support
this specification of Unicode in May 2001, but support for surrogate
pairs to represent Unicode characters with codes greater than **U+FFFF**
was not implemented.  This was when Tcl Unicode support broke away
from conformance to the Unicode Standard.  Many programming assumptions
rooted in the existence of a fixed-width, 16-bit encoding for every
Tcl string value had become deeply embedded in both Tcl's implementation
and in some of its interfaces.  The assigned Unicode characters to be gained
outside the Basic Multilingual Place remained those
of "extremely rare" interest.  This level of partial Unicode 3.1.0
support was first released in Tcl 8.4.0 in September 2002.

And then Tcl's Unicode support fell into a deep sleep.

While Tcl's support of Unicode slept, Unicode itself evolved and
had the language of its conformance standards tightened and refined.
The conception of Unicode defined as seqeunces of 16-bit code units
faded away, and the 16-bit representation, UTF-16, became just one of several
encodings to be used.  Fundamentally each distinguishable Unicode text
that exists *or that ever will exist under future revisions of Unicode*
became defined as a sequence of zero or more symbols from the
alphabet of ***unicode scalar values***.  A unicode scalar value is
associated in the Unicode character set with a code value less than
1,114,112 and also constrained to exclude the code values associated
with the surrogate extension mechanism.  The code values of unicode
scalar values are in the integer ranges 0 to 55,295 (0x0000 to 0xD7FF)
and 57,344 to 1,114,111 (0xE000 to 0x10FFFF).  The code values of
unicode scalar values are all representable as 21-bit integers.  The
21-bit integers which are not the code value of any unicode scalar
value are the ranges 55,296 to 57,343 (0xD800 to 0xDFFF)
and 1,114,112 to 2,097,151 (0x110000 to 0x1FFFFF).  (Note that 21-bit
code values are exactly what can be encoded by the lead and trail byte
scheme of UTF-8 restricted to 4-byte sequences. Note also that code
value 0x10FFFF is the largest value that can be decoded from a
surrogate pair.)

From this perspective, the Tcl 8.1 string values were no longer seen as
proper Unicode, but as a legacy system called UCS-2 which suffered from
two flaws.  It lacked support for the supplemntary planes of the full
Unicode character set beyond **U+FFFF**.  It also allowed for the
presence of symbols with codes between 55,296 and 57,343 (0xD800 to 0xDFFF).
Neither system could encode the other in direct, simple, efficient
representations.

Unicode also came to define a collection of encodings.  The simplest to
understand is UTF-32, where each unicode scalar value in a Unicode sequence
is directly represented by a 4-octet (32-bit) value matching the code.
UTF-32 offers simplicity and fixed-width representation for efficient
random access indexing.  It also offers considerable waste of storage
and transmission.  Properly defined, UTF-32 does not include representations
of symbols that are not unicode scalar values.  However, the extension
to include them is not difficult to imagine.  The name UCS-4 comes down
from the ISO 10646 effort, and has evolved to mean the same thing as UTF-32.

The UTF-16 encoding uses 2-octet (16-bit) code units in variable length
patterns to represent each unicode scalar value.  The Unicode scalar
values from ranges 0x0000 - 0xD7FF and 0xE000 - 0xFFFF are represented by
themselves in a single code unit.  The Unicode scalar values from the
range 0x10000 - 0x10FFFF are each represented by a pair of 16-bit code
units, the first from the range 0xD800 - 0xDBFF and the second from the
range 0xDC00 - 0xDFFF.  Every 2-octet code unit may appear somewhere in the
proper UTF-16 encoding of some Unicode text.  There are no code unit
values that are forbidden.  There are UCS-2 sequences that are not
valid UTF-16, precisely those UCS-2 sequences that include surrogates
not arranged in properly formed pairs.  

Note the difficulties if we try to design an encoding with 2-octet code
units to encode all sequences over the union of the UCS-2 and Unicode
alphabets.  This can certainly be done, but the result will bear little
resemblence to UTF-16.  All valid UTF-16 sequences are used up representing
valid Unicode.  To also represent the strings of UCS-2 that are not valid
Unicode, we would need to use 2-octet sequences that are not valid UTF-16.
Any such scheme will have some point of discontinuity with the legacy UCS-2
system.  Likewise, since every 2-octet code unit sequence is used in UCS-2
to represent itself, there is no room to create representation for supplemental
planes of unicode in a 2-octet encoding without introducing a discontinuity.
There will have to be some 2-octet sequence that used to mean one thing
and now means another thing at the transition, or there will have to
be some approach of preserving both systems and taking care to distinguish
at each point which is in use.  This sticking point is a major difficulty
in managing a migration in Tcl's Unicode support, since public interfaces
exist that transfer 2-octet encoded data.

Unicode also defines the UTF-8 encoding of unicode scalar values into
single-octet (8-bit) (byte-oriented) code units in variable length
patterns.  The Unicode scalar values from the range 0x0000 - 0x007F
are represented by themselves in a single code unit. Each value from the
range 0x0080 - 0x07FF is represented by a two-byte sequence of a leading
byte from the range 0xC2 - 0xDF and a trailing byte from the
range 0x80 - 0xBF.  Each value from the ranges 0x0800 - 0xD7FF and
0xE000 - 0xFFFF is represented by a three-byte sequence starting
with a leading byte from the range 0xE0 - 0xEF and two trailing bytes
as before. (Some such three-byte sequences would encode surrogates,
and are therefore not valid UTF-8 sequences).  Each value from the
range 0x10000 - 0x10FFFF is represented by a four-byte sequence
starting with a leading byte from the range 0xF0 - 0xF4 and three trailing
bytes as before. (Some such four-byte sequences would decode to a
codepoint greater than 0x10FFF, and are therefore not valid UTF-8 sequences).
Note the implication that the bytes 0xC0, 0xC1, and 0xF5 - 0xFF can never
appear in a proper UTF-8 byte sequence.  Besides those forbidden bytes,
there are many sequences also forbidden, including any trailing byte
where one does not belong, any leading byte where one does not belong,
or any multi-byte sequence that when decoded would produce a value
outside the domain for that sequence length (for example, the four-byte
sequence 0xF0 0x80 0x80 0x80 that would appear to encode **U+0000**, which
is in the domain of unicode scalar values properly encoded by a single-byte).
The last of these constraints was formally imposed by Unicode 3.1.0.
Earlier versions of Unicode explictly approved of UTF-8 decoders that accepted
overlong byte sequences, and even included such decoders in their
sample implementations.

Here the news is better when it comes to thinking about representations
of all sequences over the union of UCS-2 and Unicode alphabets.  The
three-byte sequences that might encode surrogates that are forbidden
in UTF-8 are available to encode those values of UCS-2 without interference
with UTF-8 encoding of everything else.  The WTF-8 variation of UTF-8 is one
approach in this area, though the details require careful examination.
The byte sequence 0xF0 0x90 0x80 0x80 (representing **U+10000**) and the
byte sequence 0xED 0xA0 0x80 0xED 0xB0 0x80 (representing **U+D8000 U+DC00**,
which in turn is the surrogate pair representation of **U+10000**)
are distinct, allowing the required distinction.  WTF-8 is defined to
disallow this, however, because of the lack of a continuing ability to
distinguish after conversion to UTF-16.  That remains the critical sticking
point.

Tcl 8.1 documentation began the claim that Tcl used UTF-8 to store
its string values and as the encoding with which to pass byte-oriented
string values through interfaces, but that was never strictly true.
More on that below.

Unicode 6.0.0 (October 2010) was notable as the first version to include
assignments for emoji symbols.  Their popularity on mobile devices would end
the days when interest in Unicode characters above **U+FFFF** could be
said to be rare.  

Also in October 2010 Tcl was awakened out of its Unicode support slumber,
still offering only partial Unicode 3.1.0 support.  At that point, Tcl
was brought up to date with Unicode 6.0.0, but Tcl remained without
surrogate pair support.  Since that implied Tcl was also without emoji
symbol support, this increasingly became a mark against Tcl as a langauge
to use for good Unicode programming.  The partial support of Unicode 6 was
first released in Tcl 8.5.10 in June 2011.  Since then it has been customary
to update Tcl's partial support for the new versions of the Unicode Standard
as they are released.  Tcl 8.5.19 was released February 2016 with partial
support of Unicode 8.0.  Tcl 8.6.0 was released December 2012 with partial
support of Unicode 6.2.  Tcl 8.6.6 was released July 2016 with partial
support of Unicode 9.0.

In 2017, the trunk of Tcl development was turned over to work on the
Tcl 9.0 release.  This focused attention again on how best to revise
the string representation for that new milestone.  Following the example
of earlier alphabet expansions, it appears clear that for improved Unicode
support, we need the Tcl 9 alphabet to include the set of all unicode
scalar values.  Also following prior examples, it seems desirable to
strictly grow the alphabet so that all strings representable in Tcl 8
remain representable in Tcl 9.  Achieving both would mean a Tcl 9
alphabet that is a (super?)set of the union of UCS-2 and Unicode scalar
values.  The history of Tcl
string values in releases 8.6.7 and later felt the influence of that 
focus, and are best discussed after some attention to Tcl's string
representations.

## Representations

Tcl 7 strings are represented directly as C strings.  The representation is
one-to-one and complete.  Every Tcl 7 string has a representation by exactly
one C string, and every C string represents exactly one proper Tcl 7 string.
There is no C string that can be rejected as not representing a Tcl 7 string.
This is very simple.  Also, any interface using the passing of C string values
to implement the conceptual passing of Tcl 7 string values can be designed
with this knowledge.  There is no need to provide for error handling when
the possibility of error is defined out of existence.  Because of this, many
of Tcl's interface routines that date back the longest offer no capability
to report errors in their string arguments.  The C string representation is
also a fixed width encoding of Tcl 7 strings.  This allows for efficient
indexing.

Tcl 8.0 strings are represented directly as the pair of a byte array and
a length stored as a C **signed int**.  No Tcl 8.0 string with length greater
than **INT\_MAX** can be accommodated by this representation.  Other than
that limitation, the same direct, complete, and one-to-one nature of
representation is present as for Tcl 7 strings.  A string too long is the
only error condition that needs to be considered, and that error can be
prevented by constraining size of arguments.  Again, many interfaces
offer no detection or handling mechanisms to deal with an errors or 
invalidity in string value arguments.  The Tcl 8.0 representations also
continue to offer efficient indexing via fixed-width storage.

Tcl 8.0 strings are a superset of Tcl 7 strings.  When a Tcl 7 string value
is represented in the Tcl 8.0 manner, the byte array has the same contents
as the C string representation from Tcl 7.  The only difference is whether
there must be a terminating **NUL** byte.  In many places in the Tcl 8.0
representation, such a terminating **NUL** byte was used to allow easier
interoperability with legacy routines written to Tcl 7 expectations.  Most
notably the *bytes* and *length* fields of a **Tcl\_Obj** struct implement
the Tcl 8.0 string representation and require the terminating **NULL**
to be present at *bytes*[*length*].

The expansion of the Tcl alphabet in Tcl 8.1 to all two-byte codes brought
about new string value representations in the Tcl library.  The simplest
was the creation of the **Tcl\_UniChar** type, a two-byte integer able to
contain a single symbol from the Tcl alphabet.  This representation of a
Tcl symbol is used in the new Tcl 8.1 routines **Tcl\_UniCharToUpper**,
**Tcl\_UniCharToLower**, **Tcl\_UniCharToTitle**, **Tcl\_UtfToUniChar**,
**Tcl\_UniCharToUtf** and **Tcl\_UniCharAtIndex**.  In each of these
routines a symbol is passed in as a value of type **int** and it is
passed out as a value of type **Tcl\_UniChar**.

A Tcl string is then trivially represented as an array of **Tcl\_UniChar**.
It has the familiar properties of being a simple, complete, one-to-one,
fixed-width representation of all abstract Tcl 8.1 strings, with all the
familiar comforts and benefits.  This representation is used in the new
Tcl 8.1 routines **Tcl\_UniCharLen**, **Tcl\_UniCharNcmp**,
**Tcl\_UniCharToUtfDString**, and **Tcl\_UtfToUniCharDString**.  The first
two are utility routines that can be applied to this representation.  The
second two are conversion routines between the **Tcl\_UniChar** array
representation and another representation still to be described.  None
of the public routines in Tcl 8.1 passed an argument into or out of
Tcl as a Tcl value in the **Tcl\_UniChar** array representation.

The Tcl 8.1 interface continues to pass Tcl values in and out by use
of either the C strings of Tcl 7, or the counted strings of Tcl 8.0 (often
encapsulated in a **Tcl\_Obj** struct).  These interfaces could only
represent the extended alphabet of Tcl 8.1 by way of
a reinterpretation of how the string values are encoded in the byte sequences.
The routine **Tcl\_UniCharToUtf** generates these encoded byte sequences.
Tcl 8.1 documentation claims that the byte sequence encoding is UTF-8.
This was never strictly true. Examination of the body of **Tcl\_UniCharToUtf**
reveals that it encodes something closer to the FSS-UTF encoding from
Unicode 1.1, with encoding into sequences of more than 3 bytes disabled.

The limit on byte sequence length is available from the public Tcl 
header as **TCL\_UTF\_MAX**, with the suggestion that configuration
to limits other than 3 might be available.  Tcl 8.1 did not offer any such
working customizations.  At best, any variants triggered by a value
of **TCL\_UTF\_MAX** other than 3 in the Tcl 8.1 source code could be
viewed as speculations about what might be offered in later Tcl releases.

FSS-UTF calls for 3-byte sequences to encode all codepoints in the range
U+0800 to U+FFFF, and calls for 4-byte sequences to encode all codepoints
in the range U+10000 to U+1FFFFF, and longer byte sequences to encode
codepoints up to U+7FFFFFFF, providing room for up to 31-bits of codepoints
that ISO 10646 still proposed at the time.  The UTF-8 encoding defined
in Unicode 2.0 proposed a different use for 4-byte sequences.  They would
be used to encode codepoint pairs using the surrogate extension mechanism.
The refusal of the Tcl 8.1 encoder to produce 4-byte sequences marks another
way in which Tcl's encoding was never UTF-8.

The UTF-8 spec in Unicode 2.0 was silent about the encoding of unpaired
surrogate codepoints, but the sample implementation included as an example
did encode such codepoints into 3-byte sequences.  Later revisions to
the UTF-8 spec eliminated the encoding of unpaired surrogates.  The
encoding implemented by **Tcl\_UniCharToUtf** encodes all surrogate
codepoints, paired or not, into 3-byte sequences.

Tcl 7 strings cannot contain the byte 0x00, while UTF-8 (and FSS-UTF)
specifies that **U+0000** is to be encoded by 0x00.  The early UTF-8 specs
allowed UTF-8 decoders to decode the two-byte sequence 0xC0 0x80
into **U+0000** (decoding of all overlong encodings was officially
tolerated at the time). so **Tcl\_UniCharToUtf** is implemented to use
that modified encoding of **U+0000**.

Any Tcl 8.1 string encoded in the way described above can be passed 
out of Tcl in the encoded form.  Callers of Tcl have to adjust
their expectations to treat the value properly.  Earlier versions of Tcl
would pass out arbitrary byte sequences representing strings with one byte
per symbol.  The new representation has a variable number of bytes per
symbol.  Each Tcl caller had to review its functionality and purpose to
decide what adaptations were needed.  Tcl 8.1 provided a large number
of utility routines to process byte sequences in the variable width
encoding.  Tcl 8.1 also provided routines to support processing of
arbitrary byte array values for those Tcl callers that really needed
to process arbitrary byte arrays exchanged as Tcl values.

Callers of Tcl also pass Tcl string values into the library as either
C strings or counted strings.  These callers likewise had to adjust to
the new encoding expectations.  When they pass in byte sequences in
agreement with the encoding, they can expect to gain access to the whole
extended string set of Tcl 8.1.  However, the encoding does not produce all
byte sequences.  Many byte sequences do not correspond to the proper
encoding of anything.  The Tcl 8.1 rules for decoding are
implemented in the routine **Tcl\_UtfToUniChar**.  All byte sequences
generated by the encoder are reliably decoded back to the original
symbol.  Round trips through the encoding are lossless.  In addition,
following recommendations of the time, the decoder accepted all
overlong sequences otherwise structured with appropriate sequences
of lead bytes and trail bytes.  The decoder did not decode any byte
sequences of length greater than three.  Any byte leading a sequence not
recognized by those rules would be decoded as itself, effectively making
use of ISO-8859-1 as a fallback encoding.  All of these conventions of
the decoder provided opportunities for very different byte sequences
to be seen as representations of the same string after decoding, with
all of the risks of violations of fundmental string behavior that
go with that.  The UTF-8 encoding as strictly specified benefits from
the property that sorts on the encoded form are the same as sorts on
the original sequence, but Tcl's modified approach loses that benefit.
This possibility for multiple accepted representations of the same string
is the root of most failures of string fundamentals in Tcl 8.1.

Why accept these improper byte sequences at all?  One key factor is the
large existing set of routines that accept string arguments with no
mechanism for error handling, because in earlier Tcl releases no errors
were possible.  All byte sequences were proper strings until Tcl 8.1.
That said, it is less clear why a fallback to ISO-8859-1 was considered
wiser than something like generation of U+FFFD, the REPLACEMENT character.
One factor might be the ability to pass 4-byte UTF-8 through without
conversion or loss, at the expense of it being seen as 4 symbols instead
of 2 (UTF-16) code units or one symbol.  All that said, roundtrips
from the byte-encoded format to **Tcl\_UniChar** array and back are
not lossless.  This might be another of those arguable mis-steps seen
more clearly in hindsight.

It is the responsibility of each caller of **Tcl\_UtfToUniChar** that
the byte sequence passed in is either **NUL** terminated, or has
sufficient length that decoding governed by lead byte values will not
read beyond valid memory limits.  The utility routine

>	**int** **Tcl\_UtfCharComplete**(**char** *_src_, **int** _len_)

is provided as a tool to check for this condition.  It is known by all
callers that this routine will return 1 (true) whenever
(_len_ > **TCL\_UTF\_MAX**), so many callers omit calls in that circumstance.
Callers of **Tcl\_UniCharToUtf** are expected to provide an output
buffer of at least **TCL\_UTF\_MAX** bytes, and to expect a return count
up to (but never more than) **TCL\_UTF\_MAX**.

The encoder and decoder routines allow translation back and forth between
the two Tcl 8.1 representations for strings, the variable-width byte encoding
falsely described as UTF-8, and the fixed-width encoding as elements of
a **Tcl\_UniChar** array.  The translation is one symbol at a time, either
reading or writing one **Tcl\_UniChar** element with each call.  This
nails down the conception of the array representation as UCS-2 strings, with
no decoding of multi-element sequences.  The documentation is clear about
this when it declares that *n* calls to **Tcl\_UtfNext** must produce
the same result as one call to **Tcl\_UtfAtIndex**(*n*).  Indexing and
iteration must agree.

For extensions and apps, the migration from Tcl 8.0 to Tcl 8.1 was
pretty abrupt.  The new primary string representation was a variable-width
encoding of an enlarged alphabet, requiring the use of new utility routines
to achieve basic tasks like iterating or indexing, in many cases with
a substantial performance penalty.  After conversion to the new expectations,
it was rare that the revised code could still be used with Tcl 8.0.  A
non-trivial conversion with little support for retreat or delay, and a
performance hit as a reward.  There was plenty of unhappiness expressed
about the change.

Tcl 8.2 was released just 4 months later, in August 1999.  The string
representations were unchanged, but much more caching of the
**Tcl\_UniChar** array representation was added internally to better
amortize the performance penalties of some operations on sequences
stored in a variable-width encoding.  Different strings were also
represented differently internally based on their content to offer
performance and efficiency benefits.  The result was good enough
to make the Tcl migration to international strings a success, if a bumpy one.

New interfaces were also supplied by Tcl 8.2 to allow Tcl callers to exchange
UCS-2 strings stored as **Tcl\_UniChar** arrays with the Tcl library:
**Tcl\_NewUnicodeObj**, **Tcl\_SetUnicodeObj**,
**Tcl\_GetUnicode**, **Tcl\_GetUniChar**, and **Tcl\_AppendUnicodeToObj**.
Here it is unmistakable that **Tcl\_GetUnicode** was a mis-step, re-opening
the "**NUL** as terminator" problem on the new alphabet.  The undefined
nature of **Tcl\_GetUniChar** for an index out of range was unwise.  And
overall the use of "Unicode" in the routine names that act on what came
to be understood as UCS-2 strings brings about a festival of confusion.

Tcl 8.3 (released February 2000) added no new interfaces making use of
**Tcl\_UniChar**, and string representations remained unchanged.

Tcl 8.4 (released September 2002) has already been noted not for the
changes it made, but for the changes in Unicode it declined to acknowledge.
Two new utility routines **Tcl\_UniCharNcasecmp** and **Tcl\_UniCharCaseMatch**
accepted **Tcl\_UniChar** array arguments.  The routine
**Tcl\_GetUnicodeFromObj** was also added as the properly defined
replacement for **Tcl\_GetUnicode**.  Otherwise, the universe of Tcl strings
and their representations remained unchanged.

Tcl release 8.4.4 (July, 2003) made the first stab at supporting a
value for **TCL\_UTF\_MAX** greater than 3.  Comments suggested another
possible value of 6, with a conditional typedef for a 4-byte **Tcl\_UniChar**.  However, Tcl continued to use the FSS-UTF approach to encoding longer byte
sequences in such custom builds.  In Tcl release 8.4.14 (October 2006),
comments were revised to declare for the first time a plan to make
use of UTF-16.  Nevertheless the encoder remained one patterned after
FSS-UTF.

The same state of affairs persisted through all the remaining releases of
Tcl 8.4.  It also remained effectively the same in all releases from
Tcl 8.5.0 (December 2007) through Tcl 8.5.19 (February 2016).  (A comment
was added in *tclStringObj.c* during a commit otherwise largely devoted
to formatting and whitespace issues.  The new comment also made suggestions
about claiming use of UTF-16 without making any changes in the code that 
would make it in any way conformant to UTF-16.) It also remained the same
in releases Tcl 8.6.0 (December 2012) through Tcl 8.6.4 (March 2015).  In
all these releases, Tcl's encoder into byte sequences continued to be
implemented on the old FSS-UTF model.  Setting **TCL\_UTF\_MAX** to 6
might provide representations for astral characters, but nothing would
encode them into or decode them from their UTF-16 encoding.

Tcl 8.6.5 (February 2016) for the first time broke away from FSS-UTF
in its encoder in custom builds with values of **TCL\_UTF\_MAX** greater
than 3.  Five- and six-byte sequences were no longer generated, though
the decoder continued to decode them.  The upper limit of U+10FFFF was
imposed in the encoder with larger values replaced by U+FFFD.  Several
of the classification functions were extended in custom builds to apply
to astral characters.  The other novelty in this release was the carving
out of the **TCL\_UTF\_MAX == 4** custom builds to offer astral character
support, but not through use of a 32-bit **Tcl\_UniChar**.  For the
first time, interpretation of surrogate pairs appears in some parts of
some custom builds.  Also for the first time, the callers
of **Tcl_UtfToUniChar** were required in some circumstances to manage a
dance of two calls to decode a single byte sequence, because some byte
sequences were for the first time represented by a sequence of
two **Tcl\_UniChar** values.  This was the first step toward use of
UTF-16 in the Tcl library.  Like most first attempts, it didn't get
everything right out of the gate, but improvements would continue to come.
Because these changes were only visible in custom builds, there was little
controversy about their appearance in a patch release.  Tcl 8.6.6 (July
2016) remained the same.

Tcl 8.6.7 (August 2017) was the first release since Tcl 8.1.0 to change
the decoding of string value byte sequences in the default build.  This
release brought an end to Tcl's acceptance of overlong byte sequences
(other than the modified encoding for U+0000).  This change fixes many
EIAS violations created by decoding multiple encoded sequences to mean
the same symbols.  In custom builds, Tcl 8.6.7 also ended the decoding
of the five- and six-byte sequences of FSS-UTF.




Routines where TCL\_UTF\_MAX is relevant:
Tcl\_UtfBackslash, Tcl\_UtfPrev, Tcl\_WinTCharToUtf
  All Utf.3 (sort of)
  [gets] and [read] and "unicode" string rep buffer sizing
  Encoding routines (Tcl\_ExternalToUtf,...)