ADDED doc/dev/value-history.md
Index: doc/dev/value-history.md
==================================================================
--- /dev/null
+++ doc/dev/value-history.md
@@ -0,0 +1,810 @@
+# History of Tcl string values
+
+*Speculative and likely revisionist look back at Tcl strings.*
+
+Tcl programmers are familiar with the phrase
+[Every value is a string](https://wiki.tcl-lang.org/page/everything+is+a+string).
+Discussions of this aspect of Tcl are found in many places. It is harder to
+find detailed discussions of precisely what is "a string" in Tcl, and how
+that precise definition has shifted over the years. This document is meant
+to tackle that task, based on a review of the changing interfaces and
+implementation in the Tcl source code itself.
+
+## Prehistoric Tcl (7.6 and earlier, pre-1997)
+
+**NUL** *terminated array of* **char** *(C string)*
+
+The inspirations for the original Tcl string values were clearly the
+in-memory representations of C string literals, and the
+arguments used by C programs to receive their command line.
+
+> **int** **main** ( **int** *argc*, **char** **_argv_)
+
+Every command in Tcl 7 is implemented by a **Tcl_CmdProc** with signature
+
+> **int** **Tcl_CmdProc** ( **ClientData**, _interp_, **int** _argc_, **char** *_argv_[])
+
+Each argument value _argv_[_i_] passed to a command arrives in the
+form of a (__char__ *).
+The caller is presumed to pass a non-NULL pointer to a suitably usable
+contiguous chunk of memory, interpreted as a C array of type **char**. This
+includes a duty of the caller to keep this memory allocated and undisturbed
+while the command procedure executes. (In the days of Tcl 7, this was
+trivially achieved with an assumption that all use of the Tcl library was
+single-threaded.) The elements of that array determine the string value.
+Each element with (unsigned) **char** value between 1 and 255 represents an
+element of the string, stored at the corresponding index of that string.
+The first element with value 0 (aka **NUL**) is not part of the string value,
+but marks its end.
+
+From this implementation, we see that a Tcl string in release 7.6 is a
+sequence of zero or more __char__ values from the range 1..255.
+In the C type system, each string element is a __char__.
+It is also useful to think of each string
+element as a byte. The type name **char** suggests "character", and written
+works about Tcl from that time probably refer to a Tcl string as
+a "sequence of characters". In later developments the terms "byte"
+and "character" have diverged in meaning. There is no defined limit on the
+length of this string representation; the only limit is available memory.
+
+There is a one-to-one connection between stored memory patterns and the
+abstract notion of valid Tcl strings. The Tcl string "cat" is always
+represented by a 4-byte chunk of memory storing
+(**0x63**, **0x61**, **0x74**, **0x00**).
+Any different bytes in memory represent different strings. This means
+byte comparisons are string comparisons and byte array size is string length.
+The byte values themselves are what Tcl commands operate on. From early on,
+the documentation has claimed complete freedom for each command to interpret
+the bytes that are passed to it as arguments (or "words"):
+
+> The first word is used to locate a command procedure to carry out
+> the command, then all of the words of the command are passed to
+> the command procedure. The command procedure is free to
+> interpret each of its words in any way it likes, such as an integer,
+> variable name, list, or Tcl script. Different commands interpret
+> their words differently.
+
+In practice though, it has been expected that where interpretation of a
+string element value as a member of a charset matters, the ASCII encoding
+is presumed for the byte values 1..127. This is in agreement with the
+representation of C string literals in all C compilers, and it anchors the
+character definitions that are important to the syntax of Tcl itself. For
+instance, the newline character that terminates a command in a script is
+the byte value **0x0A** . No command purporting to accept and evaluate
+an argument as a Tcl script would be free to choose something else. The
+handling of byte values 128..255 showed more variation among commands that
+took any particular note of them. Tcl provided built-in commands
+__format__ and __scan__, as well as the backslash encoding forms of
+__\\xHH__ and __\\ooo__ to manage the transformation between string element
+values and the corresponding numeric values.
+
+Each Tcl command also leaves a result value in the interpreter of evaluation,
+whether that is some result defined by the proper functioning of the
+command, or a readable message describing an error condition. In Tcl 7,
+a caller of **Tcl_Eval** has only one mechanism to retrieve that result
+value after evaluating a command: read a (__char__ *) directly from
+the _result_ field of the same **Tcl_Interp** struct passed to **Tcl_Eval**.
+The pointer found there points to the same kind of
+**NUL**-terminated array of __char__ used to pass in the argument words.
+The set of possible return values from a command is the same as the set of
+possible arguments (and set of possible values stored in a variable),
+a sequence of zero or more __char__ values from the range 1..255.
+
+It must be arranged that the pointer in *interp*->*result* points to
+valid memory readable by the caller even after the **Tcl_CmdProc** is
+fully returned. The routine **Tcl_SetResult** offers several options.
+If a command procedure already has the result value as
+a (__char__ *) _result_, it will always work to call
+
+> **Tcl_SetResult** (_interp_, _result_, **TCL_VOLATILE**);
+
+The argument **TCL_VOLATILE** instructs the Tcl library to allocate
+a large enough block of memory to copy _result_ into, makes that copy,
+and then stores a pointer to that copy in _interp_->_result_. When the
+string to be copied has length no greater than __TCL\_RESULT\_SIZE__,
+this "allocation" takes the form of using an internal _interp_ field of
+static buffer space, avoiding dynamic memory allocation for short-enough
+results. A record is kept in other fields of _interp_ recording
+that this memory must be freed whenever the result machinery must be
+re-initialized for the use of the next command procedure,
+a task performed by **Tcl_ResetResult**. This general solution to
+passing back the result value using **TCL_VOLATILE** always imposes the
+cost of copying it. In some situations there are other options.
+
+When the result value of a command originates in a static string literal
+in the source code of the command procedure, there is no need to work
+with memory management and copies. A pointer to the literal in static
+storage will be valid for the caller. The call
+
+> **Tcl_SetResult** (_interp_, **"literal"**, **TCL_STATIC**);
+
+is appropriate for this situation. The argument **TCL_STATIC** instructs
+the Tcl library to directly write the pointer to the literal into
+_interp_->_result_. It is a welcome efficiency to avoid copies when
+possible.
+
+A third option is when the command procedure has used **ckalloc**
+to allocate the storage for _result_ and then filled it with the proper
+bytes of the result value. Then the call
+
+> **Tcl_SetResult** (_interp_, _result_, **TCL_DYNAMIC**);
+
+instructs the Tcl library not to make another copy, but to take
+ownership of the allocated storage and the duty to call **ckfree**
+at the appropriate time. It is also supported for the command procedure
+to use a custom allocator and pass a function pointer for the corresponding
+custom deallocator routine.
+
+Besides **Tcl_SetResult**, a command procedure might also build up the
+result in place inside the interp result machinery by use of the
+routines **Tcl_AppendResult** or **Tcl_AppendElement**, which places more
+of the housekeeping burdens in the Tcl library. You may also see examples
+like
+
+> **sprintf** (_interp_->_result_, "%d", _n_);
+
+where the command procedure can be confident about not overflowing
+the **TCL\_RESULT\_SIZE** bytes of static buffer space.
+
+When the caller reads the result from _interp_->_result_, it is given no
+supported indication which storage protocol is in use, and no supported
+mechanism to claim ownership. This means if the caller has any need to keep
+the result value for later use, it will need to make another copy and make
+provisions for the storage of that copy. Often this takes the form of
+calling **Tcl_SetVar** to store the value in a Tcl variable (which
+performs the housekeeping of making the copy of its _newValue_ argument).
+
+This system of values has several merits. It is familiar to the intended
+audience of C programmers. The representation of each string element
+by a single byte is simple and direct and one-to-one. The result is
+a fixed-length encoding which makes random access indexing simple and
+efficient. The processing of string values takes the form of iterating
+a pointer through an array, a familiar technique that compilers and hardware
+have strong history of making efficient. The overall simplicity keeps the
+conceptual burden low, which enhances the utility for connecting software
+modules from different origins, the use of Tcl as a so-called "glue language".
+
+There are some deficits to contend with. The greatest complexity at work
+is the need to manage memory allocation and ownership. The solutions to
+these matters involve a significant amount of string copying. For any
+commands that treat arguments and produce results based on interpretations
+other than sequences of characters, parsing of argument strings into
+other representations suitable for command operations and the generation
+of a result strings from those other representations are costs that must
+be repeated again and again, since only the string values themselves carry
+information into and out of command procedures.
+
+Finally a large deficit of this value model of strings is they cannot
+include the element **NUL** within a string, even though that string
+element otherwise appears to be legal in the language. In the language
+of the day, Tcl 7 is not _binary safe_ or _8-bit clean_.
+
+
+```
+ % string length <\x01>
+ 3
+ % string length <\x00>
+ 1
+```
+
+
+Note here that Tcl backslash substitution raises no error when asked to
+generate the **NUL** byte, even though Tcl version 7.6 has no way at
+all to do anything with that byte. At some point it gets treated as
+a string terminator.
+
+Since Tcl 7 strings are C strings, C programmers of the time were familiar
+with the issue of binary safety, and several systems of the day made use of
+solutions. Electronic mail systems had to deal with the ability to pass
+arbitrary binary data through systems where binary safety of all the components
+could not be assumed. The solution was to define a number of encoding schemes.
+Tcl 7 did not take this approach. Any encoding defined to support
+the **NUL** byte would have to borrow from some byte sequence already
+representing itself. It must have been judged that the utility of the **NUL**
+was less than the disutility of complicating the value encoding that all
+commands would need to contend with, at least for general values.
+
+As Tcl gained cross-platform channel support, the inabilty to store in a
+Tcl value the arbitrary data read in from a channel started to cause more
+and more pain. In particular it was not possible to safely copy a file,
+or transmit its contents to an output channel (as one might seek to do
+when programming a web server) with code like
+
+
+```
+ fconfigure $in -translation binary
+ fconfigure $out -translation binary
+ puts -nonewline $out [read $in]
+```
+
+
+The issue was so compelling that Tcl gained a semi-secret unsupported
+command __unsupported0__ for just this purpose with functionality
+much like the command __fcopy__ that would come later.
+
+It is more speculative, but it appears the inability to pass arbitrary
+binary data through Tcl contributed to the late development of full
+support of many image formats for Tk command __image create photo -data__.
+Tk 4 never had any built-in image format that supported this function.
+
+Since Tcl I/O could read in and write out binary data, but that data could
+not pass between commands, this forced the design of many commands to accept
+filename arguments to do their own I/O rather than compose with general
+language-wide I/O facilities. As channels expanded to include sockets and
+pipes the corresponding expansion of utility could not occur in commands
+coded in this way without further revisions.
+
+The public C API for Tcl 7.6 includes 22 routines that return
+a (__char__ *) which is a Tcl string value. It includes 74 routines
+that accept at least one (__char__ *) argument that is intended to
+be a Tcl string value. It includes 6 callback signature typedefs that
+specify callback interfaces which must accept at least one (__char__ *)
+argument that is a Tcl string value. It includes one callback signature
+typedef, **Tcl_VarTraceProc**, that specfies a callback interface that
+must return a (__char__ *) that is a Tcl string value. All of these
+components of the public API are points of potential compatibility
+concern as the conception of Tcl strings shifts over time.
+
+## Tcl 8.0 (Development begun 1996, Official release 1997-1999)
+
+*Counted array of* **char**
+
+In order to remedy the deficits of the Tcl 7 value set, the implementation
+of Tcl strings had to be modified. The modification put in place by
+Tcl 8.0 was to stop treating every **NUL** byte as a marker of the end of
+a Tcl string value. Tcl would begin to permit a **NUL** byte in the internal
+portion of the __char__ arrays holding Tcl strings. Since the presence of
+a **NUL** byte would no longer (always) terminate a string, a (__char__ *)
+alone could no longer convey an arbitrary string value. New interfaces were
+defined where a (__char__ *) argument was accompanied by an __int__ argument
+that would specify how many bytes of memory at the pointer should be
+taken as the string value.
+
+As an illustration, consider the Tcl 7 public routine
+
+> **int** **Tcl_ScanElement** (const **char** * _string_, **int** * _flagPtr_ );
+
+It is a utility routine used in the task of generating the string value
+of a list out of the string values of its elements. The argument _string_
+is an element of the list passed in as a **NUL**-terminated string. It
+is not possible to successfully use this interface to pass in the string
+value of a list element that includes the **NUL** character. By expanding
+the Tcl value set, Tcl left many public interfaces incapable of dealing with
+the entire value set. These routines continue to do what they always did,
+and for some purposes that is good enough, but they are unsuitable for
+uses that must accommodate the new values supported in Tcl 8.0. This leads
+to the creation of an additional public routine
+
+> **int** **Tcl_ScanCountedElement** (const **char** * _string_, **int** _length_, **int** * _flagPtr_ );
+
+The additional argument _length_ combines with the argument _string_ to
+transmit any Tcl 8.0 string value into the routine. In most interfaces
+expanded in this way, it is conventional for the _length_ argument to
+accept the value **-1** (or sometimes any negative value) as a signal
+that _string_ is **NUL**-terminated. Another advantage of this interface
+is that it is not necessary for _string_ [ _length_ ] to contain the
+byte value **NUL** . This makes it easier to pass substrings of larger
+strings without the need to copy or overwrite.
+
+This implementation is often called a
+counted string to distinguish it from a **NUL**-terminated string.
+In Tcl 8.0, every valid Tcl string is a sequence of zero up
+to **INT_MAX** __char__ values from the range 0..255.
+In the new representation, the set of valid Tcl strings is both expanded
+and contracted. The alphabet is expanded to include **NUL**, while a
+defined limit is imposed on the length of the string sequence of elements
+from that alphabet for the first time.
+
+In the context of available memory in most computing systems
+of the day, the limitation on string length did not seem unreasonable.
+The new implementation solves the problem of storing in a Tcl value
+arbitrary binary data up to 2Gb in size. The encoding of string elements
+as __char__ array elements is still direct, simple, one-to-one, and every
+__char__ array that was a valid representation of a value in Tcl 7 remains
+valid and continues to represent the same value in Tcl 8.0, which offers a
+high degree of accommodation of client code written for Tcl 7. Every
+collection of bytes in the __char__ array represents a valid string, so
+there is no need to consider how any routine ought to respond to an invalid
+input.
+
+The public C API for Tcl 8.0 includes only 7 new routines that either
+accept or return explicitly a pair of values of type (__char__ *)
+and __int__ representing together a counted string value. A second solution
+is to include these two values as fields of a new struct, **Tcl_Obj**, and
+define new interface routines that make use of that struct.
+The **Tcl_Obj** struct introduced in Tcl 8.0 includes a (__char__ *)
+field _bytes_ and an __int__ field _length_. Taken together
+these two fields are a Tcl value represented as a counted string. The
+routine
+
+> **Tcl_Obj** * **Tcl_NewStringObj** (__char__ * _bytes_, __int__ _length_);
+
+is used to create such a struct out of the pair of arguments defining a
+counted string. The routine
+
+> __char__ * **Tcl_GetStringFromObj** ( **Tcl_Obj** * _objPtr_, __int__ * _lengthPtr_ );
+
+provides the reverse functionality, returning a (__char__ *) value and
+writing to *_lengthPtr_ an __int__ value making up the counted string
+retrieved from _objPtr_. These routines support the transmission of
+the complete binary-safe set of Tcl 8.0 values.
+
+It is mostly outside the scope of this document, but additional fields
+of the **Tcl_Obj** struct enable value sharing, and caching of conversions
+to other value representations. These other functions solve many of the
+defects of the Tcl 7 value representation other than the constrained
+universe of supported values itself. Notably these other features enable
+substantial performance gains. Much of the published advice to adopt
+interfaces using the **Tcl_Obj** struct centers on the performance rewards
+rather than the expanded capability to include **NUL** bytes.
+
+The storage of a __char__ array by a **Tcl_Obj** has some parallels with
+the storage of a __char__ array as the result of a command evaluated
+in a **Tcl_Interp**. However, the same functionality of **Tcl_SetResult**
+was not made available. To do so, something like a _freeProc_ field
+would need to be included in the **Tcl_Obj** struct, and concerns about
+excessive memory requirements to store every value in Tcl ruled the day.
+As a consequence, it is an implicit requirement that _objPtr_->_bytes_
+normally points to memory allocated by **ckalloc** and when sharing
+demands are all released it will be freed by a call to **ckfree**.
+The routine **Tcl_NewStringObj** makes such an allocation and copies
+the __char__ array in its _bytes_ argument into it. In contrast,
+the (__char__ *) value returned by **Tcl_GetStringFromObj** is a pointer
+to the same __char__ array that is stored by _objPtr_. The caller of
+**Tcl_GetStringFromObj** can sustain the validity of that memory by
+claiming a share on _objPtr_ with a call to **Tcl_IncrRefCount** and
+then later releasing that claim with a call to **Tcl_DecrRefCount**.
+One complete copy of the __char__ array contents cannot be avoided by this
+value model, but value sharing permits most subsequent copying to be avoided.
+This design also imposes consequential costs on the "Copy on Write" strategy
+when over-writing a **Tcl_Obj** struct. No alternative to complete
+allocation and copy is available, with performance consequences for
+large values that have visible impact on scripts.
+
+With a **Tcl_Obj** struct defined as a container for any Tcl 8.0 value,
+a new signature for command procedures was defined,
+
+> **int** **Tcl_ObjCmdProc** ( **ClientData**, _interp_, **int** _objc_, **Tcl_Obj** * const _objv_[]);
+
+and a new public routine **Tcl_CreateObjCommand** to create commands
+implemented using the new command procedure signature. Commands using the
+new signature have the ability to receive the complete argument set of
+Tcl 8.0 as arguments. An arbitrary Tcl 8.0 value can be returned
+using a call to **Tcl_SetObjResult**. One of these commands can be
+evaluated by use of the old routine **Tcl_Eval** or the new routine
+**Tcl_EvalObj** when the script to be evaluated itself might include
+the __NUL__ byte. The caller then retrieves the result of the evaluated
+command with a call to **Tcl_GetObjResult**. A new routine
+**Tcl_GetStringResult** allows the interpreter result to be retrieved
+as a **NUL**-terminated string, with the corresponding truncation at
+any **NUL** byte that might be in it. This is not a recommended routine,
+but it is an improvement over direct access to the _interp_->_result_
+field, which Tcl 8.0 declares deprecated. All Tcl operations that need
+to support the complete new Tcl value set had to be revised to make use
+of new facilities and interfaces.
+
+Continuing earlier illustrations, we can see that the **string** command
+was adapted in Tcl 8.0
+
+
+```
+ % string length <\x00>
+ 3
+```
+
+
+Likewise the **puts** and **read** commands are fully binary safe, so
+that
+
+
+```
+ fconfigure $in -translation binary
+ fconfigure $out -translation binary
+ puts -nonewline $out [read $in]
+```
+
+
+now faithfully copies all bytes from one channel to another. Tcl 8.0
+also introduces new commands **binary format** and **binary scan**
+that Tcl programs can use to process binary values. In Tcl 8.0, these
+commands operate directly on the counted string in _objPtr_->_bytes_.
+
+A deployed collection of extension commands written for Tcl 7 could
+still be used by Tcl 8.0. Since these commands were implemented by
+command procedures using the **Tcl_CmdProc** signature, they could
+only accept **NUL**-terminated strings as arguments. Attempts to pass
+any argument value to them that included a **NUL** byte would only
+pass the truncated value. Note that this remains within the very
+free requirements of an arbitrary Tcl command. Each command is free to
+interpret its arguments as it chooses, including the choice to
+truncate them at the first **NUL**. A Tcl 7 command would produce a
+result within the constrained value set, but that is acceptable.
+If the command records that result via direct write to _interp_->_result_
+that is deprecated and formally unsupported, but in practice tolerated
+and functional. Such an approach allowed for a gentle, even lazy, approach
+to migration to new capabilities. All of the public Tcl 7 routines that
+accept or return **NUL**-terminated strings noted in the earlier section
+continue to be supported public routines in Tcl 8.0, doing what they do,
+with all the implicit limitations. The model in place was clearly an
+expectation that all commands would migrate in time, but there was not
+a big hurry. Several routines incapable of supporting the expanded
+Tcl value model were declared deprecated as a prod toward eventual migration.
+
+Tcl 8.0 itself did not fully convert all of its built-in commands. At least
+23 **Tcl_CmdProc** command procedures persist within it. Some can be
+demonstrated.
+
+
+```
+ % regsub -all y zyz|zyz x foo
+ 2
+ % set foo
+ zxz|zxz
+ % regsub -all y zyz\x00zyz x foo
+ 1
+ % set foo
+ zxz
+```
+
+
+Others are largely masked by the use of the bytecode compiler and
+execution engine, also new in Tcl 8.0.
+In other built-in commands, the use of routines with these interface
+limitations leads to similar failures.
+
+
+```
+ % set foo <\x00>
+ <>
+ % subst {$foo}
+ <
+```
+
+
+Several routines, both public and private, and a number of established
+data structures continue to make use of **NUL**-terminated strings. Because
+of this, although the fully general set of Tcl values includes all
+binary sequences, several components of Tcl remain limited in release 8.0,
+and may not include the **NUL** byte. Examples include command names,
+namespace names, channel names, channel type names, **Tcl_ObjType** names,
+and likely more.
+
+## Tcl 8.1 (Development begun 1997, Official release 1999-)
+
+*Strings encoded in UTF-8, as modified in Java 1.1*
+
+A major re-thinking of Tcl values began in 1997 as work toward Tcl 8.1
+began. The reconciliation of Unicode 1.1 and ISO-10646 into agreement
+showed a clear path forward for software to manage text encodings under
+one organized plan. With this new future vision of text management in
+software, the notion that programmers could benefit from being shielded
+from the need to deal with encodings rapidly vanished. In contrast, the
+value of joining in support of a developing standard was apparent.
+
+The Tcl _changes_ file records an entry (in part):
+
+
+```
+6/18/97 (new feature) Tcl now supports international character sets:
+ - All C APIs now accept UTF-8 strings instead of iso8859-1 strings,
+ wherever you see "char *", unless explicitly noted otherwise.
+ - All Tcl strings represented in UTF-8, which is a convenient
+ multi-byte encoding of Unicode. Variable names, procedure names,
+ and all other values in Tcl may include arbitrary Unicode characters.
+ For example, the Tcl command "string length" returns how many
+ Unicode characters are in the argument string.
+ - For Java compatibility, embedded null bytes in C strings are
+ represented as \xC080 in UTF-8 strings, but the null byte at the end
+ of a UTF-8 string remains \0. Thus Tcl strings once again do not
+ contain null bytes, except for termination bytes.
+ - For Java compatibility, "\uXXXX" is used in Tcl to enter a Unicode
+ character. "\u0000" through "\uffff" are acceptable Unicode
+ characters.
+```
+
+
+The corresponding development patch is no longer available. The relevant
+code development fossil record for Tcl starts on September 21, 1998,
+with the source code of something close to the Tcl 8.1a2 release
+(February 24, 1998). We also have the Tcl 8.1a1 release (January 23, 1998).
+The absence of relevant entries in the _changes_ file
+suggests the "international character set" support in the Tcl 8.1a1
+release is what was committed in June, 1997. We can only speculate what
+development effort, what judgments and what referrals to what sources
+and standards played out in the development of that first implementation
+patch. Our speculation is informed, though, by what we know and can reasonably
+infer from the technical environment where this work took place.
+
+The claim that Tcl 8.1 strings are stored in UTF-8 is false, strictly
+speaking. Neither the encoding generated nor the encoding accepted by Tcl 8.1
+follow the precise specification of UTF-8. Not the specification in place
+today, nor any of the specifications in place near the time of this work.
+As the _changes_ entry indicates, a number of implementation choices are
+claimed to serve "Java compatibility". Tcl development at this time took
+place at Sun Microsystems, which was also the home of Java. It appears that
+"Java compatibility" is not a statement about any sort of interoperability,
+but about following the same conventions for the sake of programmer
+familiarity. Tcl 8.1 added the **\\u**___HHHH___ syntax as a way for
+scripts to specify Unicode characters using only the ASCII characters.
+This is the same syntax Java uses for the same thing. The "compatibility"
+at work is a reduction of cognitive burden by using the same solution to
+the same problem in two languages. Tcl copied some other deviations
+from standard UTF-8 from Java as well.
+
+To appreciate the choices made by Tcl in how to implement support for the
+"international character set", it is helpful to review some of the basics
+of encoding, and the players and offered standards of the time. Two efforts
+at defining a single Universal Coded Character Set (UCS) were active
+during the 1990s, ISO-10646 and Unicode. The common task was to define
+a mapping from each integer index value (codepoint) to a character assigned
+to the location in the character set table identified by the codepoint. The
+ISO-10646 effort conceived of a table with 4-byte (32-bit) codepoints, with
+space for up to 2^32 characters. They gave the name UCS-4 to both this
+conceptual character set and the trivial encoding of its codepoints
+as 4-byte integer values. In contrast, the Unicode effort proposed
+a character set table with up to 2^16 characters, indexed by 2-byte (16 bit)
+indices. This (potentially) smaller character set was formally known
+as UCS-2, The trivial encoding of its codepoints as 2-byte integer values
+was also called UCS-2. Less formally, but very commonly the character
+set UCS-2 was called "Unicode", and two-byte codepoint values were called
+"Unicode characters", emphasizing the much smaller table, and smaller
+codepoints of the Unicode project. The notation **U+_HHHH_** for codepoint
+values was also in use. In the mid-1990s, no codepoints outside UCS-2
+were assigned by either UCS project.
+
+At the time, a migration away from single-byte encodings like ASCII
+or ISO-8859-1 to either UCS-2 or UCS-4 was a tough sell. A doubling
+or quadrupling of memory and network traffic requirements for the sake
+of additional characters that many applications had no need to handle
+was not a welcome proposition. It was clear that a variable-length encoding
+that better managed the costs involved would be needed. Appendix F of The
+Unicode Standard, Version 1.1 (1993) contains a description of
+what was called the File System Safe UCS Transformation Format (FSS-UTF).
+FSS-UTF is a variable-length byte-oriented encoding of UCS codepoints.
+The implementation code in Tcl 8.1 more closely matches this specification
+than anything bearing the name UTF-8. The encoding rules of FSS-UTF are
+summarized in the table
+
+
+```
+ Codepoints Encoding
+ ---------- --------
+ U+0000 - U+007F 0bbbbbbb
+ U+0080 - U+07FF 110bbbbb 10bbbbbb
+ U+0800 - U+FFFF 1110bbbb (10bbbbbb)2
+
+ U+00010000 - U+001FFFFF 11110bbb (10bbbbbb)3
+ U+00200000 - U+03FFFFFF 111110bb (10bbbbbb)4
+ U+04000000 - U+7FFFFFFF 1111110b (10bbbbbb)5
+```
+
+
+FSS-UTF encodes each codepoint in the ASCII range as a single-byte,
+the same byte already used by ASCII for that character. Longer byte seqeunces
+encode higher codepoints that were less frequently used. A footnote
+in Appendix F makes the observation that encoded sequences of three bytes
+are sufficient to cover all 16-bit codepoints of UCS-2 (Unicode). The
+longer encodings of four through six bytes prepared the way to cover
+codepoints in UCS-4 up to 31 bits.
+
+(Side observation: it would be
+straightforward to continue the table of encoding rules by one more
+line, using the lead byte **0xFE** to introduce a sequence of six trail
+bytes. This would be a seven byte encoding that could encode up to 36 bits,
+easily covering the remainder of the 32-bit range of all of UCS-4. It
+seems the value of keeping byte **0xFE** out of encoded streams outweighed
+the failure to encode the upper half of UCS-4, which had no forseeable use.
+The bytes **0xFE** and **0xFF** together make up a Byte Order Mark that
+is used to determine endianness of UCS-2 data. When neither byte appears
+in UTF-8 or FSS-UTF the confidence of that interpretation is boldened.)
+
+In Tcl 8.1, encoding of Unicode codepoints is accomplished by the routine
+
+> **int** **Tcl_UniCharToUtf** ( **int** *ch*, **char** *_str_);
+
+The routine is (nearly) a direct translation of the table of encoding rules
+of FSS-UTF. However it is clear that a decision was made that Tcl would
+support "Unicode" which meant UCS-2 which meant 16-bit codepoints as
+"Unicode characters". Unicode characters require a maximum of
+three encoded bytes, and Tcl 8.1 source code uses **TCL\_UTF\_MAX**
+to represent that limit. It is useful in sizing buffers where encoded
+strings are to be written. The public header for Tcl 8.1 includes
+
+> \#define **TCL\_UTF\_MAX** 3
+
+and
+
+> typedef **unsigned short Tcl_UniChar**;
+
+However, the body of **Tcl_UniCharToUtf** includes code to
+encode up to six-byte sequences as prescribed by FSS-UTF representing
+up to 31-bit codepoints, protected by a conditional compiler directive
+
+> \#if **TCL\_UTF\_MAX** > 3 .
+
+With the default setting, Tcl is able to encode a total of 65,536
+distinct values of the *ch* codepoint argument. These are all the
+codepoints of UCS-2, the complete capacity of Unicode 1.1. When
+**Tcl_UniCharToUtf** is passed a *ch* argument outside that supported
+range, the codepoint **U+FFFD** is encoded in its place. Unicode 1.1
+assigns this codepoint the name **REPLACEMENT CHARACTER**. Unicode
+prescribes the use of **U+FFFD** to replace an incoming character
+whose value is unrepresentable. In a configuration of Tcl 8.1 with
+**TCL\_UTF\_MAX** set to 6, the **REPLACEMENT CHARACTER** would
+replace only the *ch* values in the upper half of UCS-4. Note that
+Tcl 8.1 has made a design choice here to handle unsupported inputs
+via replacement, and not via raising an error.
+
+It is clear that the Unicode support provided by Tcl 8.1 was intentionally
+limited to UCS-2, while conventions and migration supports were put in place
+so that a future (binary incompatible) version of Tcl would expand its support
+to UCS-4 with FSS-UTF encoding sequences of up to six bytes, at some point
+where the considerable cost in memory and processing burden traded off
+against the benefits of support for whatever future codepoint assignments
+outside of UCS-2 compelled support.
+
+The capacity of UCS-2 is a table of 65,536 codepoints. However, Unicode 1.1
+assigns characters to only 40,635 rows in that table. If we generate all
+possible sequences of those 40,635 assigned codepoints, and then encode
+each sequence via the FSS-UTF rules, we generate a set of byte sequences
+far smaller and more constrained that the general set of all byte sequences.
+When we claim that Tcl 8.1 strings are kept in the UTF-8 encoding, we
+imply that Tcl 8.1 strings are constrained to a much smaller set of byte
+sequences than were permitted for Tcl 8.0 strings. This raises questions
+about both compatibility and what a decoder should do with a non-conformant
+byte sequence.
+
+It is an expected feature of the Unicode character set that more characters
+are to be added over time. The number of assigned codepoints grows as new
+versions of the Unicode standard are codified and published. Because
+support for standards always lags their publication, software written in
+conformance to one version of Unicode is likely to encounter data produced
+in conformance to a later version. In light of this, the best practice is
+to accommodate and preserve unassigned codepoints as much as possible.
+Software written to support Unicode 1.1 can then accept Unicode 2 data streams,pass them through and output them again undamaged. In this way a middleware
+written to an obsolete Unicode standard can still support providers and
+clients that seek to use characters assigned only in a later standard.
+The middleware would fail to properly answer any questions about the
+later-assigned characters, such as a **string is alpha** test, but could
+still not be immediately useless.
+Unicode 1.1 left open the possibility that any codepoint in UCS-2 might
+one day be assigned. Tcl 8.1 imposes no conditions on the encoding of any
+**Tcl_UniChar** value at all.
+
+The standards specifying text encodings published in the mid-1990s were quite
+clear and explicit about the right way to do things. They were often less
+demanding and specific about how to respond in the presence of errors. The
+spirit of the Postel Robustness Principle
+
+> *Be liberal in what you accept, and conservative in what you send.*
+
+held considerable influence at the time. Many implementations chose to
+accommodate input errors, especially when that was the natural result
+of laziness. For example, the specification of FSS-UTF and all specifications
+for UTF-8 are completely clear and explicit that the proper encoding
+of the codepoint **U+0000** is the byte value **0x00**, (our old friend
+the **NUL** byte!). However, inspection of the FSS-UTF encoding rules
+quickly reveals that the multi-byte encoding strategies have room to
+encode all the codepoints that already have a proper encoding as a single
+byte. Unless we take active steps to prevent it, a decoding procedure
+will happily decode the byte sequences (**0xC0**, **0x80**) or
+(**0xE0**, **0x80**, **0x80**) as the codepoint **U+0000**. Both
+Unicode 1.1 and Unicode 2.0 include sample decoder routines that do
+exactly that. The text of Appendix A.2 of The Unicode Standard 2.0
+(which contains one definition of UTF-8 from 1996) explicitly states,
+
+> When converting from UTF-8 to Unicode values, however,
+> implementations do not need to check that the shortest encoding
+> is being used, which simplifies the conversion algorithm.
+
+Whether or not the Tcl developers were explicitly following this advice,
+the routine in Tcl 8.1 for decoding Unicode codepoints from the
+encoded byte sequences,
+
+> **int** **Tcl_UtfToUniChar** ( **const char** *_str_, **Tcl_UniChar** *_chPtr_),
+
+implements the same forgiving support for overlong byte sequences.
+
+In the context of the still incomplete transition from the Tcl 7 to
+the Tcl 8.0 conventions for string values, and the lingering difficulties
+in handling the **NUL** byte, the forgiving nature of the UTF-8 decoders
+presented a clear opportunity. If the decoder is equally happy to receive
+the byte sequence (**0xC0**, **0x80**) as a representation of **U+0000**,
+why not have the encoder encode it that way in the first place? With
+this modification, encoded byte sequences return to not having the **NUL**
+byte in them at all, and we sweep away the remnants of a difficult
+transition (while admittedly adding a new one). Tcl chose to adopt
+this modification of the encoding rules. Java 1.1, another language
+developed by Sun Microsystems at the same time, made the same modification.
+
+The modification is not without its problems. One feature of the proper
+UTF-8 encoding scheme is that a sorting of encoded byte sequences matches
+the sorting of the UCS-2 sequences they encode. Not so when custom
+modifications are in place. Likewise, byte-for-byte comparisons of
+byte sequences in a proper UTF-8 encoding reliably transfer to
+character-for-character comparisons of the UCS-2 strings they encode.
+This is especially important when testing for equality as is vital
+in searches. When any character has multiple acceptable encodings,
+the fully correct implementation of these matters becomes much more
+complex. It is also a complication that when a custom encoding
+does not match any standard specification, it becomes impossible to document
+what encoding is supported in short and precise ways. Either a long
+document spelling out the rules that are in place must be written, or
+else false claims are made about using "UTF-8" supplemented by haphazard
+revelations to the curious questioners of all the exceptions
+and transgressions that are acutally in place.
+
+That is not the end. Even when all overlong encodings of all codepoints
+of UCS-2 are accepted, there are still many, many more byte sequences
+to consider. A scheme demanding UTF-8 encoding would be expected to
+reject these other sequences, either by raising an error, or by inserting
+the **REPLACEMENT CHARACTER** in suitable places to mark the non-conformant
+input. This is not what **Tcl_UtfToUniChar** does. Instead when any
+byte is encountered that is invalid by the modified rules of UTF-8 already
+described, that byte is taken as an encoding of the code-point with the
+same value as the byte value. That is, when the modified UTF-8 encoding
+fails to be satisfied, the ISO-8859-1 encoding of Tcl 8.0 is assumed instead.
+Furthermore, this is not just a quiet accommodation, or perhaps an error
+in implementation. It is the explict documented behavior of
+**Tcl_UtfToUniChar** in Tcl 8.1.
+
+> If the input is not in proper UTF‐8 format, Tcl\_UtfToUniChar will
+> store the first byte of src in *chPtr as a Tcl\_UniChar between
+> 0x0000 and 0x00ff and return 1.
+
+The reasoning behind this encoding choice is not explicitly recorded
+in any known place. This does not appear to be a choice copied from
+Java. It seems a reasonable guess to presume this is intended to offer
+some measure of compatibility to deployed callers of Tcl 7 and Tcl 8.0,
+though the effectiveness of that strategy is questionable to say the least.
+A Tcl 8.1 that continues to treat the byte sequence (**0xC0**, **0x20**)
+as the same two bytes as Tcl 8.0, but begins to treat the byte
+sequence (**0xC0**, **0x80**) as a representation of **NUL** is not
+much of a compatibility accommodation to rely upon.
+
+many-to-one, interpretive freedom, (parse of \n)
+variable length
+
+Compat with 8.0
+ByteArrays
+
+UCS-2 sequences
+bytes v chars
+
+Two-tier encoding.
+
+Threading & memory validity
+
+API Counts?
+
+NUL naming limitations
+
+UTF-16 and surrogate pairs
+
+
+
+
+During the development of Tcl 8.1, the prevailing Unicode standards were
+Unicode 2.0 and 2.1. In these definitions of Unicode, all assigned codepoints
+were in the set U+0000 through U+FFFF, the set of characters within
+UCS-2, also called the Basic Multilingual Plane.
+
+The publication of RFC 2044 describing a UTF-8
+encoding for UCS-2 in October 1996 was also a key event.
+
+Note the end of overlong byte sequence interpretation (other than NUL)
+as bug fix
+https://core.tcl-lang.org/tcl/info/67aa9a2070
+This didn't happen at a clean minor version bump.
+
+
+
+MILESTONE: Bug 67aa9a2070 was fixed in June 2017, first released in Tcl 8.6.7
+in August 2017. This is when Tcl stopped accepting overlong byte sequences
+in its internal encoding as valid alternative ways to encode characters.
+
+MILESTONE: [8a640c771ee927c8], 2019-11-14, first released in Tcl 8.6.10
+in November 2019. This is when valid 4-byte sequences started being recognized
+and treated as something other than 4 improperly encoded ISO-8859-1 chars.
+
+