Changes On Branch dgp-fixme
Not logged in

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Changes In Branch dgp-fixme Excluding Merge-Ins

This is equivalent to a diff from 3153d51ddc to 36303870fc

2020-04-17
12:20
Add test-case for Tcl_UtfNext/Tcl_UtfPrev. About 15 of them give the wrong answer, but - at least - ... check-in: 4302a5d8b4 user: jan.nijtmans tags: core-8-branch
05:49
Merging forward the Utf changes. Needs some repair yet. Leaf check-in: 36303870fc user: dgp tags: dgp-fixme
05:16
Clean test suite for both TCL_UTF_MAX=3 and TCL_UTF_MAX=4 builds. check-in: c18f3f5870 user: dgp tags: core-8-6-branch
2020-04-16
14:54
Merge 8.7 check-in: 2603483e3a user: jan.nijtmans tags: trunk
11:14
Merge 8.7 check-in: 3153d51ddc user: jan.nijtmans tags: core-8-branch
10:32
Upgrade Xcode from 11.3 to 11.4. Let Travis test whether "make install" works when the destination d... check-in: 89b6529252 user: jan.nijtmans tags: core-8-6-branch
2020-04-15
20:12
Use TclGetBytesFromObj() in testcases in stead of Tcl_GetByteArrayFromObj(), since we only want to h... check-in: b280eb5b78 user: jan.nijtmans tags: core-8-branch

Changes to doc/Utf.3.
249
250
251
252
253
254
255

256

257
258
259
260








261
262
263
264
265
266
267
considered part of the UTF-8 string.
.PP
\fBTcl_UtfFindLast\fR corresponds to \fBstrrchr\fR for UTF-8 strings.  It
returns a pointer to the last occurrence of the Unicode character \fIch\fR
in the null-terminated UTF-8 string \fIsrc\fR.  The null terminator is
considered part of the UTF-8 string.
.PP

Given \fIsrc\fR, a pointer to some location in a UTF-8 string,

\fBTcl_UtfNext\fR returns a pointer to the next UTF-8 character in the
string.  The caller must not ask for the next character after the last
character in the string if the string is not terminated by a null
character.








.PP
\fBTcl_UtfPrev\fR is used to step backward through but not beyond the
UTF-8 string that begins at \fIstart\fR.  If the UTF-8 string is made
up entirely of complete and well-formed characters, and \fIsrc\fR points
to the lead byte of one of those characters (or to the location one byte
past the end of the string), then repeated calls of \fBTcl_UtfPrev\fR will
return pointers to the lead bytes of each character in the string, one







>
|
>
|
<
|
|
>
>
>
>
>
>
>
>







249
250
251
252
253
254
255
256
257
258
259

260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
considered part of the UTF-8 string.
.PP
\fBTcl_UtfFindLast\fR corresponds to \fBstrrchr\fR for UTF-8 strings.  It
returns a pointer to the last occurrence of the Unicode character \fIch\fR
in the null-terminated UTF-8 string \fIsrc\fR.  The null terminator is
considered part of the UTF-8 string.
.PP
\fBTcl_UtfNext\fR is used to step forward through a UTF-8 string.
If the UTF-8 string is made up entirely of complete, well-formed, and
valid character byte sequences, and \fIsrc\fR points to the lead byte
of one of those sequences, then repeated calls of \fBTcl_UtfNext\fR will

return pointers to the lead bytes of each character in the string, one
character at a time. In any other circumstance, \fBTcl_UtfNext\fR
returns \fIsrc\fR+1.  \fBTcl_UtfNext\fR will always read \fIsrc[0]\fR
and may read as many following bytes (up to a total of \fBTCL_UTF_MAX\fR)
as needed to find the end of the byte sequence. If the string is
\fBNUL\fR-terminated, \fBTcl_UtfNext\fR will not read beyond the terminating
\fBNUL\fR byte. If not, the caller must use the companion routine
\fBTcl_UtfCharComplete\fR to determine whether there is any risk
\fBTcl_UtfNext\fR might read beyond the readable memory occupied
by the string.
.PP
\fBTcl_UtfPrev\fR is used to step backward through but not beyond the
UTF-8 string that begins at \fIstart\fR.  If the UTF-8 string is made
up entirely of complete and well-formed characters, and \fIsrc\fR points
to the lead byte of one of those characters (or to the location one byte
past the end of the string), then repeated calls of \fBTcl_UtfPrev\fR will
return pointers to the lead bytes of each character in the string, one
Changes to generic/tclCmdMZ.c.
2525
2526
2527
2528
2529
2530
2531
2532
2533
2534
2535
2536
2537
2538
2539
	    int delta = 0;
	    const char *next;

	    if (!Tcl_UniCharIsWordChar(ch)) {
		break;
	    }

	    next = Tcl_UtfPrev(p, string);
	    do {
		next += delta;
		delta = TclUtfToUCS4(next, &ch);
	    } while (next + delta < p);
	    p = next;
	}
	if (cur != index) {







|







2525
2526
2527
2528
2529
2530
2531
2532
2533
2534
2535
2536
2537
2538
2539
	    int delta = 0;
	    const char *next;

	    if (!Tcl_UniCharIsWordChar(ch)) {
		break;
	    }

	    next = TclUtfPrev(p, string);
	    do {
		next += delta;
		delta = TclUtfToUCS4(next, &ch);
	    } while (next + delta < p);
	    p = next;
	}
	if (cur != index) {
Changes to generic/tclCompExpr.c.
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
    unsigned char *lexemePtr,	/* Write code of parsed lexeme to this
				 * storage. */
    Tcl_Obj **literalPtr)	/* Write corresponding literal value to this
				   storage, if non-NULL. */
{
    const char *end;
    int scanned;
    Tcl_UniChar ch = 0;
    Tcl_Obj *literal = NULL;
    unsigned char byte;

    if (numBytes == 0) {
	*lexemePtr = END;
	return 0;
    }







<







1889
1890
1891
1892
1893
1894
1895

1896
1897
1898
1899
1900
1901
1902
    unsigned char *lexemePtr,	/* Write code of parsed lexeme to this
				 * storage. */
    Tcl_Obj **literalPtr)	/* Write corresponding literal value to this
				   storage, if non-NULL. */
{
    const char *end;
    int scanned;

    Tcl_Obj *literal = NULL;
    unsigned char byte;

    if (numBytes == 0) {
	*lexemePtr = END;
	return 0;
    }
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
2117
     * We reject leading underscores in bareword.  No sensible reason why.
     * Might be inspired by reserved identifier rules in C, which of course
     * have no direct relevance here.
     */

    if (!TclIsBareword(*start) || *start == '_') {
	if (Tcl_UtfCharComplete(start, numBytes)) {
	    scanned = TclUtfToUniChar(start, &ch);
	} else {
	    char utfBytes[4];

	    memcpy(utfBytes, start, numBytes);
	    utfBytes[numBytes] = '\0';
	    scanned = TclUtfToUniChar(utfBytes, &ch);
	}
	*lexemePtr = INVALID;
	Tcl_DecrRefCount(literal);
	return scanned;
    }
    end = start;
    while (numBytes && TclIsBareword(*end)) {







|





|







2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
     * We reject leading underscores in bareword.  No sensible reason why.
     * Might be inspired by reserved identifier rules in C, which of course
     * have no direct relevance here.
     */

    if (!TclIsBareword(*start) || *start == '_') {
	if (Tcl_UtfCharComplete(start, numBytes)) {
	    scanned = TclUtfNext(start) - start;
	} else {
	    char utfBytes[4];

	    memcpy(utfBytes, start, numBytes);
	    utfBytes[numBytes] = '\0';
	    scanned = TclUtfNext(utfBytes) - utfBytes;
	}
	*lexemePtr = INVALID;
	Tcl_DecrRefCount(literal);
	return scanned;
    }
    end = start;
    while (numBytes && TclIsBareword(*end)) {
Changes to generic/tclInt.h.
4685
4686
4687
4688
4689
4690
4691








4692
4693
4694
4695
4696
4697
4698
	_count = (numBytes) - _i; \
	if (_i) { \
	    _count += Tcl_NumUtfChars((bytes) + _count, _i); \
	} \
	(numChars) = _count; \
    } while (0);









/*
 *----------------------------------------------------------------
 * Macro that encapsulates the logic that determines when it is safe to
 * interpret a string as a byte array directly. In summary, the object must be
 * a byte array and must not have a string representation (as the operations
 * that it is used in are defined on strings, not byte arrays). Theoretically
 * it is possible to also be efficient in the case where the object's bytes







>
>
>
>
>
>
>
>







4685
4686
4687
4688
4689
4690
4691
4692
4693
4694
4695
4696
4697
4698
4699
4700
4701
4702
4703
4704
4705
4706
	_count = (numBytes) - _i; \
	if (_i) { \
	    _count += Tcl_NumUtfChars((bytes) + _count, _i); \
	} \
	(numChars) = _count; \
    } while (0);

#define TclUtfPrev(src, start) \
	(((src) < (start)+2) ? (start) : \
	((unsigned char) *(src - 1)) < 0x80 ? (src)-1 : \
	Tcl_UtfPrev(src, start))

#define TclUtfNext(src)	\
	((((unsigned char) *(src)) < 0xC0) ? src + 1 : Tcl_UtfNext(src))

/*
 *----------------------------------------------------------------
 * Macro that encapsulates the logic that determines when it is safe to
 * interpret a string as a byte array directly. In summary, the object must be
 * a byte array and must not have a string representation (as the operations
 * that it is used in are defined on strings, not byte arrays). Theoretically
 * it is possible to also be efficient in the case where the object's bytes
Changes to generic/tclStringObj.c.
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
	toCopy = length;
    } else {
	if (ellipsis == NULL) {
	    ellipsis = "...";
	}
	eLen = strlen(ellipsis);
	while (eLen > limit) {
	    eLen = Tcl_UtfPrev(ellipsis+eLen, ellipsis) - ellipsis;
	}

	toCopy = Tcl_UtfPrev(bytes+limit+1-eLen, bytes) - bytes;
    }

    /*
     * If objPtr has a valid Unicode rep, then append the Unicode conversion
     * of "bytes" to the objPtr's Unicode rep, otherwise append "bytes" to
     * objPtr's string rep.
     */







|


|







1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
	toCopy = length;
    } else {
	if (ellipsis == NULL) {
	    ellipsis = "...";
	}
	eLen = strlen(ellipsis);
	while (eLen > limit) {
	    eLen = TclUtfPrev(ellipsis+eLen, ellipsis) - ellipsis;
	}

	toCopy = TclUtfPrev(bytes+limit+1-eLen, bytes) - bytes;
    }

    /*
     * If objPtr has a valid Unicode rep, then append the Unicode conversion
     * of "bytes" to the objPtr's Unicode rep, otherwise append "bytes" to
     * objPtr's string rep.
     */
2610
2611
2612
2613
2614
2615
2616
2617
2618
2619
2620
2621
2622
2623
2624

		/*
		 * Within that buffer, we trim both ends if needed so that we
		 * copy only whole characters, and avoid copying any partial
		 * multi-byte characters.
		 */

		q = Tcl_UtfPrev(end, bytes);
		if (!Tcl_UtfCharComplete(q, (int)(end - q))) {
		    end = q;
		}

		q = bytes + 4;
		while ((bytes < end) && (bytes < q)
			&& ((*bytes & 0xC0) == 0x80)) {







|







2610
2611
2612
2613
2614
2615
2616
2617
2618
2619
2620
2621
2622
2623
2624

		/*
		 * Within that buffer, we trim both ends if needed so that we
		 * copy only whole characters, and avoid copying any partial
		 * multi-byte characters.
		 */

		q = TclUtfPrev(end, bytes);
		if (!Tcl_UtfCharComplete(q, (int)(end - q))) {
		    end = q;
		}

		q = bytes + 4;
		while ((bytes < end) && (bytes < q)
			&& ((*bytes & 0xC0) == 0x80)) {
Changes to generic/tclUtf.c.
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

85
86
87
88
89
90
91
static const unsigned char totalBytes[256] = {
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
    3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,4,4,4,4,4,1,1,1,1,1,1,1,1,1,1,1
};

static const unsigned char complete[256] = {
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
/* Tcl_UtfCharComplete() might point to 2nd byte of valid 4-byte sequence */
    3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,
    3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,
/* End of "continuation byte section" */
    2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
    3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,4,4,4,4,4,1,1,1,1,1,1,1,1,1,1,1
};


/*
 *---------------------------------------------------------------------------
 *
 * TclUtfCount --
 *
 *	Find the number of bytes in the Utf character "ch".







|












|


>







62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
static const unsigned char totalBytes[256] = {
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    2,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
    3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,4,4,4,4,4,1,1,1,1,1,1,1,1,1,1,1
};

static const unsigned char complete[256] = {
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
/* Tcl_UtfCharComplete() might point to 2nd byte of valid 4-byte sequence */
    3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,
    3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,
/* End of "continuation byte section" */
    2,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
    3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,4,4,4,4,4,1,1,1,1,1,1,1,1,1,1,1
};
static int		Overlong(unsigned char *src);

/*
 *---------------------------------------------------------------------------
 *
 * TclUtfCount --
 *
 *	Find the number of bytes in the Utf character "ch".
110
111
112
113
114
115
116
117





















































118
119
120
121
122
123
124
	return 2;
    }
    if (((unsigned)(ch - 0x10000) <= 0xFFFFF)) {
	return 4;
    }
    return 3;
}






















































/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UniCharToUtf --
 *
 *	Store the given Tcl_UniChar as a sequence of UTF-8 bytes in the
 *	provided buffer. Equivalent to Plan 9 runetochar().







|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
	return 2;
    }
    if (((unsigned)(ch - 0x10000) <= 0xFFFFF)) {
	return 4;
    }
    return 3;
}

/*
 *---------------------------------------------------------------------------
 *
 * Overlong --
 *
 *	Utility routine to report whether /src/ points to the start of an
 *	overlong byte sequence that should be rejected. Caller guarantees
 *	that src[0] and src[1] are readable, and
 *
 *	(src[0] >= 0xC0) && (src[0] != 0xC1)
 * 	(src[1] >= 0x80) && (src[1] < 0xC0)
 *	(src[0] < ((TCL_UTF_MAX > 3) ? 0xF8 : 0xF0))
 *
 * Results:
 *	A boolean.
 *---------------------------------------------------------------------------
 */

static CONST unsigned char overlong[3] = {
    0x80,	/* \xD0 -- all sequences valid */
    0xA0,	/* \xE0\x80 through \xE0\x9F are invalid prefixes */
#if TCL_UTF_MAX >= 3
    0x90	/* \xF0\x80 through \xF0\x8F are invalid prefixes */
#else
    0xC0	/* Not used, but reject all again for safety. */
#endif
};

INLINE static int
Overlong(
    unsigned char *src)	/* Points to lead byte of a UTF-8 byte sequence */
{
    unsigned char byte = *src;

    if (byte % 0x10) {
	/* Only lead bytes 0xC0, 0xE0, 0xF0 need examination */
	return 0;
    }
    if (byte == 0xC0) {
	if (src[1] == 0x80) {
	    /* Valid sequence: \xC0\x80 for \u0000 */
	    return 0;
	}
	/* Reject overlong: \xC0\x81 - \xC0\xBF */
	return 1;
    }
    if (src[1] < overlong[(byte >> 4) - 0x0D]) {
	/* Reject overlong */
	return 1;
    }
    return 0;
}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UniCharToUtf --
 *
 *	Store the given Tcl_UniChar as a sequence of UTF-8 bytes in the
 *	provided buffer. Equivalent to Plan 9 runetochar().
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751

752
753
754
755
756
757
758
759

760
761
762
763
764

765
766
767
768
769
770
771

int
Tcl_NumUtfChars(
    const char *src,	/* The UTF-8 string to measure. */
    int length)			/* The length of the string in bytes, or -1
				 * for strlen(string). */
{
    Tcl_UniChar ch = 0;
    int i = 0;

    /*
     * The separate implementations are faster.
     *
     * Since this is a time-sensitive function, we also do the check for the
     * single-byte char case specially.
     */

    if (length < 0) {
	while (*src != '\0') {
	    src += TclUtfToUniChar(src, &ch);
	    i++;

	}
	if (i < 0) i = INT_MAX; /* Bug [2738427] */
    } else {
	const char *endPtr = src + length - 4;

	while (src < endPtr) {
	    src += TclUtfToUniChar(src, &ch);
	    i++;

	}
	endPtr += 4;
	while ((src < endPtr) && Tcl_UtfCharComplete(src, endPtr - src)) {
	    src += TclUtfToUniChar(src, &ch);
	    i++;

	}
	if (src < endPtr) {
	    i += endPtr - src;
	}
    }
    return i;
}







|










|
|
|
>

<




|
|
>



|
|
>







785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807

808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827

int
Tcl_NumUtfChars(
    const char *src,	/* The UTF-8 string to measure. */
    int length)			/* The length of the string in bytes, or -1
				 * for strlen(string). */
{
    const char *next;
    int i = 0;

    /*
     * The separate implementations are faster.
     *
     * Since this is a time-sensitive function, we also do the check for the
     * single-byte char case specially.
     */

    if (length < 0) {
	while ((*src != '\0') && (i < INT_MAX)) {
	    next = TclUtfNext(src);
	    i += 1 + ((next - src) > 3);
	    src = next;
	}

    } else {
	const char *endPtr = src + length - 4;

	while (src < endPtr) {
	    next = TclUtfNext(src);
	    i += 1 + ((next - src) > 3);
	    src = next;
	}
	endPtr += 4;
	while ((src < endPtr) && Tcl_UtfCharComplete(src, endPtr - src)) {
	    next = TclUtfNext(src);
	    i += 1 + ((next - src) > 3);
	    src = next;
	}
	if (src < endPtr) {
	    i += endPtr - src;
	}
    }
    return i;
}
848
849
850
851
852
853
854

855











856







857



858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874

875
876


877





878
879
880


881


882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937

938
939




940
941
942

943
944
945
946
947
948

949

950






951
952
953










954










955








956















957
958


959

960






961
962
963
964
965
966
967
968
}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UtfNext --
 *

 *	Given a pointer to some current location in a UTF-8 string, move











 *	forward one character. The caller must ensure that they are not asking







 *	for the next character after the last character in the string.



 *
 * Results:
 *	The return value is the pointer to the next character in the UTF-8
 *	string.
 *
 * Side effects:
 *	None.
 *
 *---------------------------------------------------------------------------
 */

const char *
Tcl_UtfNext(
    const char *src)		/* The current location in the string. */
{
    Tcl_UniChar ch = 0;
    int len = TclUtfToUniChar(src, &ch);


#if TCL_UTF_MAX <= 3


    if ((ch >= 0xD800) && (len < 3)) {





	len += TclUtfToUniChar(src + len, &ch);
    }
#endif


    return src + len;


}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UtfPrev --
 *
 *	The aim of this routine is to provide a way to move backward
 *	through a UTF-8 string. The caller is expected to pass non-NULL
 *	pointer arguments start and src. start points to the beginning
 *	of a string, and src >= start points to a location within (or just
 *	past the end) of the string. This routine always returns a
 *	pointer within the string (>= start).  When (src == start), it
 *	returns start. When (src > start), it returns a pointer (< src)
 *	and (>= src - TCL_UTF_MAX).  Subject to these constraints, the
 *	routine returns a pointer to the earliest byte in the string that
 *	starts a character when characters are read starting at start and
 *	that character might include the byte src[-1]. The routine will
 *	examine only those bytes in the range that might be returned.
 *	It will not examine the byte *src, and because of that cannot
 *	determine for certain in all circumstances whether the character
 *	that begins with the returned pointer will or will not include
 *	the byte src[-1]. In the scenario, where src points to the end of
 *	a buffer being filled, the returned pointer point to either the
 *	final complete character in the string or to the earliest byte
 *	that might start an incomplete character waiting for more bytes to
 *	complete.
 *
 *	Because this routine always returns a value < src until the point
 *	it is forced to return start, it is useful as a backward iterator
 *	through a string that will always make progress and always be
 *	prevented from running past the beginning of the string.
 *
 *	In a string where all characters are complete and properly formed,
 *	and the value of src points to the first byte of a character,
 *	repeated Tcl_UtfPrev calls will step to the starting bytes of
 *	characters, one character at a time. Within those limitations,
 *	Tcl_UtfPrev and Tcl_UtfNext are inverses. If either condition cannot
 *	be met, Tcl_UtfPrev and Tcl_UtfNext may not function as inverses and
 *	the caller will have to take greater care.
 *
 * Results:
 *	A pointer to the start of a character in the string as described
 *	above.
 *
 * Side effects:
 *	None.
 *
 *---------------------------------------------------------------------------
 */

const char *
Tcl_UtfPrev(
    const char *src,		/* A location in a UTF-8 string. */
    const char *start)		/* Pointer to the beginning of the string */
{

    const char *look;
    int i, byte;





    look = --src;
    for (i = 0; i < 4; i++) {

	if (look < start) {
	    if (src < start) {
		src = start;
	    }
	    break;
	}

	byte = *((unsigned char *) look);

	if (byte < 0x80) {






	    break;
	}
	if (byte >= 0xC0) {










	    if (totalBytes[byte] <= i) {










		break;








	    }















	    return look;
	}


	look--;

    }






    return src;
}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UniCharAtIndex --
 *







>
|
>
>
>
>
>
>
>
>
>
>
>
|
>
>
>
>
>
>
>
|
>
>
>


|
|











|
|
>

<
>
>
|
>
>
>
>
>
|
|
|
>
>
|
>
>









|
|
|
|
|
|
|
|


|


|
|




|
|




|
|
|
|
|
|
















>
|
|
>
>
>
>

<
<
>
|
<
|
|
<
|
>
|
>

>
>
>
>
>
>
|


>
>
>
>
>
>
>
>
>
>
|
>
>
>
>
>
>
>
>
>
>
|
>
>
>
>
>
>
>
>

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|

>
>

>
|
>
>
>
>
>
>
|







904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954

955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034


1035
1036

1037
1038

1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UtfNext --
 *
 *	The aim of this routine is to provide a way to iterate forward
 *	through a UTF-8 string. The caller is expected to pass a non-NULL
 *	pointer argument /src/ which points to a location within a string.
 *	(*src) will be read, so /src/ must not point to an unreadable
 *	location past the end of the string. If /src/ points to the
 *	beginning of a complete, well-formed and valid UTF_8 byte sequence
 *	of no more than TCL_UTF_MAX bytes, Tcl_UtfNext returns the pointer
 *	just past the end of that sequence. In any other circumstance,
 *	Tcl_UtfNext returns /src/+1.
 *
 *	Because this routine always returns a value > /src/, it is useful
 *	as a forward iterator that will always make progress. If the string
 *	is NUL-terminated, Tcl_UtfNext will not read beyond the terminating
 *	NUL character. If it is not NUL-terminated, the caller must make
 *	use of the companion routine Tcl_UtfCharComplete to test whether
 *	there is risk that Tcl_UtfNext will read beyond the end of the string.
 *	Tcl_UtfNext will never read more than TCL_UTF_MAX bytes.
 *
 *	In a string where all characters are complete and properly formed,
 *	and /src/ points to the first byte of a character, repeated
 *	Tcl_UtfNext calls will step to the starting bytes of characters, one
 *	character at a time. Within those limitations, Tcl_UtfPrev and
 *	Tcl_UtfNext are inverses. If either condition cannot be met,
 *	Tcl_UtfPrev and Tcl_UtfNext may not function as inverses and the
 *	caller will have to take greater care.
 *
 * Results:
 *	A pointer to the start of the next character in the string (or to
 *	the end of the string) as described above.
 *
 * Side effects:
 *	None.
 *
 *---------------------------------------------------------------------------
 */

const char *
Tcl_UtfNext(
    const char *src)		/* The current location in the string. */
{
    int byte = *((unsigned char *) src);
    int left = totalBytes[byte];
    const char *next = src + 1;


    while (--left) {
	byte = *((unsigned char *) next);
	if ((byte & 0xC0) != 0x80) {
	    /*
	     * src points to non-trail byte; We ran out of trail bytes
	     * before the needs of the lead byte were satisfied.
	     * Let the (malformed) lead byte alone be a character
	     */
	    return src + 1;
	}
	next++;
    }
    if (Overlong((unsigned char *)src)) {
	return src + 1;
    }
    return next;
}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UtfPrev --
 *
 *	The aim of this routine is to provide a way to move backward
 *	through a UTF-8 string. The caller is expected to pass non-NULL
 *	pointer arguments /start/ and /src/. /start/ points to the beginning
 *	of a string, and /src/ (>= /start/) points to a location within (or
 *	just past the end) of the string. This routine always returns a
 *	pointer within the string (>= /start/).  When (/src/ == /start/),
 *	it returns /start/. When (/src/ > /start/), it returns a pointer
 *	(< /src/) and (>= /src/ - TCL_UTF_MAX).  Subject to these constraints,
 *	the routine returns a pointer to the earliest byte in the string that
 *	starts a character when characters are read starting at /start/ and
 *	that character might include the byte src[-1]. The routine will
 *	examine only those bytes in the range that might be returned.
 *	It will not examine the byte (*src), and because of that cannot
 *	determine for certain in all circumstances whether the character
 *	that begins with the returned pointer will or will not include
 *	the byte src[-1]. In the scenario where /src/ points to the end of
 *	a buffer being filled, the returned pointer points to either the
 *	final complete character in the string or to the earliest byte
 *	that might start an incomplete character waiting for more bytes to
 *	complete.
 *
 *	Because this routine always returns a value < /src/ until the point
 *	it is forced to return /start/, it is useful as a backward iterator
 *	through a string that will always make progress and always be
 *	prevented from running past the beginning of the string.
 *
 *	In a string where all characters are complete and properly formed,
 *	and /src/ points to the first byte of a character, repeated
 *	Tcl_UtfPrev calls will step to the starting bytes of characters, one
 *	character at a time. Within those limitations, Tcl_UtfPrev and
 *	Tcl_UtfNext are inverses. If either condition cannot be met,
 *	Tcl_UtfPrev and Tcl_UtfNext may not function as inverses and the
 *	caller will have to take greater care.
 *
 * Results:
 *	A pointer to the start of a character in the string as described
 *	above.
 *
 * Side effects:
 *	None.
 *
 *---------------------------------------------------------------------------
 */

const char *
Tcl_UtfPrev(
    const char *src,		/* A location in a UTF-8 string. */
    const char *start)		/* Pointer to the beginning of the string */
{
    int trailBytesSeen = 0;	/* How many trail bytes have been verified? */
    CONST char *fallback = src - 1;
				/* If we cannot find a lead byte that might
				 * start a prefix of a valid UTF byte sequence,
				 * we will fallback to a one-byte back step */
    unsigned char *look = (unsigned char *)fallback;
				/* Start search at the fallback position */



    /* Quick boundary case exit. */
    if (fallback <= start) {

	return start;
    }


    do {
	unsigned char byte = look[0];

	if (byte < 0x80) {
	    /*
	     * Single byte character. Either this is a correct previous
	     * character, or it is followed by at least one trail byte
	     * which indicates a malformed sequence. In either case the
	     * correct result is to return the fallback.
	     */
	    return fallback;
	}
	if (byte >= 0xC0) {
	    /* Non-trail byte; May be multibyte lead. */

	    if ((trailBytesSeen == 0)
		/*
		 * We've seen no trailing context to use to check
		 * anything. From what we know, this non-trail byte
		 * is a prefix of a previous character, and accepting
		 * it (the fallback) is correct.
		 */

		    || (trailBytesSeen >= totalBytes[byte])) {
		/*
		 * That is, (1 + trailBytesSeen > needed).
		 * We've examined more bytes than needed to complete
		 * this lead byte. No matter about well-formedness or
		 * validity, the sequence starting with this lead byte
		 * will never include the fallback location, so we must
		 * return the fallback location. See test utf-7.17
		 */
		return fallback;
	    }

	    /*
	     * trailBytesSeen > 0, so we can examine look[1] safely.
	     * Use that capability to screen out overlong sequences.
	     */

	    if (Overlong(look)) {
		/* Reject */
		return fallback;
	    }
	    return (CONST char *)look;
	}

	/* We saw a trail byte. */
	trailBytesSeen++;

	if ((CONST char *)look == start) {
	    /*
	     * Do not read before the start of the string
	     *
	     * If we get here, we've examined bytes at every location
	     * >= start and < src and all of them are trail bytes,
	     * including (*start).  We need to return our fallback
	     * and exit this loop before we run past the start of the string.
	     */
	    return fallback;
	}

	/* Continue the search backwards... */
	look--;
    } while (trailBytesSeen < /* was TCL_UTF_MAX */ 4);

    /*
     * We've seen 4 (was TCL_UTF_MAX) trail bytes, so we know there will not be a
     * properly formed byte sequence to find, and we can stop looking,
     * accepting the fallback.
     */

    return fallback;
}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UniCharAtIndex --
 *
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
 */

int
Tcl_UniCharAtIndex(
    const char *src,	/* The UTF-8 string to dereference. */
    int index)		/* The position of the desired character. */
{
    Tcl_UniChar ch = 0;
    int fullchar = 0;
#if TCL_UTF_MAX <= 3
	int len = 0;
#endif

    while (index-- >= 0) {
#if TCL_UTF_MAX <= 3
	src += (len = TclUtfToUniChar(src, &ch));
#else
	src += TclUtfToUniChar(src, &ch);
#endif
    }
    fullchar = ch;
#if TCL_UTF_MAX <= 3
    if ((ch >= 0xD800) && (len < 3)) {
	/* If last Tcl_UniChar was a high surrogate, combine with low surrogate */
	(void)TclUtfToUniChar(src, &ch);
	fullchar = (((fullchar & 0x3FF) << 10) | (ch & 0x3FF)) + 0x10000;
    }
#endif
    return fullchar;
}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UtfAtIndex --
 *







<
<
<
|
<

<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
|







1130
1131
1132
1133
1134
1135
1136



1137

1138




1139










1140
1141
1142
1143
1144
1145
1146
1147
 */

int
Tcl_UniCharAtIndex(
    const char *src,	/* The UTF-8 string to dereference. */
    int index)		/* The position of the desired character. */
{



    int ch;






    TclUtfToUCS4(Tcl_UtfAtIndex(src, index), &ch);










    return ch;
}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UtfAtIndex --
 *
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041





1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
 */

const char *
Tcl_UtfAtIndex(
    const char *src,	/* The UTF-8 string. */
    int index)		/* The position of the desired character. */
{
    Tcl_UniChar ch = 0;
    int len = 0;

    while (index-- > 0) {
	len = TclUtfToUniChar(src, &ch);
	src += len;
    }
#if TCL_UTF_MAX <= 3





    if ((ch >= 0xD800) && (len < 3)) {
	/* Index points at character following high Surrogate */
	src += TclUtfToUniChar(src, &ch);
    }
#endif
    return src;
}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UtfBackslash --







<
<
<

|
<
|
<
>
>
>
>
>
|
|
|

<







1160
1161
1162
1163
1164
1165
1166



1167
1168

1169

1170
1171
1172
1173
1174
1175
1176
1177
1178

1179
1180
1181
1182
1183
1184
1185
 */

const char *
Tcl_UtfAtIndex(
    const char *src,	/* The UTF-8 string. */
    int index)		/* The position of the desired character. */
{



    while (index-- > 0) {
        const char *next = TclUtfNext(src);



	/*
	 * 4-byte sequences generate two UCS-2 code units in the
	 * UTF-16 representation, so in the current indexing scheme
	 * we need to account for an extra index (total of two).
	 */
	index -= ((next - src) > 3);

	src = next;
    }

    return src;
}

/*
 *---------------------------------------------------------------------------
 *
 * Tcl_UtfBackslash --
Changes to generic/tclUtil.c.
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
     * Outer loop: iterate over string to be trimmed.
     */

    do {
	const char *q = trim;
	int pInc = 0, bytesLeft = numTrim;

	pp = Tcl_UtfPrev(p, bytes);
	do {
	    pp += pInc;
 	    pInc = TclUtfToUCS4(pp, &ch1);
	} while (pp + pInc < p);

	/*
	 * Inner loop: scan trim string for match to current character.







|







1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
     * Outer loop: iterate over string to be trimmed.
     */

    do {
	const char *q = trim;
	int pInc = 0, bytesLeft = numTrim;

	pp = TclUtfPrev(p, bytes);
	do {
	    pp += pInc;
 	    pInc = TclUtfToUCS4(pp, &ch1);
	} while (pp + pInc < p);

	/*
	 * Inner loop: scan trim string for match to current character.
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
	trimLeft = TclTrimLeft(bytes, numBytes, trim, numTrim);
	numBytes -= trimLeft;

	/* If we did not trim the whole string, it starts with a character
	 * that we will not trim. Skip over it. */
	if (numBytes > 0) {
	    const char *first = bytes + trimLeft;
	    bytes = Tcl_UtfNext(first);
	    numBytes -= (bytes - first);

	    if (numBytes > 0) {
		/* When bytes is NUL-terminated, returns
		 * 0 <= trimRight <= numBytes */
		trimRight = TclTrimRight(bytes, numBytes, trim, numTrim);
	    }







|







1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
	trimLeft = TclTrimLeft(bytes, numBytes, trim, numTrim);
	numBytes -= trimLeft;

	/* If we did not trim the whole string, it starts with a character
	 * that we will not trim. Skip over it. */
	if (numBytes > 0) {
	    const char *first = bytes + trimLeft;
	    bytes = TclUtfNext(first);
	    numBytes -= (bytes - first);

	    if (numBytes > 0) {
		/* When bytes is NUL-terminated, returns
		 * 0 <= trimRight <= numBytes */
		trimRight = TclTrimRight(bytes, numBytes, trim, numTrim);
	    }
Changes to generic/tclZlib.c.
3072
3073
3074
3075
3076
3077
3078
3079


3080
3081
3082
3083
3084
3085
3086
	 * Length (cd->decompressed) == 0, toRead > 0 here.
	 *
	 * The zlib transform allows us to read at most one character from the
	 * underlying channel to properly identify Z_STREAM_END without
	 * reading over the border.
	 */

	readBytes = Tcl_ReadRaw(cd->parent, cd->inBuffer, cd->readAheadLimit);



	/*
	 * Three cases here:
	 *  1.	Got some data from the underlying channel (readBytes > 0) so
	 *	it should be fed through the decompression engine.
	 *  2.	Got an error (readBytes < 0) which we should report up except
	 *	for the case where we can convert it to a short read.







|
>
>







3072
3073
3074
3075
3076
3077
3078
3079
3080
3081
3082
3083
3084
3085
3086
3087
3088
	 * Length (cd->decompressed) == 0, toRead > 0 here.
	 *
	 * The zlib transform allows us to read at most one character from the
	 * underlying channel to properly identify Z_STREAM_END without
	 * reading over the border.
	 */

	readBytes = Tcl_ReadRaw(cd->parent, cd->inBuffer, 
		cd->readAheadLimit <= cd->inAllocated ?
		cd->readAheadLimit  : cd->inAllocated);

	/*
	 * Three cases here:
	 *  1.	Got some data from the underlying channel (readBytes > 0) so
	 *	it should be fed through the decompression engine.
	 *  2.	Got an error (readBytes < 0) which we should report up except
	 *	for the case where we can convert it to a short read.
Changes to tests/utf.test.
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
    string length [testbytestring "\xF4\x8F\xBF\xBF"]
} -result {2}
test utf-2.10 {Tcl_UtfToUniChar: lead (4-byte) followed by 3 trail, underflow} testbytestring {
    string length [testbytestring "\xF0\x8F\xBF\xBF"]
} {4}
test utf-2.11 {Tcl_UtfToUniChar: lead (4-byte) followed by 3 trail, overflow} testbytestring {
    string length [testbytestring "\xF4\x90\x80\x80"]
} {4}
test utf-2.12 {Tcl_UtfToUniChar: longer UTF sequences not supported} testbytestring {
    string length [testbytestring "\xF8\xA2\xA2\xA2\xA2"]
} {5}

test utf-3.1 {Tcl_UtfCharComplete} {
} {}








|







91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
    string length [testbytestring "\xF4\x8F\xBF\xBF"]
} -result {2}
test utf-2.10 {Tcl_UtfToUniChar: lead (4-byte) followed by 3 trail, underflow} testbytestring {
    string length [testbytestring "\xF0\x8F\xBF\xBF"]
} {4}
test utf-2.11 {Tcl_UtfToUniChar: lead (4-byte) followed by 3 trail, overflow} testbytestring {
    string length [testbytestring "\xF4\x90\x80\x80"]
} {2}
test utf-2.12 {Tcl_UtfToUniChar: longer UTF sequences not supported} testbytestring {
    string length [testbytestring "\xF8\xA2\xA2\xA2\xA2"]
} {5}

test utf-3.1 {Tcl_UtfCharComplete} {
} {}

148
149
150
151
152
153
154


155



156





































































































































































































































































157
158
159
160
161
162
163
test utf-5.1 {Tcl_UtfFindFirst} {testfindfirst testbytestring} {
    testfindfirst [testbytestring "abcbc"] 98
} {bcbc}
test utf-5.2 {Tcl_UtfFindLast} {testfindlast testbytestring} {
    testfindlast [testbytestring "abcbc"] 98
} {bc}



test utf-6.1 {Tcl_UtfNext} {



} {}






































































































































































































































































testConstraint testutfprev [llength [info commands testutfprev]]

test utf-7.1 {Tcl_UtfPrev} testutfprev {
    testutfprev {}
} 0
test utf-7.2 {Tcl_UtfPrev} testutfprev {







>
>
|
>
>
>
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
test utf-5.1 {Tcl_UtfFindFirst} {testfindfirst testbytestring} {
    testfindfirst [testbytestring "abcbc"] 98
} {bcbc}
test utf-5.2 {Tcl_UtfFindLast} {testfindlast testbytestring} {
    testfindlast [testbytestring "abcbc"] 98
} {bc}

testConstraint testutfnext [llength [info commands testutfnext]]

test utf-6.1 {Tcl_UtfNext} testutfnext {
    # This takes the pointer one past the terminating NUL.
    # This is really an invalid call.
    testutfnext {}
} 1
test utf-6.2 {Tcl_UtfNext} testutfnext {
    testutfnext A
} 1
test utf-6.3 {Tcl_UtfNext} testutfnext {
    testutfnext AA
} 1
test utf-6.4 {Tcl_UtfNext} testutfnext {
    testutfnext A\xA0
} 1
test utf-6.5 {Tcl_UtfNext} testutfnext {
    testutfnext A\xD0
} 1
test utf-6.6 {Tcl_UtfNext} testutfnext {
    testutfnext A\xE8
} 1
test utf-6.7 {Tcl_UtfNext} testutfnext {
    testutfnext A\xF4
} 1
test utf-6.8 {Tcl_UtfNext} testutfnext {
    testutfnext A\xF8
} 1
test utf-6.9 {Tcl_UtfNext} testutfnext {
    testutfnext \xA0
} 1
test utf-6.10 {Tcl_UtfNext} testutfnext {
    testutfnext \xA0G
} 1
test utf-6.11 {Tcl_UtfNext} testutfnext {
    testutfnext \xA0\xA0
} 1
test utf-6.12 {Tcl_UtfNext} testutfnext {
    testutfnext \xA0\xD0
} 1
test utf-6.13 {Tcl_UtfNext} testutfnext {
    testutfnext \xA0\xE8
} 1
test utf-6.14 {Tcl_UtfNext} testutfnext {
    testutfnext \xA0\xF4
} 1
test utf-6.15 {Tcl_UtfNext} testutfnext {
    testutfnext \xA0\xF8
} 1
test utf-6.16 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0
} 1
test utf-6.17 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0A
} 1
test utf-6.18 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xA0
} 2
test utf-6.19 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xD0
} 1
test utf-6.20 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xE8
} 1
test utf-6.21 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xF4
} 1
test utf-6.22 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xF8
} 1
test utf-6.23 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8
} 1
test utf-6.24 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8A
} 1
test utf-6.25 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0
} 1
test utf-6.26 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xD0
} 1
test utf-6.27 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xE8
} 1
test utf-6.28 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xF4
} 1
test utf-6.29 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xF8
} 1
test utf-6.30 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4
} 1
test utf-6.31 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4A
} 1
test utf-6.32 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0
} 1
test utf-6.33 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xD0
} 1
test utf-6.34 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xE8
} 1
test utf-6.35 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xF4
} 1
test utf-6.36 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xF8
} 1
test utf-6.37 {Tcl_UtfNext} testutfnext {
    testutfnext \xF8
} 1
test utf-6.38 {Tcl_UtfNext} testutfnext {
    testutfnext \xF8A
} 1
test utf-6.39 {Tcl_UtfNext} testutfnext {
    testutfnext \xF8\xA0
} 1
test utf-6.40 {Tcl_UtfNext} testutfnext {
    testutfnext \xF8\xD0
} 1
test utf-6.41 {Tcl_UtfNext} testutfnext {
    testutfnext \xF8\xE8
} 1
test utf-6.42 {Tcl_UtfNext} testutfnext {
    testutfnext \xF8\xF4
} 1
test utf-6.43 {Tcl_UtfNext} testutfnext {
    testutfnext \xF8\xF8
} 1
test utf-6.44 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xA0G
} 2
test utf-6.45 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xA0\xA0
} 2
test utf-6.46 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xA0\xD0
} 2
test utf-6.47 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xA0\xE8
} 2
test utf-6.48 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xA0\xF4
} 2
test utf-6.49 {Tcl_UtfNext} testutfnext {
    testutfnext \xD0\xA0\xF8
} 2
test utf-6.50 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0G
} 1
test utf-6.51 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xA0
} 3
test utf-6.52 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xD0
} 1
test utf-6.53 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xE8
} 1
test utf-6.54 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xF4
} 1
test utf-6.55 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xF8
} 1
test utf-6.56 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0G
} 1
test utf-6.57 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0
} 1
test utf-6.58 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xD0
} 1
test utf-6.59 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xE8
} 1
test utf-6.60 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xF4
} 1
test utf-6.61 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xF8
} 1
test utf-6.62 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xA0G
} 3
test utf-6.63 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xA0\xA0
} 3
test utf-6.64 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xA0\xD0
} 3
test utf-6.65 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xA0\xE8
} 3
test utf-6.66 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xA0\xF4
} 3
test utf-6.67 {Tcl_UtfNext} testutfnext {
    testutfnext \xE8\xA0\xA0\xF8
} 3
test utf-6.68 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0G
} 1
test utf-6.69 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xA0
} 4
test utf-6.70 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xD0
} 1
test utf-6.71 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xE8
} 1
test utf-6.71 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xF4
} 1
test utf-6.73 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xF8
} 1
test utf-6.74 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xA0G
} 4
test utf-6.75 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xA0\xA0
} 4
test utf-6.76 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xA0\xD0
} 4
test utf-6.77 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xA0\xE8
} 4
test utf-6.78 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xA0\xF4
} 4
test utf-6.79 {Tcl_UtfNext} testutfnext {
    testutfnext \xF4\xA0\xA0\xA0G\xF8
} 4
test utf-6.80 {Tcl_UtfNext - overlong sequences} testutfnext {
    testutfnext \xC0\x80
} 2
test utf-6.81 {Tcl_UtfNext - overlong sequences} testutfnext {
    testutfnext \xC0\x81
} 1
test utf-6.82 {Tcl_UtfNext - overlong sequences} testutfnext {
    testutfnext \xC1\x80
} 1
test utf-6.83 {Tcl_UtfNext - overlong sequences} testutfnext {
    testutfnext \xC2\x80
} 2
test utf-6.84 {Tcl_UtfNext - overlong sequences} testutfnext {
    testutfnext \xE0\x80\x80
} 1
test utf-6.85 {Tcl_UtfNext - overlong sequences} testutfnext {
    testutfnext \xE0\xA0\x80
} 3
test utf-6.86 {Tcl_UtfNext - overlong sequences} testutfnext {
    testutfnext \xF0\x80\x80\x80
} 1
test utf-6.87 {Tcl_UtfNext - overlong sequences} {testutfnext} {
    testutfnext \xF0\x90\x80\x80
} 4
test utf-6.88 {Tcl_UtfNext, pointing to 2th byte of 3-byte valid sequence} {testutfnext} {
    testutfnext \xE8\xA0\xA0 1
} 2

testConstraint testutfprev [llength [info commands testutfprev]]

test utf-7.1 {Tcl_UtfPrev} testutfprev {
    testutfprev {}
} 0
test utf-7.2 {Tcl_UtfPrev} testutfprev {
312
313
314
315
316
317
318





































































319
320
321
322
323
324
325
} 4
test utf-7.22 {Tcl_UtfPrev} testutfprev {
    testutfprev A\xD0\xA0\xA0\xA0
} 4
test utf-7.23 {Tcl_UtfPrev} testutfprev {
    testutfprev A\xA0\xA0\xA0\xA0
} 4






































































test utf-8.1 {Tcl_UniCharAtIndex: index = 0} {
    string index abcd 0
} {a}
test utf-8.2 {Tcl_UniCharAtIndex: index = 0} {
    string index \u4E4E\u25A 0
} "\u4E4E"







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
} 4
test utf-7.22 {Tcl_UtfPrev} testutfprev {
    testutfprev A\xD0\xA0\xA0\xA0
} 4
test utf-7.23 {Tcl_UtfPrev} testutfprev {
    testutfprev A\xA0\xA0\xA0\xA0
} 4
test utf-7.24 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xC0\x81
} 2
test utf-7.25 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xC0\x81 2
} 1
test utf-7.26 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xE0\x80\x80
} 3
test utf-7.27 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xE0\x80\x80 3
} 2
test utf-7.28 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xE0\x80\x80 2
} 1
test utf-7.29 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xF0\x80\x80\x80
} 4
test utf-7.30 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xF0\x80\x80\x80 4
} 3
test utf-7.31 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xF0\x80\x80\x80 3
} 2
test utf-7.32 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xF0\x80\x80\x80 2
} 1
test utf-7.33 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xC0\x80
} 1
test utf-7.34 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xC1\x80
} 2
test utf-7.35 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xC2\x80
} 1
test utf-7.36 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xE0\xA0\x80
} 1
test utf-7.37 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xE0\xA0\x80 3
} 1
test utf-7.38 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xE0\xA0\x80 2
} 1
test utf-7.39 {Tcl_UtfPrev -- overlong sequence}  {testutfprev} {
    testutfprev A\xF0\x90\x80\x80
} 1
test utf-7.40 {Tcl_UtfPrev -- overlong sequence}  {testutfprev} {
    testutfprev A\xF0\x90\x80\x80 4
} 1
test utf-7.41 {Tcl_UtfPrev -- overlong sequence}  {testutfprev} {
    testutfprev A\xF0\x90\x80\x80 3
} 1
test utf-7.42 {Tcl_UtfPrev -- overlong sequence}  testutfprev {
    testutfprev A\xF0\x90\x80\x80 2
} 1
test utf-7.43 {Tcl_UtfPrev -- no lead byte at start}  testutfprev {
    testutfprev \xA0
} 0
test utf-7.44 {Tcl_UtfPrev -- no lead byte at start}  testutfprev {
    testutfprev \xA0\xA0
} 1
test utf-7.45 {Tcl_UtfPrev -- no lead byte at start}  testutfprev {
    testutfprev \xA0\xA0\xA0
} 2
test utf-7.46 {Tcl_UtfPrev -- no lead byte at start}  testutfprev {
    testutfprev \xA0\xA0\xA0\xA0
} 3

test utf-8.1 {Tcl_UniCharAtIndex: index = 0} {
    string index abcd 0
} {a}
test utf-8.2 {Tcl_UniCharAtIndex: index = 0} {
    string index \u4E4E\u25A 0
} "\u4E4E"