Marpa

Artifact [20f5c8ef2d]
Login
Tcl 2016 Conference, Houston/TX, US, Nov 14-18
Send your abstracts to tclconference@googlegroups.com by Sep 12.

Artifact 20f5c8ef2d2ffab257f37a1f7d7dbd86db47fbda:

Wiki page [handling unicode] by aku 2017-06-27 18:18:44.
D 2017-06-27T18:18:44.840
L handling\sunicode
N text/x-markdown
P 2d88c6bb26e54dab973c551df9863baba966a324
U aku
W 1676
Up: [Notes](wiki?name=Notes)

# Unicode

 * Generally expect the input in UTF-8 encoding.
 * Characters are __byte__-sequences, 1 to 4 bytes long.

# Engine

 * The lexer engine takes bytes as input, not characters.

# Grammars

 * To compensate for the engine above character-based grammars are rewritten into a byte-based form where multi-byte characters are represented by rules reflecting their byte sequences.
 * Character classes become alternatives of characters, which in turn are byte sequences.
 * It is possible to optimize the above into a set of alternatives of sequences of byte-ranges.

Relevant references

 * Russ Cox's [Regular Expression Matching in the Wild](https://swtch.com/~rsc/regexp/regexp3.html), see <b>Step 3</b>.
 * Google's [RE2](https://github.com/google/re2) (also Russ Cox)
 * BurntSushi's Rust crate for [utf8-ranges](https://github.com/BurntSushi/utf8-ranges)
 * Lucene's [UTF-32 to UTF-8](https://github.com/apache/lucene-solr/blob/ae93f4e7ac6a3908046391de35d4f50a0d3c59ca/lucene/core/src/java/org/apache/lucene/util/automaton/UTF32ToUTF8.java)
 * [Stefan](http://wiki.tcl.tk/44258)'s [Tcl/Regex re-implementation](https://chiselapp.com/user/stefank/repository/tclstuff/doc/trunk/www/regex.html)

Regardless, at the bottom the engine has to support only bytes and byte-ranges, or even only bytes, with the ranges rewritten into alternations. (Finite, at most 256 for a full range [00-ff]).

As a side effect we can support the full range of unicode character classes, despite Tcl itself not supporting them.


* [Generic unicode table reader (python)](https://github.com/google/re2/blob/master/re2/unicode.py)
Z cf7419af32379b249601e21f65e1aaf6