Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Rudimentary tokenization. |
---|---|
Downloads: | Tarball | ZIP archive |
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA1: |
888382bab802c31e37c2950c188e3560 |
User & Date: | jaccarmac 2017-01-02 20:36:54.386 |
Context
2017-01-02
| ||
20:37 | Release 0.4.0 check-in: 632678c17b user: jaccarmac tags: trunk, v0.4.0 | |
20:36 | Rudimentary tokenization. check-in: 888382bab8 user: jaccarmac tags: trunk | |
2017-01-01
| ||
13:02 | Release 0.3.0 check-in: bbba264244 user: jaccarmac tags: trunk, v0.3.0 | |
Changes
Changes to main.py.
1 2 3 | #!/usr/bin/env python3 for line in open('main.st'): | > > > > > > > > > > | | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | #!/usr/bin/env python3 def tokenize(line): space_sep = line.split() parsed = [] for token in space_sep: if token.endswith("'"): parsed.append(parsed.pop() + ' ' + token) else: parsed.append(token) return parsed for line in open('main.st'): print(tokenize(line)) |