Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
| Comment: | Rudimentary tokenization. |
|---|---|
| Downloads: | Tarball | ZIP archive |
| Timelines: | family | ancestors | descendants | both | trunk |
| Files: | files | file ages | folders |
| SHA1: |
888382bab802c31e37c2950c188e3560 |
| User & Date: | jaccarmac 2017-01-02 20:36:54.386 |
Context
|
2017-01-02
| ||
| 20:37 | Release 0.4.0 check-in: 632678c17b user: jaccarmac tags: trunk, v0.4.0 | |
| 20:36 | Rudimentary tokenization. check-in: 888382bab8 user: jaccarmac tags: trunk | |
|
2017-01-01
| ||
| 13:02 | Release 0.3.0 check-in: bbba264244 user: jaccarmac tags: trunk, v0.3.0 | |
Changes
Changes to main.py.
1 2 3 |
#!/usr/bin/env python3
for line in open('main.st'):
| > > > > > > > > > > | | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
#!/usr/bin/env python3
def tokenize(line):
space_sep = line.split()
parsed = []
for token in space_sep:
if token.endswith("'"):
parsed.append(parsed.pop() + ' ' + token)
else:
parsed.append(token)
return parsed
for line in open('main.st'):
print(tokenize(line))
|