Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
| Comment: | Rudimentary tokenization. |
|---|---|
| Downloads: | Tarball | ZIP archive | SQL archive |
| Timelines: | family | ancestors | descendants | both | trunk |
| Files: | files | file ages | folders |
| SHA1: | 888382bab802c31e37c2950c188e3560dfc97a38 |
| User & Date: | jaccarmac 2017-01-02 20:36:54 |
Context
|
2017-01-02
| ||
| 20:37 | Release 0.4.0 check-in: 632678c17b user: jaccarmac tags: trunk, v0.4.0 | |
| 20:36 | Rudimentary tokenization. check-in: 888382bab8 user: jaccarmac tags: trunk | |
|
2017-01-01
| ||
| 13:02 | Release 0.3.0 check-in: bbba264244 user: jaccarmac tags: trunk, v0.3.0 | |
Changes
Changes to main.py.
1 2 3 4 |
#!/usr/bin/env python3
for line in open('main.st'):
print(line.split())
|
> > > > > > > > > > < > |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
#!/usr/bin/env python3 def tokenize(line): space_sep = line.split() parsed = [] for token in space_sep: if token.endswith("'"): parsed.append(parsed.pop() + ' ' + token) else: parsed.append(token) return parsed for line in open('main.st'): print(tokenize(line)) |