Perthtalk 3  Check-in [888382bab8]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Rudimentary tokenization.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 888382bab802c31e37c2950c188e3560dfc97a38
User & Date: jaccarmac 2017-01-02 20:36:54.386
Context
2017-01-02
20:37
Release 0.4.0 check-in: 632678c17b user: jaccarmac tags: trunk, v0.4.0
20:36
Rudimentary tokenization. check-in: 888382bab8 user: jaccarmac tags: trunk
2017-01-01
13:02
Release 0.3.0 check-in: bbba264244 user: jaccarmac tags: trunk, v0.3.0
Changes
Unified Diff Ignore Whitespace Patch
Changes to main.py.
1
2










3
4
#!/usr/bin/env python3











for line in open('main.st'):
    print(line.split())


>
>
>
>
>
>
>
>
>
>

|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/usr/bin/env python3

def tokenize(line):
    space_sep = line.split()
    parsed = []
    for token in space_sep:
        if token.endswith("'"):
            parsed.append(parsed.pop() + ' ' + token)
        else:
            parsed.append(token)
    return parsed

for line in open('main.st'):
    print(tokenize(line))