Twisted Mixing

Laurens Van Houtven

@lvh

Introduction

About me

Hi, I'm lvh.

I hack on and with Twisted.

Rackspace

About this talk

Twisted code ↔ other code
Show that you probably can use Twisted
- (Not so much that you should use Twisted)
Overview

Why should you care?

Twisted has some cool stuff in and around it
- IRC, SMTP, DNS, SSH, WebSockets…
- Running multiple things in one process
- Timed events, sane process and thread management
- Protocol and transport abstractions
- …
Twisted vs other stuff: good at different things
- Many connections, particularly if relatively silent
  - Incoming, e.g. a chat server
  - Outgoing, e.g. a scraper
- Cooperate with existing event loops
  - GUIs
  - gevent

Prerequisites

Ideally, just Python
Having played with Twisted probably helps
I will explain important concepts briefly
- Please feel free to interrupt if I forget to

Introducing Twisted

Reactor

An object that reacts to events

Reactor interfaces

Usually you call higher-level APIs!

IReactorBase: run, stop…
IReactorTime: callLater…
IReactorProcess: spawnProcess
IReactorThreads: call(In|From)Thread, …
IReactor(TCP|UDP|SSL|Multicast)
IReactor(UNIX|UNIXDatagram|Socket)
IReactorFDSet: (add|remove)(Reader|Writer), …

Deferred

An object you get now,

gets you result or failure later

Why deferreds?

Many operations take time: can't get a result right now
Blocking API
- Evaluate or raise some point in the future
- Thread can't do anything else in the mean while
Deferred API
- Get an object representing the future result now
- Actually give result (or failure) when it's available
- Thread is free to do something else

Deferred vs blocking API

Blocking read:

try:
    result = blocking_read()
    on_result(result)
except SomeError as e:
    on_failure(e)

Deferred read:

d = async_read()
d.addCallbacks(on_result, on_failure)

Inline callbacks vs blocking API

Blocking read:

try:
    result = blocking_read()
    on_result(result)
except SomeError as e:
    on_failure(e)

inlineCallbacks: Deferreds + sugar

try:
    result = yield async_read()
    on_result(result)
except SomeError as e:
    on_failure(e)

Twisted and your app

Is there even a problem?

Maybe it's trivial to get started!

SOA

Service Oriented Architecture

WSGI

Web Server Gateway Interface

twistd web --wsgi=location.of.wsgi.app

Porting your app to Twisted

No, it's not trivial
- … but cost is almost always overestimated
Clean, tested code helps
- Keep verifying behavior
- Tested code tends to be decoupled
If it's near impossible, it's probably a code smell

Writing code that works on both

Not trivial, not hard either
Example: praekelt/vumi
TODO: more examples

Demo

Flask app, served by t.w.wsgi
Real-time chat, with txsockjs

Test run, demo

Blocking code in Twisted

You can't block the reactor thread

Twisted is event-driven
- Production reactors are just event loops
- select, epoll, kqueue, IOCP, libev(ent)…
Reactor runs in a thread, calls everything else
- One thing at a time, all in the same thread
- Concurrency through asynchronous IO
Blocking the reactor thread means nothing else happens

Blocking in a callback is bad!

Blocking IO

def _getDataAtURL(url):
    return requests.get(url).json() # BLOCKS!

Blocking computation

def _compute(n):
    x = 2
    for _ in xrange(n): # BLOCKS! (for large n)
        x *= x
    send_somewhere(x)

Can't block the reactor thread!

Therefore, you have two options:

Don't block
Block another thread

Don't block

IO bound? Be asynchronous!

CPU bound? Cooperate with the event loop!

Asynchronous I/O version

treq: requests-like, but asynchronous

def _getDataAtURL(url):
    d = treq.get(url)
    d.addCallback(treq.json_content)
    return d

Cooperative version

twisted.internet.task.coiterate and friends

def _compute(n):
    x = 2
    for _ in xrange(n):
        x *= x
        yield # Yields to the reactor :)
    send_somewhere(x)

coiterate(_compute(n))

Don't block?

Avoiding blocking isn't always possible

Blocking API
- DBAPI2, …
Sometimes in C code you can't or don't want to mess with
- scrypt, …
Sometimes at a kernel/syscall level
- File IO, …

Block somewhere else

Can't block the reactor thread → block a different one!

… in the same process: deferToThread
- often used by wrappers: adbapi, txscrypt…
… in a child process: spawnProcess and friends
… in a remote process: Ampoule, PB, Foolscap, RPC…

deferToThread

Easy automagic deferreds!
Shared mutable state ☹

Twisted in blocking code

New hotness!

itamarst/crochet

Setting it up

from crochet import setup

setup()

Using it

@run_in_reactor
- Function runs in reactor thread, not calling thread
- Results in an EventualResult
EventualResult?
- Synchronous analog of Deferred
- wait(timeout=None)
- cancel(), stash()

Example

from twisted.web.client import getPage
from crochet import setup, run_in_reactor
setup()

@run_in_reactor
def download_page(url):
    return getPage(url)

result = download_page("http://www.google.com")
print result.wait()

Longer example: exchange rates

Twisted queries exchange rate every 30s
- Runs in a separate thread using crochet
Flask app serves the latest exchange rate

Demo

How does it work? Twisted part

class ExchangeRate(object):
    # ...

    @run_in_reactor
    def start(self):
        # in reactor thread because of decorator
        self._lc = LoopingCall(self._download)
        self._lc.start(30, now=True)

    def _download(self):
        # in reactor thread because of LoopingCall
        d = getPage(url)
        # ...

Twisted code looks like regular Twisted code!

(But remember the @run_in_reactor)

How does it work? Flask part

@app.route('/')
def index():
    # runs in whatever thread app.run() runs it in
    rate = EURUSD.latest_value()
    if rate is None:
        rate = "unavailable"
    return "EUR/USD rate: {0}.".format(rate)

app.run()

Flask code looks like regular Flask code!

Twisted in Gevent

Water and fire, but it works

jyio/geventreactor

Setting it up

import geventreactor

geventreactor.install()

"Blocking" code

Gevent-style "blocking", i.e. automagic suspending

Supported in most places
- Suspending the reactor greenlet: bad
- Actually blocking anything: bad
- Earlier requests.get example: probably okay

Deferreds ↔ greenlets

r = waitForDeferred(d)

d = waitForGreenlet(g)

Demo

TODO

Recap

Twisted plays well with others

It supports many protocols
- JSON-RPC, XML-RPC, HTTP/REST, AMP, whatever
- It will serve your WSGI apps
It will work side by side with existing blocking code
It can have blocking code added to it later
It will cooperate with most existing event loops
- Gevent, libev(ent), CoreFoundation…
- GUIs, like GTK, Qt…
- ZeroMQ (but please don't use ZeroMQ)
- …
Many analogs for things you already know are available
- treq is like requests
- klein is like Flask
- cyclone is Tornado on top of Twisted's reactor
- …

Conclusion

If you want to use Twisted, you probably can
That doesn't mean it's a good idea
- … although it probably is ;-)

Twisted Mixing

Laurens Van Houtven

@lvh

Introduction

About me

Rackspace

About this talk

Why should you care?

Prerequisites

Introducing Twisted

Reactor

Reactor interfaces

Deferred

Why deferreds?

Deferred vs blocking API

Inline callbacks vs blocking API

Twisted and your app

Is there even a problem?

SOA

WSGI

Porting your app to Twisted

Writing code that works on both

Demo

Blocking code in Twisted

You can't block the reactor thread

Blocking in a callback is bad!

Can't block the reactor thread!

Don't block

Asynchronous I/O version

Cooperative version

Don't block?

Block somewhere else

deferToThread

Twisted in blocking code

New hotness!

Setting it up

Using it

Example

Longer example: exchange rates

How does it work? Twisted part

How does it work? Flask part

Twisted in Gevent

Water and fire, but it works

Setting it up

"Blocking" code

Deferreds ↔ greenlets

Demo

Recap

Twisted plays well with others

Conclusion

Questions?