Twisted Mixing

Laurens Van Houtven



About me

Hi, I'm lvh.

I hack on and with Twisted.



About this talk

  • Twisted code ↔ other code
  • Show that you probably can use Twisted
    • (Not so much that you should use Twisted)
  • Overview

Why should you care?

  • Twisted has some cool stuff in and around it
    • IRC, SMTP, DNS, SSH, WebSockets…
    • Running multiple things in one process
    • Timed events, sane process and thread management
    • Protocol and transport abstractions
  • Twisted vs other stuff: good at different things
    • Many connections, particularly if relatively silent
      • Incoming, e.g. a chat server
      • Outgoing, e.g. a scraper
    • Cooperate with existing event loops
      • GUIs
      • gevent


  • Ideally, just Python
  • Having played with Twisted probably helps
  • I will explain important concepts briefly
    • Please feel free to interrupt if I forget to

Introducing Twisted


An object that reacts to events

Reactor interfaces

Usually you call higher-level APIs!

  • IReactorBase: run, stop
  • IReactorTime: callLater
  • IReactorProcess: spawnProcess
  • IReactorThreads: call(In|From)Thread, …
  • IReactor(TCP|UDP|SSL|Multicast)
  • IReactor(UNIX|UNIXDatagram|Socket)
  • IReactorFDSet: (add|remove)(Reader|Writer), …


An object you get now,

gets you result or failure later

Why deferreds?

  • Many operations take time: can't get a result right now
  • Blocking API
    • Evaluate or raise some point in the future
    • Thread can't do anything else in the mean while
  • Deferred API
    • Get an object representing the future result now
    • Actually give result (or failure) when it's available
    • Thread is free to do something else

Deferred vs blocking API

Blocking read:

    result = blocking_read()
except SomeError as e:

Deferred read:

d = async_read()
d.addCallbacks(on_result, on_failure)

Inline callbacks vs blocking API

Blocking read:

    result = blocking_read()
except SomeError as e:

inlineCallbacks: Deferreds + sugar

    result = yield async_read()
except SomeError as e:

Twisted and your app

Is there even a problem?

Maybe it's trivial to get started!


Service Oriented Architecture


Web Server Gateway Interface

twistd web

Porting your app to Twisted

  • No, it's not trivial
    • … but cost is almost always overestimated
  • Clean, tested code helps
    • Keep verifying behavior
    • Tested code tends to be decoupled
  • If it's near impossible, it's probably a code smell

Writing code that works on both

  • Not trivial, not hard either
  • Example: praekelt/vumi
  • TODO: more examples


  • Flask app, served by t.w.wsgi
  • Real-time chat, with txsockjs

Test run, demo

Blocking code in Twisted

You can't block the reactor thread

  • Twisted is event-driven
    • Production reactors are just event loops
    • select, epoll, kqueue, IOCP, libev(ent)
  • Reactor runs in a thread, calls everything else
    • One thing at a time, all in the same thread
    • Concurrency through asynchronous IO
  • Blocking the reactor thread means nothing else happens

Blocking in a callback is bad!

Blocking IO

def _getDataAtURL(url):
    return requests.get(url).json() # BLOCKS!

Blocking computation

def _compute(n):
    x = 2
    for _ in xrange(n): # BLOCKS! (for large n)
        x *= x

Can't block the reactor thread!

Therefore, you have two options:

  1. Don't block
  2. Block another thread

Don't block

IO bound? Be asynchronous!

CPU bound? Cooperate with the event loop!

Asynchronous I/O version

treq: requests-like, but asynchronous

def _getDataAtURL(url):
    d = treq.get(url)
    return d

Cooperative version

twisted.internet.task.coiterate and friends

def _compute(n):
    x = 2
    for _ in xrange(n):
        x *= x
        yield # Yields to the reactor :)


Don't block?

Avoiding blocking isn't always possible

  • Blocking API
    • DBAPI2, …
  • Sometimes in C code you can't or don't want to mess with
    • scrypt, …
  • Sometimes at a kernel/syscall level
    • File IO, …

Block somewhere else

Can't block the reactor thread → block a different one!

  • … in the same process: deferToThread
    • often used by wrappers: adbapi, txscrypt
  • … in a child process: spawnProcess and friends
  • … in a remote process: Ampoule, PB, Foolscap, RPC…


  • Easy automagic deferreds!
  • Shared mutable state ☹

Twisted in blocking code

New hotness!


Setting it up

from crochet import setup


Using it

  • @run_in_reactor
    • Function runs in reactor thread, not calling thread
    • Results in an EventualResult
  • EventualResult?
    • Synchronous analog of Deferred
    • wait(timeout=None)
    • cancel(), stash()


from twisted.web.client import getPage
from crochet import setup, run_in_reactor

def download_page(url):
    return getPage(url)

result = download_page("")
print result.wait()

Longer example: exchange rates

  • Twisted queries exchange rate every 30s
    • Runs in a separate thread using crochet
  • Flask app serves the latest exchange rate


How does it work? Twisted part

class ExchangeRate(object):
    # ...

    def start(self):
        # in reactor thread because of decorator
        self._lc = LoopingCall(self._download)
        self._lc.start(30, now=True)

    def _download(self):
        # in reactor thread because of LoopingCall
        d = getPage(url)
        # ...

Twisted code looks like regular Twisted code!

(But remember the @run_in_reactor)

How does it work? Flask part

def index():
    # runs in whatever thread runs it in
    rate = EURUSD.latest_value()
    if rate is None:
        rate = "unavailable"
    return "EUR/USD rate: {0}.".format(rate)

Flask code looks like regular Flask code!

Twisted in Gevent

Water and fire, but it works


Setting it up

import geventreactor


"Blocking" code

  • Gevent-style "blocking", i.e. automagic suspending
  • Supported in most places
    • Suspending the reactor greenlet: bad
    • Actually blocking anything: bad
    • Earlier requests.get example: probably okay

Deferreds ↔ greenlets

r = waitForDeferred(d)

d = waitForGreenlet(g)




Twisted plays well with others

  • It supports many protocols
    • JSON-RPC, XML-RPC, HTTP/REST, AMP, whatever
    • It will serve your WSGI apps
  • It will work side by side with existing blocking code
  • It can have blocking code added to it later
  • It will cooperate with most existing event loops
    • Gevent, libev(ent), CoreFoundation…
    • GUIs, like GTK, Qt…
    • ZeroMQ (but please don't use ZeroMQ)
  • Many analogs for things you already know are available
    • treq is like requests
    • klein is like Flask
    • cyclone is Tornado on top of Twisted's reactor


  • If you want to use Twisted, you probably can
  • That doesn't mean it's a good idea
    • … although it probably is ;-)