Planet Twisted

April 14, 2014

Future Foundries

Crochet 1.2.0, now with a better API!

Crochet is a library for using Twisted more easily from blocking programs and libraries. The latest version, released here at PyCon 2014, includes a much improved API for calling into Twisted from threads. In particular, a timeout is passed in - if it is hit the underlying operation is cancelled, and an exception is raised. Not all APIs in Twisted support cancellation, but for those that do (or APIs you implement) this is a really nice feature. You get high level timeouts (instead of blocking sockets' timeout-per-socket-operation) and automatic cleanup of resources if something takes too long.

#!/usr/bin/python
"""
Do a DNS lookup using Twisted's APIs.
"""
from __future__ import print_function

# The Twisted code we'll be using:
from twisted.names import client

from crochet import setup, wait_for
setup()


# Crochet layer, wrapping Twisted's DNS library in a blocking call.
@wait_for(timeout=5.0)
def gethostbyname(name):
"""Lookup the IP of a given hostname.

Unlike socket.gethostbyname() which can take an arbitrary amount
of time to finish, this function will raise crochet.TimeoutError
if more than 5 seconds elapse without an answer being received.
"""
d = client.lookupAddress(name)
d.addCallback(lambda result: result[0][0].payload.dottedQuad())
return d


if __name__ == '__main__':
# Application code using the public API - notice it works in a normal
# blocking manner, with no event loop visible:
import sys
name = sys.argv[1]
ip = gethostbyname(name)
print(name, "->", ip)

by Itamar Turner-Trauring (noreply@blogger.com) at April 14, 2014 02:52 PM

April 02, 2014

Duncan McGreggor

Hash Maps in LFE: Request for Comment

As you may have heard, hash maps are coming to Erlang in R17. We're all pretty excited about this. The LFE community (yes, we have one... hey, being headquartered on Gutland keeps us lean!) has been abuzz with excitement: do we get some new syntax for Erlang maps? Or just record-like macros?

That's still an open question. There's a good chance that if we find an elegant solution, we'll get some new syntax.

In an effort to (re)start this conversation and get us thinking about the possibilities, I've drawn together some examples from various Lisps. At the end of the post, we'll review some related data structures in LFE... as a point of contrast and possible guidance.

Note that I've tried to keep the code grouped in larger gists, not split up with prose wedged between them. This should make it easier to compare and contrast whole examples at a glance.

Before we dive into the Lisps, let's take a look at maps in Erlang:

Erlang Maps

Common Lisp Hash Tables

Racket Hash Tables

Clojure Hash Maps

Shen Property Lists

OpenLisp Hash Tables

LFE Property Lists

LFE orddicts

I summarized some very basic usability and aesthetic thoughts on the LFE mail list, but I'll restate them here:
  • Erlang syntax really is quite powerful; I continue to be impressed.
  • Clojure was by far the most enjoyable to work with... however, doing something similar in LFE would require quite a bit of additions for language or macro infrastructure. My concern here is that we'd end up with a Clojure clone rather than something distinctly Erlang-Lispy.
  • Racket had the fullest and most useful set of hash functions (and best docs).
  • Chicken Scheme was probably second.
  • Common Lisp was probably (I hate to say it) the most awkward of the bunch). I'm hoping we can avoid pretty much everything the way it was done there :-/
One of the things that makes Clojure such a joy to work with is the unified aspect of core functions and how one uses these to manipulate data structures of different types. Most other implementations have functions/macros that are dedicated to working with just maps. While that's clean and definitely has a strong appeal, Clojure reflects a great deal of elegance.

That being said, I don't think today is the day to propose unifying features for LFE/Erlang data types ;-) (To be honest, though, it's certainly in the back of my mind... this is probably also true for many folks on the mail list.)

Given my positive experience with maps (hash tables) in Racket, and Robert's initial proposed functions like map-new, map-set, I'd encourage us to look to Racket for some inspiration:
Additional thoughts:
  • "map" has a specific meaning in FPs (: lists map), and there's a little bit of cognitive dissonance for me when I look at map-*
  • In my experience, applications generally don't have too many records; however, I've known apps with 100s and 1000s of instances of hash maps; as such, the idea of creating macros for each hash-map (e.g., my-map-get, my-map-set, ...) terrifies me a little. I don't believe this has been proposed, and I don't know enough about LFE's internals (much less, Erlang's) to be able to discuss this with any certainty.
  • The thought did occur that we could put all the map functions in a module e.g., (: maps new ... ), etc. I haven't actually looked at the Erlang source and don't know how maps are implemented in R17 yet (nor how that functionality is presented to the developer). Obviously, once I have, this point will be more clear for me.
With this done, I then did a thought experiment in potential syntax additions for LFE. Below are the series of gists that demonstrate this.

Looking at this Erlang syntax:

My fingers want to do something like this in LFE:

That feels pretty natural, from the LFE perspective. However, it looks like it might require hacking on the tuple-parsing logic (or splitting that into two code paths: one for regular tuple-parsing, and the other for maps...?).

The above syntax also lends itself nicely to these:

The question that arises for me is "how would we do this when calling functions?" Perhaps one of these:

Then, for Joe's other example:

We'd have this for LFE:

Before we pattern match on this, let's look at Erlang pattern matching for tuples:

Compare this with pattern matching elements of a tuple in LFE:

With that in our minds, we turn to Joe's matching example against a specific map element:

And we could do the same in LFE like this:

I'm really uncertain about add-pair and update-pair, both the need for them and the names. Interested to hear from others who know how map is implemented in Erlang and the best way to work with that in LFE...

by Duncan McGreggor (noreply@blogger.com) at April 02, 2014 05:33 PM

Moshe Zadka

GPS Navigators, morality and the human utility function

GPS navigators used to be simple. You plug in the desired address, they calculate the quickest route. At some point, many started getting a feed of traffic information, so that they can route against big jams. Still pretty simple.

Enter Waze. Waze hyper-micro-optimizes for traffic conditions, using other users as roaming sensors and, probably, complicated AI on the back-end to predict what route would take how long. This means Waze learns from experience how fast people drive where, under what conditions — and, as far as I can tell, takes all this information into account. This means that on a residential street that is constantly being speeded to, Waze will learn that it is a fast path, and will start directing drivers through it. Drivers who are using Waze, which means they want to get to their destination as fast as possible.

There are a lot of valid reasons to want to get somewhere fast. Maybe you scheduled a meeting. Maybe someone is having an emergency. Maybe you just want to cuddle your kid before she goes to bed. Waze does not care. It tries to get users to their destination as fast as possible, no matter which residential streets they speed through. It has no concerns for safety, either the driver’s or pedestrian, neatly divulging itself of all responsibility by “just suggesting”. What if a nice street starts being higher-up on the Waze paths? And this lowers property values? Waze has no concerns for that.

I’m not picking on Waze especially. Their only fault was that they took the optimization criteria that all GPS navigators use (find fastest path) and used superior sensors and superior algorithms to implement them better. Whoever wins the “navigator wars” would have had to do at least as well, so in that sense, there is nothing special about Waze except hiring smart people who made good decisions.

But this does show how AI moves forward: smart engineers optimize the hell of whatever target function they are given. At some point, it starts having real costs that humans would understand, but an AI would not care about because the AI cares about optimizing the target function. The only way to solve it is to figure out what humans care about in terms computers can understand and make sure that any AI takes those into account — even if it is not smart enough to self-modify, it can do plenty of damage without it.

In other words, the Friendliness apocalypse, if we include not just existential risk but also a general raising of human risk factors, is not in some nebulous future until we figure out self-modification. We are in the middle of it, and we should make sure that we take it into account when building AI — even if that AI is limited to “just suggesting things”, because humans are suggestible, and have learned to trust computers’ suggestions.

by moshez at April 02, 2014 04:03 PM

March 27, 2014

Glyph Lefkowitz

Panopticon Rift

Greg Price expresses a common opinion here:

I myself had a similar reaction; despite not being particularly invested in VR specifically (I was not an Oculus backer) I felt pretty negatively about the fact that Facebook was the acquirer. I wasn't quite sure why, at first, and after some reflection I'd like to share my thoughts with you on both why this is a bad thing and also why gamers, in particular, were disturbed by it.

The Oculus Rift really captured the imagination of the gaming community's intelligentsia. John Carmack's imprimatur alone was enough to get people interested, but the real power of the Oculus was to finally deliver on the promise of all those unbelievably clunky virtual reality headsets that we've played around with at one time or another.

Virtual Boy

The promise of Virtual Reality is, of course, to transport us so completely to another place and time that we cease to even be aware of the real world. It is that experience, that complete and overwhelming sense of being in an imagined place, that many gamers are looking for when they sit down in front of an existing game. It is the aspiration to that experience that makes "immersive" such a high compliment in game criticism.

Personally, immersion in a virtual world was the beginning of my real interest in computers. I've never been the kind of gamer who felt the need to be intensely competitive. I didn't really care about rules and mechanics that much, at least not for their own sake. In fact, the term "gamer" is a bit of a misnomer - I'm more a connoisseur of interactive experiences. The very first "game" I remember really enjoying was Zork. Although Zork is a goal-directed game you can "win", that didn't interest me. Instead, I enjoyed wandering around the environments it provided, and experimenting with different actions to see what the computer would do.

Computer games are not really just one thing. The same term, "games", encompasses wildly divergent experiences including computerized Solitaire, Silent Hill, Dyad, and Myst. Nevertheless, I think that pursuit of immersion – on really, fully, presently paying attention to an interactive experience – is a primary reason that "gamers" feel the need to distinguish themselves (ourselves?) from the casual computing public. Obviously, the fans of Oculus are among those most focused on immersive experiences.

Gamers feel the need to set themselves apart because computing today is practically defined by a torrential cascade of things that we're only barely paying attention to. What makes computing "today" different than the computing of yesteryear is that computing used to be about thinking, and now it's about communication.

The advent of social media has left us not only focused on communication, but focused on constant, ephemeral, shallow communication. This is not an accident. In our present economy there is, after all, no such thing as a "social media business" (or, indeed, a "search engine business"); there are only ad agencies.

The purveyors of social media need you to be engaged enough with your friends that you won't leave their sites, so there is some level of entertainment or interest they must bring to their transactions with you, of course; but they don't want you to be so engaged that a distracting advertisement would be obviously crass and inappropriate. Lighthearted banter, pictures of shiba inus, and shallow gossip are fantastic fodder for this. Less so are long, soul-searching long-form writing explaining and examining your feelings.

An ad for cat food might seem fine if you're chuckling at a picture of a dog in a hoodie saying "wow, very meme, such meta". It's less likely to drive you through to the terminus of that purchase conversion funnel if you're intently focused on supporting a friend who is explaining to you how a cancer scare drove home how they're doing nothing with their life, or how your friend's sibling has run away from home and your friend doesn't know if they're safe.

Even if you're a highly social extrovert, the dominant emotion that this torrent of ephemeral communication produces is, at best, sarcastic amusement. More likely, it produces constant anxiety. We do not experience our better selves when we're not really paying focused attention to anything. As Community Season 5 Episode 8 recently put it, somewhat more bluntly: "Mark Zuckerberg is Fidel Castro in flip-flops."

I think that despite all the other reasons we're all annoyed - the feelings of betrayal around the Kickstarter, protestations of Facebook's creepyness, and so on - the root of the anger around the Facebook acquisition is the sense that this technology with so much potential to reverse the Balkanization of our attention has now been put directly into the hands of those who created the problem in the first place.

So now, instead of looking forward to a technology that will allow us to visit a world of pure imagination, we now eagerly await something that will shove distracting notifications and annoying advertisements literally an inch away from our eyeballs. Probably after charging us several hundred dollars for the privilege.

It seems to me that's a pretty clear justification for a few hundred negative reddit comments.

by Glyph at March 27, 2014 09:25 AM

March 20, 2014

Duncan McGreggor

lfetool T-Shirts?!

So, yeah... waaaaay too early for a T-shirt; the code has just gone from 0.1.x to 0.2.x. But, how could we resist: the project logo is retro like woodblock prints!

Earlier today, lfetool converted to using plugins (easier for contributors to play!), and hopefully later tonight YAWS + Bootstrap support will land. Regardless, there's tons more work to do, and what's more motivating for new work than a T-shirt? As you can see, we had no choice.

So let's get some. T-shirts, that is.

The lfetool logo is on the front, and Robert Virding's example Fibonacci lfescript code is on the back (with the lfetool command that generates the script at the top). Here's the front of the T-shirt:




And here's the back:



We've got a CustomInk sign-up sheet that you can add your name to if you want a shirt. I'll coordinate with folks individually, once we meet our minimum number (we're going to need to use paypal with payment upfront). Due to the colors of the source code on the back, the minimum order is 15. This will put the shirts at $22 a piece + $5 to send it to you. I've just ordered 2; 13 more go!


by Duncan McGreggor (noreply@blogger.com) at March 20, 2014 03:58 AM

March 18, 2014

Duncan McGreggor

lfetool 0.2 Released

This release of lfetool has a bunch of new interesting features -- too much to cover in a tweet, so it's getting a little blog post ;-)

Here are the high-level bits that should make users' lives better:
  • New yaws project type, provided for building basic web sites; includes the exemplar project as a dependency for generating HTML with S-expressions.
  • Every project now gets TravisCI support by default.
  • Unit tests, integration tests, and systems tests now each have their own dir.
  • Tests now get run in parallel for faster run times.
  • Added support for running on Linux and *BSD systems.
  • Added unit tests for lfetool itself (shUnit) (for checking skeleton-generation and tool options).
  • lfetool has a new logo :-)
Note that the version jumped from 0.1.2 to 0.2.0 due to the strong improvements in quality and feature set of the tool. Forth-coming features include:
  • Support for an e2 service skeleton project.
  • Support for a YAWS-RESTful service skeleton project.
  • Support for YAWS embedded in a skeleton OTP application.
  • Support for YAWS + Exemplar + Bootstrap projects
  • Support for slide decks with Exemplare + Reveal.js
If you've got more ideas for the lfetool wishlist, post a comment here or send an email to the LFE mailing list.


by Duncan McGreggor (noreply@blogger.com) at March 18, 2014 11:29 PM

March 15, 2014

Future Foundries

Signal/GC-safe cross-thread queueing in Python

I've just released a new version of Crochet, and one of the bugs fixed involves an interesting problem - reentrancy. In this particular case I'm talking about garbage collection and signal reentrancy - any function your Python program is running may be interrupted at any time (on bytecode boundaries) to do garbage collection or handle a signal. A signal handler can run arbitrary Python code, as can GC via to __del__ or weakref callbacks. Once that code finishes running control is returned to the original location in the code.

Unfortunately, due to a bug in Python, Queue.put() can deadlock in the following situation:
  1. As part of calling Queue.put(), a thread acquires the Queue's lock. This lock does not support being acquired more than once by the same thread.
  2. GC or a signal handler interrupts the function call.
  3. If the GC or signal handler code then also does Queue.put(), it will try to acquire the lock again... and since it's already locked it blocks waiting for the lock to be released.
  4. Since the signal handler/GC code is now blocked, control is never returned to original code, so lock is never released there.
  5. The thread is now deadlocked and will never recover.
Unfortunately there was no way to prevent the Queue.put() in GC; the Queue accepts log messages, and this is a GC-caused logging message coming out of code that is not under Crochet control's.

The obvious short-term solution is to reimplement a simplified Queue using Python's RLock, which allows the same thread to acquire the lock multiples times. But... RLock isn't reentrancy safe either due to another bug in Python! I could wrap OS-specific reentrant lock implementations, but that's a bigger project than I want to start.

The solution I ended up with (suggested by Jean-Paul Calderone I believe) was giving up on using threading primitives to communicate across threads. Instead I used the self-pipe trick. That is, the thread uses select() (or poll() or epoll()) to wait on one end of the pipe; to wake the thread up and tell it to check for new messages to process we simply write a byte to the other end of the pipe. Since Crochet uses Twisted, I had a pre-written event loop that already implemented self-pipe waking, and now the logging thread runs another Twisted reactor in addition to the regular reactor thread.

As far as I can tell this works, but it feels a bit like overkill. I'd welcome suggestions for other solutions.

    by Itamar Turner-Trauring (noreply@blogger.com) at March 15, 2014 12:45 PM

    March 14, 2014

    Duncan McGreggor

    The Future of LFE

    Erlang Factory

    First of all, Erlang Factory this year was just phenomenal: great talks, great energy, and none of the feared/anticipated "acquisition feeding frenzy." Instead, everyone was genuinely happy for WhatsApp and Cloudant, and with celebrations aside, they were ready to get on with the conference and dive into the tech :-)

    And gosh, there was a bunch of good tech. If you don't believe me, check out the schedule. Also on that page are the speaker pics. For talks that have video or slides pushed up, the speaker pic is visually annotated and linked.

    There's so much good stuff there -- I've definitely got my watching queue set up for the next few weeks ...

    LFE Presentation

    I gave a presentation on LFE which covered everything from motivational basics for using a Lisp in the 21st century, gave a taste of LFE in small chunks, and then took folks on a quick tour of creating projects in LFE. There was also some dessert of fun side/research projects that are currently in-progress. The slides for the presentation are here; also the slide source code is available (related demo project). I'd also like to give a shout out to the Hoplon crew for their help in making sure I could create this presentation in a Lisp (Clojure), and not HTML ;-) (It uses a Hoplon-based Reveal.js library.)

    The Good Stuff

    After the presentation, several of us chatted about Lisp and Erlang for a while. Robert and I later continued along these lines after heading over to the quiet of the ever-cool Marines Memorial 11th-floor library (complete with fireplace). Here we sketched out some of the interesting areas for future development in LFE. I'm not sure if I'm remembering everything (I've also added stuff that we didn't discuss in the library, like Sean Chalmers' recent experiments with types; blog and discussion):
    • getting the REPL to the point where full dev can happen (defining functions, macros, and records in the LFE shell)
    • adding macros (maybe just one) for easier use of Mnesia in LFE 
    • discussing the possibility of an LFE stdlib
    • gathering some of the best funcs and macros in the wild for inclusion in an LFE stdlib
    • possibly supporting both (: module func ...) and (module:func ...) syntax
    • possibly getting spec and type support in LFE
    • producing an LFE 1.0 release
    Additional efforts planned:
    • building out an LFE rebar plugin
    • examining erlang.mk as another possible option
    • starting work on an LFE Cookbook
    • creating demos of LFE on Erjang
    • creating demos of LFE-Clojure interop via JInterface
    • creating more involved YAWS/REST examples with LFE
    • explore the possibility of an SOA tutorial with LFE + YAWS 
    • async tasks in LFE with rpc lib 
    • monad tutorial for LFE (practical, not conceptual)
    • releasing a planner demo
    • finishing the genetic programming examples
    • LFE style guide
    • producing a stdlib contribution guideline
    • continued work on the LFE user guide
    I'll dump all these into github issues so they'll be easier to track.

    If this stuff is exciting to you, feel free to jump into the low-volume discussions we have on the mail list.

    More soon!


    by Duncan McGreggor (noreply@blogger.com) at March 14, 2014 08:11 PM

    March 09, 2014

    Future Foundries

    Twisted on Python 3 now pip installable

    The subset of Twisted that has been ported to Python 3 can now be pip installed. By either pointing at a version control URL or requiring Twisted 14.0 (once it's released), you can now have Twisted as a dependency for your Python 3 packages.

    Here's a slightly edited version of my Travis-CI config for Crochet, demonstrating how I run unit tests on both Python 2 and Python 3 versions of Twisted (slightly tricky because the trial test runner hasn't been ported yet):

    language: python

    env:
    - TWISTED=Twisted==13.0 RUNTESTS=trial
    - TWISTED=Twisted==13.1 RUNTESTS=trial
    - TWISTED=Twisted==13.2 RUNTESTS=trial

    python:
    - 2.6
    - 2.7
    - pypy

    matrix:
    include:
    - python: 3.3
    env: TWISTED=git+https://github.com/twisted/twisted.git RUNTESTS="python -m unittest discover"

    install:
    - pip install -q --no-use-wheel $TWISTED --use-mirrors
    - python setup.py -q install

    script: $RUNTESTS crochet.tests

    by Itamar Turner-Trauring (noreply@blogger.com) at March 09, 2014 06:27 PM

    March 08, 2014

    Ashwini Oruganti

    Kneel and Disconnect: Getting the fastest connection out of a hostname

    A Caterpillar crawled to me one day and said
    “Oh what the hell goes on inside your swollen head?
    I don’t believe that you can see as much as I
    Now close your eyes and tell me what do you spy?”

    A detailed account of the development of HostnameEndpoint follows, also leading to my talk at PyCon US 2014 in Montreal. During the course of its creation, many wise words were exchanged, thanks to exarkun and glyph (and itamar and tomprince!). I repeat them here, in hopes that it will help make you wiser too, should you aim for that goal. So well, indulge me whilst I rant on.

    Soon after I finished working on a name-resolving TCP IPv6 client endpoint, I heard of a certain crazy endpoint, described (not in as much detail as it is now, but to some extent) in a ticket. In those days, seeing as the description was not too elaborate, and the goals not too clear, any mention of working on it would lead to a discussion such as:

    <exarkun> I suspect it is obvious to anyone with more than ten years network programming experience.
    <glyph> exarkun: Completely. ashfall should hurry up and get ten years of that, then.

    This continued until one day Glyph decided to write a nice spec for it, and that’s where our story begins.

    Bornlivedie

    <kenaan> ashfall r35028 A(branches/gai-endpoint-4859/): Branching to ‘gai-endpoint-4859’ …
    <ashfall> Okay, this one is a little bit scary to start.

    The gist of this endpoint is that it takes a hostname, does a getaddrinfo lookup on it, and picks the first two addresses out of the returned list. Using some time-involving algorithm, it tries connecting to both of them.

    Doing things based on time in Twisted meant using reactor.callLater. That in turn meant using Clock as the reactor in the unit tests, since real time is slow and unpredictable, and tests are ideally fast and predictable.

    This was also the day when my appreciation of the reactor grew.

    My initial understanding of the task at hand, I remember, was that the endpoint would connect to the first address, record how long it took, then connect to the next address, record how long it took, compare the times taken, pick the one that took less time, and call connect one final time. After discussing it further with exarkun, I realized that this doesn’t actually need to record any durations, since we don’t care how long it took, as long as it was first.

    Now, the high-level objective was to establish the connection as quickly as possible, and 3 connects would take a lot of time. So we make the connection attempts in parallel, drop the one that doesn’t connect first and use the one that does connect first. So that there are two connection attempts, one (hopefully) connection success, and one discarded connection attempt. And no third connection attempt.

    And how do we do that?

    “Well, fortunately Twisted is good at doing network operations in parallel”, explained exarkun. For example:

    1
    2
    
    reactor.connectTCP('1.2.3.4', 456, f)
    reactor.connectTCP('1:2::3:4', 567, f)
    

    For my part I wondered, how is that parallel, with one reactor?

    I learnt the reactor can do many things at once, and that that’s its job, essentially. ”reactor.connectTCP returns immediately, after all. It doesn’t wait for the connection to be established first,” exarkun elaborated. “That’s why we need a factory with a buildProtocol method, so that the reactor can call it once the connection has succeeded. Because by the time it does succeed, the application will have moved on to doing something else.”

    “A secondary objective of the endpoint is to be conservative of network resources,” he added. “If the connection to 1.2.3.4 succeeds quickly enough, then it would be wasteful of network resources to try connecting to 1:2::3:4 as well. That’s why the relevant RFC talks about a 300 ms delay. If the first one is done really fast and you can avoid the second connection attempt, you should. So you won’t actually start both connection attempts at the same time. You’ll start the 2nd one a short while after the first one, if and only if the first one has not succeeded. reactor.callLater is how you would do that, and delayedCall.cancel() is how you would stop a delayed call from happening when the first one succeeds.someDeferred.cancel() is how you would make a connection attempt give up early, if the other connection attempt succeeds.”

    But now I wondered, what if 1.2.3.4 takes 254 ms and 1:2::3:4 would have taken, say, 230 ms, had we tried it (but we will never know?). And exarkun said, “[Then] Tough luck for us. We’re not going to try to solve that problem.” Turns out, it’s not as much of a problem as the problem this RFC is aimed at fixing, which is the problem where 1:2::3:4 takes 120 seconds and 1.2.3.4 takes 254 ms, and so a purely serial approach would take 120.254 seconds to succeed, whereas this one will only take 0.254 seconds to succeed. That’s a much more noticeable difference than 0.254 seconds vs 0.230 seconds.

    “I watched nine cats dance on the moon
    A flamingo stalked into my room
    It bowed its head to me and knelt
    To reveal the cards it had dealt

    Now, where does Clock come into the picture? The clock is actually a replacement for the reactor, at least in a very restricted scope. It has a callLater method, but whereas reactor.callLater(5, f) will only call f after roughly 5 seconds have passed, clock.callLater(5, f) will call f the instant you call clock.advance(5). (or any combination of clock.advance calls so that the sum of the arguments passed to all calls is 5 or more). So if you’re writing a unit test for code that does callLater(5, f), the test can complete in fewer than 5 seconds if you pass a Clock instance in for the reactor.

    Using Clock completes with more predictable results, since the reactor really does call f in roughly 5 seconds, and the variation implied by “roughly” can sometimes make a difference (in a bad way) to how the code behaves. The idea is that maybe 5 seconds is close to some other critical amount of time. Perhaps sometimes the reactor gets around to calling f very soon after 5 seconds, perhaps 5.001 seconds. That exercises one code path through the code under test. And perhaps sometimes, the reactor doesn’t get around to calling f until 5.1 seconds have elapsed, meanwhile, something else happened in those extra .099 seconds that didn’t have time to happen in the first case, and so now you’re exercising a second, different code path through the code under test. But this is a single test method, and it should only ever exercise a single code path. Since it is exercising two, now, it can have two different outcomes, and that makes it much less useful. Perhaps one of the outcomes is a success and the other is a failure, so when you run the tests, the results flip flop between good and bad, even though you’re not changing any code. Or perhaps both outcomes are success - for now. And someone breaks one of the code paths accidentally. But they run the test, and it accidentally goes through the other code path, the one they didn’t break, and so the test passes. And they think their code works.

    So, yes. Clock and writing two tests is the solution. Because with Clock, you can write a test where it appears that 5.001 seconds elapse before f is called, and a second test where it appears that 5.1 seconds elapse. (just by doing clock.advance(5.001) in one of them and clock.advance(5.1) in the other)

    Nomenclature

    An ace, three jacks, two queens, four kings
    Then turned them into burning rings
    The flames jumped out and chased four mice
    Caught by their tails they turned to ice

    The new super-smart TCP endpoint, what should it be called?

    The first name glyph suggested was HostnameEndpoint, because it connects you to a hostname, over whatever protocol is appropriate. Another suggestion was to call it TCPEndpoint, since we haven’t used that name yet. And then it was pointed out that hostnames aren’t part of TCP. Nor even IP. They’re part of DNS. 99% of this endpoint is DNS functionality. So… DNSEndpoint? But you can look up hostnames for use with SCTP or UDP via DNS as well! And this won’t let you do that. And, in fact, this endpoint does not necessarily use DNS: it could be doing name resolution via /etc/hosts, or, it could be doing name resolution using WINS… you don’t know.

    The important lesson to be learnt here is that naming things for how they work is generally worse than naming things for what they do. DNS and TCP are both implementation details (albeit kind of important details in this case); the important thing about this endpoint is that it takes a hostname and gives you a stream-oriented connection.

    So HostnameEndpoint seemed like the best and the least ambiguous choice.

    Deform to form a Star

    Reading the spec, the whole idea sounded (to me, at least) too convoluted and it wasn’t very obvious where to begin. On seeking help, exarkun suggested:

    • Make a list of features it should have.
    • Sort them from simplest to most complex. (the failure ones come before the success ones)
    • Now, write a unit test for the first thing on that list. (i.e., write the test before you have any implementation for the endpoint at all, or at least set up a TestCase subclass that mixes in that helper and implements the method for creating an endpoint)

    Part of the idea with proceeding from simplest to most complicated is that once you’ve done all the simple things, the complicated things generally end up pretty simple. They’re only complicated if you have to tackle all the complexity at once.

    We moved on to talk about what test cases to write first, because… TDD! Starting with the easy cases, I planned to begin by using the EndpointTestCaseMixin - Try making the test methods from that mixin that exercise failures pass. Then the success cases. Perhaps then try a test that exercises the case where getaddrinfo returns one IPv4 address. And then the case where it returns one IPv6 address. And when all that works, then maybe it’s time to start thinking about the interesting cases, where getaddrinfo returns an IPv4 and an IPv6 address, e.g. What if it returns IPv4 then IPv6? What if it returns IPv6 then IPv4? What if the first connection attempt succeeds in < 300ms? What if it succeeds in > 300ms, but before the second? What if the second succeeds before the first? What if they both fail? But start with the simpler cases, and work on them one at a time.

    Deferreds and Raising Exceptions

    A cloud appeared outside my door
    And through the window saw four more
    And on the back of each cloud sat
    Two rainbow smiles in wizard’s hats

    There was a part of the endpoint where I was using “except” to work with a Deferred firing with an exception. Needless to say, the unit test that was to check this was failing, and I wasn’t sure why. This led exarkun to elaborate on how Deferreds work, and where I was going wrong.

    We have Deferreds so that functions can return before they have finished the abstract task they are responsible for. They return a Deferred and the Deferred becomes responsible for indicating the result of the abstract task, once it is complete. This lets multiple tasks proceed concurrently. You can just call one function to start one task, then it will return (before the task is complete), and you can call another function to start another task (which will also return before that task is complete). Now you have two Deferreds for the two tasks, and you can use them to learn the results of those tasks.

    You learn the result by adding a callback to the Deferred. And when the result is available, the callback is called with it.

    Not all results are “successes” though. But for the same reason that these functions can’t return their result directly, neither can they raise an exception directly. By the time they return, they might not have gotten far enough in their abstract task to know that there is going to be an error. So they can still only return a Deferred. So you can also attach error callbacks, or errbacks, to Deferreds as well. An errback is just like a callback, except it is called when the Deferred gets its result and that result is not a success.

    So, to handle results from asynchronous, Deferred-returning functions, you take the returned Deferred and call addCallback(callbackFunction) on it. And to handle error results from asynchronous, Deferred-returning functions, you take the returned Deferred and call addErrback(errbackFunction) on it.

    The errback is ignored if the result is not a failure, just like an “except” statement is ignored if no exception is raised. Similarly, the callback is ignored if the result is a failure.

    Hence, “except” will not work for a Deferred that fires with exception, because the Deferred was returned. If anything was returned at all, an exception could not possibly have been raised. Because of how Python functions work - they can raise or they can return, they cannot do both.

    Returning Deferreds From Tests

    At one point, I had written a test that was returning a Deferred, and then the whole test case was timing out, never reaching completion. I learnt that since it doesn’t require any external resources, there was no reason this test needed to spin a reactor.

    I asked, then, is returning Deferreds from tests a bad thing?

    “It’s not ‘good’ or ‘bad’ really,” Glyph explained, “but if you’re going to think of it as one of those, let’s think of it as ‘bad’ by default”.

    Returning a Deferred spins the global reactor. You return Deferreds from tests that need to interface with real, external resources or connect to real sockets. In this case you absolutely do not want to do that. You want to very carefully control everything that your HostnameEndpoint is talking to. You want to fire the connection attempt success and failure in a very specific order - in several different orders, as a matter of fact. and to verify that attempts are in progress at certain times and not in progress at other times. That means you want to use Clock and you want to use MemoryReactor (possibly smashed together on one object so that you only need to have one reactor).

    Then, at precise points in your test, you want to say “this deferred should be fired by now” not “whenever this Deferred happens to fire…”. As you can see, if you do it by returning Deferreds, then your failure mode is “uh… the test didn’t finish”, and not “Oh, of course, I forgot to callback deferred X when event Y happened, and my test failed telling me that that Deferred wasn’t fired”.

    If you return an unfired Deferred from your test, then the test never ends. It just sits around waiting for that Deferred forever. where Trial defines “forever” to be 120 seconds, after which the test times out. There are times when it’s appropriate, but you won’t likely encounter them when testing Twisted itself, I was told.

    Basically if you’re originating the Deferreds that you’re testing, you should definitely not be yielding them, or returning them from your test. Tests which do return Deferreds depend on the things that return the Deferreds that they’re returning being fully tested and having no chance of timing out.

    So as a rule of thumb, only return Deferreds from a test if the test interacts with external stuff, e.g. writing tests that talk to, say, Postgres, and have to do multiple asynchronous operations. And, if you’re doing that, see if there’s a way you could reasonably interact with less external stuff first.

    This is just a special case of a general rule though. The general rule is, your tests should not depend on any more stuff than they need to. Especially, they should not depend on unpredictable external factors that may make your tests behave differently under different conditions. A real reactor is by definition an unpredictable external factor; the whole point is that it’s for talking to unpredictable external stuff.

    Glyph also shared an excellent analogy: “Think of it like an actual nuclear reactor. Before you hook your thermal regulator up to the coolant rods, you really want to have a high level of confidence that it is going to react in the appropriate way when the “uranium is too hot” wire gets some voltage. So you give that wire some voltage using a hand crank, not a critical mass.”

    0 seconds later is still “some amount of time”

    They threw five clocks down on my bed
    The chimes danced out on golden threads
    And turned to footprints on my wall
    Sequined tears began to fall”

    Now, I had a function iterateEndpoint, which iterates through a generator that yields different endpoints for different addresses returned by gai. Now, I needed to keep calling this function for each endpoint, and as per the algorithm we are implementing, we introduce a time lag of 0.3 seconds between two endpoint’s connection attempts. So, each call to iterateEndpoint needs to occur 0.3 seconds after the previous call. I needed a technique to introduce this delay in the calls. There’s reactor.callLater, which did this and I tried using it, and it didn’t work.

    By “didn’t work”, I mean that all my tests were failing, but not failing in a way that would help me identify what’s wrong. But mostly, MemoryReactor.tcpClients was empty, when it should hold some host address, i.e., (in terms of real endpoint) a connection wasn’t being established.

    I blamed callLater for this, because there was at least one connection attempt when I hadn’t used callLater and introduced the delay, whereas now it wasn’t even trying the first endpoint’s host address.

    Then I made the delay in the looped calls 0 seconds to check, and I learnt that I needed to “advance” the clock for this to work. Even when the delay was 0 seconds. And this just wasn’t make any sense.

    That was when glyph started quoting Morpheus and said, “Free your mind!”

    “Advancing the clock means calling Clock.advance()”, he explained further. “Specifically when I say ‘free your mind’ I mean ‘stop thinking about everything in terms of real stuff happening’. Everything here is just a method calling another method. Some of those methods represent things (like the passage of time on a clock) but in a test it’s all a simulation. I’d also say that ultimately when it’s not in the test it’s just a higher fidelity simulation but that’s a thorny philosophical issue we probably shouldn’t get into. At any rate it’s always just methods calling other methods. So your LoopingCall specifies a function to call to advance to the next connection attempt; it asks an IReactorTime to call that thing via callLater.

    But then, I wondered, if I set the time delay between two consecutive calls to the looped code as 0 seconds, why would I need to advance the clock now?

    “Probably the easiest way to understand the answer to that is to look at the implementation of Clock”, Glyph said patiently. “Particularly the implementation of Clock.callLater. When you call callLater, it just remembers the function you asked it to run, puts it in a list, and sorts the list. Sorting a list does not call your function. Similarly, a “real” reactor’s callLater will never run your function reentrantly. 0 seconds later is still ‘some amount of time’ later, even if it’s not a very big amount of time. the behavior of Clock is that it only invokes application code when ‘time passes’ which is to say when you call .advance(). Real reactors do the same thing and only invoke timed calls when you get back to the main loop.

    But… it’s zero. It’s supposed to mean “nothing”! I protested.

    “You’re thinking too hard about it”, said Glyph. “It’s a number. The behavior of Clock is such that it remembers a callable, sorts them by number, and then calls them. It stuffs it on the stack of callLaters and re-evaluates that when the reactor gets control again. The magnitude of the number isn’t important except in that negative numbers aren’t allowed because that’s non-sensical. In actual fact it is actually an orderable object, ‘number’ is a convention that we apply to it. callLater means ‘when time advances, no sooner than N seconds later, call this function’.”

    “So, when the reactor is told to perform a task 0 seconds later, does it not do that immediately?”, I asked, still not quite convinced.

    “Well of course it does it immediately. But not reentrantly. ‘As soon as possible but no sooner’.” was Glyph’s answer. “‘immediately’ is defined by some time-keeping thing, so in order to figure out when “immediately” occurs, it has to consult some time-keeping thing. And the point in the code - not in time - where it does that consultation is the reactor loop for reactors and the advance() method for Clock. Practically speaking, callLater is not a hard realtime API, because Python isn’t a hard realtime system, so callLater(n, f) cannot physically run f precisely n seconds after its invocation; it can’t even get a zero-error concept of when its own invocation was. clocks keep moving while computers are computing. so if you say callLater(1, f) really you’re likely for f to get run at 1.000000001 seconds later, or something close to that. so if you say callLater(0, f) you’re going to get 0.000000001 seconds later, if you’re comparing against real time. if you’re using a task.Clock(), time is frozen, so you don’t get to 0 seconds later until you allow time to advance.”

    “The most important thing though is that you understand the (very simple) implementation within Clock. there’s nothing complicated going on in there, no magic. it’s almost as simple as l = []; l.append(f); for g in l: g(). All the stuff about how time actually works is useful to understand but ultimately peripheral to the very simple rule that functions don’t get called until you call them.”

    “One last time”, I said, now almost convinced, “what exactly does ‘reentrant’ mean here?”

    “It means that def forever(): reactor.callLater(0, forever) will not cause a ‘maximum recursion depth exceeded’ error, because callLater does not call forever below its stack frame. forever() doesn’t re-enter itself upon the call to callLater, hence, callLater does not invoke its argument reentrantly. In other words, it doesn’t mutually recurse.”

    The Calculus of Testing

    Now, in my tests, I was just popping the result I want from MemoryReactor.tcpClients, and matching it against a value. So, the test ensured that the connection attempts were being made, but they weren’t checking the algorithm.

    Turns out, you can’t check an algorithm with a unit test. “Some people think you can check an algorithm with formal logic, but they’re probably wrong in practice. An algorithm is too complicated to check in any general way.”, explained exarkun.

    So I can only hope that the implementation of the algorithm is correct?

    “No, you can test that the output of the algorithm is correct for some number of inputs”, said he. “And you can hope that your selection of inputs exercises all meaningful branches (and branch combinations) in the algorithm.”

    “The idea with unit tests is that you constrain the domain of possible incorrect outputs”, Glyph added. “You test what you believe to be a representative sample of each ‘interesting’ region of behavior for your algorithm, and by ‘interesting’ I mean each range of input that requires more code.”

    A general lesson I learnt was that I was trying really hard to understand how the whole thing fits together all at once, but it helps to compartmentalize. It helps to just figure out what functions call what other functions; later, once you know what’s going on, learn why they’re getting called in that way. You can do a whole lot without knowing why. And in some cases there isn’t really a “why”, it’s just an arbitrary decision because the reactor has a bunch of work to do and it has to choose some order, so it does things in whatever order we thought to implement first.

    And just like that, we had our endpoint written in a respectful, TDD-way, with well-tested parts, etc. But it still wasn’t perfect. Later in the reviews, exarkun suggested that connect could be implemented more simply - probably as an independent object with instance state (and perhaps as an explicit state machine) rather than a collection of nested functions with shared, closed-over mutable objects. That led me to file another ticket, and work on getting a finite state machine into Twisted. But that’s another story!

    The caterpillar gasped at me and said
    “My god if that’s what’s going on inside your head
    You can see so much more than I
    I think it’s time to turn into a butterfly.”

    March 08, 2014 02:24 PM

    February 24, 2014

    Glyph Lefkowitz

    Unyielding

    The Oak and the Reed by Achille Michallon

    … that which is hard and stiff
    is the follower of death
    that which is soft and yielding
    is the follower of life …

    – the Tao Te Ching, chapter 76

    Problem: Threads Are Bad

    As we know, threads are a bad idea, (for most purposes). Threads make local reasoning difficult, and local reasoning is perhaps the most important thing in software development.

    With the word “threads”, I am referring to shared-state multithreading, despite the fact that there are languages, like Erlang and Haskell which refer to concurrent processes – those which do not implicitly share state, and require explicit coordination – as “threads”.

    My experience is mainly (although not exclusively) with Python but the ideas presented here should generalize to most languages which have global shared mutable state by default, which is to say, quite a lot of them: C (including Original Recipe, Sharp, Extra Crispy, Objective, and Plus Plus), JavaScript, Java, Scheme, Ruby, and PHP, just to name a few.

    With the phrase “local reasoning”, I’m referring to the ability to understand the behavior (and thereby, the correctness) of a routine by examining the routine itself rather than examining the entire system.

    When you’re looking at a routine that manipulates some state, in a single-tasking, nonconcurrent system, you only have to imagine the state at the beginning of the routine, and the state at the end of the routine. To imagine the different states, you need only to read the routine and imagine executing its instructions in order from top to bottom. This means that the number of instructions you must consider is n, where n is the number of instructions in the routine. By contrast, in a system with arbitrary concurrent execution – one where multiple threads might concurrently execute this routine with the same state – you have to read the method in every possible order, making the complexity nn.

    Therefore it is – literally – exponentially more difficult to reason about a routine that may be executed from an arbitrary number of threads concurrently. Instead, you need to consider every possible caller across your program, understanding what threads they might be invoked from, or what state they might share. If you’re writing a library desgined to be thread-safe, then you must place some of the burden of this understanding on your caller.

    The importance of local reasoning really cannot be overstated. Computer programs are, at least for the time being, constructed by human beings who are thinking thoughts. Correct computer programs are constructed by human beings who can simultaneously think thoughts about all the interactions that the portion of the system they’re developing will have with other portions.

    A human being can only think about seven things at once, plus or minus two. Therefore, although we may develop software systems that contain thousands, millions, or billions of components over time, we must be able to make changes to that system while only holding in mind an average of seven things. Really bad systems will make us concentrate on nine things and we will only be able to correctly change them when we’re at our absolute best. Really good systems will require us to concentrate on only five, and we might be able to write correct code for them even when we’re tired.

    Aside: “Oh Come On They’re Not That Bad”

    Those of you who actually use threads to write real software are probably objecting at this point. “Nobody would actually try to write free-threading code like this,” I can hear you complain, “Of course we’d use a lock or a queue to introduce some critical sections if we’re manipulating state.”

    Mutexes can help mitigate this combinatorial explosion, but they can’t eliminate it, and they come with their own cost; you need to develop strategies to ensure consistent ordering of their acquisition. Mutexes should really be used to build queues, and to avoid deadlocks those queues should be non-blocking but eventually a system which communicates exclusively through non-blocking queues effectively becomes a set of communicating event loops, and its problems revert to those of an event-driven system; it doesn’t look like regular programming with threads any more.

    But even if you build such a system, if you’re using a language like Python (or the ones detailed above) where modules, classes, and methods are all globally shared, mutable state, it’s always possible to make an error that will affect the behavior of your whole program without even realizing that you’re interacting with state at all. You have to have a level of vigilance bordering on paranoia just to make sure that your conventions around where state can be manipulated and by whom are honored, because when such an interaction causes a bug it’s nearly impossible to tell where it came from.

    Of course, threads are just one source of inscrutable, brain-bending bugs, and quite often you can make workable assumptions that preclude you from actually having to square the complexity of every single routine that you touch; for one thing, many computations don’t require manipulating state at all, and you can (and must) ignore lots of things that can happen on every line of code anyway. (If you think not, when was the last time you audited your code base for correct behavior in the face of memory allocation failures?) So, in a sense, it’s possible to write real systems with threads that perform more or less correctly for the same reasons it’s possible to write any software approximating correctness at all; we all need a little strength of will and faith in our holy cause sometimes.

    Nevertheless I still think it’s a bad idea to make things harder for ourselves if we can avoid it.

    Solution: Don’t Use Threads

    So now I’ve convinced you that if you’re programming in Python (or one of its moral equivalents with respect to concurrency and state) you shouldn’t use threads. Great. What are you going to do instead?

    There’s a lot of debate over the best way to do “asynchronous” programming - that is to say, “not threads”, four options are often presented.

    1. Straight callbacks: Twisted’s IProtocol, JavaScript’s on<foo> idiom, where you give a callback to something which will call it later and then return control to something (usually a main loop) which will execute those callbacks,
    2. “Managed” callbacks, or Futures: Twisted’s Deferred, JavaScript’s Promises/A[+], E’s Promises, where you create a dedicated result-that-will-be-available-in-the-future object and return it for the caller to add callbacks to,
    3. Explicit coroutines: Twisted’s @inlineCallbacks, Tulip’s yield from coroutines, C#’s async/await, where you have a syntactic feature that explicitly suspends the current routine,
    4. and finally, implicit coroutines: Java’s “green threads”, Twisted’s Corotwine, eventlet, gevent, where any function may switch the entire stack of the current thread of control by calling a function which suspends it.

    One of these things is not like the others; one of these things just doesn’t belong.

    Don’t Use Those Threads Either

    Options 1-3 are all ways of representing the cooperative transfer of control within a stateful system. They are a semantic improvement over threads. Callbacks, Futures, and Yield-based coroutines all allow for local reasoning about concurrent operations.

    So why does option 4 even show up in this list?

    Unfortunately, “asynchronous” systems have often been evangelized by emphasizing a somewhat dubious optimization which allows for a higher level of I/O-bound concurrency than with preemptive threads, rather than the problems with threading as a programming model that I’ve explained above. By characterizing “asynchronousness” in this way, it makes sense to lump all 4 choices together.

    I’ve been guilty of this myself, especially in years past: saying that a system using Twisted is more efficient than one using an alternative approach using threads. In many cases that’s been true, but:

    1. the situation is almost always more complicated than that, when it comes to performance,
    2. “context switching” is rarely a bottleneck in real-world programs, and
    3. it’s a bit of a distraction from the much bigger advantage of event-driven programming, which is simply that it’s easier to write programs at scale, in both senses (that is, programs containing lots of code as well as programs which have many concurrent users).

    A system that presents “implicit coroutines” – those which may transfer control to another concurrent task at any layer of the stack without any syntactic indication that this may happen – are simply the dubious optimization by itself.

    Despite the fact that implicit coroutines masquerade under many different names, many of which don’t include the word “thread” – for example, “greenlets”, “coroutines”, “fibers”, “tasks” – green or lightweight threads are indeed threads, in that they present these same problems. In the long run, when you build a system that relies upon them, you eventually have all the pitfalls and dangers of full-blown preemptive threads. Which, as shown above, are bad.

    When you look at the implementation of a potentially concurrent routine written using callbacks or yielding coroutines, you can visually see exactly where it might yield control, either to other routines, or perhaps even re-enter the same routine concurrently. If you are using callbacks – managed or otherwise – you will see a return statement, or the termination of a routine, which allows execution of the main loop to potentially continue. If you’re using explicit coroutines, you’ll see a yield (or await) statement which suspends the coroutine. Because you can see these indications of potential concurrency, they’re outside of your mind, in your text editor, and you don’t need to actively remember them as you’re working on them.

    You can think of these explicit yield-points as places where your program may gracefully bend to the needs of concurrent inputs. Crumple zones, or relief valves, for your logic, if you will: a single point where you have to consider the implications of a transfer of control to other parts of your program, rather than a rigid routine which might transfer (break) at any point beyond your control.

    Like crumple zones, you shouldn’t have too many of them, or they lose their effectiveness. A long routine which has an explicit yield point before every single instruction requires just as much out-of-order reasoning, and is therefore just as error-prone as one which has none, but might context switch before any instruction anyway. The advantage of having to actually insert the yield point explicitly is that at least you can see when a routine has this problem, and start to clean up and consolidate the mangement of its concurrency.

    But this is all pretty abstract; let me give you a specific practical example, and a small theoretical demonstration.

    The Buggiest Bug

    Brass Cockroach - Image Credit GlamourGirlBeads http://www.etsy.com/listing/62042780/large-antiqued-brass-cockroach1-ants3074

    When we wrote the very first version of Twisted Reality in Python, the version we had previously written in Java was already using green threads; at the time, the JVM didn’t have any other kind of threads. The advantage to the new networking layer that we developed was not some massive leap forward in performance (the software in question was a multiplayer text adventure, which at the absolute height of its popularity might have been played by 30 people simultaneously) but rather the dramatic reduction in the number and severity of horrible, un-traceable concurrency bugs. One, in particular, involved a brass, mechanical cockroach which would crawl around on a timer, leaping out of a player’s hands if it was in their inventory, moving between rooms if not. In the multithreaded version, the cockroach would leap out of your hands but then also still stay in your hands. As the cockroach moved between rooms it would create shadow copies of itself, slowly but inexorably creating a cockroach apocalypse as tens of thousands of pointers to the cockroach, each somehow acquiring their own timer, scuttled their way into every player’s inventory dozens of times.

    Given that the feeling that this particular narrative feature was supposed to inspire was eccentric whimsy and not existential terror, the non-determinism introduced by threads was a serious problem. Our hope for the even-driven re-write was simply that we’d be able to diagnose the bug by single-stepping through a debugger; instead, the bug simply disappeared. (Echoes of this persist, in that you may rarely hear a particularly grizzled Twisted old-timer refer to a particularly intractable bug as a “brass cockroach”.)

    The original source of the bug was so completely intractable that the only workable solution was to re-write the entire system from scratch. Months of debugging and testing and experimenting could still reproduce it only intermittently, and several “fixes” (read: random, desperate changes to the code) never resulted in anything.

    I’d rather not do that ever again.

    Ca(sh|che Coherent) Money

    Despite the (I hope) entertaining nature of that anecdote, it still might be somewhat hard to visualize how concurrency results in a bug like that, and the code for that example is far too sprawling to be useful as an explanation. So here's a smaller in vitro example. Take my word for it that the source of the above bug was the result of many, many intersecting examples of the problem described below.

    As it happens, this is the same variety of example Guido van Rossum gives when he describes why chose to use explicit coroutines instead of green threads for the upcoming standard library asyncio module, born out of the “tulip” project, so it's happened to more than one person in real life.

    Photo Credit: Ennor https://www.flickr.com/photos/ennor/441394582/sizes/l/

    Let’s say we have this program:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    def transfer(amount, payer, payee, server):
        if not payer.sufficient_funds_for_withdrawl(amount):
            raise InsufficientFunds()
        log("{payer} has sufficient funds.", payer=payer)
        payee.deposit(amount)
        log("{payee} received payment", payee=payee)
        payer.withdraw(amount)
        log("{payer} made payment", payer=payer)
        server.update_balances([payer, payee])
    

    (I realize that the ordering of operations is a bit odd in this example, but it makes the point easier to demonstrate, so please bear with me.)

    In a world without concurrency, this is of course correct. If you run transfer twice in a row, the balance of both accounts is always correct. But if we were to run transfer with the same two accounts in an arbitrary number of threads simultaneously, it is (obviously, I hope) wrong. One thread could update a payer’s balance below the funds-sufficient threshold after the check to see if they’re sufficient, but before issuing the withdrawl.

    So, let’s make it concurrent, in the PEP 3156 style. That update_balances routine looks like it probably has to do some network communication and block, so let’s consider that it is as follows:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    @coroutine
    def transfer(amount, payer, payee, server):
        if not payer.sufficient_funds_for_withdrawl(amount):
            raise InsufficientFunds()
        log("{payer} has sufficient funds.", payer=payer)
        payee.deposit(amount)
        log("{payee} received payment", payee=payee)
        payer.withdraw(amount)
        log("{payer} made payment", payer=payer)
        yield from server.update_balances([payer, payee])
    

    So now we have a trivially concurrent, correct version of this routine, although we did have to update it a little. Regardless of what sufficient_funds_for_withdrawl, deposit and withdrawl do - even if they do network I/O - we know that we aren’t waiting for any of them to complete, so they can’t cause transfer to interfere with itself. For the sake of a brief example here, we’ll have to assume update_balances is a bit magical; for this to work our reads of the payer and payee’s balance must be consistent.

    But if we were to use green threads as our “asynchronous” mechanism rather than coroutines and yields, we wouldn’t need to modify the program at all! Isn’t that better? And only update_balances blocks anyway, so isn’t it just as correct?

    Sure: for now.

    But now let’s make another, subtler code change: our hypothetical operations team has requested that we put all of our log messages into a networked log-gathering system for analysis. A reasonable request, so we alter the implementation of log to write to the network.

    Now, what will we have to do to modify the green-threaded version of this code? Nothing! This is usually the point where fans of various green-threading systems will point and jeer, since once the logging system is modified to do its network I/O, you don’t even have to touch the code for the payments system. Separation of concerns! Less pointless busy-work! Looks like the green-threaded system is winning.

    Oh well. Since I’m still a fan of explicit concurrency management, let’s do the clearly unnecessary busy-work of updating the ledger code.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    @coroutine
    def transfer(amount, payer, payee, server):
        if not payer.sufficient_funds_for_withdrawl(amount):
            raise InsufficientFunds()
        yield from log("{payer} has sufficient funds.", payer=payer)
        payee.deposit(amount)
        yield from log("{payee} received payment", payee=payee)
        payer.withdraw(amount)
        yield from log("{payer} made payment", payer=payer)
        yield from server.update_balances([payer, payee])
    

    Well okay, at least that wasn’t too hard, if somewhat tedious. Sigh. I guess we can go update all of the ledger’s callers now and update them too…

    …wait a second.

    In order to update this routine for a non-blocking version of log, we had to type a yield keyword between the sufficient_funds_for_withdrawl check and the withdraw call, between the deposit and the withdraw call, and between the withdraw and update_balances call. If we know a little about concurrency and a little about what this program is doing, we know that every one of those yield froms are a potential problem. If those log calls start to back up and block, a payer may have their account checked for sufficient funds, then funds could be deducted while a log message is going on, leaving them with a negative balance.

    If we were in the middle of updating lots of code, we might have blindly added these yield keywords without noticing that mistake. I've certainly done that in the past, too. But just the mechanical act of typing these out is an opportunity to notice that something’s wrong, both now and later. Even if we get all the way through making the changes without realizing the problem, when we notice that balances are off, we can look only (reasoning locally!) at the transfer routine and realize, when we look at it, based on the presence of the yield from keywords, that there is something wrong with the transfer routine itself, regardless of the behavior of any of the things it’s calling.

    In the process of making all these obviously broken modifications, another thought might occur to us: do we really need to wait before log messages are transmitted to the logging system before moving on with our application logic? The answer would almost always be “no”. A smart implementation of log could simply queue some outbound messages to the logging system, then discard if too many are buffered, removing any need for its caller to honor backpressure or slow down if the logging system can’t keep up. Consider the way syslog says “and N more” instead of logging certain messages repeatedly. That feature allows it to avoid filling up logs with repeated messages, and decreases the amount of stuff that needs to be buffered if writing the logs to disk is slow.

    All the extra work you need to do when you update all the callers of log when you make it asynchronous is therefore a feature. Tedious as it may be, the asynchronousness of an individual function is, in fact, something that all of its callers must be aware of, just as they must be aware of its arguments and its return type.

    In fact you are changing its return type: in Twisted, that return type would be Deferred, and in Tulip, that return type is a new flavor of generator. This new return type represents the new semantics that happen when you make a function start having concurrency implications.

    Haskell does this as well, by embedding the IO monad in the return type of any function which needs to have side-effects. This is what certain people mean when they say Deferreds are a Monad.

    The main difference between lightweight and heavyweight threads is that it is that, with rigorous application of strict principles like “never share any state unnecessarily”, and “always write tests for every routine at every point where it might suspend”, lightweight threads make it at least possible to write a program that will behave deterministically and correctly, assuming you understand it in its entirety. When you find a surprising bug in production, because a routine that is now suspending in a place it wasn’t before, it’s possible with a lightweight threading system to write a deterministic test that will exercise that code path. With heavyweight threads, any line could be the position of a context switch at any time, so it’s just not tractable to write tests for every possible order of execution.

    However, with lightweight threads, you still can’t write a test to discover when a new yield point might be causing problems, so you're still always playing catch-up.

    Although it’s possible to do this, it remains very challenging. As I described above, in languages like Python, Ruby, JavaScript, and PHP, even the code itself is shared, mutable state. Classes, types, functions, and namespaces are all shared, and all mutable. Libraries like object relational mappers commonly store state on classes.

    No Shortcuts

    Despite the great deal of badmouthing of threads above, my main purpose in writing this was not to convince you that threads are, in fact, bad. (Hopefully, you were convinced before you started reading this.) What I hope I’ve demonstrated is that if you agree with me that threading has problematic semantics, and is difficult to reason about, then there’s no particular advantage to using microthreads, beyond potentially optimizing your multithreaded code for a very specific I/O bound workload.

    There are no shortcuts to making single-tasking code concurrent. It's just a hard problem, and some of that hard problem is reflected in the difficulty of typing a bunch of new concurrency-specific code.

    So don’t be fooled: a thread is a thread regardless of its color. If you want your program to be supple and resilient in the face of concurrency, when the storm of concurrency blows, allow it to change. Tell it to yield, just like the reed. Otherwise, just like the steadfast and unchanging oak tree in the storm, your steadfast and unchanging algorithms will break right in half.

    by Glyph at February 24, 2014 10:05 AM

    February 11, 2014

    The Moon Research Blog

    Twisted Names – now with a (private) EDNS(0) message parser

    This is the first in a series of articles describing my progress towards introducing EDNS(0) and DNSSEC support to Twisted Names.

    In this article, I'll describe two new private classes designed to parse EDNS(0) messages.

    But first a little background.

    The EDNS protocol extensions depend on the use of a special DNS record which crucially contains the advertised maximum datagram size, extended return codes, a DNSSECOK flag, and a payload which contains zero or more extensible headers.

    This special record is called an OPT record. It isn't a real record though; it doesn't convey any domain names or domain records. It's a hack to work around the lack of space in a standard 512 byte DNS message. All the header flags and fields that couldn't be shoehorned into the 96bit header section of a DNS message are instead encoded into an OPT "pseudo" record.

    EDNS clients and servers append one OPT record to the "additional" records in a sent DNS message. This record is then extracted and interpreted by an EDNS aware receiver.

    Twisted already has a class for parsing DNS records and it already has a class for parsing DNS messages. I've now created some wrappers which add the EDNS special behaviour to those existing APIs.

    The first wrapper is a class called twisted.names.dns._OPTHeader, which wraps an instance of twisted.names.dns.RRHeader. This allows us to re-use the existing record parsing code, and means that the code in _OPTHeader is purely concerned with the special behaviour of an OPT record. Which seems neat! It also made the unit tests much more focused.

    The second wrapper is a class called twisted.names.dns._EDNSMessage which is in trunk but not yet in a released version. This wraps an instance of twisted.names.dns.Message. And again, this allows us to re-use the existing parsing code while clearly separating the EDNS specific behaviour.

    So Twisted Names can now interpret OPT records and EDNS messages, but we've decided to keep these private for the time being; until we are satisfied with the APIs.

    This first version of _EDNSMessage has been written as a drop in replacement for dns.Message. But as a result it has inherited some of the ugliness of that class, namely:

    Tom has some ideas about introducing a new Interface for message constructorsand I've done some work on improving the abbreviated names, but I got stuck trying to figure out how to introduce the changes while maintaining backwards compatibility.

    I'd originally thought those things were blockers, because I intended that users would pass curried message constructors / factories to the various protocols and client APIs, as a way of configuring / overriding EDNS flags and fields in their requests. But that's not a good API (as Tom points out in another comment).

    So I've decided that those cleanups would be nice to have, but probably not necessary right now.

    A much simpler strategy – and my current priority – is to just switch all Twisted names code (client, protocols, server) to use the new dns._EDNSMessage class in place of dns.Message. This shouldn't introduce any backwards incompatibilities and I'll explicitly disable the use EDNS during encoding (for now). That way Twisted Names clients and servers should continue to behave exactly as before.

    But I'll then be free to start work on a DNS client which can be configured to send EDNS(0) requests and which is capable of negotiating with EDNS(0) servers. That's the key requirement of the first project milestone.

    (twisted.names.server and the twistd dns plugin will remain non-EDNS for the timebeing.)

    In fact I've already started that work, but first I had to fix a bug in the way twisted.names.server generates response messages. My proposed fix includes new functions and methods for generating response messages from a request message. (Previously, the request message was being re-used as the basis for the response, and inevitably, client specific flags and fields were leaking into the response.)

    Happily, these new APIs will also allow me to explicitly disable EDNS(0) in twisted.names.server, until such time as we implement server side EDNS(0) support.

    So that's all for now. In my next post I hope to be able to report that Twisted Names is using the new _EDNSMessage class internally and summarise some of the other work that I've done on Twisted Names in support of the EDNS(0) and DNSSEC project.

    Thanks once again to The NLnet Foundation for funding this project.

    by Richard Wall (noreply@blogger.com) at February 11, 2014 04:50 PM

    February 03, 2014

    The Moon Research Blog

    The Bristol Twisted Sprint – a belated thankyou

    I'd like to thank Luke, Rob and everyone at HybridCluster for organising the Bristol Twisted Sprint in December last year and for their hospitality on the day.

    I got quite a bit of work done and it was a rare opportunity to meet Tom and JP, who had flown in from the Americas, and Hynek who had flown in from Berlin.

    Earlier that week I'd been amazed to receive an email from Tom with code-reviews of three of my EDNS branches. He'd done them during his flight from Canada! Not sure I'd have had the stamina, given the lack of elbow room and the temptation of in-flight entertainment and complimentary booze. But Tom's a pro and it meant that we had lots to discuss on the day of the sprint. Tom had raised some concerns about my proposed mechanism for setting EDNS parameters from the high level twisted.names.client API – with which I agree. But I'm still struggling to implement the alternative mechanism that we discussed on the day. More on that in a later blog post.

    It was also nice to see Hynek, who showed me an interesting custom DNS server which he'd written despite the lack of documentation and despite having to subclass and override the private _lookup method of twisted.names.authority. He laid it on pretty thick, and guilt–tripped me into writing some decent documentation for twisted.names.server. Conscience is what motivates me, so I'm pleased to report that we now have a little more narrative documentation for Twisted Names. And it looks pretty nice too, thanks to khorn's and glyph's work on porting the Twisted documentation to ReST and Sphinx.

    JP adopted a "headmasterly" role; supervising things from a lectern at the front of the room (an improvised standing desk), and intimidating us (the pupils) with astonishing Typespeed scores between builds – the keyboard noise was deafening! But I'm exaggerating, JP was as helpful and as thoughtful as usual. Switching context effortlessly to assist us with all areas of Twisted — from development environment tips, to Twisted OpenSSL integration, to EDNS, to PyPy tests and benchmarking, to Tun/Tap APIs. Very impressive!

    Let's hope there are more UK Twisted sprints in 2014.

    by Richard Wall (noreply@blogger.com) at February 03, 2014 12:32 PM

    February 01, 2014

    The Moon Research Blog

    Twisted Names EDNS — a progress update (or lack thereof)

    Apologies to those of you who have been eagerly awaiting news of EDNS and DNSSEC support in Twisted Names. I haven't really got a good excuse; just that I'm a bad communicator and I've been distracted by some mundane (and some exciting) non-computer activities. Sorry.

    So, a progress report is long overdue and progress has been a little slower than expected, but let me try and compensate by promising a series of short updates and demos which I'll publish over the next few weeks, demonstrating what hasbeen achieved so far.

    Stay tuned!

    by Richard Wall (noreply@blogger.com) at February 01, 2014 09:43 PM

    January 24, 2014

    Glyph Lefkowitz

    Unyielding

    The Oak and the Reed by Achille Michallon

    … that which is hard and stiff
    is the follower of death
    that which is soft and yielding
    is the follower of life …

    – the Tao Te Ching, chapter 76

    Problem: Threads Are Bad

    As we know, threads are a bad idea, (for most purposes). Threads make local reasoning difficult, and local reasoning is perhaps the most important thing in software development.

    With the word “threads”, I am referring to shared-state multithreading, despite the fact that there are languages, like Erlang and Haskell which refer to concurrent processes – those which do not implicitly share state, and require explicit coordination – as “threads”.

    My experience is mainly (although not exclusively) with Python but the ideas presented here should generalize to most languages which have global shared mutable state by default, which is to say, quite a lot of them: C (including Original Recipe, Sharp, Extra Crispy, Objective, and Plus Plus), JavaScript, Java, Scheme, Ruby, and PHP, just to name a few.

    With the phrase “local reasoning”, I’m referring to the ability to understand the behavior (and thereby, the correctness) of a routine by examining the routine itself rather than examining the entire system.

    When you’re looking at a routine that manipulates some state, in a single-tasking, nonconcurrent system, you only have to imagine the state at the beginning of the routine, and the state at the end of the routine. To imagine the different states, you need only to read the routine and imagine executing its instructions in order from top to bottom. This means that the number of instructions you must consider is n, where n is the number of instructions in the routine. By contrast, in a system with arbitrary concurrent execution – one where multiple threads might concurrently execute this routine with the same state – you have to read the method in every possible order, making the complexity nn.

    Therefore it is – literally – exponentially more difficult to reason about a routine that may be executed from an arbitrary number of threads concurrently. Instead, you need to consider every possible caller across your program, understanding what threads they might be invoked from, or what state they might share. If you’re writing a library desgined to be thread-safe, then you must place some of the burden of this understanding on your caller.

    The importance of local reasoning really cannot be overstated. Computer programs are, at least for the time being, constructed by human beings who are thinking thoughts. Correct computer programs are constructed by human beings who can simultaneously think thoughts about all the interactions that the portion of the system they’re developing will have with other portions.

    A human being can only think about seven things at once, plus or minus two. Therefore, although we may develop software systems that contain thousands, millions, or billions of components over time, we must be able to make changes to that system while only holding in mind an average of seven things. Really bad systems will make us concentrate on nine things and we will only be able to correctly change them when we’re at our absolute best. Really good systems will require us to concentrate on only five, and we might be able to write correct code for them even when we’re tired.

    Aside: “Oh Come On They’re Not That Bad”

    Those of you who actually use threads to write real software are probably objecting at this point. “Nobody would actually try to write free-threading code like this,” I can hear you complain, “Of course we’d use a lock or a queue to introduce some critical sections if we’re manipulating state.”

    Mutexes can help mitigate this combinatorial explosion, but they can’t eliminate it, and they come with their own cost; you need to develop strategies to ensure consistent ordering of their acquisition. Mutexes should really be used to build queues, and to avoid deadlocks those queues should be non-blocking but eventually a system which communicates exclusively through non-blocking queues effectively becomes a set of communicating event loops, and its problems revert to those of an event-driven system; it doesn’t look like regular programming with threads any more.

    But even if you build such a system, if you’re using a language like Python (or the ones detailed above) where modules, classes, and methods are all globally shared, mutable state, it’s always possible to make an error that will affect the behavior of your whole program without even realizing that you’re interacting with state at all. You have to have a level of vigilance bordering on paranoia just to make sure that your conventions around where state can be manipulated and by whom are honored, because when such an interaction causes a bug it’s nearly impossible to tell where it came from.

    Of course, threads are just one source of inscrutable, brain-bending bugs, and quite often you can make workable assumptions that preclude you from actually having to square the complexity of every single routine that you touch; for one thing, many computations don’t require manipulating state at all, and you can (and must) ignore lots of things that can happen on every line of code anyway. (If you think not, when was the last time you audited your code base for correct behavior in the face of memory allocation failures?) So, in a sense, it’s possible to write real systems with threads that perform more or less correctly for the same reasons it’s possible to write any software approximating correctness at all; we all need a little strength of will and faith in our holy cause sometimes.

    Nevertheless I still think it’s a bad idea to make things harder for ourselves if we can avoid it.

    Solution: Don’t Use Threads

    So now I’ve convinced you that if you’re programming in Python (or one of its moral equivalents with respect to concurrency and state) you shouldn’t use threads. Great. What are you going to do instead?

    There’s a lot of debate over the best way to do “asynchronous” programming - that is to say, “not threads”, four options are often presented.

    1. Straight callbacks: Twisted’s IProtocol, JavaScript’s on<foo> idiom, where you give a callback to something which will call it later and then return control to something (usually a main loop) which will execute those callbacks,
    2. “Managed” callbacks, or Futures: Twisted’s Deferred, JavaScript’s Promises/A[+], E’s Promises, where you create a dedicated result-that-will-be-available-in-the-future object and return it for the caller to add callbacks to,
    3. Explicit coroutines: Twisted’s @inlineCallbacks, Tulip’s yield from coroutines, C#’s async/await, where you have a syntactic feature that explicitly suspends the current routine,
    4. and finally, implicit coroutines: Java’s “green threads”, Twisted’s Corotwine, eventlet, gevent, where any function may switch the entire stack of the current thread of control by calling a function which suspends it.

    One of these things is not like the others; one of these things just doesn’t belong.

    Don’t Use Those Threads Either

    Options 1-3 are all ways of representing the cooperative transfer of control within a stateful system. They are a semantic improvement over threads. Callbacks, Futures, and Yield-based coroutines all allow for local reasoning about concurrent operations.

    So why does option 4 even show up in this list?

    Unfortunately, “asynchronous” systems have often been evangelized by emphasizing a somewhat dubious optimization which allows for a higher level of I/O-bound concurrency than with preemptive threads, rather than the problems with threading as a programming model that I’ve explained above. By characterizing “asynchronousness” in this way, it makes sense to lump all 4 choices together.

    I’ve been guilty of this myself, especially in years past: saying that a system using Twisted is more efficient than one using an alternative approach using threads. In many cases that’s been true, but:

    1. the situation is almost always more complicated than that, when it comes to performance,
    2. “context switching” is rarely a bottleneck in real-world programs, and
    3. it’s a bit of a distraction from the much bigger advantage of event-driven programming, which is simply that it’s easier to write programs at scale, in both senses (that is, programs containing lots of code as well as programs which have many concurrent users).

    A system that presents “implicit coroutines” – those which may transfer control to another concurrent task at any layer of the stack without any syntactic indication that this may happen – are simply the dubious optimization by itself.

    Despite the fact that implicit coroutines masquerade under many different names, many of which don’t include the word “thread” – for example, “greenlets”, “coroutines”, “fibers”, “tasks” – green or lightweight threads are indeed threads, in that they present these same problems. In the long run, when you build a system that relies upon them, you eventually have all the pitfalls and dangers of full-blown preemptive threads. Which, as shown above, are bad.

    When you look at the implementation of a potentially concurrent routine written using callbacks or yielding coroutines, you can visually see exactly where it might yield control, either to other routines, or perhaps even re-enter the same routine concurrently. If you are using callbacks – managed or otherwise – you will see a return statement, or the termination of a routine, which allows execution of the main loop to potentially continue. If you’re using explicit coroutines, you’ll see a yield (or await) statement which suspends the coroutine. Because you can see these indications of potential concurrency, they’re outside of your mind, in your text editor, and you don’t need to actively remember them as you’re working on them.

    You can think of these explicit yield-points as places where your program may gracefully bend to the needs of concurrent inputs. Crumple zones, or relief valves, for your logic, if you will: a single point where you have to consider the implications of a transfer of control to other parts of your program, rather than a rigid routine which might transfer (break) at any point beyond your control.

    Like crumple zones, you shouldn’t have too many of them, or they lose their effectiveness. A long routine which has an explicit yield point before every single instruction requires just as much out-of-order reasoning, and is therefore just as error-prone as one which has none, but might context switch before any instruction anyway. The advantage of having to actually insert the yield point explicitly is that at least you can see when a routine has this problem, and start to clean up and consolidate the mangement of its concurrency.

    But this is all pretty abstract; let me give you a specific practical example, and a small theoretical demonstration.

    The Buggiest Bug

    Brass Cockroach - Image Credit GlamourGirlBeads http://www.etsy.com/listing/62042780/large-antiqued-brass-cockroach1-ants3074

    When we wrote the very first version of Twisted Reality in Python, the version we had previously written in Java was already using green threads; at the time, the JVM didn’t have any other kind of threads. The advantage to the new networking layer that we developed was not some massive leap forward in performance (the software in question was a multiplayer text adventure, which at the absolute height of its popularity might have been played by 30 people simultaneously) but rather the dramatic reduction in the number and severity of horrible, un-traceable concurrency bugs. One, in particular, involved a brass, mechanical cockroach which would crawl around on a timer, leaping out of a player’s hands if it was in their inventory, moving between rooms if not. In the multithreaded version, the cockroach would leap out of your hands but then also still stay in your hands. As the cockroach moved between rooms it would create shadow copies of itself, slowly but inexorably creating a cockroach apocalypse as tens of thousands of pointers to the cockroach, each somehow acquiring their own timer, scuttled their way into every player’s inventory dozens of times.

    Given that the feeling that this particular narrative feature was supposed to inspire was eccentric whimsy and not existential terror, the non-determinism introduced by threads was a serious problem. Our hope for the even-driven re-write was simply that we’d be able to diagnose the bug by single-stepping through a debugger; instead, the bug simply disappeared. (Echoes of this persist, in that you may rarely hear a particularly grizzled Twisted old-timer refer to a particularly intractable bug as a “brass cockroach”.)

    The original source of the bug was so completely intractable that the only workable solution was to re-write the entire system from scratch. Months of debugging and testing and experimenting could still reproduce it only intermittently, and several “fixes” (read: random, desperate changes to the code) never resulted in anything.

    I’d rather not do that ever again.

    Ca(sh|che Coherent) Money

    Despite the (I hope) entertaining nature of that anecdote, it still might be somewhat hard to visualize how concurrency results in a bug like that, and the code for that example is far too sprawling to be useful as an explanation. So here's a smaller in vitro example. Take my word for it that the source of the above bug was the result of many, many intersecting examples of the problem described below.

    As it happens, this is the same variety of example Guido van Rossum gives when he describes why chose to use explicit coroutines instead of green threads for the upcoming standard library asyncio module, born out of the “tulip” project, so it's happened to more than one person in real life.

    Photo Credit: Ennor https://www.flickr.com/photos/ennor/441394582/sizes/l/

    Let’s say we have this program:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    def transfer(amount, payer, payee, server):
        if not payer.sufficient_funds_for_withdrawl(amount):
            raise InsufficientFunds()
        log("{payer} has sufficient funds.", payer=payer)
        payee.deposit(amount)
        log("{payee} received payment", payee=payee)
        payer.withdraw(amount)
        log("{payer} made payment", payer=payer)
        server.update_balances([payer, payee])
    

    (I realize that the ordering of operations is a bit odd in this example, but it makes the point easier to demonstrate, so please bear with me.)

    In a world without concurrency, this is of course correct. If you run transfer twice in a row, the balance of both accounts is always correct. But if we were to run transfer with the same two accounts in an arbitrary number of threads simultaneously, it is (obviously, I hope) wrong. One thread could update a payer’s balance below the funds-sufficient threshold after the check to see if they’re sufficient, but before issuing the withdrawl.

    So, let’s make it concurrent, in the PEP 3156 style. That update_balances routine looks like it probably has to do some network communication and block, so let’s consider that it is as follows:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    @coroutine
    def transfer(amount, payer, payee, server):
        if not payer.sufficient_funds_for_withdrawl(amount):
            raise InsufficientFunds()
        log("{payer} has sufficient funds.", payer=payer)
        payee.deposit(amount)
        log("{payee} received payment", payee=payee)
        payer.withdraw(amount)
        log("{payer} made payment", payer=payer)
        yield from server.update_balances([payer, payee])
    

    So now we have a trivially concurrent, correct version of this routine, although we did have to update it a little. Regardless of what sufficient_funds_for_withdrawl, deposit and withdrawl do - even if they do network I/O - we know that we aren’t waiting for any of them to complete, so they can’t cause transfer to interfere with itself. For the sake of a brief example here, we’ll have to assume update_balances is a bit magical; for this to work our reads of the payer and payee’s balance must be consistent.

    But if we were to use green threads as our “asynchronous” mechanism rather than coroutines and yields, we wouldn’t need to modify the program at all! Isn’t that better? And only update_balances blocks anyway, so isn’t it just as correct?

    Sure: for now.

    But now let’s make another, subtler code change: our hypothetical operations team has requested that we put all of our log messages into a networked log-gathering system for analysis. A reasonable request, so we alter the implementation of log to write to the network.

    Now, what will we have to do to modify the green-threaded version of this code? Nothing! This is usually the point where fans of various green-threading systems will point and jeer, since once the logging system is modified to do its network I/O, you don’t even have to touch the code for the payments system. Separation of concerns! Less pointless busy-work! Looks like the green-threaded system is winning.

    Oh well. Since I’m still a fan of explicit concurrency management, let’s do the clearly unnecessary busy-work of updating the ledger code.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    @coroutine
    def transfer(amount, payer, payee, server):
        if not payer.sufficient_funds_for_withdrawl(amount):
            raise InsufficientFunds()
        yield from log("{payer} has sufficient funds.", payer=payer)
        payee.deposit(amount)
        yield from log("{payee} received payment", payee=payee)
        payer.withdraw(amount)
        yield from log("{payer} made payment", payer=payer)
        yield from server.update_balances([payer, payee])
    

    Well okay, at least that wasn’t too hard, if somewhat tedious. Sigh. I guess we can go update all of the ledger’s callers now and update them too…

    …wait a second.

    In order to update this routine for a non-blocking version of log, we had to type a yield keyword between the sufficient_funds_for_withdrawl check and the withdraw call, between the deposit and the withdraw call, and between the withdraw and update_balances call. If we know a little about concurrency and a little about what this program is doing, we know that every one of those yield froms are a potential problem. If those log calls start to back up and block, a payer may have their account checked for sufficient funds, then funds could be deducted while a log message is going on, leaving them with a negative balance.

    If we were in the middle of updating lots of code, we might have blindly added these yield keywords without noticing that mistake. I've certainly done that in the past, too. But just the mechanical act of typing these out is an opportunity to notice that something’s wrong, both now and later. Even if we get all the way through making the changes without realizing the problem, when we notice that balances are off, we can look only (reasoning locally!) at the transfer routine and realize, when we look at it, based on the presence of the yield from keywords, that there is something wrong with the transfer routine itself, regardless of the behavior of any of the things it’s calling.

    In the process of making all these obviously broken modifications, another thought might occur to us: do we really need to wait before log messages are transmitted to the logging system before moving on with our application logic? The answer would almost always be “no”. A smart implementation of log could simply queue some outbound messages to the logging system, then discard if too many are buffered, removing any need for its caller to honor backpressure or slow down if the logging system can’t keep up. Consider the way syslog says “and N more” instead of logging certain messages repeatedly. That feature allows it to avoid filling up logs with repeated messages, and decreases the amount of stuff that needs to be buffered if writing the logs to disk is slow.

    All the extra work you need to do when you update all the callers of log when you make it asynchronous is therefore a feature. Tedious as it may be, the asynchronousness of an individual function is, in fact, something that all of its callers must be aware of, just as they must be aware of its arguments and its return type.

    In fact you are changing its return type: in Twisted, that return type would be Deferred, and in Tulip, that return type is a new flavor of generator. This new return type represents the new semantics that happen when you make a function start having concurrency implications.

    Haskell does this as well, by embedding the IO monad in the return type of any function which needs to have side-effects. This is what certain people mean when they say Deferreds are a Monad.

    The main difference between lightweight and heavyweight threads is that it is that, with rigorous application of strict principles like “never share any state unnecessarily”, and “always write tests for every routine at every point where it might suspend”, lightweight threads make it at least possible to write a program that will behave deterministically and correctly, assuming you understand it in its entirety. When you find a surprising bug in production, because a routine that is now suspending in a place it wasn’t before, it’s possible with a lightweight threading system to write a deterministic test that will exercise that code path. With heavyweight threads, any line could be the position of a context switch at any time, so it’s just not tractable to write tests for every possible order of execution.

    However, with lightweight threads, you still can’t write a test to discover when a new yield point might be causing problems, so you're still always playing catch-up.

    Although it’s possible to do this, it remains very challenging. As I described above, in languages like Python, Ruby, JavaScript, and PHP, even the code itself is shared, mutable state. Classes, types, functions, and namespaces are all shared, and all mutable. Libraries like object relational mappers commonly store state on classes.

    No Shortcuts

    Despite the great deal of badmouthing of threads above, my main purpose in writing this was not to convince you that threads are, in fact, bad. (Hopefully, you were convinced before you started reading this.) What I hope I’ve demonstrated is that if you agree with me that threading has problematic semantics, and is difficult to reason about, then there’s no particular advantage to using microthreads, beyond potentially optimizing your multithreaded code for a very specific I/O bound workload.

    There are no shortcuts to making single-tasking code concurrent. It's just a hard problem, and some of that hard problem is reflected in the difficulty of typing a bunch of new concurrency-specific code.

    So don’t be fooled: a thread is a thread regardless of its color. If you want your program to be supple and resilient in the face of concurrency, when the storm of concurrency blows, allow it to change. Tell it to yield, just like the reed. Otherwise, just like the steadfast and unchanging oak tree in the storm, your steadfast and unchanging algorithms will break right in half.

    by Glyph at January 24, 2014 08:28 AM

    January 21, 2014

    Glyph Lefkowitz

    And Now For Something Completely Different

    It seems that the constant reminders of all the wonderful new features that my previous web publishing host had in store for me finally motivated me to set up a somewhat more do-it-yourself publishing arrangement, as behooves someone of my particular talents and skills.

    Desk

    I’m using Pelican now, and from what I’ve seen of it so far, it’s a promising tool. I particularly like their approach to publishing multiple content types; for content that I’m writing new, I can use markdown, but for content that I want to process automatically, it accepts HTML just fine, which means that the conversion process from my previous host has been relatively painless. (Relatively painless, that is. Remember to calibrate pain on the Extract-Transform-Load scale, which starts at 8 or so.)

    One interesting consequence of this change is that you may now access this blog securely if that’s the sort of thing you are interested in.

    Another change is that I’ve made the conscious decision to eliminate the comments section. While over the years I have been lucky to receive many interesting and engaging comments on my posts, I think that the best feedback I've gotten has always been from people writing in their own spaces. So I've mainly removed the temptation of the comment box to prevent the motivation to write something thougthful from being burned away in a quick riposte.

    If you would like to comment on something I’ve said here, now or in the future, just send me an email and I’ll be happy to either post your comments inline, in a new post, or ideally, link to your own published response elsewhere.

    Or, of course, ignore it entirely, as is my right as the lord of this particular digital fiefdom.

    Sadly, this change makes some of the older posts imported to this blog read slightly oddly, as they invite you to “comment below”, but nobody’s been doing that for some time, so I think that overall this change is likely to provoke a better quality of discussion and use of time. The full indictment of how You Kids Today ruined the Internet with Your Horrible Comments, and Back In My Day Things Were Better And So Forth will have to wait for a future article. (Or you could go read any New York Times article with the word “millennials” in the title, as it will say basically the same thing.)

    I hope you enjoy the new format.

    by Glyph at January 21, 2014 10:01 AM

    January 13, 2014

    Duncan McGreggor

    Prefix Operators in Haskell

    I wanted to name this post something a little more catchy, like "McCarthy's Dream: Haskell as Lisp's M-Expressions" but I couldn't quite bring myself to do it. If s-expressions had greater support in Haskell, I would totally have gone for it, but alas, they don't.

    However, there is still reason to celebrate: many Haskell operators do support prefix notation! This was a mind-blower to me, since I hadn't  heard about this until last night...

    At the Data Day Texas conference this past Saturday, O'Reilly/StrataConf had a fantastic booth. Among the many cool give-aways they were doing, I obtained a coupon for a free ebook and another for 50% off. Upon returning home and having made my free book decision, I was vacillating between an OCaml book and the famous Haskell one. I've done a little Haskell in the past but have never touched OCaml, so I was pretty tempted.

    However, something amazing happened next. I stumbled upon a page that was comparing OCaml and Haskell, which led to another page... where Haskell prefix notation was mentioned. I know many Haskellers who might read this would shrug, or say "yeah, we know", but this was quite a delightful little discovery for me :-)

    I don't remember the first page I found, but since then, I've come across a couple more resources:
    That's pretty much it, though. (But please let me know if you know of or find any other resources!)

    As such, I needed to do a lot more exploration. Initially, I was really excited and thought I'd be able to convert all Haskell forms to s-expressions (imports, lets, etc.), but I quickly found this was not the case. But the stuff that did work is pretty cool, and I saved it in a series of gists for your viewing pleasure :-)

    Addition

    The first test was pretty simple. Finding success, I thought I'd try something I do when using a Lisp/Scheme interpreter as a calculator. As you can see below, that didn't work (the full traceback is elided). Searching on Hoogλe got me to the answer I was looking for, though. Off to a good start:


    More Operators

    I checked some basic operators next, and a function. Everything's good so far:


    Lists

    Here are some basic list operations, including the ever-so-cool list difference operator, (\\). Also, I enjoyed the cons (:) so much that I made a t-shirt of it :-) (See the postscript below for more info.)


    List Comprehensions

    I'm not sure if one can do any more prefixing than this for list comprehensions:


    Functions

    Same thing for functions; nothing really exciting to see. (Btw, these examples were lifted from LYHGG.)


    Lambdas

    Things get a little more interesting with lambda expressions, especially when a closure is added for spice:


    Function Composition

    Using the compose operator in prefix notation is rather... bizarre :-) It looks much more natural as a series of compositions in a lambda. I also added a mostly-standard view of the compose operator for comparison:


    Monads

    I've saved the best for last, an example of the sort of thing I need when doing matrix operations in game code, graphics, etc. The first one is standard Haskell...


    Wow. Such abstract. So brains

    Note that to make the prefix version anywhere close to legible, I added whitepsace. (If you paste this in ghci, you'll see a bunch of prompts, and then the result. If you're running in the Sublime REPL, be sure to scroll to the right.)

    And that pretty much wraps up what I did on Sunday afternoon :-)

    Other Resources

    Post Script

    If you're in the San Antonio / Austin area (or SF; I'll be there in March) and want to go in on a t-shirt order, let me know :-) We need at least six for the smallest order (tri-blend Next Level; these are a bit pricey, but they feel GREAT). I'll collect payment before I make an order.


    by Duncan McGreggor (noreply@blogger.com) at January 13, 2014 07:52 PM

    December 31, 2013

    Duncan McGreggor

    Joe Armstrong's Favorite Erlang Program... in LFE

    It kind of shocked me to discover recently that a new post hasn't been pushed out on this blog since April. Looking at the drafts I've got backlogged -- 30 and counting -- I also realized that none of these were going to get finished in a single day. Of those 30, probably 6 will ever see the light of day, and all of those are drafts in various states of completion for the Lambda Calculus series.

    There's a great Tiny Scheme example I want to write about, some cool Clojure stuff I've played with, and bunch of functional programming work I'd like to share, etc., etc., but again, none of these are anything I can complete in a few minutes (allowing me to do the other things that I'd like to do today!).

    I'd given up, when I remembered that there was something short, sweet, and a bit of ol' fun that I wanted to share! Mr. Erlang himself recently blogged about it, and I wanted to convert it to LFE: Joe Armstrong's Favorite Erlang Program.

    This little puppy is quite delightful -- and Joe shares a great little story about how he deployed it on Planet Lab in his "aside" section of that post :-)

    After you read his post, come back and take a look at this code:
    That's how you do it in LFE! (Also, did you notice that Github now knows how to colorize *.lfe files? That happened here.)

    Ain't it just the purdiest thing you ever saw?

    This code has also been submitted for inclusion in the LFE examples (thus the "examples/" below). Let's do a quick sanity check:
    And now, let's run the example!

    Happy New Year, everyone :-D


    by Duncan McGreggor (noreply@blogger.com) at December 31, 2013 11:08 PM

    December 19, 2013

    Jack Moffitt

    Building Rust Code - Using Make Part 2

    This series of posts is about building Rust code. In the last post I showed some nice abstractions for using Make with Rust. Today I'll show an improved version of this integration.

    New Rust Compiler Flags

    After landing the pkgid attribute work there was much community discussion about how that feature could be improved. The net result was:

    • pkgid was renamed to crate_id it's being used to identify a crate and not a package, which is a grouping of crates. Actually, a package is still a pretty fluid concept in Rust right now.
    • The crate_id attribute can now override the inferred name of the crate with new syntax. A crate_id of github.com/foo/rust-bar#bar:1.0 names the crate bar which can be found at github.com/foo/rust-bar. Previously the crate name was inferred to be the last component of the path, rust-bar.
    • The compiler has several new flags to print out this information, saving tooling the bother of parsing it out and computing crate hashes itself. You can use --crate-id, --crate-name, and --crate-file-name so get the value of the crate_id attribute, the crate's name, and output filenames the compiler will produce.

    These changes made a good thing even better.

    Magical Makefiles Version 2

    The Makefile hasn't changed much, but here is a much simpler rust.mk that the new compiler flags enable:

    define RUST_CRATE
    
    _rust_crate_dir = $(dir $(1))
    _rust_crate_lib = $$(_rust_crate_dir)lib.rs
    _rust_crate_test = $$(_rust_crate_dir)test.rs
    
    _rust_crate_name = $$(shell $(RUSTC) --crate-name $$(_rust_crate_lib))
    _rust_crate_dylib = $$(shell $(RUSTC) --crate-file-name --lib $$(_rust_crate_lib))
    
    .PHONY : $$(_rust_crate_name)
    $$(_rust_crate_name) : $$(_rust_crate_dylib)
    
    $$(_rust_crate_dylib) : $$(_rust_crate_lib)
    $$(RUSTC) $$(RUSTFLAGS) --dep-info --lib $$<
    
    -include $$(patsubst %.rs,%.d,$$(_rust_crate_lib))
    
    ifneq ($$(wildcard $$(_rust_crate_test)),"")
    
    .PHONY : check-$$(_rust_crate_name)
    check-$$(_rust_crate_name): $$(_rust_crate_name)-test
        ./$$(_rust_crate_name)-test
    
    $$(_rust_crate_name)-test : $$(_rust_crate_test)
        $$(RUSTC) $$(RUSTFLAGS) --dep-info --test $$< -o $$@
    
    -include $$(patsubst %.rs,%.d,$$(_rust_crate_test))
    
    endif
    
    endef
    

    No more nasty sed scripts necessary, but of course, the crate hash is still computable if you want to do it yourself for some reason.

    by Jack Moffitt (jack@metajack.im) at December 19, 2013 02:33 PM

    December 12, 2013

    Jack Moffitt

    Building Rust Code - Using Make

    This series of posts is about building Rust code. In the first post I covered the current issues (and my solutions) around building Rust using external tooling. This post will cover using Make to build Rust projects.

    The Example Crate

    For this post, I'm going to use the rust-geom library as an example. It is a simple Rust library used by Servo to handle common geometric tasks like dealing with points, rectangles, and matrices. It is pure Rust code, has no dependencies, and includes some unit tests.

    We want to build a dynamic library and the test suite, and the Makefile should be able to run the test suite by using make check. As much as possible, we'll use the same crate structure that rustpkg uses so that once rustpkg is ready for real use, the transition to it will be painless.

    Makefile Abstractions

    Did you know that Makefiles can define functions? It's a little clumsy, but it works and you can abstract a bunch of the tedium away. I'd never really noticed them before dealing with the Rust and Servo build systems, which use them heavily.

    By using shell commands like shasum and sed, we can compute crate hashes, and by using Make's eval function, we can dynamically define new targets. I've created a rust.mk which can be included in a Makefile that makes it really easy to build Rust crates.

    Magical Makefiles

    Let's look at a Makefile for rust-geom which uses rust.mk.

    include rust.mk
    
    RUSTC ?= rustc
    RUSTFLAGS ?=
    
    .PHONY : all
    all: rust-geom
    
    .PHONY : check
    check: check-rust-geom
    
    $(eval $(call RUST_CRATE, .))
    

    It includes rust.mk, sets up some basic variables that control the compiler and flags, and then defines the top level targets. The magic bit comes the call to RUST_CRATE which takes a path to where a crate's lib.rs and test.rs are located. In this case the path is the current directory, ..

    RUST_CRATE finds the pkgid attribute in the crate and uses this to compute the crate's name, hash, and the output filename for the library. It then creates a target with the same name as the crate name, in this case rust-geom, and a target for the output file for the library. It uses the Rust compiler's support for dependency information so that it will know exactly when it needs to recompile things.

    If the crate contains a test.rs file, it will also create a target that compiles the tests for the crates into an executable as well as a target to run the tests. The executable will be named after the crate; for rust-geom it will be named rust-geom-test. The check target is also named after the crate, check-rust-geom.

    The files lib.rs and test.rs are the files rustpkg itself uses by default. This Makefile does not support the pkg.rs custom build logic, but if you need custom logic, it is easy enough to modify this example. One benefit of following in rustpkg's footsteps here is that this same crate should be buildable with rustpkg without modification.

    Behind the Scenes

    rust.mk is a a little ugly, but not too bad. It defines a few helper functions like RUST_CRATE_PKGID and RUST_CRATE_HASH which are used by the main RUST_CRATE function. The syntax is a bit silly because of the use of eval and the need to escape $s, but it shouldn't be too hard to follow if you're already familiar with Make syntax.

    RUST_CRATE_PKGID = $(shell sed -ne 's/^#[ *pkgid *= *"(.*)" *];$$/\1/p' $(firstword $(1)))
    RUST_CRATE_PATH = $(shell printf $(1) | sed -ne 's/^([^#]*)\/.*$$/\1/p')
    RUST_CRATE_NAME = $(shell printf $(1) | sed -ne 's/^([^#]*\/){0,1}([^#]*).*$$/\2/p')
    RUST_CRATE_VERSION = $(shell printf $(1) | sed -ne 's/^[^#]*#(.*)$$/\1/p')
    RUST_CRATE_HASH = $(shell printf $(strip $(1)) | shasum -a 256 | sed -ne 's/^(.{8}).*$$/\1/p')
    
    ifeq ($(shell uname),Darwin)
    RUST_DYLIB_EXT=dylib
    else
    RUST_DYLIB_EXT=so
    endif
    
    define RUST_CRATE
    
    _rust_crate_dir = $(dir $(1))
    _rust_crate_lib = $$(_rust_crate_dir)lib.rs
    _rust_crate_test = $$(_rust_crate_dir)test.rs
    
    _rust_crate_pkgid = $$(call RUST_CRATE_PKGID, $$(_rust_crate_lib))
    _rust_crate_name = $$(call RUST_CRATE_NAME, $$(_rust_crate_pkgid))
    _rust_crate_version = $$(call RUST_CRATE_VERSION, $$(_rust_crate_pkgid))
    _rust_crate_hash = $$(call RUST_CRATE_HASH, $$(_rust_crate_pkgid))
    _rust_crate_dylib = lib$$(_rust_crate_name)-$$(_rust_crate_hash)-$$(_rust_crate_version).$(RUST_DYLIB_EXT)
    
    .PHONY : $$(_rust_crate_name)
    $$(_rust_crate_name) : $$(_rust_crate_dylib)
    
    $$(_rust_crate_dylib) : $$(_rust_crate_lib)
        $$(RUSTC) $$(RUSTFLAGS) --dep-info --lib $$<
    
    -include $$(patsubst %.rs,%.d,$$(_rust_crate_lib))
    
    ifneq ($$(wildcard $$(_rust_crate_test)),"")
    
    .PHONY : check-$$(_rust_crate_name)
    check-$$(_rust_crate_name): $$(_rust_crate_name)-test
        ./$$(_rust_crate_name)-test
    
    $$(_rust_crate_name)-test : $$(_rust_crate_test)
        $$(RUSTC) $$(RUSTFLAGS) --dep-info --test $$< -o $$@
    
    -include $$(patsubst %.rs,%.d,$$(_rust_crate_test))
    
    endif
    
    endef
    

    If you wanted, you could add the crate's target and the check target to the all and check targets within this function, simplifying the main Makefile. You could also have it generate an appropriate clean-rust-geom target as well.

    It's not going to win a beauty contest, but it will get the job done nicely.

    Next Up

    In the next post, I plan to show the same example, but using CMake.

    by Jack Moffitt (jack@metajack.im) at December 12, 2013 04:42 PM

    December 11, 2013

    Jack Moffitt

    Building Rust Code - Current Issues

    As rustpkg is still in its infancy, most Rust code tends to be built with make, other tools, or by hand. I've been working on updating Servo's build system to something a bit more reliable and fast, and so I've been giving a lot of thought to build tooling with regards to Rust.

    In this post, I want to cover what the current issues are with building Rust code, especially with regards to external tooling. I'll also describe some recent work I did to address these issues. In the future, I want to cover specific ways to integrate Rust with a few different build tools.

    Current Issues

    Building Rust with existing build tools is a little difficult at the moment. The main issues are related to Rust's attempt to be a better systems language than the existing options.

    For example, Rust uses a larger compilation unit than C and C++ compilers, and existing build tools are designed around single file compilation. Rust libraries are output with unpredictable names. And dependency information must be done manually.

    Compilation Unit

    Many programming languages compile one source file to one output file and then collect the results into some final product. In C, you compile .c files to .o files, then archive or link them into .lib, .a, .dylib, and so on depending on the platform and whether you are building an executable, static library, or shared library. Even Java compiles .java inputs to one or more .class outputs, which are then normally packaged into a .jar.

    In Rust, the unit of compilation is the crate, which is a collection of modules and items. A crate may consist of a single source file or an arbitrary number of them in some directory hierarchy, but its output is a single executable or library.

    Using crates as the compilation unit makes sense from a compiler point of view, as it has more knowledge during compilation to work from. It also makes sense from a versioning point of view as all of the crate's contents goes together. Using crates as the compilation unit allows for cyclic dependencies between modules in the same crates, which is useful to express some things. It also means that separate declaration and implementation pieces are not needed, such as the header files in C and C++.

    Most build tools assume a model similar to that of a typical C compiler. For example, make has pattern rules that can take and input to and output based on on filename transformations. These work great if one input produces one output, but they don't work well in other cases.

    Rust still has a main input file, the one you pass to the compiler, so this difference doesn't have a lot of ramifications when using existing build tools.

    Output Names

    Compilers generally have an option for what to name their output files, or else they derive the output name with some simple formula. C compilers use the -o option to name the output; Java just names the files after the classes they contain. Rust also has a -o option, which works like you expect, except in the case of libraries where it is ignored.

    Libraries in Rust are special in order to avoid naming collisions. Since libraries often end up stored centrally, only one library can have a given name. If I create a library called libgeom it will conflict with someone else's libgeom. Operating systems and distributions end up resolving these conflicts by changing the names slightly, but it's a huge annoyance. To avoid collisions, Rust includes a unique identifier called the crate hash in the name. Now my Rust library libgeom-f32ab99 doesn't conflict with libgeom-00a9edc.

    Unfortunately, the current Rust compiler computes the crate hash by hashing the link metadata, such as name and version, along with the link metadata of its dependencies. This results in a crate hash that only the Rust compiler is realistically able to compute, making it seem pseudo-random. This causes a huge problem for build tooling as the output filename for libraries in unknown.

    To work around this problem when using make, the Rust and Servo build systems use a dummy target called libfoo.dummy for a library called foo, and after running rustc to build the library, it creates the libfoo.dummy file so that make has some well known output to reason about. This workaround is a bit messy and pollutes the build files.

    Here's an example of what a Makefile looks like with this .dummy workaround:

    RUSTC ?= rustc
    
    SOURCES = $(find . -name '*.rs')
    
    all: librust-geom.dummy
    
    librust-geom.dummy: lib.rs $(SOURCES)
        @$(RUSTC) --lib $<
        @touch $@
    
    clean:
        @rm -f *.dummy *.so *.dylib *.dll
    

    While this works, it also has some drawbacks. For example, if you edit a file during a long compile, the libfoo.dummy will get updated after the compile is finished, and rerunning the build won't detect any changes. The timestamp of the input file will be older than the final output file that the build tool is checking. If the build system knew the real output file name, it could compare the correct timestamps, but that information has been locked inside the Rust compiler.

    Dependency Information

    Build systems need to be reliable. When you edit a file, it should trigger the correct things to get rebuilt. If nothing changes, nothing should get rebuilt. It's extremely frustrating if you edit a file, rebuild the library, and find that your code changes aren't reflected in the new output for some reason or that the library is not rebuilt at all. Reliable builds need accurate dependency information in order to accomplish this.

    There's currently no way for external build tools to get dependency information about Rust crates. This means that developers tend to list dependencies by hand which is pretty fragile.

    One quick way to approximate dependency info is just to recursively find every *.rs in the crate's source directory. This can be wrong for multiple reasons; perhaps the include! or include_str! macros are used to pull in files that aren't named *.rs or conditional compilation may omit several files.

    This is similar to dealing with header dependencies by hand when working with C and C++ code. C compilers have options to generate dependency info to deal with this, which used by tools like CMake.

    The price of inaccurate or missing dependency info is an unreliable build and a frustrated developer. If you find yourself reaching for make clean, you're probably suffering from this.

    Making It Better

    It's possible to solve these problems without sacrificing the things we want and falling back to doing exactly what C compilers do. By making the output file knowable and handling dependencies automatically we make make build tool integration easy and the resulting builds reliable. This is exactly what I've been working on the last few weeks.

    Stable and Computable Hashes

    The first thing we need is to make the crate hash stable and easily computable by external tools. Internally, the Rust compiler uses SipHash to compute the crate hash, and takes into account arbitrary link metadata as well as the link metadata of its dependencies. SipHash is not something easily computed from a Makefile and the link metadata is not so easy to slurp and normalize from some dependency graph.

    I've just landed a pull request that replaces the link metadata with a package identifier, which is a crate level attribute called pkgid. You declare it like #[pkgid="github.com/mozilla-servo/rust-geom#0.1"]; at the top of your lib.rs. The first part, github.com/mozilla-servo, is a path, which serves as both a namespace for your crate and a location hint as to where it can be obtained (for use by rustpkg for example). Then comes the crate's name, rust-geom. Following that is the version identifier 0.1. If no pkgid attribute is provided, one is inferred with an empty path, a 0.0 version, and a name based on the name of the input file.

    To generate a crate hash, we take the SHA256 digest of the pkgid attribute. SHA256 is readily available in most languages or on the command line, and the pkgid attribute is very easy to find by running a regular expression over the main input file. The first eight digits of this hash are used for the filename, but the full hash is stored in the crate metadata and used as part of the symbol hashes.

    Since the crate hash no longer depends on the crate's dependencies, it is stable so long as the pkgid attribute doesn't change. This should happen very infrequently, for instance when the library changes versions.

    This makes the crate hash computable by pretty much any build tool you can find, and means rustc generates predictable output filenames for libraries.

    Dependency Management

    I've also got a pull request, which should land soon, to enable rustc to output make-compatible dependency information similar to the -MMD flag of gcc. To use it, you give rustc the --dep-info option and for an input file of lib.rs it will create a lib.d which can be used by make or other tools to learn the true dependencies.

    The lib.d file will look something like this:

    librust-geom-da91df73-0.0.dylib: lib.rs matrix.rs matrix2d.rs point.rs rect.rs side_offsets.rs size.rs
    

    Note that this list of dependencies will include code pulled in via the include! and include_str! macros as well.

    Here's an example of a handwritten Makefile using dependency info. Note that this uses a hard-coded output file name, which works because crate hash is stable unless the pkgid attribute is changed:

    RUSTC ?= rustc
    
    all: librust-geom-851fed20-0.1.dylib
    
    librust-geom-851fed20-0.1.dylib: lib.rs
        @$(RUSTC) --dep-info --lib $<
    
    -include lib.d
    

    Now it will notice when you change any of the .rs files without needed to explicitly list them, and this will get updated as your code changes automatically. A little Makefile abstraction on top of this can make it quite nice and portable.

    Next Up

    In the next few posts, I'll show examples of integrating the improved Rust compiler with some existing build systems like make, CMake, and tup.

    (Update: the next post covers building Rust with Make.)

    by Jack Moffitt (jack@metajack.im) at December 11, 2013 10:31 AM

    November 26, 2013

    The Moon Research Blog

    EDNS(0) and DNSSEC Coming Soon to Twisted

    I'm happy to announce that I've won some funding from The NLnet Foundation - DNS Security Fund, to add EDNS(0)and DNSSEC (and possibly DANE) support to Twisted.

    The current project plancontains the list of milestones and tickets.

    If there are any DNS enthusiasts / experts reading this, I'd appreciate your feedback; on the plan and on the implementation (as it evolves). Please get in touch.

    Tom Prince has kindly agreed to do some "express" code and design reviews for me. That'll be a great help given the current length of the review queue.

    But the more eyes on the code the better, so please consider helping out with code reviews if you can. (I'm happy to trade reviews if you've got your own branch / patch waiting to be reviewed.)

    I've been given a head start in this project by the patches contributed by Bob Novasand Phil Mayers, so thank you both. I hope you'll be able monitor what I'm doing and steer me in the right direction.

    Thanks also to Itamar who encouraged me to apply for the funding and to Tom Prince and everyone who helped me draft the proposal.

    I'll be working on this at the Twisted Sprint in Bristol, UK on December 7th; where I'll be delighted to discuss the project and demonstrate what I've been up to.

    Hope to see you there!

    by Richard Wall (noreply@blogger.com) at November 26, 2013 01:30 PM

    November 08, 2013

    Twisted Matrix Laboratories

    Twisted 13.2.0 Released

    On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 13.2!

    The highlights of this release are:
    • Twisted now includes a HostnameEndpoint implementation which uses IPv4 and IPv6 in parallel, speeding up the connection by using whichever connects first (the 'Happy Eyeballs'/RFC 6555 algorithm). (#4859)
    • Improved support for Cancellable Deferreds by kaizhang, our GSoC student. (#4320, #6532, #6572, #6639)
    • Improved Twisted.Mail documentation by shira, our Outreach Program for Women intern. (#6649, #6652)
    • twistd now waits for the application to start successfully before exiting after daemonization. (#823)
    • SSL server endpoint string descriptions now support the specification of chain certificates. (#6499)
    • Over 70 closed tickets since 13.1.0.

    For more information, check the NEWS file.

    You can find the downloads on PyPi (or alternatively the Twisted Matrix Downloads page).

    Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

    by HawkOwl (noreply@blogger.com) at November 08, 2013 08:18 PM

    October 19, 2013

    Future Foundries

    Announcing Crochet v1.0: Use Twisted Anywhere!

    It's been a busy 6 months since I first released Crochet, and now it's up to v1.0. Along the way I've expanded the documentation quite a bit and moved it to Sphinx, fixed a whole bunch of bug reports from users, added some new APIs and probably introduced some new bugs. What is Crochet, you ask?

    Crochet is an MIT-licensed library that makes it easier for blocking or threaded applications like Flask or Django to use the Twisted networking framework. Crochet provides the following features:

    • Runs Twisted's reactor in a thread it manages.
    • The reactor shuts down automatically when the process' main thread finishes.
    • Hooks up Twisted's log system to the Python standard library loggingframework. Unlike Twisted's built-in logging bridge, this includes support for blocking Handler instances.
    • A blocking API to eventual results (i.e. Deferred instances). This last feature can be used separately, so Crochet is also useful for normal Twisted applications that use threads.
    You can download Crochet at: http://pypi.python.org/pypi/crochet

    Documentation can be found on Read The Docs.

    Bugs and feature requests should be filed at the project Github page.

    by Itamar Turner-Trauring (noreply@blogger.com) at October 19, 2013 03:57 PM

    October 18, 2013

    Twisted Matrix Laboratories

    Announcing code.twistedmatrix.com

    I'm please to announce that there is a new git mirror hosted on twistedmatrix.com infrastructure.

    git clone https://code.twistedmatrix.com/git/Twisted
    or

    https://git.twistedmatrix.com/Twisted
    For those that prefer bzr, the bzr mirror is also available as

    bzr branch https://code.twistedmatrix.com/bzr/Twisted/trunk Twisted
    The build bots are now using these mirrors for checking out code.

    by Tom Prince (noreply@blogger.com) at October 18, 2013 05:11 PM

    October 10, 2013

    Ralph Meijer

    Logitech Ultrathin Touch Mouse

    Logitech is my brand of choice for input devices. Unfortunately, though, Logitech seems to focus on their unifying receiver for most of their stuff, to the detriment of their Bluetooth offering. Every now and then, they do come out with a nice Bluetooth device, usually targetting ultrabooks or tablets. Last month I stumbled upon the new Logitech Ultrathin Touch Mouse (t630). As usual, it is marketed for Windows compatibility, with Linux officially not supported. They do have a second model targetted to Mac users with the t631, but I suspect the only difference is its color.

    Fortunately, this device mostly works fine on my Ubuntu 13.04 laptops. Plural, because this tiny mouse can be set up to pair with two devices, switchable with a switch on the bottom. The only problem is that, out-of-the-box, gnome-bluetooth cannot reconnect with the device when it has been powered down or switched back from the other channel. It turns out that Logitech might not be following standards, and requires repairing every time. In my search for similar cases, I found a bug report for another device that has had similar issues, and the solution presented there also works for the Ultrathin Touch Mouse.

    The trick is to tell gnome-bluetooth to always send the pincode (0000, as usual) upon connecting. For this, it needs an entry in /usr/share/gnome-bluetooth/pin-code-database.xml like this:

    <!-- Logitech Ultrathin Touch Mouse -->
    <device oui="00:1F:20:" name="Ultrathin Touch Mouse" pin="0000"/>

    I filed a bug report to have this included by default. After adding the entry, add the mouse as a new input device and it should work as expected.

    On to the mouse' features. Besides detecting motion with its bottom laser, the surface is a touch pad that can be depressed as a whole. Pressing in the top left and top right corner will trigger the left and right mouse button events (button 1 and 3). To do a middle-click, you have to press in the center of the touch pad, not at the top middle, as you'd expect. Vertical and horizontal scrolling can be done with swipe gestures, respectively up/down and left/right. This will trigger buttons 4 through 7.

    On top of that, there are some additional gestures, which Logitech has pictured in a handy overview. First, there is a two-finger left or right swipe for doing previous and next actions. In X11 this will trigger buttons 8 and 9, and Firefox, for example, will respond to move back and forth in a tab's history. The other three gestures generate keyboard events, instead of the usual mouse events. A double-finger double-tap yields a press and release of the Super_L key. In Unity this brings up the dash home by default. Finally there are swipes from the left edge and from the right edge. The former triggers Ctrl_L Super_L Tab, which switches between the two last used tabs in Firefox, the latter Alt_L Super_L XF86TouchpadOff, which doesn't have a default action bound to it, as far as I can tell. Logitech also mentions the single-finger double tap, but that doesn't seem to register any event in the input handler.

    The mouse can be charged with via its micro-USB connector, also on the bottom, with a convenient short USB cable in the box. The micro-USB connector on that cable is also angled so the mouse doesn't have to be upright when charging. The battery state is reported to the kernel, but there is another bug in upower that will make batteries in bluetooth input devices show up as laptop batteries.

    Having used the mouse for a few days now, I like it a lot. It is really tiny, but not in a bad way (for me). The two-finger swipe gestures are a bit tricky to get right, but I don't really use them anyway. I also tried hooking it up to my Nexus 7, and that works nicely. All-in-all a great little device, especially while travelling.

    by ralphm at October 10, 2013 12:00 PM

    Using ElasticSearch and Logstash to Serve Billions of Searchable Events for Customers

    Kicking off the revival of this publication, I recently did a guest post on our use of Elasticsearch at Mailgun. Since I have joined the Mailgun team at Rackspace in May, my primary project was to reimplement the Mailgun customer logs so that we can serve billions of searchable events. Head over to the HackerNews page for some additional details.

    by ralphm at October 10, 2013 11:47 AM

    September 27, 2013

    Zhang Kai (GSoC)

    Final Report

    Summer is over!

    Well, at least the Google Summer of Code is over. I had a great time working on Twisted. Here is what I do:

    My project, Deferred Cancellation, involves adding cancellation support to Twisted APIs that return a uncancellable Deferred as much as possible. In this summer, I added cancellation support to 8 APIs, including:

    • twisted.mail.smtp.sendmail #6572
    • twisted.mail.pop3client.POP3Client #6588
    • twisted.mail.imap4.IMAP4Client #6613
    • twisted.internet.defer.DeferredList #6639
    • twisted.names.dns.DNSMixin._query #6644
    • twisted.internet.task.LoopingCall.start #6656
    • twisted.web.client.readBody #6686
    • twisted.internet.defer.DeferredFilesystemLock.deferUntilLocked #6720

    When working on the project I found two bugs and fixed them:

    • The “_loopFinshed” in TimerService.volatile should be changed to “_loopFinished” #6657
    • twisted.names.test.test_dns.TestTCPController should override the messageReceived method. #6655

    Also we added timeout implementation to Deferred, based on cancellation support. #5786

    In total, there are 11 tickets, 4 of them have already be merged into trunk. I will keep tracking the rest of the tickets and continue to work on Twisted.

    It’s my first time working on a open source project, and I’ve had a great time. Thanks to all the people who helped me during this summer. I’ve learned so much from this project!

    September 27, 2013 04:46 PM

    September 08, 2013

    Stacey Sern

    Relaying

    My latest Twisted adventure began with a comment I came across in relay.py:

    1
    2
    3
    4
    5
    
    class RelayerMixin:
    
        # XXX - This is -totally- bogus
        # It opens about a -hundred- -billion- files
        # and -leaves- them open!
    

    This seemed like a worthy problem to investigate so that, at the very least, I could write a ticket to track the issue.

    The first challenge was to set up a smart host configuration with Twisted. A smart host is a mail server which accepts mail to any address and then determines the mail exchange for the address and connects to it to relay the mail. Unlike an open relay, a smart host imposes restrictions on the source of messages. While some may accept mail only from authenticated senders, Twisted’s default is to relay any mail received over a Unix socket or from localhost.

    It was easy enough to run a smart host on my development machine. I just had to invoke twistd mail with the relay option and specify a directory to hold messages to be relayed:

    1
    
    twistd -n mail --relay=/tmp/mail_queue
    

    The smart host uses DNS to look up mail exchanges and contacts them via SMTP on port 25. Because my ISP does not allow outgoing traffic on port 25 and because I did not want to relay test messages to real mail servers, I needed to make some changes to the Twisted source so that the email messages would be relayed to a Twisted mail server that I ran on a second computer. I modified relaymanager.py to relay to port 8025 and to use a hosts file for DNS resolution.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    
    class SmartHostSMTPRelayingManager:
        ...
        # PORT = 25
        PORT = 8025
        ...
        def _checkStateMX(self):
            ...
            if self.mxcalc is None:
                # self.mxcalc = MXCalculator()
                from twisted.names.client import createResolver
                resolver = createResolver(None, None, b"/tmp/hosts")
                self.mxcalc = MXCalculator(resolver)
    

    The hosts file maps example.com and example.net to the IP address of the computer running the target mail server.

    1
    2
    
    10.224.77.149 example.com
    10.224.77.149 example.net
    

    I configured that server to run on the default port, 8025, and accept mail for a few users on the domains example.com and example.net:

    1
    2
    
    twistd -n mail -d example.com=/tmp/example.com -u jim=pwd -u nat=pwd
    -d example.net=/tmp/example.net -u joe=pwd -u bob=pwd
    

    When I used telnet on the development machine to send mail to the smart host running on the same machine and addressed it to one of the configured users on example.com or example.net, the smart host relayed it to the mail server on the second machine.

    Now that I had a usable configuration, I wanted to explore the implications of the comment that RelayerMixin opened a large number of files and never closed them. RelayerMixin is used to introduce a set of functions for relaying mail to another class, a relayer, through inheritance. On initialization, the relayer calls one of the RelayerMixin functions, loadMessages, with a list of the pathnames of messages which it is responsible for relaying. loadMessages opens each message file and stores the file object in a list. I hypothesized that if I sent a lot of messages to the smart host at once, its relayers would open files for all the messages and hit the operating system limit for open files.

    I wrote a short program to send the SMTP commands for a series of messages to the smart host running on port 8025 of the same machine. The messages are randomly destined to one of two addresses on each of the two domains served by the mail server on the other machine.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    
    from twisted.internet import protocol, reactor
    from twisted.test.proto_helpers import LineSendingProtocol
    from twisted.internet.defer import Deferred
    from random import randint
    
    NUM_MESSAGES = 250
    
    addresses = ['joe@example.net', 'bob@example.net',
                 'jim@example.com', 'nat@example.com']
    num_addrs = len(addresses) - 1
    
    msgs = ['helo']
    
    for i in range(0, NUM_MESSAGES):
        origin = 'foo@example.com'
        destination = addresses[randint(0, num_addrs)]
        msgs.append('mail from: <{}>'.format(origin))
        msgs.append('rcpt to: <{}>'.format(destination))
        msgs.append('data'),
        msgs.append('from {} to {}'.format(origin, destination)),
        msgs.append('hi {}'.format(destination)),
        msgs.append('.'),
    
    msgs.append('quit')
    client = LineSendingProtocol(msgs)
    
    done = Deferred()
    f = protocol.ClientFactory()
    f.protocol = lambda: client
    f.clientConnectionLost = lambda *args: done.callback(None)
    
    def finished(reason):
        reactor.stop()
    
    done.addCallback(finished)
    
    reactor.connectTCP('127.0.0.1', 8025, f)
    reactor.run()
    

    As I increased the number of messages sent, I expected to eventually see an exception occur when too many files were opened but that did not occur no matter how many messages were sent. From the server log, I observed that instead of opening one connection to the mail server for each domain and sending all the queued messages for that domain, the smart host was repeatedly connecting to the mail server and sending no more than a few messages at a time. That explained why the limit on open files was not being reached. The relayers were being handed only a few messages at a time so there was no need to open a lot of files at once.

    This strategy for allocating work to relayers did not seem very efficient so I started exploring further. SmartHostSMTPRelayingManager, which implements the smart host functionality, has a function, checkState, which is called periodically to see if there are messages waiting to be relayed and if there is capacity to create new relayers. If so, it calls _checkStateMX to create relayers and allocate messages to them. It turns out that _checkStateMX contains a subtle bug which is the cause of the allocation behavior.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    
    def _checkStateMX(self):
        nextMessages = self.queue.getWaiting()
        nextMessages.reverse()
    
        exchanges = {}
        for msg in nextMessages:
            from_, to = self.queue.getEnvelope(msg)
            name, addr = rfc822.parseaddr(to)
            parts = addr.split('@', 1)
            if len(parts) != 2:
                log.err("Illegal message destination: " + to)
                continue
            domain = parts[1]
    
            self.queue.setRelaying(msg)
            exchanges.setdefault(domain, []).append(self.queue.getPath(msg))
            if len(exchanges) >= (self.maxConnections - len(self.managed)):
                break
    

    _checkStateMX asks the relay queue for a list of waiting messages. Then it loops through the messages, grouping them by target domain. Eventually, each group will be handed off to a relayer. The problem is that _checkStateMX breaks out of the loop as soon as it has at least one message for the maximum number of domains it can concurrently contact. That value, maxConnections, is an optional parameter to SmartHostSMTPRelayingManager.__init__. Its default value is 2.

    As _checkStateMX loops through the waiting messages, it creates a list of messages for the first domain it sees and keeps adding messages for that domain to the list. When it sees a second domain, it creates another list for that domain but since it has hit the limit on connections, it breaks out of the loop. So, any other messages in the queue for either domain must wait to be sent even though they could be handled by the same relayers. Instead of breaking out of the loop when it reaches the connection limit, _checkStateMX should continue to add messages to the lists for the domains it has already seen and ignore messages for other domains.

    With the understanding of how messages are allocated to relayers, I was now easily able to trigger an exception for too many open files by sending a large number of messages to one domain instead of splitting them between two.

    As a result of this exploration, I filed and submitted fixes for two issue tickets, a defect ticket for the handling of open files by RelayerMixin, and an enhancement ticket to improve how messages are allocated to relayers.

    September 08, 2013 07:27 PM

    August 30, 2013

    Allen Short

    On Readability

    Programs must be written for people to read, and only incidentally for machines to execute. — Abelson & Sussman, Structure and Interpretation of Computer Programs

    Code readability gets talked about a lot these days. I haven't yet heard from anyone who's opposed to it. Unfortunately, learning to read code is a skill rarely discussed and even more rarely taught. As the SICP quote above points out, readability is perhaps the most important quality to aim for when writing. But what is code readability, exactly?

    I propose there are three essential levels of code readability:

    • "What is this code about?"
    • "What is this code supposed to do?"
    • "What does this code do?"

    The first level is important when you're skimming code, trying to develop a picture of its overall structure. Good organization into modules tends to help a lot with this; if you have modules named utilthen this is harder than it has to be. Supporting docs describing architecture and organization can assist with this as well, along with usage examples if the code you're reading is a library.

    The second level is what you encounter once you've found some code and you want to use or modify it. Maybe it's a library with weak or missing documentation and you're trying to discover how a function wants to be called or which methods to override in a class you need to inherit from. Good style, good class/function names, docstrings, and comments can all be very helpful in making your code readable for this case. There's been some research which associates poor quality identifier names and obvious bugs, so even if your code works well, poor naming can make it look like code that doesn't.

    However, neither of these are the most important sense in which readability matters. They're more about communicating intent, so that later readers of your code can figure out what you were thinking. Sometimes it's a way to share hard-won insight that leaves indelible scars. But the final level is what matters most and matters longest.

    To understand a program you must become both the machine and the program. — Alan Perlis, Epigrams In Programming #23

    The most important level of readability is being able to look at code and understand what it actually does when run. All the factors discussed above — naming, style, documentation — cannot help with this task at all. In fact, they can be actively harmful to this. Even if the comment accurately described the code when it was written, there's no reason it has to now. The most obvious time you'll need to engage in this level of code reading is debugging — when the code looks like it does the right thing, but actually doesn't.

    Language design plays a big role in supporting or detracting from the creation of readable code. As the link above shows, C provides a myriad of features that either fight against or outright destroy readability. So when picking a language, include readability as a factor in your decision making — and not just how the syntax looks. Mark Miller provides some excellent thoughts on how to design a language for readability in his notes on The Power of Irrelevance. There are also several people studying how to build tools to help us read code more effectively; Clarity In Code touches on some of the issues being considered in that field.

    But given the constraints of the language you're currently using, what can we do to improve readability in our code? The key is promoting local reasoning. The absolute worst case for readability occurs when you have to understand all the code to understand any of the code. So we want to preserve as many barriers between portions of the program as are needed to prevent this.

    Since local reasoning is good, global state is bad. The more code that can affect the behavior of the function or class you're looking at right now, the more work it takes to actually discern what it'll do, and when. Similarly, threads (and coroutines) destroy readability. since they destroy the ability to understand control flow locally in a single piece of code. Other forms of "magic" like call stack inspection, adding behavior to a class from other modules, metaclass shenanigans, preprocessor hacks, or macros all detract from local reasoning as well.

    Since this perspective on programming is so little discussed or taught, it leads to a communications gap between inexperienced and veteran programmers. Once an experienced programmer has spent enough time picking apart badly designed code, it can become the dominant factor in his assessment of all the code he sees. I've experienced this myself fairly often: code that could be easily made a little more readable makes me itch; code that can't easily be fixed that was written with no thought for readability can be quite upsetting. But writing readable code is extra work, so people who haven't spent dozens of hours staring at a debugger prompt are sometimes baffled by the strong emotions these situations inspire. Why is it such a big deal to make that variable global? It works, after all.

    Every program has at least one bug and can be shortened by at least one instruction — from which, by induction, it is evident that every program can be reduced to one instruction that does not work. — Ken Arnold

    When considering how to write readable code, choice of audience matters a lot. Who's going to read what you're writing? When? For writing prose, we do this all the time. We use quite different style in a chat message or email than in a blog post, and a different style again in a formal letter or article. The wider your audience and the longer the duration you expect the message to be relevant, the more work you put into style, clarity, and readability. The same applies to code. Are you writing a one-off script that's only a few dozen lines long? Using global variables and one-letter identifiers is probably not going to hurt you, because you will most likely delete the code rather than read it again. Writing a library for use in more than one program? Be very careful about using any global state at all. (Also pay special attention to the names you give your classes/modules/functions; they may be very difficult to change later.) If new programmers were taught this idea as well as they're taught how to, e.g., override methods in a class or invoke standard libraries, it would be a lot easier for both new programmers and those mentoring them to relax.

    So when you do set out to write readable code, consider your audience. There are some obvious parties to consider. If you're writing a library, your users will certainly read some of your code, even if it's just examples of how to use your library. Anyone who wants to modify your code later will need to read it. If your code is packaged by an OS distribution or other software collection, the packager will often need to read parts of your code to see how it interacts with other elements of the system. Security auditors will want to read your code to know how it handles the authority it's granted or the secrets it protects. And not least, you're writing for your future self! So even if those other people don't exist or don't matter to you — make life easier on that last guy. He'll thank you for it.

    by Allen Short (noreply@blogger.com) at August 30, 2013 07:08 PM

    August 27, 2013

    Zhang Kai (GSoC)

    Weekly Report

    Summary

    I have merged branches/documenta-4320 into trunk. Now the howto page of deferred has documentation about cancellation.

    I have merged branches/timerservice-typo-6657 into trunk. Now the twisted.application.internet.TimerService can be pickled correctly.

    I have merged branches/deferredlist-cancellation-6639 into trunk. Now twisted.internet.defer.DeferredList has cancel() method.

    I have finished the patch for ticket #6656(twisted.internet.task.LoopingCall.start should return a cancellable Deferred) and submitted it for review.

    I have revised the patch for ticket #5786(Add timeout implementation to Deferred, based on cancellation support) according to the comments and submitted it for review.

    Plan for the next week

    In the next week, I will write an email to solicit opinions on raising exception from cancel(). Also, I will revised my patches according to the comments.

    August 27, 2013 03:20 PM

    August 21, 2013

    Jack Moffitt

    Seven Web Frameworks in Seven Weeks in Beta

    I'm happy to announce that my new book Seven Web Frameworks in Seven Weeks: Adventures in Better Web Apps is now available in beta from Pragmatic Programmers. My co-author Fred Daoud and I cover a wide variety of frameworks in different styles and languages. The book covers Sinatra (Ruby), CanJS (JavaScript), AngularJS (JavaScript), Ring (Clojure), Webmachine (Erlang), Yesod (Haskell), and Immutant (Clojure). This first beta contains the first five of those.

    I hope you enjoy it!

    by Jack Moffitt (jack@metajack.im) at August 21, 2013 10:06 AM

    August 15, 2013

    Stacey Sern

    Unit Tests

    One of the first Twisted mail issue tickets I tackled involved what seemed to be the simple matter of adding missing unit tests but led to some refactoring of Twisted code. The ticket highlights the missing unit tests for the startMessage and exists functions of both the AbstractMaildirDomain and DomainQueuer classes.

    AbstractMaildirDomain is a base class which is meant to be subclassed to create mail domains where the emails are stored in the Maildir format. Most of the functions it provides are placeholders that need to be overridden in subclasses. However, it does provide implementations for two functions, exists and startMessage. exists checks whether a user exists in the domain or an alias of it and, if so, returns a callable which returns a MaildirMessage to store a message for the user. Otherwise, it raises an SMTPBadRcpt exception. startMessage returns a MaildirMessage which stores a message for the user.

    The existing unit tests for AbstractMaildirDomain were minimal, checking just that it fully implemented the IAliasableDomain interface. Additional test cases were needed to verify that:

    • startMessage returns a MaildirMessage
    • for a valid user, exists returns a callable which returns a MaildirMessage and that the callable returns distinct messages when called multiple times
    • for an invalid user, exists raises SMTPBadRcpt

    AbstractMaildirDomain could not be used directly in testing exists and startMessage because both call a function which has only a placeholder in the base class. So for testing purposes, I created a subclass of AbstractMaildirDomain, TestMaildirDomain, which overrides the placeholder functions.

    Since each test would need a TestMaildirDomain to exercise, I wrote a setUp function which runs prior to the test and creates a TestMaildirDomain as well as a temporary Maildir directory for it to use. I also added a tearDown function which runs after each test to remove the temporary directory.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    
    class AbstractMaildirDomainTestCase(unittest.TestCase):
    
        def setUp(self):
            self.root = self.mktemp()
            self.service = mail.mail.MailService()
            self.domain = TestMaildirDomain(self.service, self.root)
            self.domain.addUser('user',"")
    
        def tearDown(self):
            shutil.rmtree(self.root)
    

    The tests for startMessage and exists turned out to be pretty straightforward:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    
    def test_startMessage(self):
        msg = self.domain.startMessage('user@')
        self.assertTrue(isinstance(msg, maildir.MaildirMessage))
    
    def test_exists(self):
        f = self.domain.exists(mail.smtp.User("user@", None, None, None))
        self.assertTrue(callable(f))
        msg1 = f()
        self.assertTrue(isinstance(msg1, mail.maildir.MaildirMessage))
        msg2 = f()
        self.assertTrue(msg1 != msg2)
        self.assertTrue(msg1.finalName != msg2.finalName)
    
    def test_doesntexist(self):
        self.assertRaises(mail.smtp.SMTPBadRcpt, self.domain.exists,
                mail.smtp.User("nonexistentuser@", None, None, None))
    

    Like AbstractMaildirDomain, DomainQueuer acts as a domain, but instead of saving emails for users, it puts messages in a queue for relaying. Its exists and startMessage functions are meant to be used in the same way as those of AbstractMaildirDomain. It seemed like it would be an easy matter to adapt the unit tests for AbstractMaildirDomain to work for DomainQueuer.

    It turned out, however, that the test for exists on DomainQueuer failed because a function was being called on a variable set to None. Something clearly hadn’t been properly configured for the test.

    Here’s the DomainQueuer code being exercised in the test:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    
    class DomainQueuer:
    
        def exists(self, user):
            if self.willRelay(user.dest, user.protocol):
                ...
            raise smtp.SMTPBadRcpt(user)
    
        def willRelay(self, address, protocol):
            peer = protocol.transport.getPeer()
            return (self.authed or isinstance(peer, UNIXAddress) or
                peer.host == '127.0.0.1')
    

    exists calls willRelay with the protocol which was passed into it as part of the user argument. willRelay tries to call getPeer on the transport instance variable of the protocol. But, the minimal User object passed to exists, which worked for the AbstractMaildirDomain test, is not sufficient for the DomainQueuer test. willRelay needs the protocol to determine whether it should relay the message.

    I had two thoughts about how to get around this problem, but I wasn’t sure either of them was satisfactory. One option would be to provide the User object with a fake protocol and fake transport so that willRelay could be configured to return the desired value. That seemed to be a very roundabout way to solve the problem of getting willRelay to return a specific boolean value.

    A more direct way to get willRelay to return the desired value would be to monkeypatch it. That is, as part of the unit test, to replace the DomainQueuer.willRelay function with one that returns the desired value. The problem with monkeypatching is that, even though willRelay is part of the public interface of DomainQueuer, it could change in the future. For example, it could received additional optional arguments. Then, the unit test which used the monkeypatched version might not reflect the behavior of the real version of willRelay.

    I got some great advice about how to approach this problem and unit testing in general on the Twisted developers IRC channel. The idea behind unit testing is to test each unit of code independently. Here we can consider exists and willRelay to be two different units. But the way willRelay is implemented means that it is difficult to test exists independent of it. I was advised that sometimes the best thing to do when adding unit tests is to refactor the code itself. So, I attempted to do that without changing the public interface of DomainQueuer.

    I introduced a base class, AbstractRelayRules, whose purpose is to encapsulate the rules for determining whether a message should be relayed. Then, I defined a subclass, DomainQueuerRelayRules, which contains the default rules for the DomainQueuer class.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    
    class AbstractRelayRules(object):
        """
        A base class for relay rules which determine whether a message
        should be relayed.
        """
    
        def willRelay(self, address, protocol, authorized):
            """
            Determine whether a message should be relayed.
            """
            return False
    
    
    class DomainQueuerRelayRules(object):
        """
        The default relay rules for a DomainQueuer.
        """
    
        def willRelay(self, address, protocol, authorized):
            """
            Determine whether a message should be relayed.
    
            Relay for all messages received over UNIX sockets or from 
            localhost or when the message originator has been authenticated.
            """
            peer = protocol.transport.getPeer()
            return (authorized or isinstance(peer, UNIXAddress) or
                peer.host == '127.0.0.1')
    

    In DomainQueuer, I changed the __init__ function to take a new optional parameter specifying the rules to be used to determine whether to relay. When relayRules is not provided, the default DomainQueuerRelayRules is used. I also changed the willRelay function to ask the relayRules whether to relay instead of determining that itself. Existing code which creates a DomainQueuer without providing the extra argument works in the exact same way as the old code.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    
    class DomainQueuer:
    
        def __init__(self, service, authenticated=False, relayRules=None):
            ...
            self.relayRules = relayRules
            if not self.relayRules:
                self.relayRules = DomainQueuerRelayRules()
    
    
        def willRelay(self, address, protocol):
            """
            Determine whether a message should be relayed according
            to the relay rules.
            """
            return self.relayRules.willRelay(address, protocol, self.authed)
    

    Finally, I created another subclass of AbstractRelayRules to be used for test purposes. TestRelayRules can be configured with a boolean value which determines whether relaying is allowed or not.

    1
    2
    3
    4
    5
    6
    7
    
    class TestRelayRules(mail.relay.AbstractRelayRules):
    
        def __init__(self, relay):
            self.relay = relay
    
        def willRelay(self, address, protocol, authorized):
            return self.relay
    

    Now, the unit tests for DomainQueuer.exists using TestRelayRules are quite simple.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    
    class DomainQueuerTestCase(unittest.TestCase):
    
        def test_exists(self):
            dq = mail.relay.DomainQueuer(self.service, False, TestRelayRules(True))
            f = dq.exists(mail.smtp.User("user@", None, None, None))
            self.assertTrue(callable(f))
            msg1 = f()
            self.assertTrue(isinstance(msg1, mail.mail.FileMessage))
            msg2 = f()
            self.assertTrue(msg1 != msg2)
            self.assertTrue(msg1.finalName != msg2.finalName)
    
        def test_doesntexist(self):
            dq = mail.relay.DomainQueuer(self.service, False, TestRelayRules(False))
            self.assertRaises(mail.smtp.SMTPBadRcpt, dq.exists,
                    mail.smtp.User("user@", None, None, None))
    

    Refactoring to decouple the rules for relaying messages from the DomainQueuer certainly made the unit testing code much cleaner. As a general matter, difficulty in writing unit tests may highlight dependencies in the code which should be refactored.

    August 15, 2013 11:04 AM

    August 12, 2013

    Zhang Kai (GSoC)

    Weekly Report

    In the last week, I revised my patch for ticket 6657(The “_loopFinshed” in TimerService.volatile should be changed to “_loopFinished”). Now instead of testing the implementation detail, we test that TimerService is pickleable by testing the pickle result.

    I also revised my patch for ticket 6656(twisted.internet.task.LoopingCall.start should return a cancellable Deferred). I added a utility method to LoopingCall so that cancel(), stop() and reset() can share the code. Then I added a test for cancellation after stop() is called.

    After revised my patches, I looked at ticket 4632(ability to cascade canceling inlineCallbacks’s deferred). inlineCallbacks helps one write Deferred-using code that looks like a regular sequential function. When one call anything that results in a Deferred, one can simply yield it, the generator will automatically be resumed when the Deferred's result is available. Ticket 4632 is aimed to add the ability to cascade canceling inlineCallbacks's Deferred to inlineCallbacks. The work is almost done by tracktor and glyph. Some docstring are needed, and the code need to be revised to meet the code standard.

    I also began to work on ticket 4320(Deferred cancellation documentation). I have added some documentation about how to use Deferred cancellation from itamar’s blog posts to the howto doucmentation. Because there are already some documentation in the branche, so I need to do some adjust before I submit it for review.

    Plan for the next week

    In the next week, I will continue revise my patches according to the comments and merge the branches that have passed review to the trunk. Also I will try to find a new ticket to work on.

    August 12, 2013 12:35 PM

    August 05, 2013

    Zhang Kai (GSoC)

    Weekly Report

    In the last week, I submitted my patch for ticket 6644. When working on it, I found a bug of twisted.names.test.test_dns.TestTCPController`.

    twisted.names.test.test_dns.TestTCPController pretends to be a DNS query processor for a twisted.names.dns.DNSProtocol. It inherits from twisted.names.test.test_dns.TestController, a testing DNS query processor for a twisted.names.dns.DNSDatagramProtocol. However the messageReceived method for a DNSProtocol is different from the method for a DNSDatagramProtocol. There should be no addr parameter in the messageReceived method for a DNSProtocol. So TestTCPController should override the messageReceived method.

    I have written a patch for it and submitted the patch for review(ticket 6655).

    I also finished my patch for adding cancellation support for the Deferred returned by twisted.internet.task.LoopingCall.start(ticket 6656). Calling twisted.internet.task.LoopingCall.start will start running a target function every interval seconds. It will return a A Deferred whose callback will be invoked with self when self.stop is called, or whose errback will be invoked when the function raises an exception or returned a Deferred that has its errback invoked[1]. If a call of the target function hasn’t returned yet when cancelling the Deferred, we should cancel the running call. If a call of the target function is scheduled when cancelling the Deferred, we should cancel the scheduled call. The running flag should be set to False.

    The patch is finished. However, due to a bug of twisted.application.internet.TimerService, the new patch won’t pass the test. More specifically, the twisted.test.test_application.TestInternet2.testPickledTimer will fail.

    The bug is due to a typo. The “_loopFinshed” in TimerService.volatile should be “_loopFinished”. So when pickling a TimerService it will actually try to pickle _loopFinished(the Deferred returned by LoopingCall.start), which it shouldn’t. In the patch for ticket 6656, a reference to LoopingCall is added to the Deferred returned by LoopingCall.start. So when trying to pickle _loopFinished, it will try to pickle an instance of LoopingCall, which is unpickleable. This makes the test fail.

    I’ve submitted a patch to fix this bug(ticket 6657).

    Plan for the next week

    In the next week, I will revise all my patches according to the comments. I will also look at ticket 4632 and make it ready for review.

    [1] https://twistedmatrix.com/trac/browser/trunk/twisted/internet/task.py#L146

    August 05, 2013 11:15 AM

    August 01, 2013

    Stacey Sern

    Types

    Much of the work of documenting the Twisted Mail API has involved searching through the Python code to determine the types for parameters and return values. It often involves comparing functions in different classes which inherit from the same base class or implement the same interface. In some cases, I’ve resorted to looking at unit tests or example code to see how objects are used. After a recent experience while tracking down types, I’m more convinced than ever of the value of the API documentation.

    I was documenting the alias module, which contains classes for redirecting mail from one user to another user, to a file, to a process, and to a group of aliases. Four different classes inherit from the base class AliasBase and implement the interface IAlias, which contains the function createMessageReceiver. The class hierarchy looks like this:

    twisted.mail.alias.AliasBase
        twisted.mail.alias.AddressAlias
        twisted.mail.alias.AliasGroup
        twisted.mail.alias.FileAlias
        twisted.mail.alias.ProcessAlias
    

    I was trying to determine the return value of IAlias.createMessageReceiver. The return value was clear for three of the four classes that implement IAlias because the object to be returned was created in the return statement.

    FileAlias -> FileWrapper 
    ProcessAlias -> MessageWrapper   
    AliasGroup -> MultiWrapper 
    

    The objects returned are all message receivers which implement the smtp.IMessage interface. They deliver a message to the appropriate place: a file, a process or a group of message receivers. It seemed pretty clear that the return value of the createMessageReceiver function in the IAlias interface should be smtp.IMessage. However, there was one more class that implemented the interface, AddressAlias, and the return value from that wasn’t so clear.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    
    class AddressAlias(AliasBase):
        implements(IAlias)
    
        def __init__(self, alias, *args):
            AliasBase.__init__(self, *args)
            self.alias = smtp.Address(alias)
    
        def createMessageReceiver(self):
            return self.domain().exists(str(self.alias))
    

    AddressAlias.createMessageReceiver returns the result of a call to exists on the result of a call to domain. domain is a base class function which returns an object which implements the IDomain interface. Fortunately, the IDomain interface was documented. It returns a callable which takes no arguments and returns an object implementing IMessage. Unfortunately, this return value didn’t match the pattern of the other three classes implementing IAlias.createMessageReceiver, all of which return an object implementing IMessage.

    Although messy, it was possible that the return value of IAlias.createMessageReceiver was either an smtp.IMessage provider or a callable which takes no arguments and returns an smtp.IMessage provider. Or, it might have been a mistake.

    At this point, I fortuitously happened to be looking at this code in an old branch and noticed a difference. There, the AddressAlias.createMessageReceiver function appeared as follows:

    1
    2
    
    def createMessageReceiver(self):
        return self.domain().startMessage(str(self.alias))
    

    After some investigation, I found a ticket that had been fixed earlier this year to remove calls to the deprecated IDomain.startMessage function. In the old code, startMessage also returns an IMessage provider. So, it seemed that a bug had been introduced in the switch from startMessage to exists.

    The result of the call to exists must be invoked to get the proper message receiver to return. The code should read:

    1
    2
    
    def createMessageReceiver(self):
        return self.domain().exists(User(self.alias), None, None, None)()
    

    I filed a ticket in the issue tracking system and subsequently submitted a fix. While reworking the unit tests, I relied heavily on the API documentation I had written for the alias module. I think it’s safe to say that had the API been fully documented when the original change was made, this error would have been easy to spot during code review or to avoid in the first place.

    August 01, 2013 06:42 PM

    July 30, 2013

    Thomas Vander Stichele

    morituri 0.2.2 “my bad” released

    The 0.2.1 release contained a bug causing “rip offset” find to fail. That’s annoying for new users, so I spent some time repenting in brown paper bag hell, and fixed a few other bugs besides. Hence, my bad.

    I can understand that you didn’t all mass-flattr the 0.2.2 release – you tried it and you saw the bug! Shame on me.

    Well, it’s fixed now, so feel free to pour in your flattr love if you use morituri! Just follow this post to my blog and hit the button.

    The 0.2.2 packages are in the Fedora 17-18-19 repositories. Enjoy!

    flattr this!

    by Thomas at July 30, 2013 09:23 PM

    July 29, 2013

    Zhang Kai (GSoC)

    Weekly Report

    In the last week, I began to revise my patch for ticket 6639(Add cancellation support to twisted.internet.defer.DeferredList).

    1. The deferredList is private variable now.
    2. The DeferredList.__init__() will only iterate over deferredList once. Since the deferredList could be a generator or some other single-use iterator, iterating over deferredList might cause problems.
    3. The docstring about errback after cancellation is removed now. We are not sure about the errback. All of the passed-in Deferreds can callback instead of errback on cancellation.
    4. The test_cancelDeferredListFiredWithNonFailure and test_cancelDeferredListFiredWithFailure is removed now. When all the Deferreds in the list have already fired, there is no observable difference between calling the cancel method for the Deferreds and not. When a Deferred has fired, the cancel method itself will do nothing.

    In the last week, I also began to add cancellation support to twisted.names.dns.DNSMixin._query(ticket 6644).

    The DNSMixin is a DNS protocol mixin shared by UDP and TCP implementations. The _query method sends out a message with the given queries and give up when timeout.[1]

    When a query is sent by DNSMixin._query, a cancelCall is created to cancel the query when timeout. The resultDeferred and the cancelCall is then saved in DNSMixin.liveMessages with the query id as the key for later processing. To cancel a query, we need to remove the resultDeferred and cancelCall from DNSMixin.liveMessages and cancel the cancelCall.

    Also, if the queried message arrived after the cancellation, we need to make sure the message is safely dropped on the floor. This is ensured by the implementation of DNSDatagramProtocol and DNSProtocol.

    Plan for the next week

    In the next week, I will begin to add cancellation support to twisted.internet.task.LoopingCall.start. I will also look at ticket 4632(ability to cascade canceling inlineCallbacks’s deferred) and make it ready for review.

    [1] https://twistedmatrix.com/trac/browser/trunk/twisted/names/dns.py#L1844

    July 29, 2013 06:20 AM

    July 25, 2013

    Ashwini Oruganti

    The Reactor Part 2: Because It Reacts

    “I’ve seen the past, I’ve seen the future
    Beyond dimension and into empty space
    Finding questions, never answers
    living time behind another face”

    Porcupine Tree It Will Rain for a Million Years

    Once Glyph had asked me “Do you know why the reactor is called a reactor?”
    I could only think of one very simple (too simple, perhaps) answer and concluded that maybe I’m just not wise enough to come up with something more insightful. But as it turned out, it was as ingenuous as I’d thought: The reactor is called so… because it reacts.

    It runs in the background and listens for events. When an event takes place, the reactor performs the action(s) it has been asked to. Since the events involve I/O, the reactor uses the select system call to wait for I/O. The events that the reactor knows about are network, file system, and timer events, and it abstracts away platform-specific behavior giving us interfaces that make it easier to respond to these events.

    The Imported Stuff

    But then, people often have notions like: “I’ll use Twisted when I have something complicated; this is simple, so I’ll just use import socket”. That, I’m afraid, is amongst the very first mistakes ever made - it’s nothing but a wrong turn, as we discussed in Part 1. Telling Twisted what you want your code to do is more reliable (and easier!) than trying to implement the low-level system calls yourself. So when it’s said “you wouldn’t have to worry about low-level interfaces” what’s actually meant is that you shouldn’t (read “aren’t supposed to”) be dealing with the low-level APIs directly. Twisted’s implementation takes care of using the underlying non-blocking APIs correctly and handling obscure edge cases, so use it and don’t import socket your code.

    Talking to Twisted

    Once it has been decided to use Twisted, the next obstruction is usually “How do I get this thing to do anything”? The answer to that is pretty simple: you ask it to.

    By itself, the reactor (and even Twisted, for that matter) does nothing. If you ask it to listen for events, it will obediently do so. If you, however, do not instruct what to do when an event occurs, it will, very faithfully, do nothing. That being clear, the next step should probably be about conveying our instructions to the reactor. Twisted uses a Deferred as a means of “talking” to the event-loop. You want to tell Twisted what to do when an event occurs? And what do to when another event occurs after that first event? How do you even keep track of all these events that haven’t occurred yet? You do that with callbacks.

    As it has been said, time and again, “If you’re used to writing code that runs in a straight line, and only calls functions which return immediately with results, the idea of waiting for something to call you back might be confusing. But there’s nothing magical, no ‘voodoo’ about callbacks.” In fact that confusion might drive you to come up with flawed assumptions and conclusions in an attempt to make sense out of things. For example, “Deferreds run in the reactor”. I’d say Deferreds are not running anywhere. A Deferred is just a generic data structure that helps you manage callbacks, and a callback is just you instructing the reactor what you wish to have happen when a certain event (whose occurrence it’s tracking) takes place. The reactor does not depend on Deferreds in any way, and the reverse is true as well; you can take Deferreds out of Twisted and still use it - there’s nothing special about them.

    Reactor and Threads

    Here’s a thing about Twisted and threads: “You can only ever have a single reactor, in a single thread, in any given Twisted process. If you try to have more, nothing will work.”

    The whole point is, you shouldn’t have to. The reactor can be made to look out for various different events in the same thread (yes, everything in one thread), after all, that’s what an “event-driven” and “asynchronous” framework should be able to do, right?

    For example, if you want to listen on 3 different ports with 3 different protocols:

    1
    2
    3
    4
    5
    
    from twisted.internet import reactor
    reactor.listenTCP(4321, FirstProtocolFactory())
    reactor.listenTCP(5432, SecondProtocolFactory())
    reactor.listenTCP(6543, ThirdProtocolFactory())
    reactor.run()

    Also, most Twisted APIs are not thread-safe, but if you really, really want to run the reactor in a thread, there’s nothing stopping you:

    1
    2
    3
    4
    
    from twisted.internet import reactor
    from threading import Thread
    
    Thread(target=reactor.run, args=(False,)).start()

    Note that I’m explicitly saying this isn’t a very good idea unless you have a very good reason to do this, and now all the Twisted APIs can only be used in this thread, and . I showed you this example to only prove that you can use Twisted any way you want; there’s nothing magical or special about it and it does not restrict your options - for that matter, it actually gives you a wider array of (better!) choices.

    Starting the Engine, Stopping the Engine

    So let’s say you have no other choice and you simply must use threads in your Twisted code. In such cases, it is important to understand that the reactor runs in the main thread, and that if you want to invoke a Twisted API from any other thread, you should use reactor.callFromThread, which sends an object to call to the reactor thread. Also, if you are writing a blocking program, that doesn’t mean you should try to start and stop the reactor for every request. And I quote, “…reactor.run() should be called only once in your whole program. Don’t think of it as ‘start this one request I have’, think of it as ‘start all of Twisted’.” The same goes for reactor.stop() - identify an exit point, and call it there.

    Many or One; Which One?

    In the next part, we will talk about the reactor being a singleton, evaluate the “global” reactor, discuss the mess that installing reactors can be, and see how best to write unit tests for code that uses reactors.

    July 25, 2013 01:31 PM

    July 22, 2013

    Zhang Kai (GSoC)

    Weekly Report

    In the past week, I revised my patch for ticket 6588(adding cancellation support to POP3Client) and submitted it for review. I also revised my patch for ticket 6613(adding cancellation support to IMAP4Client) and submitted it for review.

    In the last meeting, we talked about adding cancellation support to twisted.internet.defer.DeferredList.

    DeferredList is a tool for collecting the results of several Deferreds. It tracks a list of Deferreds for their results, and makes a single callback when they have all completed(of if the fireOnOneCallback/fireOnOneErrback flag is set, the DeferredList will fire when the first Deferred in the list fires with a non-failure/failure result)[1].

    Cancelling a DeferredList means cancelling every Deferred in the list. Basically, what we need to do is to call the cancel method for each Deferred in the list. There are several things need to be noticed.

    The original implementation of DeferredList doesn’t save the list of the Deferreds. It gathers the results of the Deferreds by adding callback/errback to the Deferreds. To call the cancel method for each Deferred we need to store the list in DeferredList. We need to copy the list instead of just referring to it since the list may be changed later.

    We need to make sure that we call the cancel method for every Deferred. Because the cancel method may raise exceptions, we need to catch the exceptions, log about it, then call the cancel method for the next Deferred.

    If the DeferredList has already fired, we should do nothing when cancelling the DeferredList. The special case is when the fireOnOneCallback/fireOnOneErrback flag is set. In this case, the DeferredList will fire when one Deferred in the list fires. Even in this case there are other Deferreds that haven’t fired yet, we still shouldn’t cancel them since the DeferredList itself has already fired.

    Also, the twisted.internet.defer.gatherResults uses DeferredList to return a list with the results of the given Deferreds[2]. So adding cancellation support to DeferredList will also add cancellation support to gatherResults. We need to write tests for that too.

    Plan for the next week

    In the next week, I will finish the patch for adding cancellation support to DeferredList.

    [1] http://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py#L743 [2] http://twistedmatrix.com/trac/browser/trunk/twisted/internet/defer.py#L853

    July 22, 2013 05:15 AM