Planet Twisted

I recently came across an excellent tool called KeyCombiner that helps you practice keyboard shortcuts (3 sets for free, $29/6 months for more sets). I spent some time to create a set for Amazing Marvin, my current todo manager of choice.

The shareable URL to use in KeyCombiner is https://keycombiner.com/collecting/collections/shared/f1f78977-0920-4888-a86d-d00a7201502e

I generated it from the printed PDF version of Marvin’s keyboard guide and a bunch of manual editing, in a google sheet.

Keyboard shortcuts are great timesavers and help reduce friction, but it’s getting harder to learn them properly, and this tool has been a great help for some other apps, and for figuring out common shortcuts across apps, and for picking custom shortcuts (in other apps) that don’t conflict. If this is a problem you recognize, give KeyCombiner a try.

Twitter is horrible. Everyone already knows this. ^{1 2 3}

But, Twitter can also be good, sometimes, after a fashion.

Throughout the pandemic, I have personally found Twitter to be a helpful tool for self-regulation. The little hits of dopamine on demand throughout the day have allowed me to suppress and modulate some truly unpleasant intrusive thoughts, during times when I have had neither the executive function nor sufficient continuous uninterrupted time allocated to focus on other, more useful things. Twitter has allowed me to anesthetize the internal doom-sayer during the absolutely most mind-shatteringly stressful period of my — and, presumably, most living humans’ — entire life.

Like any anesthetic, however, there comes a point where administering additional doses is more harmful than beneficial, even if the pain it’s suppressing is still there. It’s time for me to take a break, and it seems like it would be wise to take one long enough for new habits to form.

To that end, I’ll be taking the entirety of June off from Twitter; depending on how that goes, I might see you back there on 2021-07-01, or, should I find the fortitude in the meanwhile, never.

The “I’m taking a break from social media” genre of post is certainly a bit self-indulgent⁴, so it behooves me to say why I’m bothering to post about this rather than just, you know, doing it.

There are three reasons:

1. Changing times: I’m naturally distractable so I tend to keep an eye on my social media usage. I periodically look at how much time I’m spending, the benefits I’m getting, and the problems it’s causing. For most of the pandemic I could point to at least one or two useful actions per week that I’d taken because of something I’d learned on Twitter. Sometimes I’d learn about risk modeling or health precautions, emerging understanding of impacts of isolation on mental health, and ways to participate to address the exhausting, non-stop political upheaval of 2020/2021. But now I’m mostly just agonizing over the lack of any useful guidance for parents with young children who cannot yet get vaccinated for COVID-19 at this late stage of the crisis, and getting directionlessly more angry about the state of the world. The benefits have slowly evaporated over the last few weeks but the costs remain.⁵

2. Accountability: simply deleting the app, logging out of the website, etc, is clearly not enough to stay away, so an audience who can notice me posting and say “stop posting” should hopefully be enough to keep me honest. Please do note that I will still be allowing certain automated systems to post on my behalf, though. This post, for example, and any other posts I put on my blog, will show up in my Twitter feed automatically, I don’t post those manually.

3. A gentle prompt for others: maybe you’re having similar issues. Maybe you’d like to join me. During the pandemic I’ve found that many types of unpleasant mental states that I’ve described are more relatable than usual. Some so much so that they’ve got whole articles about jargon to describe them, like “disenfranchised stress”⁶ and “vicarious trauma”⁷. Feel free to ignore this: I’m not saying you should join me. Just that if you’ve already been thinking you should, you can take this as a challenge to do the same.

In the meanwhile, I’ll try to do some longer-form writing, particularly writing that isn’t about social media.

If you’d like to get in touch, I won’t be replying to DMs, so feel free to send me an email directly. If you want to interact in real time, I am still on IRC, as glyph on irc.libera.chat. Feel free to drop by #glyph and say hi.

It is possible to work with Python quite a bit and not be aware of some of the subtler details of package management. Since Python is a popular “glue” language, one of its core strengths is integrating with libraries written in other languages: from database drivers written in C, numerical algorithms written in Fortran, to cryptographic algorithms written in Rust. In all these cases, one way to avoid error-prone and frustrating installation errors in the target environment is to distribute pre-built code. However, while source code can be made portable, making the build output portable is a lot more complicated.

Note: This post focuses specifically about binary wheels on Linux. Binary wheels exist for other platforms, but those are beyond the current scope.

The Python manylinux project, composed of three PEPs, two software repositories, and support in pip, addresses how to accomplish that. These problems are hard, and few other ecosystems solve them as well as Python. The solution has many moving parts, developed over the course of ten years. Unfortunately, this means that understanding all of those is not easy.

While this post cannot make it easy, it can at least make it easier, by making sure all the details are in one place.

Some of you read my previous post on typing.Protocols and probably wondered: “what about zope.interface?” I’ve advocated strongly for it in the past — but now that we have Mypy and Protocols, is it simply a relic of an earlier time? Can we entirely replace it with Protocol?

Let’s have a look.

Typing in 2 dimensions

In the previous post I discussed structural versus nominal typing. In Mypy’s type system, most classes are checked nominally whereas Protocol is checked structurally. However, there’s another way that Protocol is distinct from a normal class: normal classes are concrete types, and Protocols are abstract.

Abstract types:

1. cannot be instantiated: every instance of an abstract type is an instance of some concrete sub-type, and
2. do not include (complete) implementation logic.

Concrete types:

1. can be instantiated: they are complete descriptions of a type, and
2. must include all their own implementation logic.

Protocols and Interfaces are both abstract, but Interfaces are nominal. The highest level distinction between the two is that when you have a problem that requires an abstract type, but nominal checking is preferable to structural, Interfaces are a better solution.

Python’s built-in Abstract Base Classes are technically abstract-and-nominal as well, but they’re in a strange halfway space; they’re formally “abstract” because they can’t be instantiated, but they’re partially concrete in that they can contain any amount of implementation logic themselves, and thereby making an object which is a subtype of multiple ABCs drags in all the usual problems of the conflicting namespaces within multiple inheritance.

Theoretically, there’s a way to treat ABCs as purely abstract — which is to use ABCMeta.register — but as of this writing (March 2021) it doesn’t work with Mypy, so within the context of “static typing in Python” we presently have to ignore it.

Practicalities

The first major advantage that Protocol has is that since it is now built in to Python itself, there’s no reason not to use it. When Protocol didn’t even exist, regardless of all the advantages of adding explicit abstract types to your project with zope.interface, it did still have the small down-side of requiring a new dependency, with all the minor headaches that might imply.

beyond the theoretical distinctions, there’s a question of how well tooling supports zope.interface. There are some clear gaps; there is not a ton of great built-in IDE support for zope.interface; less-sophisticated linters will sometimes still complain that Interfaces don’t take self as their first argument. Indeed, Mypy itself does this by default — although more on that in a moment. Less mainstream performance-focused type-checkers like Pyre and Pyright don’t support zope.interface, either, although their lack of support for zope.interface is just a part of a broader problem of their lack of extensibility; they also can’t support SQLAlchemy or the Django ORM without special-casing in the tools themselves.

But what about Mypy itself — if we have to discount ABCMeta.register due to practical tooling deficiencies even if they provide a built-in way to declare a nominal-but-abstract type in principle, we need to be able to use zope.interface within Mypy as well for a fair comparison with Protocol. Can we?

Luckily, yes! Thanks to Shoobx, there’s a fairly actively maintained Mypy plugin that supports zope.interface which you can use to statically check your Interfaces.

However, this plugin does have a few key limitations as of this writing (Again, March 2021), which makes its safety guarantees a bit lower-quality than Protocol.

The net result of this is that Protocols have the “home-field advantage” in most cases; out of the box, they’ll work more smoothly with your existing editor / linter setup, and as long as your project supports Python 3.6+, at worst (if you can’t use Python 3.7, where Protocol is built in to typing) you have to take a type-check-time dependency on the typing_extensions package, whereas with zope.interface you’ll need both the run-time dependency of zope.interface itself and the Mypy plugin at type-checking time.

So in a situation where both are roughly equivalent, Protocol tends to win by default. There are undeniably big areas where Interfaces and Protocols overlap, and in plenty of them, using Protocol is a fine idea. But there are still some clear places that zope.interface shines.

First, let’s look at a case which Interfaces handle more gracefully than Protocols: opting out of matching a simple shape, where the shape doesn’t fully describe its own meaning.

Where Interfaces work best: hidden and complex meanings

The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information.

Alan Perlis, “Epigrams in Programming”, Epigram 34.

The place where structural typing has the biggest advantage is when the type system is expressive enough to fully encode the meaning of the desired behavior within the structure of the type itself. Consider a Protocol which describes an object that can add some integers together:

1
2
3

class Math(Protocol):
    def add_integers(addend1: int, addend2: int) -> int:
        ...

It’s fairly unambiguous what adherents to this Protocol should do, and anyone implementing such a thing should be able to clearly tell that the method is supposed to add a couple of integers together; there’s nothing hidden about the structure of the integers, no constraints the type system won’t let us specify. It would be quite surprising if anything that didn’t have the intended behavior would match this Protocol.

A the other end of the spectrum, we might have a plugin Interface that has a lot of hidden structure. For this example, we have an Interface called IPlugin containing a method with an easy-to-conflict-with name (“name”) overloaded with very specific constraints on its return type: the string must contain the dotted-path name of a Python object in an import-able module (like, for example, "os.path.join").

1
2
3

class IPlugin(Interface):
    def name() -> str:
        "Return the fully-qualified Python identifier of the thing to load."

With Protocols, you can work around these limitations, by manually making it harder to match; adding elements to the structure that embed names relevant to its semantics and thereby making the type behave more as if it were nominally typed.

You could make the method’s name long and ugly instead (plugin_name_to_load, let’s say) or add unused additional attributes (yep_i_am_a_plugin = Literal[True]) in order to reduce the risk of accidental matches, but these workarounds look hacky, and they have to be manually namespaced; if you want to mark it as having semantics associated with your specific plugin system, you have to embed the name of that system in your attributes themselves; here we’re just saying “plugin” but if we want to be truly careful, we have to embed the whole name of our project in there.

With Interfaces, the maintainer of each implementation must explicitly opt in, by choosing whether to specify that they are an @implementer(IPlugin). Since they had to import IPlugin from somewhere, this annotation carries with it a specific, namespaced declaration of semantic intent: “I know what the Interface IPlugin means, and I promise that I can provide it”.

This is the most salient distinction between Protocols and Interfaces: if you have strong reasons to want adherents to the abstract type to opt in, you want an Interface; if you want them to match automatically, you want a Protocol.

Runtime support

Interfaces also provide a more nuanced set of runtime checks.

You can say that an object directlyProvides an interface, allowing for some level of (at least runtime) type safety, and ask if IPlugin is .providedBy some object.

You can do most of this with Protocol, but it’s awkward. The @runtime_checkable decorator allows your Protocol to make isinstance(x, MyProtocol) work like IMyInterface.providedBy(x), but:

1. you’re still missing directlyProvides; the runtime checking is all by type, not by the individual properties of the instance;
2. it’s not the default, so if you’re not the one defining the Protocol, there’s no guarantee you’ll be able to use it.

With Interfaces, there’s also no mandatory relationship between the implementer (i.e. the type whose instances fit the specified shape) and the provider (the specific object which can fit the specified shape). This means you get features like classProvides and moduleProvides “for free”.

Interfaces work particularly well for communication between frameworks and application code. For example, let’s say you’re evolving the meaning of an Interface implemented by applications over time — EventHandler, EventHandler2, EventHandler3 — which have similarly named and typed methods, but subtly different expectations on their lifecycle or when precisely the methods will be called. A framework facing this problem can use a series of Interfaces, and check at runtime to see which of these the application implements, and be secure in the knowledge that the application has properly intentionally adopted the new interface, and doesn’t just happen to have a matching method name against an older version.

Finally, zope.interface gives you adaptation and adapter registries, which can be a useful mechanism for doing things like templating, like a much more powerful version of singledispatch from the standard library.

Adapter registries are nuanced, complex tools and unfortunately an example that captures the full utility of their power would itself be commensurately complex. However, the core of adaptation is the idea that if you have an arbitrary object x, and you want a provider of the interface IY, you can do the following:

1

y = IY(x, None)

This performs a multi-stage check:

1. If x already provides IY (either via implementer, provider, directlyProvides, classProvides, or moduleProvides), it’s simply returned; so you don’t need to special-case the case where you’ve already got what you want.
2. If x has a __conform__(interface) method, it’ll be called with IY as the interface, and if __conform__ returns anything non-None that result will be returned from the call to IY.
3. If IY has a specially-defined __adapt__ method, it can implement its own logic for this hook directly.
4. Each globally-registered function in zope.interface’s adapter_hooks will be invoked to find a function that can transform x into an IY provider. Twisted has its own global registry in this list, which is what registerAdapter manipulates.

But from the perspective of the caller, you can just say “I want an IY”.

With Protocols, you can emulate this with functools.singledispatch by making a function which returns your Protocol type and registers various types to do conversion. The place that adapter registries have an advantage is their central nature and consistent idiom for converting to the target type; you can use adaptation for any Interface in the same way, and any type can participate in adaptation in the ways listed above via flexible mechanisms depending on where it makes sense to put your implementation, whereas any singledispatch function to convert to a Protocol needs to be bespoke per-Protocol.

Describing and restricting existing shapes

There are still several scenarios where Protocol’s semantics apply more cleanly.

Unlike Interfaces, Protocols can describe the types of things that already exist. To see when that’s an advantage, consider a sprawling application that uses tons of libraries and manipulates 3D spatial data points.

There’s a convention among these disparate libraries where they all represent a “point” as an object with .x, .y, and .z attributes which are all floats. This is a natural enough shape, given the domain, that lots of your libraries just fit it by accident. You want to write functions that can work with data output by any of these libraries as long as it plausibly looks like your own concept of a Point:

1
2
3
4

class Point(Protocol):
    x: float
    y: float
    z: float

In this case, the thing defining the Protocol is your application; the thing implementing the Protocol is your collection of libraries. Since the libraries don’t and can’t know about the application — the dependency arrow points the other way — they can’t reference the Protocol to note that they implement it.

Using Protocol, you can also restrict an existing type to preserve future flexibility.

For example, let’s say we’re implementing a “mailbox” type pattern, where some systems deliver messages and other systems retrieve them later. To avoid mix-ups, the system that sends the messages shouldn’t retrieve them and vice versa - receivers only receive, and senders only send. With Protocols, we can describe this without having any new custom concrete types, like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from typing import Protocol, TypeVar

T_co = TypeVar("T_co", covariant=True)
T_con = TypeVar("T_con", contravariant=True)

class Sender(Protocol[T_con]):
    def add(self, item: T_con) -> None:
        "Put an item in the slot."

class Receiver(Protocol[T_co]):
    def pop(self) -> T_co:
        "Retrieve an item from the PO box."

All of that code is just telling Mypy our intentions; there’s no behavior here yet.

The actual implementation is even shorter:

1
2
3

from typing import Set

mailbox: Set[int] = set()

Literally no code of our own - set already does the job we described. And how do we use this?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

def send(sender: Sender[int]) -> None:
    sender.add(3)

def receive(receiver: Receiver[int]) -> None:
    receiver.pop()
    receiver.add(3)
    # Mypy stops us making this mistake:
    # "Receiver[int]" has no attribute "add"

send(mailbox)
receive(mailbox)

For its initial implementation, this system requires nothing beyond types available in the standard library; just a set. However, by treating their parameter as a Sender and a Receiver respectively rather than a Set, send and receive prevent themselves from using any functionality from the set passed in aside from the one method that their respective roles are supposed to “see”. As a result, Mypy will now tell us if any code which receives the sender object tries to remove objects.

This allows us to use existing data structures in libraries without the usual attendant problem of advertising to all clients that every tiny implementation detail of those existing structures is an intended part of the public interface. Python has always tried to make these sort of distinctions by leaving certain things undocumented or saying narratively which things you should rely on, but it’s always hit-or-miss (usually miss) whether library consumers will see those admonitions or not; by making it a feature of the programming environment, Mypy makes it harder to ignore.

Conclusions

In modern Python code, when you have an abstract collection of behavior, you should probably consider using a Protocol to describe it by default. However, Interface is also staying up to date with modern Python tooling by with Mypy support, and it can be worthwhile for more sophisticated consumers that want support for nominal typing, or that want to draw on its reach adaptation and component registration feature-set.

A story about looking for a universe, and finding a pi(e)

This is fine. You need not feel shame. Many want to create a universe. But it is good you are being careful. A universe with sentient beings is a big moral responsibility.

It is good to start with something small. The smallest. Move up from there. So there is one point, that can move in one dimension. Just interesting enough to be exciting, but little chances of messing anything serious up.

It is good if the way the point moves can be described by a function. At each point in time, \(t\), the point is at a place, \(x\). This mapping is a function.

Right now there are no sentient beings in the universe. But inevitably, you will want to create a universe with such beings. Beings that want the universe to be predictable. Better start practicing now!

This means you want \(f\) to have a simple mathematical description. The concept of death is morally complicated, but you want to allow for the potential of beings with a limited lifespan. This means that figuring out how \(f\) changes should not rely on \(t\).

One way to have a predictable function that does not depend on \(t\) is to define \(f\) with a position-independent differential equation (PIDE): an equation that involves \(f\) and its derivatives.

You are just starting out, so why not have the simplest PIDE?

Any simpler and your universe will be constant! Solving differential equations is hard. A solution probably exists, right? Hopefully, one that is more interesting than the constant zero function.

Yes, it definitely solves it, but that sounds like a really boring universe. If a solution that is not \(0\) at \(0\) exists, \(f\), then

is also a solution since derivatives are linear. The function \(f/f(0)\) is an interesting solution. Call it \(e\).

For a constant \(c\), \(e(x + c)/e(c)\) solves the equation and is \(1\) at \(0\). Differential equations have a unique solution with the same starting condition, so

or, equivalently

As a result, with a little induction,

or

If you use the convention that

, you get

Your universe is one dimensional, but you cannot help thinking about two dimensions. Is there a way to get a second dimension for "free"? You decide to leave the details for later, and for now, just see if \(e\) can be extended to the complex numbers. This is sure to come handy later.

Since \(e' = e\), \(e'' = e\) nd so on. In particular

This means that the Taylor series for \(e\) looks like

This converges for every \(x\). Because it converges absolutely everywhere, it can be extended to complex number with the same formula, and \(e' = e\) over the complex numbers as well.

If \(t\) is a real number,

and so

Nifty, for real \(t\), you get that \(e(it)\) is on the unit circle. But where on the unit circle?

Where \(d\) represents the rest of the series.

We can estimate \(|d|\) as follows:

and

Now

and \(t \to \operatorname{Re} e(ti)\) is a continuous function. This means there must be a minimal \(t\) such that

This is fun! It is not obvious how this ties into the original goal of building universes, but it definitely cannot hurt! This is an interesting number. You decide to give it a name.

The minimal \(t\) such that \(\operatorname{Re} e(ti) = 0\) will be \(\rho\). Since \(||e(\rho i)|| = 1\), this means that

and so

Looks like \(4 \rho\) is even more interesting than \(\rho\), maybe it should have its own name. How about \(\tau\)? With this new symbol, we get

So \(e\), the tentative universe-evolution function, has a period of \(\tau i\). You did not expect it. It is good you started with a simple universe, there are many things to learn before creating an interesting one.

You had a special name for \(\rho\) and for \(4 \rho\), it seems almost rude not to have a name for their geometric mean. All this universe creation is hungry work, though. It would be so much easier to think if you had a piece of...

Back to the topic at hand, you decide to call the geometric mean of \(\rho\) and \(\tau\), \(\tau / 2\), \(\pi\):

Time to relax and eat a nice piece of pie. You definitely deserved it. Whether it is savory or sweet, a pie is delicious. Enjoy it. Savor it. The universe will be waiting for you, right here.

Satisfied and with a full tummy, you get back to the universe. You gave \(\rho\), \(\pi\), and \(\tau\) names.

Any idea what their approximate value is?

You know that \(0 < \rho < 2\), but this is a pretty wide gap of ignorance. Calculating \(e(it)\), for \(t\) in that range, seems to converge quickly.

Just a handful of terms gave you something accurate to within \(2/315\). Time to leave abstract thinking, and crank up your universe simulation machine. You want to have a sense of the values involved here.

import math

def approximate_re_e_i(t):
    return sum((-1) ** n * t ** (2*n) / math.factorial(2 * n) for n in range(0, 10))

With a decent approximation of $ \operatorname{Re} e(it) $, you look for where the function is zero using binary search. It might not be the fastest way, but your universe simulator can handle it.

def find_zero():
    low, high = 0, 2
    while (high - low) > 0.001:
        midpoint = (high + low) / 2
        value = approximate_re_e_i(midpoint)
        if value < 0:
            high = midpoint
        else:
            low = midpoint
    return (high + low) / 2

Now it is time to activate the simulator, and find the values.

rho = find_zero()
tau = 4 * rho
pi = tau / 2

Wonderful progress on the universe for today. A great day. A wonderful day. A day you want to celebrate.

But when?

With \(\rho <2\), you know that \(\pi <8\). Sounds like the integer part of it could be the month, maybe?

month = int(pi)

All that is left is to choose a day. There are thirty days in a month, so hopefully two digits will do here.

rest = pi - month
day = int(rest * 100)

import datetime

year = datetime.date.today().year

celebration = datetime.date(year=year, month=month, day=day)

print("Celebrate on", celebration)

Celebrate on 2021-03-14

Enjoy your pie!

The widely used Python package cryptography changed their build system to use Rust for low-level code which caused an emotional GitHub thread. Enthusiasts of 32-bit hardware from the 1990s aside, there was a vocal faction that stipulated adherence to Semantic Versioning from the maintainers – claiming it would’ve prevented all grief. I will show you not only why this is wrong, but also how relying on Semantic Versioning hurts you.

Many people have written about the logistical challenges of food in a conference. You trade off not just, as Chris points out, expensive food versus terrible food, but also the challenges of serving the food to everyone at once.

One natural method of crowd control is the buffet line. People shuffling slowly through the line, picking food items, allows for a natural choke-point that avoids overwhelming table and staff availability. It is unpleasant to have to walk slowly, at the pace of the slowest decision maker, while hungry.

As humans do, one tries to make the best of a bad situation. All of the people in the conference share some common interests, and many of them have interesting tales besides. A common way of entertaining yourself in the line is to strike up a conversation with the random person before or ahead of you. Indeed, this has led me to hear some fascinating things: tales of incidents, new libraries, or just interesting perspectives.

With a global pandemic looming, responsible folks have either cancelled conferences or led virtual conferences. Virtual conferences, especially while a global pandemic ravages the world, are nowhere as good as the real thing.

One of my favorite things in conferences is the so-called hallway track, where we stand and chat about common interests. Friendly and inclusive people stand in the "pac-man" shape, so that people can join the conversation. I have learned a lot from these random conversations.

As humans do, one tries to make the best of a bad situation. While we are stuck at home, at least lunch time is easy. When you want to eat, order a delivery or step into the kitchen and food, chosen by you, is available. No shuffling. No waiting.

So far, no conference has tried to have a virtual buffet line, where people are forced to virtually wait in a line before eating. True, the random conversations are gone, but they have always been a coping mechanism, not the intent. If the pandemic continues, however, I am not sure this will remain true.

Conferences have already tried to "recreate" many of the constraints foisted upon physical conferences by the uncaring laws of physics in order to make them feel more "real". This rarely helps the "realism" but often creates new, unexpected problem.

One conference platform allows for "virtual coffee tables" where 2-10 people (depending on the table) can sit. Once the table is "full", nobody else can join the conversation. Table-mates can speak via text, video, or audio.

The reason real hallway tables are set for 2-10 people is because of physical constraints and avaialbility of furniture. There was no careful design of which combination of 2-10-sized tables makes for an "optimal" experience.

Further, this is not even a good recreation. With real tables, space is somewhat negotiable. An extra person can fit in if the seated people will let them. People can see the conversation. People can trade-off a subtle "how eaves-droppy" they want to be. You can stand next to the table for a long time, but possibly perceived as weird. You can pass by quickly, catch a whiff of the conversation. You can hear from afar, but only distorted highlights

These things mean, for example, someone seated at a table trying to harass a table-mate chances being seen and caught by random people. While we hope that this is not the only thing preventing people from harassing, this is a useful social enforcement tool. However, the "virtual tables" are more like "virtual isolation rooms". Stuck inside one with an unpleasant person means they can say and do what they will with no fear of witnesses.

How does Code of Conduct enforcement happens? How do vulnerable demographics feel about that?

Attempting to recreate a physical experience in a virtual world is doomed to failure, unless you have sophisticated science-fictional-level virtual reality and physics simulation. However, as a culture, we have adapted to video chats, video webinars, text chats and more. We figured out social conventions and norms, and how to enforce them.

When designing a virtual conference, concentrating on "physical fidelity" is a fool's errand. Instead, figure out what kind of pleasant virtual experiences you want to supply, how to enforce those norms you want to enforce, and how to communicate expected standards to the attendees.

Just like physical conferences can be different, virtual conferences can be different. Pre-recorded or live talks, video distribution platforms, chatting platforms, and more, need to be chosen carefully. Optimize for a good conference, not a conference that feels like an in-person conference.

There are many wordy articles on configuring your web server’s TLS ciphers. This is not one of them. Instead, I will share a configuration that scores a straight “A” on Qualys’s SSL Server Test in 2020.

How to ensure that your tests run code that you think they are running, and how to measure your coverage over multiple tox runs (in parallel!).

DRY, or Don't Repeat Yourself is frequently touted as a principle of software development. "Copy-pasta" is the derisive term applied to a violation of it, tying together the concept of copying code and pasta as description of software development bad practices (see also spaghetti code).

It is so uniformly reviled that some people call DRY a "principle" that you should never violate. Indeed, some linters even detect copy-paste so that it can never sneak into the code. But copy-paste is not a comic-book villain, and DRY does not come bedecked in primary colors to defeat it.

It is worthwhile to know why DRY started out as a principle. In particular, some for some modern software development practices, violating DRY is the right thing to do.

The main problem with repeating a code chunk is that if a bug is found, there is more than one place where it needs to be fixed. On the surface of it, this seems like a reasonable criticism. All code has bugs, those bugs will be fixed, why not minimize the cost of fixing them?

As with all engineering decisions, following DRY is a trade-off. DRY leads to the following issues:

• Loss of locality
• Overgeneralized code
• Coordination issues
• Ownership issues

I’ve often heard Henry Ford quoted as saying:

“If I had asked people what they wanted, they would have said faster horses.”

Despite the fact that he probably didn’t actually say that, it does neatly encapsulate a certain approach to product development. And it’s one that the modern technology industry loves to lionize.

There’s a genre of mythologized product development whereby wholly unique and novel products spring, fully-formed, Athena-like, from the foreheads of Zeusian industrialists like Jobs, or Musk, or Bezos. This act of creation requires no input from customers. Indeed, the myths constructed about the iconic products associated with these industrialists often gloss over or outright ignore the work of their hundreds of thousands of employees, not to mention long years of iteration along with legions of early-adopter customers.

Ford’s other major area of contribution to public discourse was, of course, being a big ol’ Nazi, just writing so much Nazi stuff that he was one of Hitler’s heroes.¹

This could be a coincidence, of course; lots of prominent thinkers in the past were absolutely hideous racists, anti-semites, slave owners and worse; these terrible ideas were often products of the time, and the people who held them sometimes nevertheless had other ideas worth examining.

But I think that this sentiment reflects a wider underlying infatuation with authoritarian ideology. At its core, the idea is that the uniquely gifted engineer is just better than their users, fundamentally smarter, more able to discern their true needs, more aware of the capabilities of the technology that we alone are familiar with. Why ask the little people, they can’t possibly know what they really need.

While we may blithely quote this sort of thing, when you look at the nuts and bolts of the technology industry, the actual practice of the industry has matured past it. Focus groups and user research are now cornerstones of interaction design. We know that it’s pure hubris to think that we can predict the way that users react with; you can’t just wing it.

But, I hadn’t heard a similarly pithy encapsulation of an empathetic approach that keeps the user in the loop and doesn’t condescend to them, until today. The quote came up, and my good friend Tristan Seligmann responded with this:

If you ask your users questions that they don’t have the skills to answer — like “how can we improve your horse?” — they will give you bad answers; but the solution to this is to ask better questions, not to ask no questions.

Tristan Seligmann

That, fundamentally, is the work-product of a really good engineer. Not faster horses or faster cars:

Better questions.

Computers work on binary code. If statements take one path: true, or false. For computers, bright lines and clear borders make sense.

Humans are more complicated. What's an adult? When are you happy? How mature are you? Humans have fuzzy feelings with no clear delination.

I was more responsible as a ten year old than as a three year old. At 13, I reached the age when I was responsible for following Jewish law myself. At 18, the legal system trusted me to could drink alcohol and drive, and trusted me that I will keep the two activities distinct. In the US, you cannot become a senator before you are 30.

At what age are you responsible "enough"?

Software is written by humans, not computers. Humans with feelings, hopes, and dreams. We cry, we strive, we feel accomplished at times, and disappointed at others.

If you were designing a version system for computers, SemVer, or "Semantic Versioning", would make perfect sense. Each part number in a three-part version number is given a specific, distinct, definition:

• Increment MAJOR for backwards incompatible changes.
• Increment MINOR for changes which add functionality.
• Increment PATH for bugfix-only backwards compatible release.

But software is not made by computers. It is made by humans.

How do you ship a product on schedule? One useful approach is applying the You Ain’t Gonna Need It principle, or YAGNI for short: leave out all the things that seem nice-to-have, but you have no proof you actually need.

But beyond the things you don’t need, there are still plenty of features you pretty clearly do need… but are not blockers on releasing your product. So beyond YAGNI, there’s also YDNIY: You Don’t Need It Yet.

Let’s see an example of this principle in practice, visualize the principle as a flowchart, and then compare it to another popular acronymed concept, the Minimum Viable Product.

A real world example: shipping a new memory profiler

In March 2020 I shipped the initial release of a new memory profiler for Python, Fil.

Here’s how it changed over time in terms of features, from May to August 2020:

• 0.3.0, initial release: Installable via pip packaging tool, runs only on Linux, only profiles complete program runs.
• 0.3.3: Support for an additional memory allocation API.
• 0.4.0: Support for out-of-memory situations.
• 0.5.0: macOS support.
• 0.6.0: Support for mmap() allocation API.
• 0.7.0: Support for C++ memory allocation API.
• 0.9.0: Much faster and lower overhead in some use cases, added support for yet another memory allocation API.
• 0.10.0: Support for running inside Jupyter notebooks, and native support for installing via the Conda packaging tool.

All of the features I added in later releases were clearly necessary from the start; YAGNI did not apply. Lots of people use macOS, the target audience of data scientists and scientists often use Conda and Jupyter, all those memory allocation APIs are used in the real world, and so on.

But even a tool that only runs complete programs on Linux, and only tracks the most popular memory allocation APIs, is still useful to some people.

If I had waited until all those features were implemented to ship an initial release, all the people who used the profiler during the first four months of its existence would have had to keep using worse tools. And with every release, the number of people for whom the tool is useful has grown.

Unlike YAGNI, YDNIY doesn’t mean you don’t implement a feature—you just delay it so that you can release something now.

The YAGNI and YDNIY algorithm

Features that are not clearly necessary can be dropped based on the YAGNI principle. And if the product is still useful without the feature, you can delay that feature based on the YDNIY principle.

In flowchart form:

YAGNI and YDNIY vs. MVP

The Minimum Viable Product, or MVP, is another acronym that might seem like it means the same thing as YDNIY. But as defined by its originator, Eric Ries, an MVP has a different goal, and actually adds on more work.

Specifically, Ries defines an MVP as “that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.” He goes on to explain that a “MVP is quite annoying, because it imposes extra overhead. We have to manage to learn something from our first product iteration. In a lot of cases, this requires a lot of energy invested in talking to customers or metrics and analytics.”

To put it another way, the goal of the MVP is to learn about users or customers, whereas the goal of YAGNI and YDNIY is to get something useful into users’ hands as quickly as possible.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

One of my favorite features within Twisted — but also one of the least known — is LoopingCall.withCount, which can be used in applications where you have some real-time thing happening, which needs to keep happening at a smooth rate regardless of any concurrent activity or pauses in the main loop. Originally designed for playing audio samples from a softphone without introducing a desync delay over time, it can also be used to play animations while keeping track of their appropriate frame.

LoopingCall is all around a fun tool to build little game features with. I’ve built a quick little demo to showcase some discoveries I’ve made over a few years of small hobby projects (none of which are ready for an open-source release) over here: DrawSnek.

This little demo responds to 3 key-presses:

1. q quits. Always a useful thing for full-screen apps which don’t always play nice with C-c :).
2. s spawns an additional snek. Have fun, make many sneks.
3. h introduces a random “hiccup” of up to 1 full second so you can see what happens visually when the loop is overburdened or stuck.

Unfortunately a fully-functioning demo is a bit lengthy to go over line by line in a blog post, so I’ll just focus on a couple of important features for stutter- and tearing-resistant animation & drawing with PyGame & Twisted.

For starters, you’ll want to use a very recent prerelease of PyGame 2, which recently added support for vertical sync even without OpenGL mode; then, pass the vsync=1 argument to set_mode:

1
2
3
4
5

screen = pygame.display.set_mode(
    (640 * 2, 480 * 2),
    pygame.locals.SCALED | pygame.locals.FULLSCREEN,
    vsync=1
)

To allow for as much wall-clock time as possible to handle non-drawing work, such as AI and input handling, I also use this trick:

1
2
3
4
5
6
7

def drawScene():
    screen.fill((0, 0, 0))
    for drawable in self.drawables:
        drawable.draw(screen)
    return deferToThread(pygame.display.flip)

LoopingCall(drawScene).start(1 / 62.0)

By deferring pygame.display.flip to a thread¹, the main loop can continue processing AI timers, animation, network input, and user input while blocking and waiting for the vertical blank. Since the time-to-vblank can easily be up to 1/120th of a second, this is a significant amount of time! We know that the draw won’t overlap with flip, because LoopingCall respects Deferreds returned from its callable and won’t re-invoke you until the Deferred fires.

Drawing doesn’t use withCount, because it just needs to repeat about once every refresh interval (on most displays, about 1/60th of a second); the vblank timing is what makes sure it lines up.

However, animation looks like this:

1
2
3

def animate(self, frameCount):
    self.index += frameCount
    self.index %= len(self.images)

We move the index forward by however many frames it’s been, then be sure it wraps around by modding it by the number of frames.

Similarly, the core² of movement looks like this:

1
2
3

def move(self, frameCount):
    self.sprite.x += frameCount * self.dx
    self.sprite.y += frameCount * self.dy

Rather than moving based on the number of times we’ve been called, which can result in slowed-down movement when the framerate isn’t keeping up, we jump forward by however many frames we should have been called at this point in time.

One of these days, maybe I’ll make an actual game, but in the meanwhile I hope you all enjoy playing with these fun little basic techniques for using Twisted in your game engine.

One of the wonderful things about Python is the ease with which you can start writing a script - just drop some code into a .py file, and run python my_file.py. Similarly it’s easy to get started with modularity: split my_file.py into my_app.py and my_lib.py, and you can import my_lib from my_app.py and start organizing your code into modules.

However, the details of the machinery that makes this work have some surprising, and sometimes very security-critical consequences: the more convenient it is for you to execute code from different locations, the more opportunities an attacker has to execute it as well...

Python needs a safe space to load code from

Here are three critical assumptions embedded in Python’s security model:

1. Every entry on sys.path is assumed to be a secure location from which it is safe to execute arbitrary code.
2. The directory where the “main script” is located is always on sys.path.
3. When invoking python directly, the current directory is treated as the “main script” location, even when passing the -c or -m options.

If you’re running a Python application that’s been installed properly on your computer, the only location outside of your Python install or virtualenv that will be automatically added to your sys.path (by default) is the location where the main executable, or script, is installed.

For example, if you have pip installed in /usr/bin, and you run /usr/bin/pip, then only /usr/bin will be added to sys.path by this feature. Anything that can write files to that /usr/bin can already make you, or your system, run stuff, so it’s a pretty safe place. (Consider what would happen if your ls executable got replaced with something nasty.)

However, one emerging convention is to prefer calling /path/to/python -m pip in order to avoid the complexities of setting up $PATH properly, and to avoid dealing with divergent documentation of how scripts are installed on Windows (usually as .exe files these days, rather than .py files).

This is fine — as long as you trust that you’re the only one putting files into the places you can import from — including your working directory.

Your “Downloads” folder isn’t safe

As the category of attacks with the name “DLL Planting” indicates, there are many ways that browsers (and sometimes other software) can be tricked into putting files with arbitrary filenames into the Downloads folder, without user interaction.

Browsers are starting to take this class of vulnerability more seriously, and adding various mitigations to avoid allowing sites to surreptitiously drop files in your downloads folder when you visit them.¹

Even with mitigations though, it will be hard to stamp this out entirely: for example, the Content-Disposition HTTP header’s filename* parameter exists entirely to allow the the site to choose the filename that it downloads to.

Composing the attack

You’ve made a habit of python -m pip to install stuff. You download a Python package from a totally trustworthy website that, for whatever reason, has a Python wheel by direct download instead of on PyPI. Maybe it’s internal, maybe it’s a pre-release; whatever. So you download totally-legit-package.whl, and then:

1
2

~$ cd Downloads
~/Downloads$ python -m pip install ./totally-legit-package.whl

This seems like a reasonable thing to do, but unbeknownst to you, two weeks ago, a completely different site you visited had some XSS JavaScript on it that downloaded a pip.py with some malware in it into your downloads folder.

Boom.

Demonstrating it

Here’s a quick demonstration of the attack:

1
2
3
4
5

~$ mkdir attacker_dir
~$ cd attacker_dir
~/attacker_dir$ echo 'print("lol ur pwnt")' > pip.py
~/attacker_dir$ python -m pip install requests
lol ur pwnt

PYTHONPATH surprises

Just a few paragraphs ago, I said:

If you’re running a Python application that’s been installed properly on your computer, the only location outside of your Python install or virtualenv that will be automatically added to your sys.path (by default) is the location where the main executable, or script, is installed.

So what is that parenthetical “by default” doing there? What other directories might be added?

Anything entries on your $PYTHONPATH environment variable. You wouldn’t put your current directory on $PYTHONPATH, would you?

Unfortunately, there’s one common way that you might have done so by accident.

Let’s simulate a “vulnerable” Python application:

1
2
3
4
5

# tool.py
try:
    import optional_extra
except ImportError:
    print("extra not found, that's fine")

Make 2 directories: install_dir and attacker_dir. Drop this in install_dir. Then, cd attacker_dir and put our sophisticated malware there, under the name used by tool.py:

1
2

# optional_extra.py
print("lol ur pwnt")

Finally, let’s run it:

1
2

~/attacker_dir$ python ../install_dir/tool.py
extra not found, that's fine

So far, so good.

But, here’s the common mistake. Most places that still recommend PYTHONPATH recommend adding things to it like so:

1

export PYTHONPATH="/new/useful/stuff:$PYTHONPATH";

Intuitively, this makes sense; if you’re adding project X to your $PYTHONPATH, maybe project Y had already added something, maybe not; you never want to blow it away and replace what other parts of your shell startup might have done with it, especially if you’re writing documentation that lots of different people will use.

But this idiom has a critical flaw: the first time it’s invoked, if $PYTHONPATH was previously either empty or un-set, this then includes an empty string, which resolves to the current directory. Let’s try it:

1
2
3

~/attacker_dir$ export PYTHONPATH="/a/perfectly/safe/place:$PYTHONPATH";
~/attacker_dir$ python ../install_dir/tool.py
lol ur pwnt

Oh no! Well, just to be safe, let’s empty out $PYTHONPATH and try it again:

1
2
3

~/attacker_dir$ export PYTHONPATH="";
~/attacker_dir$ python ../install_dir/tool.py
lol ur pwnt

Still not safe!

What’s happening here is that if PYTHONPATH is empty, that is not the same thing as it being unset. From within Python, this is the difference between os.environ.get("PYTHONPATH") == "" and os.environ.get("PYTHONPATH") == None.

If you want to be sure you’ve cleared $PYTHONPATH from a shell (or somewhere in a shell startup), you need to use the unset command:

1
2

~/attacker_dir$ python ../install_dir/tool.py
extra not found, that's fine

Setting PYTHONPATH used to be the most common way to set up a Python development environment; hopefully it’s mostly fallen out of favor, with virtualenvs serving this need better. If you’ve got an old shell configuration that still sets a $PYTHONPATH that you don’t need any more, this is a good opportunity to go ahead and delete it.

However, if you do need an idiom for “appending to” PYTHONPATH in a shell startup, use this technique:

1
2

export PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}new_entry_1"
export PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}new_entry_2"

In both bash and zsh, this results in

1
2

$ echo "${PYTHONPATH}"
new_entry_1:new_entry_2

with no extra colons or blank entries on your $PYTHONPATH variable now.

Finally: if you’re still using $PYTHONPATH, be sure to always use absolute paths!

Related risks

There are a bunch of variant unsafe behaviors related to inspecting files in your Downloads folder by doing anything interactive with Python. Other risky activities:

• Running python ~/Downloads/anything.py (even if anything.py is itself safe) from anywhere - as it will add your downloads folder to sys.path by virtue of anything.py’s location.
• Jupyter Notebook puts the directory that the notebook is in onto sys.path, just like Python puts the script directory there. So jupyter notebook ~/Downloads/anything.ipynb is just as dangerous as python ~/Downloads/anything.py.

Get those scripts and notebooks out of your downloads folder before you run ’em!

But cd Downloads and then doing anything interactive remains a problem too:

• Running a python -c command that includes an import statement while in your ~/Downloads folder
• Running python interactively and importing anything while in your ~/Downloads folder

Remember that ~/Downloads/ isn’t special; it’s just one place where unexpected files with attacker-chosen filenames might sneak in. Be on the lookout for other locations where this is true. For example, if you’re administering a server where the public can upload files, make extra sure that neither your application nor any administrator who might run python ever does cd public_uploads.

Maybe consider changing the code that handles uploads to mangle file names to put a .uploaded at the end, avoiding the risk of a .py file getting uploaded and executed accidentally.

Mitigations

If you have tools written in Python that you want to use while in your downloads folder, make a habit of preferring typing the path to the script (/path/to/venv/bin/pip) rather than the module (/path/to/venv/bin/python -m pip).

In general, just avoid ever having ~/Downloads as your current working directory, and move any software you want to use to a more appropriate location before launching it.

It’s important to understand where Python gets the code that it’s going to be executing. Giving someone the ability to execute even one line of arbitrary Python is equivalent to giving them full control over your computer!

Why I wrote this article

When writing a “tips and tricks” article like this about security, it’s very easy to imply that I, the author, am very clever for knowing this weird bunch of trivia, and the only way for you, the reader, to stay safe, is to memorize a huge pile of equally esoteric stuff and constantly be thinking about it. Indeed, a previous draft of this post inadvertently did just that. But that’s a really terrible idea and not one that I want to have any part in propagating.

So if I’m not trying to say that, then why post about it? I’ll explain.

Over many years of using Python, I’ve infrequently, but regularly, seen users confused about the locations that Python loads code from. One variety of this confusion is when people put their first program that uses Twisted into a file called twisted.py. That shadows the import of the library, breaking everything. Another manifestation of this confusion is a slow trickle of confused security reports where a researcher drops a module into a location where Python is documented to load code from — like the current directory in the scenarios described above — and then load it, thinking that this reflects an exploit because it’s executing arbitrary code.

Any confusion like this — even if the system in question is “behaving as intended”, and can’t readily be changed — is a vulnerability that an attacker can exploit.

System administrators and developers are high-value targets in the world of cybercrime. If you hack a user, you get that user’s data; but if you hack an admin or a dev, and you do it right, you could get access to thousands of users whose systems are under the administrator’s control or even millions of users who use the developers’ software.

Therefore, while “just be more careful all the time” is not a sustainable recipe for safety, to some extent, those of us acting on our users’ behalf do have a greater obligation to be more careful. At least, we should be informed about the behavior of our tools. Developer tools, like Python, are inevitably power tools which may require more care and precision than the average application.

Nothing I’ve described above is a “bug” or an “exploit”, exactly; I don’t think that the developers of Python or Jupyter have done anything wrong; the system works the way it’s designed and the way it’s designed makes sense. I personally do not have any great ideas for how things could be changed without removing a ton of power from Python.

One of my favorite safety inventions is the SawStop. Nothing was wrong with the way table saws worked before its invention; they were extremely dangerous tools that performed an important industrial function. A lot of very useful and important things were made with table saws. Yet, it was also true that table saws were responsible for a disproportionate share of wood-shop accidents, and, in particular, lost fingers. Despite plenty of care taken by experienced and safety-conscious carpenters, the SawStop still saves many fingers every year.

So by highlighting this potential danger I also hope to provoke some thinking among some enterprising security engineers out there. What might be the SawStop of arbitrary code execution for interactive interpreters? What invention might be able to prevent some of the scenarios I describe below without significantly diminishing the power of tools like Python?

Stay safe out there, friends.

Acknowledgments

Thanks very much to Paul Ganssle, Nathaniel J. Smith, Itamar Turner-Trauring and Nelson Elhage for substantial feedback on earlier drafts of this post.

Any errors remain my own.

I have written before about my Inbox Zero methodology. This is still what I practice, but there is a lot more that helps me.

The concept behind "Universal Binary" is that the only numbers that make sense asymptotically are zero, one, and infinity. Therefore, in order to prevent things from going off into infinity, there needs to be processes that keep everything to either zero or one.

One:

• TODO list.
• Calendar.
• Time tracker.

Zero:

• "Notes"
• E-mails in inbox
• Non-pinned open tabs
• Floating tasks

Most bugs you encounter have been encountered by others before you; most programming problems you face have been faced by others as well. And many of those people have written down details about what they’ve learned—in issue trackers, documentation, and blog posts.

All you have to do is find this information.

Typing a phrase in to your search engine of choice will sometimes take you straight to the right answer. But quite often, the results aren’t helpful.

No need to give up, though: there are still plenty of ways you can productively keep searching.

Use site-specific search too

It’s easy to believe that search engines have all the answers right at the top, but they actually hide quite a lot of content deep in their results. And some obscure content never gets indexed at all, which is unfortunate when it’s the obscure content that you need to find.

So instead of just using a search engine, use the local search engine of the project issue tracker, the documentation, StackOverflow, and so on.

For example, let’s saying you’re using Eliot, a somewhat obscure Python logging library I maintain, and you want to use it with the Pandas library. Unfortunately, you get an error, so you search Google for the text of the error: eliot dataframe is not json serializable. Now, there is an actual issue in Eliot’s GitHub issue tracker with this exact error message—but as of August 2020 Google doesn’t return it, probably because it didn’t bother to index that page.

But if you were to use the search form on the Eliot GitHub project’s issues page, you would find the issue that mentions this particular error. In this case, as in many others, the search engine isn’t actually indexing everything: you have no choice but to use the local search engine.

Local search engines often have the additional benefit of allowing more structured search, for example:

• An issue tracker might let you search by open/closed status, labels, or the affected version.
• StackOverflow questions are tagged with particular technologies by the person submitting the question.

You still want to use a search engine

A software project’s documentation and issue tracker are a great place to start searching, but sometimes you’ll find solutions elsewhere.

For example, if you have a problem with library A, it might be that project B had the same issue, and you can find a workaround in their issue tracker. Or perhaps someone wrote a handy blog post on the issue—or they might have other related content. And that related content can also be useful.

Read results that don’t answer your question

Often you’ll encounter results that solve a similar but not identical problem. Read those pages anyway.

First, because you’ll learn more about the shape of the problem, broad approaches, and how the underlying software works.

Second, because you might find suggestions of new places to search.

Third, because this will give you an opportunity to apply your close reading skills and learn more domain-specific jargon. You can then use this jargon to widen, narrow, and vary your search.

Note: This article is an excerpt from my book, The Secret Skills of Productive Programmers, which also has a chapter on close reading.

Narrow and widen your search

Let’s say you’ve tried an initial search engine search, and you got a huge swath of unrelated results. For example, if I use Google in private mode to search for eliot, I get many entries about the poet T.S. Eliot.

Given too many results, you need to focus in: add a keyword or two that will help narrow the results to those you care about. In this example, searching for eliot logging find the actual Python library; searching for eliot python also helps.

Again, the jargon you’ve found along the way will help you know what to add.

If your search is too specific, you can do the opposite, removing some unnecessary keywords.

Try lots of variations by using jargon

Even if your initial searches don’t work, you shouldn’t give up: now is the time to start using synonyms and alternative phrasings. You are using a certain phrase to describe the problem, but other people might conceptualize it a different way, and use different phrases.

If you can rephrase the search you are likely to find many results you haven’t seen before. For example, let’s say you’re using the Pandas dataframe library for Python and you’re running out of memory. If you search for pandas too much memory, pandas out of memory, pandas large files, and pandas out of core will give you some overlapping results, but each returns some results you won’t get from other phrases.

The last term comes from “out-of-core computation”, a computer science term for algorithms that process data that doesn’t fit in memory. How might you learn about that phrase? By collecting jargon as you go along.

Search for errors the right way

When searching for errors, you need to copy/paste enough that you’re identifying the specific error, but not too much such that the search engine can’t find any matching results. For example:

Traceback (most recent call last):
  File "/home/itamarst/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/itamarst/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/itamarst/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/itamarst/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/itamarst/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/itamarst/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "flask1.py", line 18, in index
    return _counter + "\n"
TypeError: unsupported operand type(s) for +: 'Counter' and 'str'

Different languages will have different formatting, but the basic idea is that the lines that includes directories are specific to your computer. Searching for /home/itamarst is not going to get good results!

Searching for the last line might work:

TypeError: unsupported operand type(s) for +: 'Counter' and 'str'

Or maybe the last two lines, if that line is from code I downloaded and didn’t write myself:

    return _counter + "\n"
TypeError: unsupported operand type(s) for +: 'Counter' and 'str'

Or perhaps I want to understand the generic error, rather than this particular instance:

TypeError: unsupported operand type(s) for +:

Or perhaps I think this is a problem in the Flask library rather than my code, in which case I might search for:

flask TypeError: unsupported operand type(s) for +: 'Counter' and 'str'

Typically the important information will be either at the beginning or the end of the error traceback or stacktrace.

(Thanks to Jason Swett for suggesting this technique.)

Finally, be careful

One issue with searching for random solutions on the web is that the proposed solution is sometimes wrong or broken. I’ve seen people propose insecure solutions on StackOverflow, and then get upvoted by other people who don’t know any better.

Just because the solution seems to work doesn’t mean it’s correct: you still have to think, do some additional research to validate the proposal, and probably write some tests too.

Want more ways to become a more productive programmer? This article is an excerpt from my book, The Secret Skills of Productive Programmers.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

I suffer from ADHD.

Photo by Taylor Young on Unsplash

I want to be clear: when I say I suffer from this disorder, I am making a self-diagnosis. I’ve obliquely referred to suffering from ADHD in previous posts, but rarely at any length. The main reason for my avoidance of the topic is that it still makes me super uncomfortable to write publicly about a “self-diagnosis”, since there’s a tremendous amount of Internet quackery thanks to amateur diagnosticians.

This despite the fact that I’ve known for the past 15 years that I have ADHD.

I am absolutely not trying to set myself up as a maverick unlicensed freelance psychiatrist here. If you think you might have ADHD, or any other ailment, whether mental or physical, call your primary care physician. Don’t email me.

At the same time, for me, this diagnosis is not really ambiguous or in a gray area. This is me looking down and noticing I’ve only got one arm, and diagnosing myself as a one-armed person. I’ve taken numerous ADHD screening questionnaires and reliably scored well into the range of “there is no ambiguity whatsoever, you absolutely have ADHD”, so I feel confident to describe myself as having it.

Terminology aside, this post is about a set of cognitive and metacognitive issues that I have, and some tools that I found useful to remedy them. I think others might find those same tools useful in similar situations. So if you’re also uncomfortable with the inherently unreliable nature of self-diagnosis, or the clinical specificity of the term “ADHD” — and I absolutely don’t blame you if you are — I invite you to read “ADHD” as a shorthand for some character traits that I informally believe fit that label, and not a robust clinical analysis of myself or anyone else.

With that extended disclaimer out of the way, I’ll get started on the post itself; and where better to do that than at the start of my own challenges.

The ‘Laziness’ model

Photo by Zosia Korcz on Unsplash

Throughout my childhood, I was labeled an “underachiever”. I performed well on tests and didn’t do homework. I was frequently told by adults — especially my teachers — that I was “brilliant” but “lazy”.

Was I lazy? Is there even such a thing as “laziness”? Here’s a spoiler for you — “no”¹ — but I didn’t know that at the time. All I knew was that I couldn’t seem to do certain things — boring things: homework, long division, and cleaning up my room, for a few examples. I couldn’t seem to do the things that my peers found routine and trivial.

This is a common enough experience that it shows up clearly even in systematic reviews and meta-analyses of adult sufferers of ADHD. Everybody tells you you’re lazy, and so you believe it. It sure looks like laziness from the outside!

In retrospect, that’s the interesting problem with this false diagnosis: “from the outside”. Assuming for the moment that laziness does in fact exist and is a salient character flaw, what would the experience of the interiority of such laziness actually feel like?

It seems unlikely that it would feel like I what I actually felt at the time:

1. Frequently, suddenly remembering, in contexts where it wouldn’t help — walking to school, in an unrelated class, while walking to work — that I had to Do The Thing.

2. Anxiously, yearningly, often desperately wishing I could Do The Thing.

3. Trying to Do The Thing at the responsible time, finding that my mind would wander and I would lose several hours of time... sitting for hours, literally bored to tears, while I attempted and failed to Do The Thing.

4. At long last, finally managing to start. Once I was truly exhausted and starting to panic, I’d drink a gallon of heavily-caffeinated and very sugary soda at 2 in the morning and finally finally find that I suddenly had the ability to Do The Thing, and white-knuckle my way through an all-nighter to finish The Thing. (This step was more common after I got to my late teens; before that, The Thing just wouldn’t get Done.)

Sitting up night after night destroying my mental and physical health, depriving myself of sleep, focusing with every ounce of my will on tasks that I absolutely hated doing but was forcing myself to complete at all costs: it doesn’t seem to line up with the popular conception of what “laziness” might be like! Yet, I absolutely believed that I was lazy. If I were not lazy, surely Doing The Thing wouldn’t be so difficult!

I took pains at the start of this post to point out that mental health diagnosis is usually best left to professionals. I think that at this point in the story I should emphasize that “I’m lazy” is also itself a self-diagnosis, and — at least in every case where I’ve ever heard it used — a much worse one than “I have ADHD”.

If you are not a licensed psychologist or psychiatrist, any time you decide with certainty that someone (even yourself!) has an intrinsic, persistent character flaw, you’re effectively diagnosing them. If you decide that they’re inherently lazy, or selfish, or arrogant, you’re effectively diagnosing them with a sort of personality disorder of your own invention.

So, although I didn’t see it at the time, laziness didn’t seem to describe me terribly well. What description fits better?

The ‘Attention Deficit’ model

Photo by Tom Bradley on Unsplash

In my late 20s, my Uncle Joel gave me a gift that changed my life: the book “Driven to Distraction: Recognizing and Coping with Attention Deficit Disorder”² by Edward M. Hallowell. The life-changing aspect of this book was not so much that it showed that there were other people “like me”, or that my problem had a name, but that it gave me a different, and more accurately predictive, model to understand my own behavior.

In other words, it allowed me to see — for the first time — that the scarcest resource limiting my efficacy wasn’t the will to do the work, but rather the ability to focus. With this enhanced understanding, I could select a more effective strategy for dealing with the problem.

I did select such a strategy! It worked very well — albeit with some caveats. I’ll get to those in a moment.

Although my limiting factor was the ability to pay attention, the problem that prevented me from recognizing this was one of metacognition — the way I was thinking about how I think.

My early model of my own mind was that I was a lazy person who just needed to do what I had assumed everyone else must be doing: forcing myself to do the tasks that I was having trouble completing. If I really wanted to get them done, then what possible other reason could there be for me to not do them?

The ‘laziness’ model didn’t generate particularly good predictions. For any given project at school, it would predict that I would not try very hard to do it, since the very dictionary definition of ‘lazy’ is “unwilling to work or use energy”. The observed behavior, by contrast, was constant, panicked, intense (albeit failed, or at least highly inefficient) uses of significant amounts of energy.

The main reason to have a model of a thing is to make predictions about that thing. If the predictions that a model gives you are consistently wrong, then the model isn’t directly useful. At that point, it’s time to discard it and find a better one. At the very least, it’s time to revise the model in question until it starts giving you more accurate, actionable information.

The ‘laziness’ model is wrong, but worse than that, it’s harmful. What it routinely predicts, regardless of context, is that I need more negative self-talk, more ‘motivation’ in the form of vicious self-criticism, more forcing myself to “just do it”. All of these things, particularly when performed habitually, cause real, significant harm.

If I gave myself the most negative self-talk I could muster, the most vicious criticism, and really put Maximum Effort into forcing myself to do the thing I wanted done... if it didn’t work, of course that just meant that I needed to engage in even more self-abuse! I could always try harder!

This is the worst way that a model can be inaccurate: an unfalsifiable, self-reinforcing prediction. I could never demonstrate to myself that I’d really been as unkind to myself as was possible; there was always room for escalation. Psychologically, it’s also the worst kind of behavioral advice, which is the kind that generates a self-reinforcing negative feedback loop.

Once I started putting my newfound knowledge into practice, the difference between interventions predicated on an understanding of the problem as “lack of usable attention span” and those based on “lack of willpower” was night and day. I stopped trying to white-knuckle my way through all of my challenges and developed non-judgmental ways to remind myself to do things.

I knew that I, personally, was never going to spontaneously remember to do things at the right time, so I developed ways of letting computers remind me. I knew that I’d never be able to stick with routine, repetitive tasks, so I made a unified list of all the tedious administrative tasks I need to perform. I can’t keep important dates and times in mind, so I rely completely upon my calendar.

Even given these successes, “it worked!” is a colossal oversimplification. Today, it’s about 15 years later, and I’m still sifting through the psychological rubble wrought by the destructive, maladaptive coping mechanisms that I just described, and still trying to find better ways to remain effective when I’m feeling distracted... which is most of the time.

Simply having a better model at the coarsest level is just the first step. Instantiating that model in a working, fleshed out technological system is a ton of work in its own right.³ But it’s work that starts having little successes, which is a lot easier to build on and maintain momentum with than the same failure repeated day after day.

Given that I was starting — nearly from scratch — at 25, and had a lifetime worth of bad habits to unlearn, constructing a workable system that addressed my personal organizational needs still took the better part of a decade.

So as I move into the next, slightly more prescriptive section here, I don’t want to give anybody the idea that I think this is easy.

Don’t give up!

Listen up, Simon. Don’t believe in yourself. Believe in me! Believe in the Kamina who believes in you!

Kamina, Episode 1,
Tengen Toppa Gurren Lagann

At the start of this post, I specifically mentioned that I hadn’t wanted to write at length about ADHD due to my discomfort with self-diagnosis. So, you might be wondering: what was it that overcame this resistance and prompted me to finally write about my own experiences with ADHD?

The original inspiration was a pattern of complaints about suffering from ADHD I see periodically — mainly on Twitter — that look roughly like this:

• “ADHD means never being on time for a meeting and having no excuse, forever.”
• “It’s great to have ADHD and never be able to complete a routine task. Sigh.”
• “I can’t take out the trash and my roommates just can’t understand that this is just part of who I am and I will never get better.”
• “Why can’t neurotypicals understand that I’m just never going to “get stuff done” like they can. It’s exhausting.”

These are paraphrased and anonymized on purpose; I really don’t want to direct any negative attention towards someone specific, particularly someone just venting about struggles.

Of course, no blog post in mid-2020 would be complete without some reference to the ... situation. The original inspiration for this post predates the dawn of the new hell-world we all now inhabit, but, to say the least, COVID-19⁴ has presented some new challenges to the coping mechanisms I’m writing about here. (Still, I know that I’m considerably better off than the average American in this mess.)

The message I’m trying to get across here is hopeful — others suffering with executive-function deficits similar to mine might be able to do what I did and fix a lot of their problems with this one weird trick! — and the constant drumbeat of despair all around us right now makes that sort of message feel more urgent.

Posts like the ones I described above seem to represent a recurring pattern of despair, and they make me sad. Not because I can’t identify with them; I have absolutely had these feelings. Not even because they’re wrong, exactly: it really is harder for folks with ADHD to handle some of these situations, and the struggle really is lifelong.

They make me sad because they’re expressing a fatalistic perspective; a fixed mindset⁵ that precludes any hope of future improvement. The through line that I have seen from all of these posts is a familiar, specific kind of despair; a thought I’ve had myself multiple times:

When somebody that I care about asks me, ‘Can you do the dishes later?’, I want to say ‘yes’ and have them believe me. I want to be able to believe myself, and I don’t think I will ever be able to.

Unlike myself when I was younger, the authors of these posts already have a name for their problem: ADHD. Sometimes they’ve even tried some amount of therapy or even medication.

Even so, they’re still buying in to the maladaptive strategy of “just try harder”. Since they already know that ADHD is, at least in part, a structural brain difference, they despair of ever being able to actually do that though, which leaves “giving up” as the only viable strategy.

Don’t give up! I believe in you!

Different problems, different tools

I have had another lifelong problem since when I was young: I am severely nearsighted. Yet, I never developed any psychological hangups around that; nobody ever told me that I needed to buckle down and just squint harder. This problem was socially quite well understood, so… I got glasses. Then I could see, as long as I consistently used those glasses.

Nobody ever expected me to be able to see without glasses.

Photo by NordWood Themes on Unsplash

Calendars, to-do lists, and systems like Getting Things Done are the corrective lenses for the ADHD brain.⁶.

If a to-do list is a corrective lens for ADHD, one of the major issues around understanding how to use it is that the mass-market literature around to-do lists assumes a certain level of neurotypicality. Assistive devices may frequently be useful to non-disabled people, but their relationship to and use of such affordances is very different.

Through the Looking-Glass ...

Let’s stretch this lens metaphor into absurdity.

In our metaphorical world, ADHD is myopia, and so most — or at least many — folks are “sight-typical”. Productivity systems are our “lenses”.

If nearsightedness were as poorly understood as ADHD, and you were nearsighted, you wouldn’t be able to pop on down to Lenscrafters and pick up a pair of spectacles. You might realize that the problem was with your eyes, and think, “lenses might help me see farther”. Many kinds of lenses might be commercially available in such a world! Lenses for telescopes, cameras, microscopes...

The way that someone with 20/20 vision might use a lens to see farther is to use a telescope to see something really far away. But you, my hypothetically-nearsighted friend, don’t need a powerful zoom lens to take surveillance photographs from a helicopter. Even if you could make such lenses work to correct your vision, you wouldn’t want to carry a pair of 2-kilogram DSLR zoom lenses everywhere you go. You want eyeglasses, which are something different.

Photo by James Bold on Unsplash

The lenses in eyeglasses are — while operating on fundamentally the same principles of optics as the lenses in a telescope or a microscope — constructed and packaged in a completely different way. But most importantly, the way you use them is to wear them every day, not to deploy them on special occasions in the rare event where you need to do something extreme, but all the time, every day, in the same way.

... and What I Found There

Photo by Glenn Carstens-Peters on Unsplash

A person with nominal executive function might use the occasional free-floating to-do list to track a big, complex project with a lot of small interrelated tasks. Most folks in the modern information-driven economy routinely need to do projects that are too complex to easily memorize all the required steps. Even doing your own personal taxes has enough steps to require at least a little bit of tracking.

Such a person could make a to-do list for that one project — their telescope, if you will — put it in a place where they’d remember to look at it when they’re working on that project, and then remember to check things off when they’re done.

They could have one to-do list on the fridge for groceries, a note on their phone for stuff to get for their spouse, and a wiki page outlining some tasks at work. They would probably have enough free-floating executive function to remember which list maps to which project and when each project is relevant, and remember to check each one at the appropriate time.

I spent a lot of time trying to make disconnected to-do lists like this work for me. They never have. Even when I’m feeling particularly productive there is a cycle of list-generation, that goes like this:

1. When I want to work on the project in question, I can’t remember where the to-do list is, but I need to figure out what I need to do again.

2. So I go and write a new to-do list, spend a bunch of time rewriting the one I’d already written but can’t quickly find. Then I do some work on the project, check off a few things, and put the list away.

3. Later, I’ll find both lists, both half checked off, and now I waste a bunch of time trying to figure out which one is the right one.

4. Repeat this process a few times, and now I have a dozen lists. The lists themselves start generating more work than the actual project, because now I am constantly re-making and finding lists, trying to figure out which one is the most up to date.

This is the simplest case, but the real problem happens at a higher level: one of the biggest problems caused by any executive function deficit like ADHD is the difficulty of task initiation.

The more irrelevant distractions I can see while I’m trying to work out what to do next, the harder that decision becomes. And there’s nothing quite so distracting as the detritus of a thousand half-finished to-do lists.

One List To Rule Them All

What I’ve found works for me is a single, primary to-do list that I can obsessively check in with every minute of every day, which subsumes every other list related to every other project in my life.

I’m hardly the only person to have this insight — if you start engaging with the “productivity” noosphere, reading all the books, listening to the podcasts, this is a recurring theme. You don’t just have an ‘app’ or a ‘list’, you have to have a System. It has to be reliable; you have to know you’re going to keep checking it, or it’s worthless for storing your commitments. But unfortunately this is frequently buried under a lot of other technical complexity about the fiddly details of how to set up one system or the other. It’s very easy to miss the forest for the trees.

Having ADHD means that I routinely forget what I’ve decided to do over the course of only a minute or two after I’ve decided to do it. Just this week, I had to remind myself no fewer than three times to write down “buy more olive oil” because I kept remembering that we were running low when I was in the kitchen and by the time I finished washing my hands to put it into my phone I’d already forgotten why I did that and went back to making dinner.

I need to write everything down. I’m not going to remember five or six, or even two or three places to check for what to do next. I need to have one place to check what comes next, and then build the habit of constantly going back to it, both to add new things and to see what needs to be done.

Technology can help. Technology might even be necessary — it is for me.

But if you’re considering trying this out for the first time, be mindful that piles of to-do apps can be just as distracting as piles of paper. The important thing is to clearly, singularly decide on the one place which is the ‘root’ of your task tracking system.

You can even do this with a pen and paper. Carry the same, single notebook with you everywhere, and make it absolutely clear that it is your primary list, which is where you have to put any references to other lists. Some people have a lot more success with something tactile, to engage all the senses.

For me personally, the high-tech portion of this strategy is indispensable. I use a combination of OmniFocus for things that have to be done and Apple’s built-in calendar application for places I have to be at a particular time.⁷

OmniFocus⁸ defines the core gameplay loop of my life. Rather than having to cultivate and retain an elaborate series of interlocking habits and rituals to remain functional, I have a single root habit which triggers every other habit.

That habit? Consulting the unified “what should I do next” perspective in OmniFocus. Every time I am even marginally distracted, I check that view again.

Any time I have trouble initiating a task, I start breaking down the top task in that list into smaller and smaller “next physical action”. I don’t even rely on myself to do this; since I know I’ll forget to break things down, I frequently make tasks that look like this:

• thing I want to do
• plan the thing I want to do
  • break down the planning task into tiny actions and write them down here
  • break down the task itself into tiny actions and write them down here

To reduce distraction, I routinely close down any windows that are not necessary for whatever I’m currently working on. Particularly, I routinely sweep to get rid of browser tabs, asking (as I would with an email) “does this window represent a task I should do?”. If yes, it goes in the task system, if no, I close it so it won’t distract me further.

To facilitate this clean-up, on every computer that I use, I have a global hot-key set up to turn the thing that I’m looking at — some selected text, an image, an email message, a browser tab, a chat message at work — into a task that I can look at later.

Everything I have to do on a regular basis is in this system as a recurring task; for example:

• taking out the trash
• doing the dishes
• logging in to Jira at work to look for assigned tasks
• checking my email
• brushing my teeth

Yes, even basic personal hygiene is in here. Not because I’ll necessarily forget, or that it takes a lot of energy, but I don’t want to waste one iota of brainpower I could be devoting to my current task to worrying about whether I might need to do something else later. If I don’t see ‘brush teeth’ in my “what should I do next” view, then I know, with certainty, that I don’t need to be thinking about tooth-brushing right now.

The “what should I do next” view is available on all of my computers, on my tablet, on my phone, and it even dominates my watch-face; I check it more often than I check the time:

No single feature is a hard requirement of my system; I could get along without any one of them in a pinch. However, the way that they combine to constantly reinforce what the next thing I need to do is in any given context, at any given time, means that I need to expend less energy trying to consciously hang on to all the context.

Limitations and Risks

I don’t want to give an overly rosy view of this strategy. Getting a single unified to-do system that works for you is not the same as getting a brain that can remember to do stuff. So here are some caveats:

1. Implementing and maintaining such a system is never easy. It just takes tasks like ‘making sure I renew my passport before I need to travel’, ‘show up on time for the meeting’ and ‘buy a gift at least a week before the wedding’ from totally impossible to possible to do at least somewhat reliably with a sustainable level of effort. The main thing that I believe is possible for everyone is being able to commit to simple future tasks.
2. Building enough data about one’s own habits and procrastination triggers also takes time, and to make such a system effective, one needs to do that work as well. (A passive time-tracking tool like Screen Time on your phone or RescueTime on your workstation can be quite illuminating — and surprising.)
3. The initial wave of relief I felt when I started tracking tasks masked a gradual increase in my general anxiety over time. Checking and re-checking the ‘what to do next’ list can become a bit of an anxious compulsion, a safety behavior that doesn’t always help me plan my day. As one builds the habit of routinely checking the list, it’s important to avoid developing constant anxiety about the list as the only motivation to do so.
4. Similarly, it is important to learn to under-commit. Not only does one need to avoid putting an unrealistic amount of stuff into the system, everybody (but especially everybody with ADHD!) needs non-trivial chunks of unstructured, unplanned time, where the system will clearly say ‘nothing to do now, just relax’. The “poor self-observation” and “time blindness” symptoms of ADHD ensure that properly estimating things before committing is a constant challenge that never really goes away either.
5. This strategy definitely won’t be sufficient for some folks. ADHD is a spectrum and there’s no precise mechanism to calibrate where you are on it. Some folks will respond really well to this strategy, some folks will need medication before it helps to a useful degree.

Finishing up (about finishing up)

If you’re suffering from ADHD and despairing that you will never finish a task or be on time to an appointment: you can. It’s possible to do it at least pretty reliably. I believe if you commit to one and only one task tracking system, and consistently use it every single day, all the time, you can commit to tasks and get them done.

If you do it consistently enough, it will eventually become muscle memory, and not something you need to consciously remember to do every day.

It’s still never going to be easy to Do The Thing, even if your digital brain can perfectly remember what The Thing is right now.

At the very least, it was possible for me to learn to trust myself when I say that I will do something in the future, by designing a system around my own limited attention, and if I can do it, I think you can too.

Acknowledgments

This was a big one! I’d like to particularly thank my Uncle Joel, without whom this post (and many of my other achievements) would not be possible for the reasons described above, as well as Moshe Zadka, Amber Brown, Tom Most, and Eevee for extensive feedback on previous drafts of this post.

Additionally, I’d like to thank David Reid for introducing me to many of the tools and techniques that I still use every day, and Cory Benfield, Jonathan Lange, and Hynek Schlawack for many illuminating conversations over the years about the specifics and detailed mechanics of the tools whose use I describe in this post.

Any errors, of course, remain my own.

The Labyrinth is a children’s movie. The main character is 16 years old, and solving a logic puzzle that will literally decide if she lives or dies. In fiction, characters are faced with realistic challenges: ones they can solve, even if they have to make an effort.

So, it makes sense that the designer of the eponymous labyrinth did not consult logicians Richard Smullyan, George Boolos (no relation to the inventor of boolean algebra), and John McCarthy (yes, the same person who invented Lisp and suggested that a “2-month, 10-(person) study” would make significant headway in the study of Artificial Intelligence). Those three would suggest that the designer use The Hardest Logic Puzzle Ever.

There are persistent rumors of the movie getting a reboot, or perhaps a sequel. Like any good sequel, the protagonists should face newer and bigger challenges. In the interests of helping the screen writers for the sequel/reboot, here is my explanation of the “Hardest Logic Puzzle Ever”, together with clear code.

from __future__ import annotations

Do you remember movies from your childhood that just did not age that well? You cringe at some tasteless joke, you say “I can’t believe it was acceptable to show this back then”? I think we can agree we want to future-proof the script for a bit.

Using modern-style annotations will make our code much easier to maintain.

import attr

This movie will come out in the 2020s, and we should use 2020-era code to model it. The attrs library is almost always the right solution to implement classes.

import random

The protagonist will face unknown challenges. Randomness is a state of knowledge. From her perspective, the true state of the world is random.

from zope import interface
from typing import Callable, Mapping, Tuple

This is already a hard logic puzzle, no reason to make it harder on ourselves. Clear interface and type hints will make the logic easier to understand.

This is why zope.interface is appropriate for The Hardest Logic Puzzle Ever. We also need the Callable interface, since our protagonist will be asking the Gods questions in the form of functions, and Mapping, since she will eventually need to answer the question “which God is which”.

The Tuple type will most be used by the careful protagonist in her internal type hints. This is high stakes code, and she wants to make sure she gets it right.

In the HLPE (Hardest Logic Puzzle Ever), the ones who answer the questions are Gods. Just like in the Marvel Cinematic Universe, they might not be literal “Gods”, but clearly powerful aliens. As aliens, they have their own language. The words for “yes” and “no” are “da” and “ja” – but we do not know which is which.

I suggest that this will not be revealed in the script. This is good fodder for endless fan discussions later on Reddit. As such, the best way is to make sure we do not know the answer ourselves: make it a random language!

import enum

@enum.unique
class GodWords(enum.Enum):
    ja = "ja"
    da = "da"

def make_god_language() -> Mapping[bool, GodWords]:
    words = list(GodWords)
    random.shuffle(words)
    ret = {}
    return {True: words.pop(), False: words.pop()}

Shuffling the words for “yes” and “no” means that the .pop() call will get a random one. Now we have the language: it maps an abstract concept (a Python Boolean) to a string.

In the HLPE, there are three Gods, called “A”, “B”, and “C”. They can be asked any question that refers to the Gods.

@enum.unique
class GodNames(enum.Enum):
    A = "A"
    B = "B"
    C = "C"

class IGod(interface.Interface):
    def ask(question: Question) -> GodWords:
        """Ask"""

But what questions is the protagonist allowed to ask? Questions are something the audience-identification protagonist will ask. For this, a Protocol is appropriate. (Also, those are more convenient for describing functions, which we do not want to annotate with explicit implementation declarations.)

from typing_extensions import Protocol

class Question(Protocol):
    def __call__(self, gods: Dict[GodName, IGod]) -> bool:
        pass

Because of how annotations work, at this point we need not know what GodName or IGod are.

The simplest of the Gods is the one who speaks always truly (in the God language).

@interface.implementer(IGod)
@attr.s(auto_attribs=True, frozen=True)
class TrueGod:
    _gods: Mapping[GodNames, IGod]
    _language: Mapping[bool, GodWords]

    def ask(self, question: Question) -> GodWords:
        return self._language[bool(question(self._gods))]

The next God always lies.

@attr.s(auto_attribs=True, frozen=True)
class FalseGod:
    _gods: Mapping[GodNames, IGod]
    _language: Mapping[bool, GodWords]

    def ask(self, question: Question) -> GodWords:
        return self._language[not bool(question(self._gods))]

But how to implement Random? This is a harder question than it seems.

The original statement just said “whether Random speaks truly or falsely is a completely random matter”. What does that mean? If you ask it two questions, can it answer truthfully to one and lie to the other?

Boolos wrote a “clarification”: “Whether Random speaks truly or not should be thought of as depending on the flip of a coin hidden in (their) brain: if the coin comes down heads, (they) speaks truly; if tails, falsely.” This clarification fails to elucidate much: it does not answer, for example, the question above.

Finally, based on the suggested solution, and assuming that the obvious simpler solutions do not work, Raben and Raben suggested suggested the clear guideline: “Whether Random says ja or da should be thought of as depending on the flip of a coin hidden in his brain: if the coin comes down heads, he says ja; if tails, he says da.”

This is the guideline I have chosen to implement here.

@attr.s(auto_attribs=True, frozen=True)
class RandomGod:
    _gods: Mapping[GodNames, IGod]
    _language: Mapping[bool, GodWords]

    def ask(self, question: Question) -> GodWords:
        return self._language[random.choice([True, False])]

So much back and forth discussion, over two millenia, for such simple code.

I’m sure the God-like aliens would have used code to describe the puzzle from the beginning, avoiding the messiness of natural language.

Accessing the God classes themselves would be the height of hubris, but the goal of the protagonist is to answer “which God is which”. We will build a special enumeration of the Gods’ identities, to be used in the solution.

GodIdentities = enum.Enum('God Identities',
                          {klass.__name__: klass.__name__
                           for klass in [TrueGod, FalseGod, RandomGod]})

With the Gods’ personalities implemented, we can write the code that creates a random world that complies with the terms of the puzzle. Three Gods, known as “A”, “B”, and “C”, assigned to the names randomly, and speaking in a randomly generated language.

def make_gods() -> Mapping[GodNames, IGod]:
    language = make_god_language()
    gods = {}
    god_list = [klass(gods=gods, language=language)
                for klass in [TrueGod, FalseGod, RandomGod]]
    random.shuffle(god_list)
    for name in GodNames:
        gods[name] = god_list.pop()
    return gods

The code so far corresponds to the first part of the scene: where the protagonist comes to the place of the Gods, learns the local rules, and realizes that she must either solve the puzzle correctly, or fail.

This time she is granted the chance to ask three questions. However, there are many more unknowns: there are 6 possible assignments for the Gods, and two possible meanings for the language. Frankly, I doubt that I would be able to solve the puzzle on my feet, when I am afraid for my life.

But luckily, I have Wikipedia and time, and so I have written up the code to find a solution here.

Part of the solution is to ask a God a question about themselves: “if I asked you SOME QUESTION, would you say ja”. While no question asked of Random is useful (the answer is random) for either True or False, this question would result in “ja” the right answer is “yes”, and “da* if the right answer is”no". This means that wrapping questions like this means we have to care neither the identity of the God, nor about the details of the God language.

This makes it a useful abstraction!

def if_asked_a_question_would_you_say_ja_is_ja(
        god_name: GodNames,
        gods: Mapping[GodNames, IGod],
        question: Callable[[IGod, Mapping[GodNames, IGod]], bool],
    ) -> bool:
    you = gods[god_name]
    def add_you(gods):
        return question(you, gods)
    return you.ask(lambda gods: you.ask(add_you) == GodWords.ja) == GodWords.ja

This would be a wonderful chance for a flash-“forward” into a hypothetical scene: the movie goes into black-and-white, and the protagonist voice-overs: if “A” is “True”, what would happen if I ask them “is 1 equal 1”? What would happen if I ask them “if I asked you, ‘is 1 equal 1?’, would you answer ’ja? What if”A" is “False”?

def hypothetical():
    language = make_god_language()
    gods = {}
    hypothetical_true_god = TrueGod(gods=gods, language=language)
    hypothetical_false_god = FalseGod(gods=gods, language=language)
    gods[GodNames.A] = hypothetical_true_god
    gods[GodNames.B] = hypothetical_false_god
    for (name, identity) in [(GodNames.A, GodIdentities.TrueGod), (GodNames.B, GodIdentities.FalseGod)]:
        for question in [lambda x,y: 1==1, lambda x, y: 1==0]:
            objective_value = question(None, None)
            print(
                f"{identity}, asked {objective_value} question:",
                if_asked_a_question_would_you_say_ja_is_ja(name, gods, question)
            )
hypothetical()
del hypothetical

God Identities.TrueGod, asked True question: True
God Identities.TrueGod, asked False question: False
God Identities.FalseGod, asked True question: True
God Identities.FalseGod, asked False question: False

Movie goes back to color. A smile spreads on the protagonist’s face. She knows how to solve this.

The next challenge is to find a God that is not Random. If you ask God B about whether A is Random, then if the answer is “ja”, then it means either the correct answer is that A is Random or it means B is Random. Either way, C is not Random.

For similar reasons, if B answers “da”, then “A” is not Random.

Either way, we have found a non-Random God. Now we know that we can find the truth from them by using the wrapper!

So we ask them whether they’re True. Now we ask them about whether “B” is Random.

So we know:

• One God who is not Random (who we will call “the interlocutor”, since we’ll spend the rest of the conversation with them)
• Whether the interlocutor is True
• Whether B is Random

def ask_questions(gods: Mapping[GodNames, IGod]) -> Tuple[GodNames, bool, bool]:
    is_a_random_according_to_b = if_asked_a_question_would_you_say_ja_is_ja(
        GodNames.B,
        gods,
        lambda you, gods: isinstance(gods[GodNames.A], RandomGod)
    )
    interlocutor = GodNames.C if is_a_random_according_to_b else GodNames.A
    is_interlocutor_true = if_asked_a_question_would_you_say_ja_is_ja(
        interlocutor,
        gods,
        lambda you, gods: isinstance(you, TrueGod)
    )
    is_b_random = if_asked_a_question_would_you_say_ja_is_ja(
        interlocutor,
        gods,
        lambda you, gods: isinstance(gods[GodNames.B], RandomGod)
    )
    return interlocutor, is_interlocutor_true, is_b_random

Once again, the movie goes into a black and white flash-forward as the protagonist plans her move.

def hypothetical():
    for experiment in range(1, 5):
        print("Experiment", experiment)
        gods = make_gods()
        interlocutor, is_interlocutor_true, is_b_random = ask_questions(gods)
        print(f"I think {interlocutor} is {is_interlocutor_true}God. They're {type(gods[interlocutor]).__name__}.")
        print(f"B is {'' if is_b_random else 'not '}RandomGod. They're {type(gods[GodNames.B]).__name__}.")
hypothetical()
del hypothetical

Experiment 1
I think GodNames.C is FalseGod. They're FalseGod.
B is not RandomGod. They're TrueGod.
Experiment 2
I think GodNames.A is FalseGod. They're FalseGod.
B is not RandomGod. They're TrueGod.
Experiment 3
I think GodNames.A is TrueGod. They're TrueGod.
B is RandomGod. They're RandomGod.
Experiment 4
I think GodNames.A is FalseGod. They're FalseGod.
B is not RandomGod. They're TrueGod.

Now that we know the answers, it is time to put them all together. We first record whether the interlocutor is True or False. If B is Random, we mark them as such. If not, we know that neither the interlocutor or B is Random, so the other God must be Random.

Now that we know two Gods’ identities, the last name that remains belongs to the last possible identity.

def analyze_answers(interlocutor: GodNames, is_interlocutor_true: bool, is_b_random: bool):
    solution = {}
    solution[interlocutor] = (GodIdentities.TrueGod if is_interlocutor_true
                              else GodIdentities.FalseGod)
    [other_god] = set(GodNames) - set([interlocutor, GodNames.B])
    random_god = GodNames.B if is_b_random else other_god
    solution[random_god] = GodIdentities.RandomGod
    [last_god] = set(GodNames) - set(solution.keys())
    [last_god_value] = set(GodIdentities) - set(solution.values())
    solution[last_god] = last_god_value
    return solution

Putting the questioning and the analysis together is straightforward.

def find_solution(gods: Mapping[GodNames, IGod]) -> Mapping[GodNames, GodIdentities]:
    interlocutor, is_interlocutor_true, is_b_random = ask_questions(gods)
    return analyze_answers(interlocutor, is_interlocutor_true, is_b_random)

Our little checker returns both a description of the situation, as well as whether the solution was correct.

def check_solution() -> Tuple[Mapping[str, str], Mapping[str, str], bool]:
    gods = make_gods()
    solution = find_solution(gods)
    reality = sorted([(name.value, type(god).__name__) for name, god in gods.items()])
    deduced = sorted([(name.value, identity.value) for name, identity in solution.items()])
    return reality, deduced, reality == deduced

The last step is to test our solution multiple times. Again, remember that there are 6 * 2 = 12 possible situations. However, if “B” is Random, we can get two different answers. This means the total number of options for a path is more than 12, but less than 24.

If we run the solution for a 1000 times, the probability that a given path will not be taken is less than (23/24)**1000 * 24. How much is it?

(23/24)**1000 * 24

7.885069901070409e-18

Good enough!

Now we can see if the script works. We lack Hollywood’s professional script doctors, but we do have a powerful Python interpreter.

for i in range(1000):
    reality, deduced, correct = check_solution()
    if not correct:
        raise ValueError("Solution is incorrect", reality, deduced)
    if i % 200 == 0:
        print("Solved correctly for", reality)

Solved correctly for [('A', 'RandomGod'), ('B', 'TrueGod'), ('C', 'FalseGod')]
Solved correctly for [('A', 'RandomGod'), ('B', 'FalseGod'), ('C', 'TrueGod')]
Solved correctly for [('A', 'RandomGod'), ('B', 'TrueGod'), ('C', 'FalseGod')]
Solved correctly for [('A', 'TrueGod'), ('B', 'FalseGod'), ('C', 'RandomGod')]
Solved correctly for [('A', 'TrueGod'), ('B', 'RandomGod'), ('C', 'FalseGod')]

We printed out every 200th situation, to have some nice output!

Python confirms it. Hollywood should buy our script.

Thanks to Glyph Lefkowitz for his feedback on the Labyrinth post, some of which inspired changes in this post. Thanks to Mark Williams for his feedback on an early draft. Any mistakes or issues that remain are my responsibility.

Get it?
Quack quack quack quack
Quack quack quack quack

Weird Al Yancovic,
“I Want A New Duck”

Mypy makes most things better

Mypy is a static type checker for Python. If you’re not already familiar, you should check it out; it’s rapidly becoming a standard for Python projects. All the cool kids are doing it. With Mypy, you get all the benefits of high-level dynamic typing for rapid experimentation, and all the benefits of rigorous type checking to complement your tests and improve reliability.¹ The best of both worlds!

Mypy can change how you write Python code. In most cases, this is for the better. For example, I have opined on numerous occasions about how bad None is. But this can significantly change with Mypy. Now, when you return None, you can say -> Optional[str] and rest assured that all your callers will be quickly, statically checked for places where they might encounter an AttributeError on that None, which makes this a more appealing option than risking raising a runtime exception (which Mypy can’t check).

But sometimes, things can get worse

But in some cases, as you add type annotations, they can make your code more brittle, especially if you annotate with the most initially obvious types. Which is to say, you define some custom classes, and then say that the parameters to and return values from your functions and methods are simply instances of those classes you just defined.

Most Mypy tutorials give you a bunch of examples with str, int, List[int], maybe an Optional[float] or two, and then leave you to your own devices when it comes to defining your own classes; yet, huge amounts of real-world applications are custom classes.

So if you’re new to Mypy, particularly if you’re applying it to a large existing codebase, it’s quite natural to write a little code like this:

1
2
3
4
5
6

from dataclasses import dataclass
@dataclass
class Duck:
    quiet: bool = False
    def quack(self) -> None:
        print("Quack." if self.quiet else "QUACK!")

and then, later, write some code like this:

1
2
3
4

def duck_war(aggressor: Duck, defender: Duck) -> None:
    aggressor.quack()
    defender.quack()
    print("The only winning move is not to play.")

In untyped, pre-Mypy python, in addition to being a poignant message about the futility of escalating violence, duck_war is a very flexible function, regardless of where it’s defined. It can take anything with a quack() method.

But, while the strictness of the Mypy type-check here brings a level of safety — no None accidentally masquerading as a Duck here — it also adds a level of brittleness. Tests which use carefully-constructed fakes will now fail to type check, because duck_war technically insists upon only precisely instances of Duck and nothing else.

A few sub-optimal answers to this question

So when you want something else that has slightly different behavior — when you want a new Duck², so to speak — what do you do?

There are a couple of anti-patterns you might arrive at to work around this as you begin your Mypy journey:

1. add # type: ignore comments to all your tests, or remove their type signatures so they don’t get type checked. This solution throws the baby out with the bath water, as it eliminates any safety that any of these callers experience.
2. add calls to cast(Duck, ...) around any things which you know are “enough like” a Duck for your purposes. This is much more fine-grained and targeted (and can be a great hack for working with libraries who provide type stubs which are too specific) but this also trades off a bit too much safety, since nothing at the point of the cast verifies anything unless you build your own ad-hoc system to do so.
3. subclassing Duck. This can also be expedient, and not too bad if you have to; Mypy removes some of the sharpest edges from Python inheritance, providing some guard rails around overriding methods, but it remains a bad idea for all the usual reasons.

The good answer: typing.Protocol

Mypy has a feature, typing.Protocol , that provides a straightforward way to describe any object that has a quack() method.

You can do this like so:

1
2
3
4
5

from typing import Protocol

class Ducky(Protocol):
    def quack(self) -> None:
        "Quack."

Now, with only a small modification to its signature — while leaving the implementation the same — duck_war can now support anything sufficiently duck-like:

1
2

def duck_war(aggressor: Ducky, defender: Ducky) -> None:
    ...

In addition to making it possible for other code — for example, a unit test — to pass its own implementation of Ducky into duck_war without subclassing or tricking the type checker, this change also improves the safety of duck_war’s implementation itself. Previously, when it took a Duck, it would have been equally valid for duck_war to access the .quiet attribute of Duck as it would have been to access .quack, even though .quiet is ostensibly an internal implementation detail.

Now, we could add an underscore prefix to quiet to make it “private”, but the type checker will still happily let you access it. So a Protocol allows you to clearly reveal your intention about what types you expect your arguments to be.

Why isn’t everything like this?

Unfortunately, typing.Protocol began its life as typing_extensions.Protocol: a custom extended feature of the type system that wasn’t present in Mypy initially, and isn’t in the standard library until Python 3.8. Built-in types like Iterable and Sequence are type-checked as if they’re Protocols by being slightly special within Mypy, but it’s not clear to the casual user how this is happening.

However, other types, like io.TextIO, don’t quite behave this way, and some early-adopter projects for Mypy have types that are either too strict or too permissive because they predated this.

So I really wanted to write this post to highlight the Protocol style of describing types and encourage folks to use it.

In conclusion

The concept I’ve described above is not new in the world of type theory.

The way that typing works in Mypy with most types — builtins, custom classes, and abstract base classes — is known as nominal typing. Nominal as in “based on names”; if the object you have directly references the name of the type it’s being compared to, by being an instance of it, then it matches.

In other words: if it’s named “Duck”, it’s a duck. There are some advantages to nominal typing³, but this brittleness is not very Pythonic!

In contrast, the type of type-checking accomplished by Protocol is known as structural typing.⁴ Whether the caller matches a Protocol depends on the structure of your object — in other words, what methods and attributes it has.

In even other-er words - if it .quack()s like a duck, it is a duck.

If you’re just starting to use Mypy — particularly if you’re building a library that exports types that users are expected to implement — consider using Protocol to describe those types. With Protocol, while you get much-improved safety from type-checking, you don’t lose the wonderful flexibility and easy testability that duck typing has always given you in Python.

GitHub’s own CI called GitHub Actions has been out of closed beta for a while and offers generous free quotas and a seamless integration with the rest of the site. Let’s have a look at how to use it for an open source Python package.

The choice in parameters for back-off configuration is important. It can be the difference between a barely noticable blip in service quality and an hours-long site outage. In order to explore the consequences of the choice, I wrote a little fictional ditty about a fictional website.

I hope you enjoy escaping into this fictional reality as much as I enjoyed writing about it.

Your recipe site is different. After all, recipe sites are a dime-a-dozen. With today's modern technology, any kid can put a quick mock-up together with Django, React, and MongoDB to store recipes and retrieve them by various attributes.

In order to make your recipe site stand above the rest, you made sure it uses really cutting edge techniques. From details of the web requests coming in, using sophisiticated language parsing and machine learning algorithm, with just a few words about the user's likes and dislikes, you find the perfect recipe just for them.

HackerNews called it "just a bunch of buzzwords", of course. But once the graphs went up into the right, with 50% month-over-month growth rates, everyone explained that they knew that this one was different. Popularity sky-rocketed, the engineers worked on scaling up the site, and though it was not the world's most sophisticated microservice architecture, it was medium-service architecture, at least.

The web front end would call the machine learning cluster, running on special GPU machines, to get the appropriate keywords by which to look up the recipe. Maybe not a the kind of 50-microservices-architecture that takes three whiteboards to explain, but at least it was easy enough to scale up horizontally. You hired a great Site Reliability Engineer, who built a sophisticated continuous delivery machine. As your machine learning team fine-tuned the model, it would slowly roll out into the cluster, running continous A/B tests that would immediately roll back the change if the model performed worse than before.

The SRE also made sure the web front-end would back off of a malfunctioning machine learning node, as those do sometimes. Instead of immediately reconnecting, it would try reconnecting at intervals starting at 5 milliseconds and increasing by powers of 2: 10ms, 20ms, and so on.

Amazing.

You will never forget what happened next. A new model was rolled out. It looked great. Performed wonderfully. The only problem was that it had a hidden time-bomb. The model had a small overfit that happend to match the sum of the user's ID and the time. This, in itself, is the kind of overfit that happens every day. Unfortunately, if that value exceeded a threshold, it triggered a bug in the machine learning library. When that happened, the node would become flakey, and start dropping random connections. Even worse, for the specific input used in the health checks, it would work: all nodes appeared healthy in the monitoring dashboard, and to the service discovery framework.

A nightmare scenario.

One by one, nodes started dropping off, as users with higher IDs connected and the seconds ticked on mercilessly. The front end started backing off: by 5ms, then 10ms, then 20ms.

The exponential function explodes fast, so of course the backoff (with jitter, as is common practice) had an upper limit. When the Site Reliability Engineer set the upper limit, there was a lot of things on their mind. Decisions are hard to make. Sometimes, one neuron firing makes all the difference. But neurons are small, and electrons are smaller.

When we look at the firing of a neuron, we can no longer imagine the world is as Maxwell and Newton imagined it to be. Quantum effects must be taken into account. As per the obviously correct Many-Worlds interpretation of Quantum Mechanics, the one world had split into two that would never interact again.

In one world, the SRE had set the maximum to 1 second. In the other, to 10 minutes. For a while, these worlds seemed to parallel each other closely: sure, a few bits on in a YAML file were different, but what storm could come from such a small butterfly?

It was not until the disasterous model roll-out that the worlds would diverge wildly. In the world where the limit was set to one second, the front ends were hammering the nodes mercilessly. Rolling back the model would mean that a node would start getting too many connections, and before a few seconds had passed, it would fall back down.

Eventually, in this world, the whole front-end had to be brought down, the model fully rolled back, and only then the front-end brought back up. This was too bad, because the front-end had a bad-but-working model built-in, and some recipes would still be found while the "outage" was happening. Sure, they were not the "wow you read my mind" quality, but they were still decent results.

Bringing down the whole front-end cluster, and bringing it up again, turned the outage from a blip in recipe click-through rates into a three-hour, news-coverage-worthy outage.

In the other world, where the back-off had a maximum of ten minutes, things progressed much more smoothly. As a machine learning node's model was rolled back, it came back up, and machines started connecting to it. The first one to be brought up did go down from connection overload -- but took ten minutes to do so, enough time to ascertain that this solution was correct. A few more nodes were rolled back, and as the connections grew, the roll back flowed through the fleet.

The outage has been managed, and other than a few irate customers posting on Twitter about looking for vegan recipes and getting a "meat lover's delight", the outage mostly went unnoticed.

Back-off is important. Exponential back-off with jitter and a maximum is almost always the right solution. Yes, the exponent matters a little. The initial back off matters a little. But the maximum matters a lot. Set the maximum to "human time frames": a few minutes (1-30) is a good balance.

Even in a big cluster, an extra action every 10 minutes will probably not cause serious downstream repercussions. But even the most entitled customer can be mollified by a support technician for five minutes doing "everything possible to find a solution" while waiting the clock out on the 10 minute back-off clock.

Choose maximums for back-off carefully, or the site you bring down may be your own.

Thanks to Avy Faingezicht for his feedback on an earlier draft. Any mistakes or issues that remain are my reponsibility.

There should be one — and preferably only one — obvious way to do it.

Tim Peters, “The Zen of Python”

Moshe wrote a blog post a couple of days ago which neatly constructs a wonderful little coding example from a scene in a movie. And, as we know from the Zen of Python quote, there should only be one obvious way to do something in Python. So my initial reaction to his post was of course to do it differently — to replace an __init__ method with the new @dataclasses.dataclass decorator.

But as I thought about the code example more, I realized there are a number of things beyond just dataclasses that make the difference between “toy”, example-quality Python, and what you’d do in a modern, professional, production codebase today.

So let’s do everything the second, not-obvious way!

There’s more than one way to do it

Larry Wall, “The Other Zen of Python”

Getting started: the __future__ is now

We will want to use type annotations. But, the Guard and his friend are very self-referential, and will have lots of annotations that reference things that come later in the file. So we’ll want to take advantage of a future feature of Python, which is to say, Postponed Evaluation of Annotations. In addition to the benefit of slightly improving our import time, it’ll let us use the nice type annotation syntax without any ugly quoting, even when we need to make forward references.

So, to begin:

1

from __future__ import annotations

Doors: safe sets of constants

Next, let’s tackle the concept of “doors”. We don’t need to gold-plate this with a full blown Door class with instances and methods - doors don’t have any behavior or state in this example, and we don’t need to add it. But, we still wouldn’t want anyone using using this library to mix up a door or accidentally plunge to their doom by accidentally passing "certian death" when they meant certain. So a Door clearly needs a type of its own, which is to say, an Enum:

1
2
3
4
5

from enum import Enum

class Door(Enum):
    certain_death = "certain death"
    castle = "castle"

Questions: describing type interfaces

Next up, what is a “question”? Guards expect a very specific sort of value as their question argument and we if we’re using type annotations, we should specify what it is. We want a Question type that defines arguments for each part of the universe of knowledge that these guards understand. This includes who they are themselves, who the set of both guards are, and what the doors are.

We can specify it like so:

1
2
3
4
5
6
7

from typing import Protocol, Sequence

class Question(Protocol):
    def __call__(
        self, guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        ...

The most flexible way to define a type of thing you can call using mypy and typing is to define a Protocol with a __call__ method and nothing else¹. We could also describe this type as Question = Callable[[Guard, Sequence[Guard], Door], bool] instead, but as you may be able to infer, that doesn’t let you easily specify names of arguments, or keyword-only or positional-only arguments, or required default values. So Protocol-with-__call__ it is.

At this point, we also get to consider; does the questioner need the ability to change the collection of doors they’re passed? Probably not; they’re just asking questions, not giving commands. So they should receive an immutable version, which means we need to import Sequence from the typing module and not List, and use that for both guards and doors argument types.

Guards and questions: annotating existing logic with types

Next up, what does Guard look like now? Aside from adding some type annotations — and using our shiny new Door and Question types — it looks substantially similar to Moshe’s version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from dataclasses import dataclass

@dataclass
class Guard:
    _truth_teller: bool
    _guards: Sequence[Guard]
    _doors: Sequence[Door]

    def ask(self, question: Question) -> bool:
        answer = question(self, self._guards, self._doors)
        if not self._truth_teller:
            answer = not answer
        return answer

Similarly, the question that we want to ask looks quite similar, with the addition of:

1. type annotations for both the “outer” and the “inner” question, and
2. using Door.castle for our comparison rather than the string "castle"
3. replacing List with Sequence, as discussed above, since the guards in this puzzle also have no power to change their environment, only to answer questions.
4. using the [var] = value syntax for destructuring bind, rather than the more subtle var, = value form

1
2
3
4
5
6
7
8
9

def question(guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]) -> bool:
    [other_guard] = (candidate for candidate in guards if candidate != guard)

    def other_question(
        guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        return doors[0] == Door.castle

    return other_guard.ask(other_question)

Eliminating global state: building the guard post

Next up, how shall we initialize this collection of guards? Setting a couple of global variables is never good style, so let’s encapsulate this within a function:

1
2
3
4
5
6
7

from typing import List

def make_guard_post() -> Sequence[Guard]:
    doors = list(Door)
    guards: List[Guard] = []
    guards[:] = [Guard(True, guards, doors), Guard(False, guards, doors)]
    return guards

Defining the main point

And finally, how shall we actually have this execute? First, let’s put this in a function, so that it can be called by things other than running the script directly; for example, if we want to use entry_points to expose this as a script. Then, let's put it in a "__main__" block, and not just execute it at module scope.

Secondly, rather than inspecting the output of each one at a time, let’s use the all function to express that the interesting thing is that all of the guards will answer the question in the affirmative:

1
2
3
4
5
6

def main() -> None:
    print(all(each.ask(question) for each in make_guard_post()))


if __name__ == "__main__":
    main()

Appendix: the full code

To sum up, here’s the full version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

from __future__ import annotations
from dataclasses import dataclass
from typing import List, Protocol, Sequence
from enum import Enum


class Door(Enum):
    certain_death = "certain death"
    castle = "castle"


class Question(Protocol):
    def __call__(
        self, guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        ...


@dataclass
class Guard:
    _truth_teller: bool
    _guards: Sequence[Guard]
    _doors: Sequence[Door]

    def ask(self, question: Question) -> bool:
        answer = question(self, self._guards, self._doors)
        if not self._truth_teller:
            answer = not answer
        return answer


def question(guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]) -> bool:
    [other_guard] = (candidate for candidate in guards if candidate != guard)

    def other_question(
        guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        return doors[0] == Door.castle

    return other_guard.ask(other_question)


def make_guard_post() -> Sequence[Guard]:
    doors = list(Door)
    guards: List[Guard] = []
    guards[:] = [Guard(True, guards, doors), Guard(False, guards, doors)]
    return guards


def main() -> None:
    print(all(each.ask(question) for each in make_guard_post()))


if __name__ == "__main__":
    main()

Acknowledgments

I’d like to thank Moshe Zadka for the post that inspired this, as well as Nelson Elhage, Jonathan Lange, Ben Bangert and Alex Gaynor for giving feedback on drafts of this post.

In the 1986 movie Labyrinth, a young girl (played by Jennifer Connelly) is faced with a dilemma. The adorable Jim Henson puppets explain to her that one guard always lies, and one guard always tells the truth. She needs to figure out which door leads to the castle at the center of the eponymous Labyrinth, and which one to certain death (dun-dun-dun!).

I decided that like any reasonable movie watcher, I need to implement this in Python.

First, I implemented two guards: one who always tells the truth, and one who always lies. The guards know who they are, and what the doors are, but can only answer True or False.

import dataclasses
from typing import List

guards = [None, None]
doors = ["certain death", "castle"]

@dataclasses.dataclass
class Guard:
    _truth_teller: bool
    _guards: List
    _doors: List[str]

    def ask(self, question):
        answer = bool(question(self, self._guards, self._doors))
        if not self._truth_teller:
            answer = not answer
        return answer

guards[0] = Guard(True, guards, doors)
guards[1] = Guard(False, guards, doors)

This being a children’s movie, the girl defeats all odds and figures out what to ask the guard: “would he (points to the other guard) tell me that this (points to the door on the left) door leads to the castle?”

def question(guard, guards, doors):
    other_guard, = (candidate for candidate in guards if candidate != guard)
    def other_question(ignored, guards, doors):
        return doors[0] == "castle"
    return other_guard.ask(other_question)

What would the truth-teller answer?

guards[0].ask(question)

True

And the liar?

guards[1].ask(question)

True

No matter who she asks, now she can count on a lie. After a short exposition, she confidently walks through the other door. It’s a piece of cake!

Thanks to Rina Arstain and Veronica Hanus for their feedback on an earlier draft. Thanks to Glyph Lefkowitz for the idea to use dataclasses. All mistakes and issues that remain are my responsibility.

Some related work in the field of formalizing logic puzzles:

• Glyph wrote a response post about getting the code to be professional-level Python.
• Hillel Wayne wrote about how to solve logic puzzles with Alloy.

How do you setup your dev environment? Depending on your language there are many choices of editor, package manager, build tool, linter, on and on. And every article you find will have a different combination of suggested tools, each of which claiming that their list is The Right Way To Do Things.

So which do you choose?

The short answer: it doesn’t matter. Your choice of dev environment is meaningless.

The slightly less flippant answer is that, yes, there are some contraints on which tools you should pick, but otherwise you should just pick something and move on.

Let’s see why dev environments don’t matter that much in the end, and what limited constraints you should apply when choosing your tools.

Learning how to cook

Imagine you’re training to become a chef. You will need to learn how to use a knife correctly, to chop and dice safely and quickly.

And yes, you need a sharp knife. But when you’re starting out, it doesn’t matter which knife you use: just pick something sharp and good enough, and move on. After all, the knife is just a tool.

The people eating the food you cook don’t care about which knife you used: they care how the food tastes and looks.

After six months in the kitchen, you’ll start understanding how you personally use a knife, what cuisines you want to pursue, what techniques you want to vary. And then you’ll have the knowledge to pick a specific knife or knives exactly suited to your needs.

But remember: the people eating your food still won’t care which knife you used.

Choosing a dev environment

When you use a website, you don’t care which build tool the programmer used. When you run an app, you don’t care which editor they used. You want the software to work, to do what it says, to be easy to use, to get out of your way—and you don’t care how they did it.

And that applies just as much to the users of your code: they don’t care which tools you used.

And when you’re starting out, whether programming in general or a new language or framework, you don’t know how you will like to work. So instead of obsessing over finding the ideal development environment and toolchain, just pick tools that are good enough:

• Popular: So you can easily find help.
• Easy to get going: Your goal is ship useful code, and as a beginner time spent fiddling with your dev environment won’t help with that.

Once you have enough experience, you will start developing opinions. You might become choosy about which tools you use, or end up customizing them to your needs. You might even write an article about your particular dev environment and favorite tools.

But however strong your preferences are, chances are that given tools you don’t quite like as much, you will still do just fine. If you know what you’re doing, you can chop vegetables with any sharp knife, even if it’s not your favorite.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

(I have shown this technique in my mailing list. If this kind of thing seems interesting, why not subscribe?)

Imagine you want to log something that is, potentially, expensive to calculate. For example, in DEBUG mode, you would like to count the classes of the objects in gc.get_objects() and log those counts: this is often a useful technique for diagnosing reference leaks. This is pretty heavy to calculate, and logging it always sounds wasteful.

The logging module has lazy interpolation.

A line like

logging.debug("total number: %r", 2500)

will not bother calculating the string interpolation total number: 2500 unless the logging level is set to output it.

However, calling logging.debug("total number: %r", get_object_counts()) will still call get_object_counts() regardless. It is not the cost of calculating the interpolation, but that of calculating the counts themselves that you would like to avoid.

One way to piggy-back on the lazy interpolation to do lazy evaluation is to write a custom class with a lazy __repr__():

class OnDemand:
    def __init__(self, callable):
        self.callable = callable
    def __repr__(self):
        return repr(self.callable())

This defines a class, OnDemand, which will only call the function when it needs its repr.

This allows us to write code like:

logging.debug("total number: %r", OnDemand(get_object_counts))`

Now, get_object_counts() will not be called at all: notice that the logging line does not call it. This makes the lazy evaluation explicit: we explicitly delay evaluation with OnDemand.

Here is a simple example of how it would work:

>>> def get_object_counts():
...     print("doing something expensive")
...     return 5
...
>>> debug_logging.debug("result is %r", OnDemand(get_object_counts))
doing something expensive
DEBUG:debug:result is 5
>>> warnonly_logging.debug("result is %r", OnDemand(get_object_counts))
>>> warnonly_logging.debug("result is %r", get_object_counts())
doing something expensive

Note that when the warnonly_logging is called with a .debug() method, the expensive calculation is not done: not only is the log message ignored, but the function that calculates the value is not done.

The last line shows that without the careful usage of OnDemand, the expensive calculation is done.

This should make it easy to sprinkle heavy calculations in logging.debug statements. Coupled with easy ways to trigger logging level changes (which are beyond the scope of this article) this gives a powerful way to get insight into your program's innards.

(Thanks to Adi Stav, Avy Faingezicht, Chris Withers, Dave Briccetti, and Lucas Wiman on their feedback on an early draft of this article. Any mistakes or issues that remain are my responsibility.)

When you’re looking for a new programming job, how do you explain your value? The usual approach is a long list of technologies, but this leaves out a critical skill: your ability to solve problems.

If you can convey your level of skill at problem solving, you can get:

• More job offers.
• Jobs with technologies you don’t know.
• A higher salary by getting slotted into a higher pay grade.

Often just a few extra words can make a big difference in demonstrating your skills. Let’s see how you can do it.

Three levels of problem solving skills

As I discussed elsewhere in more detail, problem solving comes in three stages, each approximately corresponding to a particular career stage (these names are due to Randall Koutnik):

1. Staff or principal software engineers are Finders: they find new problems.
2. Senior software engineers are Solvers: they solve already-identified problems.
3. Junior software engineers are Implementers: they implement already-identified solutions.

The earlier you are in the problem-solving process, the more productive you are, and therefore the more valuable as an employee.

As a result, you need to communicate how advanced your skill is across these three levels to demonstrate your productivity. Everything from your resume to the stories you tell in interviews should communicate your level of skill.

Explaining your skill level

Explaining your skill level involves telling stories that use the correct words and sufficient information to demonstrate your skill. I’m going to use resumes as an example here, but you should ensure you do this in interviews as well—if you’re practicing with a friend, make sure they’re checking for this, it’s easy to leave the information out.

Consider the following entry from a resume:

Moved deployment from manually-managed hosts to a new Kubernetes cluster.

This experience entry uses an implementation-level verb: “moved”. Similarly, “coded”, “tested”, “wrote”, “fixed”, “optimized"—these are all about implementation. And maybe it’s implementation was tricky and difficult, and it’s good to convey that, but if you also solved the problem or identified the problem it’s impossible to tell from this phrasing.

Any task you write about in your resume and talk about in interviews was the result of someone identifying a problem, and someone coming up with the solution. If it was you, make sure you say that.

Let’s say in our example above you were the one tasked with figuring out an alternative to manually-managed hosts. If so, you need to add additional context and verbs that convey that:

Investigated alternatives to manually-managed hosts, decided on Kubernetes, and moved the deployment to a new cluster.

Now we can clearly see you solved the problem.

If you were the one who identified the problem, again you need to make sure you explicitly call that out:

Identified manually-managed hosts as an operational problem, and got management buy-in to change to a better system. Subsequently investigated alternatives, decided on Kubernetes, and moved the deployment to a new cluster.

Now we can see all the value you provided.

If you only identified a problem, that’s fine too, just say so:

Noticed a critical customer-facing bug that was impacting many users; after the team responsible for that area fixed it, they reported a $300,000 increase in revenue as a result.

Communicating value at different career stages

Now that you’ve seen how to phrase your skill level, let’s see how this works at different career stages.

Implementers

When you’re at the start of your career you will mostly be implementing other people’s solutions. However, in one or two cases you might be starting to solve problems, or even find problems. Make sure your resume highlights those instances, however small.

Sometimes this will happen in non-programming contexts. For example, I knew one early stage software engineer who made very insightful suggestions about hiring. Mention it anyway.

If you had another career before switching to programming, you’ll likely have plenty of examples. Make sure to highlight those even if they’re unrelated to coding; those problem-finding and solving skills will at least partially transfer.

Solvers

If you can solve problems on your own, you want to both:

1. Communicate this fact.
2. Highlight the places where you did identify problems, even if it’s happened in only a few cases.

Review your resume and make sure all the relevant entries explicitly talk about the ways in which you came up with the solution. If you already have a suitable job title, like senior software engineer, then getting the phrasing right isn’t quite as important—but it’s still worth doing.

If you still have a junior job title but your skills have progressed, it’s doubly important to ensure you’re highlighting your problems solving skills.

Once you’ve done that, try to expand on any places where you were involved in identifying problems.

Finders

Your goal as a Finder is to ensure you’re not confused with a Solver. It’s very easy to phrase things in a way that doesn’t make clear you identified the problem—after all, identifying the problem may only have a taken a few minutes.

But those initial few minutes where you noticed something needs to be done are quite often the most value parts of the whole process, so make sure you explicitly talk about finding the problem. This is even more important if your current job title doesn’t reflect your actual level of skill.

What to do next

Even if submitting resumes isn’t the best way to find a job, you still need one, and writing it is a good way to rehearse for an interview.

So get your resume out, and for each experience entry make sure it’s clear whether you implemented the solution, solved the problem, and/or found the problem. This will take you an hour, no more, and at the end of this process you’ll have a much easier time communicating some of your most valuable non-technological skills.

And if you’d like to learn some more of those skills, check out my new book, The Secret Skills of Productive Programmers.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

Another week has passed, and another 3 million people in the US have filed for unemployment. While the current situation hasn’t impacted programming jobs quite as much, it’s just a matter of time before the economic damage hits most everywhere. There will be layoffs, and plenty of them, and occasionally whole companies shutting down.

So even if your job is secure now, you might still lose it in the future. How can you prepare? What can you do to reduce your future risks?

The first thing you need to do is come up with a plan, which is what this article is all about. In particular, you will want to:

• Try to make sure you have the necessary financial resources.
• Make your future job hunt easier, by building a network, making sure your skills are up-to-date, and making sure you have visible public proof of your skills.
• Come up with a series of fallback plans if things don’t go well.

Let’s go over these one-by-one.

Money in the bank

If you lose your job, you lose your paycheck—but you still have to pay your bills. And after the dot-com bust, the last big tech recession, it took years for all the jobs to come back

If you have at least six months of living expenses in cash, that’s a good start. If not, it’s best to think about how to get there.

There are two sides to this:

1. If possible, you need to cut your expenses, which will both allow you to save and reduce how much money you need for each unsalaried month. See this more detailed article.
2. Ensuring your financial assets, if you have any, aren’t correlated with your job.
   1. If you own stock in your own company, you are making a double bet: if the company goes down, you will lose money and your job.
   2. If you work for a startup that needs to raise money soon, a crashing stock market will also greatly reduce the viability of your current job.
   3. More broadly, if you own stocks and to a lesser extent corporate bonds, how correlated are they with your ability to keep a job?
   4. Even more broadly, how much of your net worth is tied to the tech industry, or the economy as a whole?

In short, you want cash on hand, and plenty of it.

Making your future job hunt easier

Searching for a job will be much easier if you:

• Know lots of people.
• Have useful skills.
• Can visibly demonstrate you have those skills.

Let’s cover those one by one.

Knowing lots of people

Applying for a job by sending in your resume is the hardest way to get hired. It’s much easier if you know someone who can vouch for you, can get you past the initial screen, or can fill you in on what the hiring manager really wants.

So the more people you know, the better off you are. Elsewhere I have a guest post about (social) networking, but that can take time and is harder during a pandemic. But there are still a few easy things you can do in the short term:

• Join a public Slack or two for the technology area you specialize in. You can help answer people’s questions, see when people mention they’re hiring, and more broadly get a better sense of the zeitgeist, which is useful for building your skills (see below).
• Keep the contact info for former co-workers. This can be done via LinkedIn, for example, and often there will be an ex-employee Slack. If there isn’t one, you can start it—especially if your company is having initial rounds of layoffs. This too can often be very educational, as former employees might be more forthcoming.
• Find ways to help other people. Can you teach useful skills? Join a local mutual aid organization?

Build useful skills

If you’ve been working at the same job for a while, it’s easy for your technical skills to get a little stale. Unless you’re working at the right place, hang out with the right people, or do the right things, you might not be aware of the latest technology, or you might be using out-of-date practices.

So you’ll want to update your skills a little. As always, doing this extensively outside of your job may not be possible, so try to:

1. Spend an hour a week, ideally during work hours, getting up-to-date on the latest technologies. The goal here is breadth, not depth: sign up for a newsletter for your technology stack (he’s a partial list), skim the topics at a relevant conference, maybe watch a talk or two. I cover learning for breadth here, but the basic idea is that knowing a tool exists and what it does can take very little time, and is quite valuable on its own: both on the job, but also in interviews (“I haven’t used it myself, but I believe tool X is how you would solve this”).
2. Try to learn more technologies on the job, because that is the best place to do so.

Create visible proof of skills

Having skills is one thing, proving you have them is another. It is therefore quite useful during a job hunt to have some visible, public proof you have these skills. For example:

Open source: When I moved to the US in my 20s, my work on an open source project made it much easier for me to get job interviews, and eventually job offers. It wasn’t just that my resume said that I knew computer networking, I could point to a publicly available project used by real people and say “I worked on that”.

Even if you share code that isn’t widely used, it can still be useful as proof of skill.

Conference talks: Speaking publicly about a particular skill, technology, or project is a great way to get public proof of skills. With conferences moving online, speaking at conferences is now much easier. You don’t have to travel or pay for you travel, and you don’t have to get approval from your manager to lose work. If there’s a topic where you know enough to help someone else, look for conferences on the topic and submit a proposal.

Blogging: Have something to share, or learning something new? Write it down and share it publicly. Writing well is an immensely useful skill in general, so this will also count as improving your skills. You can write for your own blog, or you can propose a blog post on your company’s tech blog, if they have one.

Fallback plans

In an ideal world you would lose your job, start a job search, find a new job within a month, and everything will be fine. Sadly we don’t always live in an ideal world.

So if you live in a country like the US that a shitty social net it’s worth coming up with a series of fallback plans, if only for your own peace of mind.

For example, how can you make your money last longer?

1. As soon as you lose your job, apply for unemployment.
2. Cut additional costs.
3. If time stretches on and you still don’t have a job, figure out ways to reduce housing costs. Are you young and have the ability to move back in with your parents? Have more room than you need and the option of adding roommates? All in a pandemic-safe way, of course.
4. Any ways you can make money some other way, if it’s really taking too long?

If you can’t find a job immediately, you will have probably have more time to upgrade your skills.

1. Which skills are worth working on?
2. What’s the best way to improve them?

You’ll also want to meet more people who can help you find a job.

1. Can you go to online meetups?
2. Find more places to interact with people online?

Write this all down, and when you’re worried you’ll at least have the comfort of knowing there will be some things you can do if and when your job goes away.

We’re all in this together

As with most big problems, there is only so much you can do as an individual: to meaningfully improve the situations we need to work together, whether in mutual aid groups or via political organization. On the other hand, you need to ensure that you as an individual are doing OK; you can’t help others if you’re collapsing under your own troubles.

And since this can all be overwhelming, start with a few simple actions:

1. Cut an expense or two.
2. Get in touch with some old co-workers.
3. Sign up for a newsletter.
4. Start writing down your fallback plans.

And then, once you have things under control emotionally, when you have a plan and you know what you’re doing next, start thinking about how you can help others, and work with other people to improve things for everyone.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

Some projects have the policy that all tests must have an explanatory comment – including all of mine. At first, I found that baffling. If that’s you right now, this article is for you.

Numbers in Python come in all shapes and forms. The reason different kind of representations of numbers exist is because they all have different trade-offs. These trade-offs are often surprising!

>>> 1 + 2 - 2 - 1
0
>>> 0.1 + 0.2 - 0.2 - 0.1
2.7755575615628914e-17

>>> a = 2**-53
>>> (a + a) + 1 == a + (a + 1)
False

>>> print(set(type(p) for p in primes))
>>> one = fractions.Fraction(1)
>>> before = datetime.now()
>>> res = sum(one/p for p in primes[:10000])
>>> after = datetime.now()
>>> print("It took", after-before)
>>> print("Size of output", len(str(res)))
>>> print("Approximate value", float(res))
{<class 'int'>}
It took 0:01:16.033260
Size of output 90676
Approximate value 2.7092582487972945

>>> print(set(type(p) for p in primes))
>>> before = datetime.now()
>>> res = sum(1/p for p in primes[:10000])
>>> after = datetime.now()
>>> print("It took", after-before)
>>> print("Size of output", len(str(res)))
>>> print("Approximate value", float(res))
{<class 'int'>}
It took 0:00:00.000480
Size of output 17
Approximate value 2.709258248797317

Approximate value 2.7092582487972945
Approximate value 2.709258248797317
                    1234567891234

>>> getcontext().prec = 6
>>> Decimal(1) / Decimal(7)
Decimal('0.142857')
>>> getcontext().prec = 28
>>> Decimal(1) / Decimal(7)
Decimal('0.1428571428571428571428571429')

>>> getcontext().prec = 6
>>> # 6853 lines elided
... with localcontext() as ctx:
...     ctx.prec = 10
...     Decimal(1) / Decimal(7)
...
Decimal('0.1428571429')

(This article is based on the one I originally published on OpenSource.com.)

In 2010, the My Little Pony franchise was rebooted with the animated show My Little Pony: Friendship is Magic. The combination of accessibility to children with the sophisticated themes the show tackled garnered a following that cut across ages. I was swept up in the wave and discovered there is a lot to learn about DevOps from the show.

The show begins with Twilight Sparkle reading obscure documentation, only to realize that Equestria, where the show is set, is due to suffer a calamity. Though Nightmare Moon has been imprisoned for a thousand years, there is a prophecy she will return.

Lesson 1: Technical debt matters.

Document technical debt. Pay attention to the signs of risk no matter how infrequently they occur. Have a plan to resolve it.

Twilight Sparkle goes to her manager with the news, only to be told that it is not a current priority. She is sent to Ponyville to prepare for the coming celebration, instead.

Lesson 2: Communication with management is key.

Twilight Sparkle communicated her priority but did not convince her management that it was more important than the celebration.

We all need to make clear what the business case is for resolving critical issues. It is also not straightforward to explain technical debt in business terms. If management does not agree on the severity, find new ways to communicate the risk, and team up with others who speak that language.

As the prophecy has foreseen, Nightmare Moon returns and declares eternal night. Twilight quickly understands that she cannot resolve the issue by herself, and she recruits the ponies who will become, with her, the "Mane Six." They each stand for a different element of harmony — Applejack stands for Honesty, Fluttershy for Kindness, Pinkie Pie for Laughter, Rarity for Generosity, Rainbow Dash for Loyalty, and Twilight Sparkle herself for Magic.

Lesson 3: Few are the issues that can be resolved by one person.

When facing an outage, reach out to other people with complementary skills who can help you. It is best if they are different than you: different backgrounds leads to differing perspectives, and that can lead to better problem-solving.

Lesson 4: When resolving an outage, honest communication is key.

Throughout the struggle against the eternal night, the Mane Six have to speak openly and honestly about what's not working. Their blameless communication is part of problem-solving.

Lesson 5: When resolving an outage, kindness to yourself and to others is crucial.

Though tempers flare hot in the land of Equestria, we all benefit from coming back to working together.

Lesson 6: Laughter is important.

Even when everything comes crashing down, remember to take a break, drink a glass of water, and take a deep breath. Stressing out does not help anything.

Lesson 7: Be generous.

Even if you are not on-call right now, if your help is needed to resolve a problem, help out as you hope your colleagues will do for you.

Lesson 8: Be loyal.

An outage is not a time to settle rivalries between teams. Focus on how to collaborate and resolve the outage as a team.

Lesson 9: Though people skills are important, you have to understand the technology on a deep level.

Keep your tech skills sharp. Expertise is not only the ability to learn; it is knowing when that information is needed. Part of being an expert is practice.

After the issue is resolved, Princess Celestia realizes that the Mane Six are crucial to the long-term survival of Equestria, and tells Twilight Sparkle to stay in Ponyville and keep researching the magic of friendship.

Lesson 10: After an outage is resolved, conduct a review, take concrete lessons, and act on them.

I could go on, episode by episode, detailing lessons relevant for DevOps, but I will wrap up with one of my favorite ones.

In the "Winter Wrap-Up" episode, all the ponies in Ponyville help in preparing for the spring. As per tradition, they do not use magic, leaving Twilight Sparkle to wonder how she can contribute. Eventually, she realizes that she can help by making a checklist to make sure everything is done in the right order.

Lesson 11: When automation is impossible or inadvisable, write a solid checklist, and follow it. Do not depend on your memory.

Twilight Sparkle and the Mane Six overcome great obstacles as a team, and now have a system to improve as a team. I hope you, too, can help bring a collaborative DevOps culture to your work.

John H. Conway passed away ten days ago, and I think it's only now I can write a proper eulogy.

I was first introduced to his work, if not his name, when I was at the end of elementary school. I am sure everyone has heard about the Game of Life, but did you know it had a 1D version? The 1D version is significantly simpler, but has the advantage that on a grid paper, you can just play with yourself manually by putting a generation on each line.

This was 12 year old me's "fidget spinner", how I kept myself calm in classes. Starting with an initial configuration and letting it evolve.

Later on, when I went to college, I got to borrow his amazing book, "On Numbers and Games". Now, I am definitely the sort of person who reads math books for fun, but most of them are not fun. They are dry, poorly written, and make leaps all the time. "ONAG" was the exact opposite. It's a short, delightful book, that tries to get across the thinking, the intuition, the methods, and, yes, the joy.

Fast forward a decade or two, and again I found myself enamored with another one of his inventions: the Look-and-Say sequence. My old interview coding question was getting too popular on the interview-question-sites, and I was getting worried. Writing code for the look-and-say sequence is reasonably straightforward, but does require basic skills: looping while keeping a bunch of state variables.

Then I read about his work on the look-and-say sequence, and was utterly amazed and delighted by it. Atoms and decay and asymptotic growth!

Throughtout his career, I think what made his things special is that he embodies the truest mathematician spirit, which is also the truest geek spirit: starting out with something simple, and then nerding out about it until you have built a whole universe.

Whether it is a place where guns shoot spaceships at 3/8 the speed of light, an algebraic field so vast it includes all other ordered fields and also all infinities, or a concept of numbers atomically decaying, he was a master at whipping out mathematicially consistent fictional worlds.

Goodbye John H. Conway, you were taken from us too soon.

This article was written during abnormal circumstances, with much of the planet under lockdown due to the COVID-19 pandemic. Parents with children at home have far less time, and pretty much everyone is feeling stressed and distracted.

Under more normal circumstances there are only so many hours in the day to do your job; now it’s even worse. And yet work needs to get done: code needs to get written, features need to be shipped, bugs need to be fixed.

Faced with an ever growing list of tasks, how do you get everything done?

The short answer is, you can’t. You will never get everything done.

What you can do, though, is choose the right work, the most valuable work, the most useful work, the work with most leverage. Choose the right work and you can gets orders of magnitude improvement in your output.

Let’s see how.

The goal: increased output

Your output as a programmer is based both on your productivity and on how much time you work:

Output = Productivity × Time Worked

The first thing to notice is that there is a hard limit on how much increasing your working hours can help. After all, there are only 168 hours in a week.

If you never slept, ate, or did anything but work—and this will literally kill you—you can work 4.2× as much as a 40-hour workweek, and that’s it. And even with smaller increases in work hours, the gains quickly decline. As you work more hours you’ll become fatigued and make more mistakes; beyond a certain point those extra work hours will decrease your productivity, canceling out any gains.

What is output for a programmer?

Since increasing working hours isn’t really an option, the key to increasing your output is increasing your productivity. Productivity is the output you produce in each fixed unit of time, for example:

Productivity = Output per week

If you’re going to improve your productivity, you need to understand how to measure output.

The obvious measure is how much code you write: the more code, the better. This measure is obvious, popular, and completely wrong.

All other things being equal, is it better to implement the same feature with 10 lines of code, or 10,000 lines of code? If we measure output by code produced, the latter solution is better, but in most cases a 10 line solution is preferable to 10,000 lines. More code means higher maintenance costs, not to mention more opportunities for defects.

Your job as a programmer is not writing code, your job is solving problems: software is a tool, a means to an end. Software becomes valuable because of the problems it solves.

As a rough measure, your output as a programmer can be measured by the problems you solve: the more significant the problems you can solve, the better.

If you work for a business, significance eventually translates directly or indirectly into monetary terms: money made or money saved. In other areas you can come up with domain-specific concrete measures of usefulness: number of people served, carbon emissions reduced, number of scientists using your software, and so on.

Note: If the problems you solve produce negative value you will become anti-productive: the better you are at your job, the more damage you will cause.

If making money hurts people or the environment, your work may be productive for your employer but anti-productive for society as a whole. So make sure you’re carefully considering the ethical consequences of your actions as a worker.

How to increase productivity

Given the above, here’s how you can increase your productivity:

1. Find the most significant problem you can work on.
2. Come up with the most efficient solution to that problem.
3. Implement the solution with minimum wasted time.

Let’s go through these steps one by one, and see why they’re key to productivity.

1. Find the most significant problem

Let’s consider our formula for productivity again:

Productivity = Significant problems solved / Week

There are many problems you could be working on, so first you have to choose one. If you could solve either of these problems, should you be working on:

1. Implementing a particular missing feature; this will increase revenue by $50,000.
2. Fixing a bug that was decreasing customer retention; this will increase revenue by $1,000,000.

All other things being equal, the second problem is obviously the one you should be focusing on. Even if it takes 10× as long to solve and implement that bug fix, it should still be the highest priority:

Productivity of #1 =    $50,000 /  1 Week  =  $50,000 / Week
Productivity of #2 = $1,000,000 / 10 Weeks = $100,000 / Week

Here’s the issue: in order to fix that expensive bug and improve customer retention, you need to know the problem exists. If no one ever notices that customers are leaving, if no one ever finds that bug, if no one realizes the connection between the two—then that problem will never be solved.

And that’s why finding problems is the first and most valuable step in increasing productivity.

2. Come up with an efficient solution

Once you’ve identified the most significant problem—or once your manager assigns you a problem they identified—you need to come up with a solution.

Which solution do you think is better?

1. Takes 1000 lines of code and 4 weeks to implement.
2. Takes 100 lines of code and 3 days to implement.

All other things being equal, the second solution is obviously better. But again, you need to find that solution.

If you only ever find that first solution, then no matter how efficiently you implement it, no matter how focused you are, no matter how much you manage to speed things up—you’re still implementing a much less efficient solution.

And that’s why identifying better solution is the second most valuable step in increasing productivity.

3. Implement the solution without wasting a time

Once you’ve identified a problem and chosen the solution, there is only so much leverage you have to improve productivity. You obviously want to avoid getting stuck and spinning your wheels, because wasted time reduces your productivity.

But given a particular solution, there’s only so much waste you can reduce, only so fast you can go:

Wasted time → $50,000 / 2 weeks = $25,000 / week
No waste    → $50,000 / 1 week  = $50,000 / week

Efficient implementation is the last and least valuable way of increasing productivity.

Technological skills aren’t enough

While you get the most increased productivity from identifying problems and the least from efficient implementation, your career as a programmer progresses in the opposite direction:

1. Junior engineers implement solutions.
2. Senior engineers find solutions and implement them.
3. Principal or staff engineers identify problems, find solutions, and implement them.

So becoming more productive isn’t just about helping your employer’s bottom line, it’s also about learning the skills that will give you more pay and more influence.

Critically, technological skills are necessary but not sufficient to increase your productivity:

• Your JavaScript skills don’t matter if you can never meet deadlines.
• Your testing skills don’t matter if you can’t convince your manager of the value of testing.
• Your software architecture skills don’t matter if no one has ever heard of your product.

Why these skills are “secret”

Most discussions of programming productivity tend to end up focusing purely on technology, coding, and design skills, and skip over these problem-solving skills. Of course, this isn’t a conspiracy of silence, no one is deliberately hiding the existence of the skills.

My guess is that experienced programmers still have to learn new technologies, so they’re more likely to realize the need to explain those particular skills. But having learned them once, they apply skills like timeboxing, or considering multiple different solutions to a problem, without even noticing they’re doing it. And so they end up talking about problem-solving skills rather less, and about technological skills rather more.

How do you learn these skills?

This article is an excerpt from my book, The Secret Skills of Productive Programmers, covering the non-technical skills you need to get better at identifying problems, solving problems, and implementing them on schedule.

Elsewhere on this site you’ll find many free articles on building up your skills.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

The Twisted Requests (treq) package is an HTTP client built on the popular Twisted library that is used for web requests. Async libraries offer the ability to do large amounts of network requests in parallel with relatively little CPU impact. This can be useful in HTTP clients that need to make several requests before they have all the information they need.

This post shows an example of a problem like this, and how to solve it using treq.

I enjoy playing the real-time strategy game Clash Royale. Clash Royale is a mobile strategy player-vs-player game where players play cards in an arena to win. Each card has different strengths and weaknesses, and different players prefer different cards. Clash Royale remembers which card a player plays the most; this is their "favorite" card. Players come together in clans where they can help each other. Supercell, Clash Royale's developer, released an HTTP-based API where different statistics can be queried.

How can we write a program that will output the most popular favorite cards in a clan?

If you want to follow along, you will need to register an account. If you register an account, create an API token via the Clash Royale developer portal. Then choose "Create New Key" under your profile, and enter a name, description, and a valid IP address. (An exact address is required.) Since you should never save an API key in your code, keep it as a separate file in ~/.crtoken:

$ ls ~/.crtoken
/home/moshez/.crtoken

To make it easier to see what is going on, let's start with this introductory program that prints Hello world, and then we'll talk through what it does:

import collections, json, os, sys, urllib.parse
from twisted.internet import task, defer
import treq

with open(os.path.expanduser("~/.crtoken")) as fpin:
    token = fpin.read().strip()

def main(reactor):
    print("Hello world")
    return defer.succeed(None)

task.react(main, sys.argv[1:])

This imports many more modules than we need for the "Hello world" example. We will need these modules for the final version of the program, which will accomplish the more complex task of asynchronously querying an API. After the import, the program reads the token from the file and stores it in the variable token. (We are not going to do anything with the token right now, but it's good to see that code.) Next there is a main function that accepts a Twisted reactor. A reactor is sort of like an interface to the machinery of the Twisted package. In this case the function main is sent as a parameter to task.react, and, which will call main with the reactor and any arguments we give -- the command-line arguments, in this case.

The main function returns a defer.succeed(None). This is how it returns a value of the right type: a deferred value, but one that already has been "fired" or "called." Because of that, the program will exit immediately after printing Hello world, as we need.

Next, we will look at the concepts of async functions and ensureDeferred:

async def get_clan_details(clan):
     print("Hello world", clan)

def main(reactor, clan):
    return defer.ensureDeferred(get_clan_details(clan))

task.react(main, sys.argv[1:])

In this program, which should start with the same imports, we moved all the logic to the async function get_clan_details. Just like a regular function, an async function has an implicit return None at the end. However, async functions, sometimes called co-routines, are a different type than Deferred. In order to let Twisted, which has existed since Python 1.5.2, use this modern feature, we must adapt the co-routine using ensureDeferred.

While we could write all the logic without using co-routines, using the async syntax will allow us to write code that is easier to understand, and we will need to move a lot less of the code into embedded callbacks.

The next concept to introduce is that of await. Later, we will await a network call, but for simplicity, right now, we will await on a timer. Twisted has a special function, task.deferLater, which will call a function with given parameters after some time has passed.

The following program will take five seconds to complete:

async def get_clan_details(clan, reactor):
     out = await task.deferLater(
         reactor,
         5,
         lambda clan: f"Hello world {clan}",
         clan
     )
     print(out)

def main(reactor, clan):
    return defer.ensureDeferred(get_clan_details(clan, reactor))

task.react(main, sys.argv[1:])

A note about types: task.deferLater returns a Deferred, as do most Twisted functions that do not have the value already available. When running the Twisted event loop, we can await on both Deferred values and co-routines.

The function task.deferLater will wait five seconds and then call our lambda, calculating the string to print out.

Now we have all the Twisted building blocks needed to write an efficient clan-analysis program!

Since we will be using the global reactor, we no longer need to accept the reactor as a parameter in the function that calculates these statistics. The way to use the token is as a "bearer" token in the headers:

async def get_clan_details(clan):
    headers={b'Authorization': b'Bearer '+token.encode('ascii')}

We want clan tags to be sent, which will be strings. Clan tags begin with #, so they must be quoted before they're put in URLs. This is because # has the special meaning "URL fragment":

async def get_clan_details(clan):
     # ...
     clan = urllib.parse.quote(clan)

The first step is to get the details of the clan, including the clan members:

async def get_clan_details(clan):
     # ...
     res = await treq.get("https://api.clashroyale.com/v1/clans/" + clan,
                          headers=headers)

Notice that we have to await the treq.get call. We have to be explicit about when to wait and get information since it is an asynchronous network call. Just using the await syntax to call a Deferred function does not let us take full power of asynchronicity (we will see how to do it later).

Next, after getting the headers, we need to get the content. The treq library gives us a helper method that parses the JSON directly:

async def get_clan_details(clan):
     # ...
     content = await res.json()

The content includes some metadata about the clan, which is not interesting for our current purposes, and a memberList field that contains the clan members. Note that while it has some data about the players, the current favorite card is not part of it. It does include the unique "player tag" that we can use to retrieve further data.

We collect all player tags, and, since they also begin with #, we URL-quote them:

async def get_clan_details(clan):
     # ...
     player_tags = [urllib.parse.quote(player['tag'])
                    for player in content['memberList']]

Finally, we come to the real power of treq and Twisted: generating all requests for player data at once! That can really speed up tasks like this one, which queries an API over and over again. In cases of APIs with rate-limiting, this can be problematic.

There are times when we need to be considerate to our API owners and not run up against any rate limits. There are techniques to support rate-limiting explicitly in Twisted, but they are beyond the scope of this post. (One important tool is defer.DeferredSemaphore.)

async def get_clan_details(clan):
     # ...
     requests = [treq.get("https://api.clashroyale.com/v1/players/" + tag,
                          headers=headers)
                 for tag in player_tags]

Remember that requests do not return the JSON body directly. Earlier, we used await so that we did not have to worry about exactly what the requests return. They actually return a Deferred. A Deferred can have an attached callback that will modify the Deferred. If the callback returns a Deferred, the final value of the Deferred will be the value of the returned Deferred.

So, to each deferred, we attach a callback that will retrieve the JSON of the body:

async def get_clan_details(clan):
     # ...
     for request in requests:
         request.addCallback(lambda result: result.json())

Attaching callbacks to Deferreds is a more manual technique, which makes code that is harder to follow but uses the async features more efficiently. Specifically, because we are attaching all the callbacks at the same time, we do not need to wait for the network calls, which potentially can take a long time, to indicate how to post-process the result.

From Deferreds to values

We cannot calculate the most popular favorite cards until all results have been gathered. We have a list of Deferreds, but what we want is a Deferred that gets a list value. This inversion is exactly what the Twisted function defer.gatherResults does:

async def get_clan_details(clan):
     # ...
     all_players = await defer.gatherResults(requests)

This seemingly innocent call is where we use the full power of Twisted. The defer.gatherResults function immediately returns a deferred that will fire only when all the constituent Deferreds have fired and will fire with the result. It even gives us free error-handling: if any of the Deferreds error out, it will immediately return a failed deferred, which will cause the await to raise an exception.

Now that we have all the players' details, we need to munch some data. We get to use one of Python's coolest built-ins, collections.Counter. This class takes a list of things and counts how many times it has seen each thing, which is exactly what we need for vote counting or popularity contests:

async def get_clan_details(clan):
     # ...
     favorite_card = collections.Counter([player["currentFavouriteCard"]["name"]
                                          for player in all_players])

Finally, we print it:

async def get_clan_details(clan):
     # ...
     print(json.dumps(favorite_card.most_common(), indent=4))

So, putting it all together, we have:

import collections, json, os, sys, urllib.parse
from twisted.internet import task, defer
import treq

with open(os.path.expanduser("~/.crtoken")) as fpin:
    token = fpin.read().strip()


async def get_clan_details(clan):
     headers = headers={b'Authorization': b'Bearer '+token.encode('ascii')}
     clan = urllib.parse.quote(clan)
     res = await treq.get("https://api.clashroyale.com/v1/clans/" + clan,
                          headers=headers)
     content = await res.json()
     player_tags = [urllib.parse.quote(player['tag'])
                    for player in content['memberList']]
     requests = [treq.get("https://api.clashroyale.com/v1/players/" + tag,
                          headers=headers)
                 for tag in player_tags]
     for request in requests:
         request.addCallback(lambda result: result.json())
     all_players = await defer.gatherResults(requests)
     favorite_card = collections.Counter([player["currentFavouriteCard"]["name"]
                                          for player in all_players])
     print(json.dumps(favorite_card.most_common(), indent=4))

def main(reactor, clan):
    return defer.ensureDeferred(get_clan_details(clan))

task.react(main, sys.argv[1:])

Thanks to the efficiency and expressive syntax of Twisted and treq, this is all the code we need to make asynchronous calls to an API. If you were wondering about the outcome, my clan's list of favorite cards is Wizard, Mega Knight, Valkyrie, and Royal Giant, in descending order.

(This post is based on the article I wrote for opensource.com)

It has been a long time since I learned how to program, and it is easy to forget some of the hard-won lessons in the beginning. Easy until I try to teach people to program. There is a lot of accidental and inherent complexity in programming, but I am ready for that: I remember to explain how carefully to follow the syntax, and the kind of syntactical gotchas that are easy to fail.

But there is one metalesson that is much easier to forget, and much harder to learn, and to teach. Humans are used to small mistakes having reasonably small consequences. But even in cases with catatrophic consequences, the consequences look related to the mistakes.

However, in programming, small mistakes can lead not just to big consequences, but to weird consequences. A missing comma might mean that things work fine in the testing environment, but in production, every third request gets a slightly wrong result.

This really stumps people. They copy code somewhat imperfectly from the board, or make a small mistake when they edit it to change from "Hello world" to "Goodbye world", and suddenly, a completely unrelated part of the program starts going haywire.

This happens to old hands too. The number of times I have edited code and ran the tests, only to discover a clearly unrelated test failing, is not small. The difference that comes with experience is that I take a deep breath, think "I've got this", and start down the troubleshooting path.

The troubleshooting can include any number of things: I might go in with a debugger, add print statements, do a bisect-diff to figure out what caused the problem, try random things to see what happens, or just trace the execution path carefully.

The troubleshooting process does not matter as much as what comes before it: the deep breath. This is my time to accept the problem has happened, and that I am in for a process which can take two hours, and at the end of which my entire productivity will be "added missing semicolon". Sometimes a breath is not enough, and I need to get up and get some tea. But the most important, and almost invisible step, at the beginning is to step back, remember that this is, weirdly, part of the job, and to become comfortable doing it.

If you want to be any kind of software developer, accept it now. Much of your life will be seeing weird consequences, and tracing it back to a small mistake. Eventually, like everyone, you will succeed. Flush with victory, make a note of your success somewhere: anywhere where the overhead of writing it is low, be it an e-mail to yourself or a note-keeping app.

If you ever decide to teach, or write a blog, this is an unending source of content.

(Thanks to Veronica Hanus for her feedback on an early draft. All issues and mistakes that remain are my responsibility.)

• curve25519-sha256 key exchange algorithm support in Conch.
• "openssh-key-v1" key format support in Conch.
• Security fixes to twisted.web, including preventing request smuggling attacks and rejecting malformed headers. CVE-2020-10108 and CVE-2020-10109 were assigned for these issues, see the NEWS file for full details.
• twist dns --secondary now works on Python 3.
• The deprecation of twisted.news.
• ...and various other fixes, with 28 tickets closed in total. 

This was originally sent to my newsletter. I send one e-mail, always about Python, every other Sunday. If this blog post interests you, consider subscribing.

The underappreciated else keyword in Python has three distinct uses.

if anonymize:
    print("Hello world")
else:
    print("Hello, name")

for x in numbers:
    if x % 2 == 1:
        print("Found", x)
        break
else:
    print("No odd found")

try:
    before, after = things
except ValueError:
    part1 = things[0]
    part2 = 0
    after = 0
else:
    part1, part2 = before

I’m missing a key part from the public Python discourse and I would like to help to change that.

Since this topic keeps coming up, I’d like to briefly share my thoughts on Python package metadata because it’s – as always – more complex than it seems.