Planet Twisted

The choice in parameters for back-off configuration is important. It can be the difference between a barely noticable blip in service quality and an hours-long site outage. In order to explore the consequences of the choice, I wrote a little fictional ditty about a fictional website.

I hope you enjoy escaping into this fictional reality as much as I enjoyed writing about it.

Your recipe site is different. After all, recipe sites are a dime-a-dozen. With today's modern technology, any kid can put a quick mock-up together with Django, React, and MongoDB to store recipes and retrieve them by various attributes.

In order to make your recipe site stand above the rest, you made sure it uses really cutting edge techniques. From details of the web requests coming in, using sophisiticated language parsing and machine learning algorithm, with just a few words about the user's likes and dislikes, you find the perfect recipe just for them.

HackerNews called it "just a bunch of buzzwords", of course. But once the graphs went up into the right, with 50% month-over-month growth rates, everyone explained that they knew that this one was different. Popularity sky-rocketed, the engineers worked on scaling up the site, and though it was not the world's most sophisticated microservice architecture, it was medium-service architecture, at least.

The web front end would call the machine learning cluster, running on special GPU machines, to get the appropriate keywords by which to look up the recipe. Maybe not a the kind of 50-microservices-architecture that takes three whiteboards to explain, but at least it was easy enough to scale up horizontally. You hired a great Site Reliability Engineer, who built a sophisticated continuous delivery machine. As your machine learning team fine-tuned the model, it would slowly roll out into the cluster, running continous A/B tests that would immediately roll back the change if the model performed worse than before.

The SRE also made sure the web front-end would back off of a malfunctioning machine learning node, as those do sometimes. Instead of immediately reconnecting, it would try reconnecting at intervals starting at 5 milliseconds and increasing by powers of 2: 10ms, 20ms, and so on.

Amazing.

You will never forget what happened next. A new model was rolled out. It looked great. Performed wonderfully. The only problem was that it had a hidden time-bomb. The model had a small overfit that happend to match the sum of the user's ID and the time. This, in itself, is the kind of overfit that happens every day. Unfortunately, if that value exceeded a threshold, it triggered a bug in the machine learning library. When that happened, the node would become flakey, and start dropping random connections. Even worse, for the specific input used in the health checks, it would work: all nodes appeared healthy in the monitoring dashboard, and to the service discovery framework.

A nightmare scenario.

One by one, nodes started dropping off, as users with higher IDs connected and the seconds ticked on mercilessly. The front end started backing off: by 5ms, then 10ms, then 20ms.

The exponential function explodes fast, so of course the backoff (with jitter, as is common practice) had an upper limit. When the Site Reliability Engineer set the upper limit, there was a lot of things on their mind. Decisions are hard to make. Sometimes, one neuron firing makes all the difference. But neurons are small, and electrons are smaller.

When we look at the firing of a neuron, we can no longer imagine the world is as Maxwell and Newton imagined it to be. Quantum effects must be taken into account. As per the obviously correct Many-Worlds interpretation of Quantum Mechanics, the one world had split into two that would never interact again.

In one world, the SRE had set the maximum to 1 second. In the other, to 10 minutes. For a while, these worlds seemed to parallel each other closely: sure, a few bits on in a YAML file were different, but what storm could come from such a small butterfly?

It was not until the disasterous model roll-out that the worlds would diverge wildly. In the world where the limit was set to one second, the front ends were hammering the nodes mercilessly. Rolling back the model would mean that a node would start getting too many connections, and before a few seconds had passed, it would fall back down.

Eventually, in this world, the whole front-end had to be brought down, the model fully rolled back, and only then the front-end brought back up. This was too bad, because the front-end had a bad-but-working model built-in, and some recipes would still be found while the "outage" was happening. Sure, they were not the "wow you read my mind" quality, but they were still decent results.

Bringing down the whole front-end cluster, and bringing it up again, turned the outage from a blip in recipe click-through rates into a three-hour, news-coverage-worthy outage.

In the other world, where the back-off had a maximum of ten minutes, things progressed much more smoothly. As a machine learning node's model was rolled back, it came back up, and machines started connecting to it. The first one to be brought up did go down from connection overload -- but took ten minutes to do so, enough time to ascertain that this solution was correct. A few more nodes were rolled back, and as the connections grew, the roll back flowed through the fleet.

The outage has been managed, and other than a few irate customers posting on Twitter about looking for vegan recipes and getting a "meat lover's delight", the outage mostly went unnoticed.

Back-off is important. Exponential back-off with jitter and a maximum is almost always the right solution. Yes, the exponent matters a little. The initial back off matters a little. But the maximum matters a lot. Set the maximum to "human time frames": a few minutes (1-30) is a good balance.

Even in a big cluster, an extra action every 10 minutes will probably not cause serious downstream repercussions. But even the most entitled customer can be mollified by a support technician for five minutes doing "everything possible to find a solution" while waiting the clock out on the 10 minute back-off clock.

Choose maximums for back-off carefully, or the site you bring down may be your own.

Thanks to Avy Faingezicht for his feedback on an earlier draft. Any mistakes or issues that remain are my reponsibility.

There should be one — and preferably only one — obvious way to do it.

Moshe wrote a blog post a couple of days ago which neatly constructs a wonderful little coding example from a scene in a movie. And, as we know from the Zen of Python quote, there should only be one obvious way to do something in Python. So my initial reaction to his post was of course to do it differently — to replace an __init__ method with the new @dataclasses.dataclass decorator.

But as I thought about the code example more, I realized there are a number of things beyond just dataclasses that make the difference between “toy”, example-quality Python, and what you’d do in a modern, professional, production codebase today.

So let’s do everything the second, not-obvious way!

There’s more than one way to do it

Getting started: the __future__ is now

We will want to use type annotations. But, the Guard and his friend are very self-referential, and will have lots of annotations that reference things that come later in the file. So we’ll want to take advantage of a future feature of Python, which is to say, Postponed Evaluation of Annotations. In addition to the benefit of slightly improving our import time, it’ll let us use the nice type annotation syntax without any ugly quoting, even when we need to make forward references.

So, to begin:

1

from __future__ import annotations

Doors: safe sets of constants

Next, let’s tackle the concept of “doors”. We don’t need to gold-plate this with a full blown Door class with instances and methods - doors don’t have any behavior or state in this example, and we don’t need to add it. But, we still wouldn’t want anyone using using this library to mix up a door or accidentally plunge to their doom by accidentally passing "certian death" when they meant certain. So a Door clearly needs a type of its own, which is to say, an Enum:

1
2
3
4
5

from enum import Enum

class Door(Enum):
    certain_death = "certain death"
    castle = "castle"

Questions: describing type interfaces

Next up, what is a “question”? Guards expect a very specific sort of value as their question argument and we if we’re using type annotations, we should specify what it is. We want a Question type that defines arguments for each part of the universe of knowledge that these guards understand. This includes who they are themselves, who the set of both guards are, and what the doors are.

We can specify it like so:

1
2
3
4
5
6
7

from typing import Protocol, Sequence

class Question(Protocol):
    def __call__(
        self, guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        ...

The most flexible way to define a type of thing you can call using mypy and typing is to define a Protocol with a __call__ method and nothing else¹. We could also describe this type as Question = Callable[[Guard, Sequence[Guard], Door], bool] instead, but as you may be able to infer, that doesn’t let you easily specify names of arguments, or keyword-only or positional-only arguments, or required default values. So Protocol-with-__call__ it is.

At this point, we also get to consider; does the questioner need the ability to change the collection of doors they’re passed? Probably not; they’re just asking questions, not giving commands. So they should receive an immutable version, which means we need to import Sequence from the typing module and not List, and use that for both guards and doors argument types.

Guards and questions: annotating existing logic with types

Next up, what does Guard look like now? Aside from adding some type annotations — and using our shiny new Door and Question types — it looks substantially similar to Moshe’s version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from dataclasses import dataclass

@dataclass
class Guard:
    _truth_teller: bool
    _guards: Sequence[Guard]
    _doors: Sequence[Door]

    def ask(self, question: Question) -> bool:
        answer = question(self, self._guards, self._doors)
        if not self._truth_teller:
            answer = not answer
        return answer

Similarly, the question that we want to ask looks quite similar, with the addition of:

1. type annotations for both the “outer” and the “inner” question, and
2. using Door.castle for our comparison rather than the string "castle"
3. replacing List with Sequence, as discussed above, since the guards in this puzzle also have no power to change their environment, only to answer questions.
4. using the [var] = value syntax for destructuring bind, rather than the more subtle var, = value form

1
2
3
4
5
6
7
8
9

def question(guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]) -> bool:
    [other_guard] = (candidate for candidate in guards if candidate != guard)

    def other_question(
        guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        return doors[0] == Door.castle

    return other_guard.ask(other_question)

Eliminating global state: building the guard post

Next up, how shall we initialize this collection of guards? Setting a couple of global variables is never good style, so let’s encapsulate this within a function:

1
2
3
4
5
6
7

from typing import List

def make_guard_post() -> Sequence[Guard]:
    doors = list(Door)
    guards: List[Guard] = []
    guards[:] = [Guard(True, guards, doors), Guard(False, guards, doors)]
    return guards

Defining the main point

And finally, how shall we actually have this execute? First, let’s put this in a function, so that it can be called by things other than running the script directly; for example, if we want to use entry_points to expose this as a script. Then, let's put it in a "__main__" block, and not just execute it at module scope.

Secondly, rather than inspecting the output of each one at a time, let’s use the all function to express that the interesting thing is that all of the guards will answer the question in the affirmative:

1
2
3
4
5
6

def main() -> None:
    print(all(each.ask(question) for each in make_guard_post()))


if __name__ == "__main__":
    main()

Appendix: the full code

To sum up, here’s the full version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

from __future__ import annotations
from dataclasses import dataclass
from typing import List, Protocol, Sequence
from enum import Enum


class Door(Enum):
    certain_death = "certain death"
    castle = "castle"


class Question(Protocol):
    def __call__(
        self, guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        ...


@dataclass
class Guard:
    _truth_teller: bool
    _guards: Sequence[Guard]
    _doors: Sequence[Door]

    def ask(self, question: Question) -> bool:
        answer = question(self, self._guards, self._doors)
        if not self._truth_teller:
            answer = not answer
        return answer


def question(guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]) -> bool:
    [other_guard] = (candidate for candidate in guards if candidate != guard)

    def other_question(
        guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        return doors[0] == Door.castle

    return other_guard.ask(other_question)


def make_guard_post() -> Sequence[Guard]:
    doors = list(Door)
    guards: List[Guard] = []
    guards[:] = [Guard(True, guards, doors), Guard(False, guards, doors)]
    return guards


def main() -> None:
    print(all(each.ask(question) for each in make_guard_post()))


if __name__ == "__main__":
    main()

Acknowledgments

I’d like to thank Moshe Zadka for the post that inspired this, as well as Nelson Elhage, Jonathan Lange, Ben Bangert and Alex Gaynor for giving feedback on drafts of this post.

In the 1986 movie Labyrinth, a young girl (played by Jennifer Connelly) is faced with a dilemma. The adorable Jim Henson puppets explain to her that one guard always lies, and one guard always tells the truth. She needs to figure out which door leads to the castle at the center of the eponymous Labyrinth, and which one to certain death (dun-dun-dun!).

I decided that like any reasonable movie watcher, I need to implement this in Python.

First, I implemented two guards: one who always tells the truth, and one who always lies. The guards know who they are, and what the doors are, but can only answer True or False.

import dataclasses
from typing import List

guards = [None, None]
doors = ["certain death", "castle"]

@dataclasses.dataclass
class Guard:
    _truth_teller: bool
    _guards: List
    _doors: List[str]

    def ask(self, question):
        answer = bool(question(self, self._guards, self._doors))
        if not self._truth_teller:
            answer = not answer
        return answer

guards[0] = Guard(True, guards, doors)
guards[1] = Guard(False, guards, doors)

This being a children’s movie, the girl defeats all odds and figures out what to ask the guard: “would he (points to the other guard) tell me that this (points to the door on the left) door leads to the castle?”

def question(guard, guards, doors):
    other_guard, = (candidate for candidate in guards if candidate != guard)
    def other_question(ignored, guards, doors):
        return doors[0] == "castle"
    return other_guard.ask(other_question)

What would the truth-teller answer?

guards[0].ask(question)

True

And the liar?

guards[1].ask(question)

True

No matter who she asks, now she can count on a lie. After a short exposition, she confidently walks through the other door. It’s a piece of cake!

Thanks to Rina Arstain and Veronica Hanus for their feedback on an earlier draft. Thanks to Glyph Lefkowitz for the idea to use dataclasses. All mistakes and issues that remain are my responsibility.

Some related work in the field of formalizing logic puzzles:

• Glyph wrote a response post about getting the code to be professional-level Python.
• Hillel Wayne wrote about how to solve logic puzzles with Alloy.

How do you setup your dev environment? Depending on your language there are many choices of editor, package manager, build tool, linter, on and on. And every article you find will have a different combination of suggested tools, each of which claiming that their list is The Right Way To Do Things.

So which do you choose?

The short answer: it doesn’t matter. Your choice of dev environment is meaningless.

The slightly less flippant answer is that, yes, there are some contraints on which tools you should pick, but otherwise you should just pick something and move on.

Let’s see why dev environments don’t matter that much in the end, and what limited constraints you should apply when choosing your tools.

Learning how to cook

Imagine you’re training to become a chef. You will need to learn how to use a knife correctly, to chop and dice safely and quickly.

And yes, you need a sharp knife. But when you’re starting out, it doesn’t matter which knife you use: just pick something sharp and good enough, and move on. After all, the knife is just a tool.

The people eating the food you cook don’t care about which knife you used: they care how the food tastes and looks.

After six months in the kitchen, you’ll start understanding how you personally use a knife, what cuisines you want to pursue, what techniques you want to vary. And then you’ll have the knowledge to pick a specific knife or knives exactly suited to your needs.

But remember: the people eating your food still won’t care which knife you used.

Choosing a dev environment

When you use a website, you don’t care which build tool the programmer used. When you run an app, you don’t care which editor they used. You want the software to work, to do what it says, to be easy to use, to get out of your way—and you don’t care how they did it.

And that applies just as much to the users of your code: they don’t care which tools you used.

And when you’re starting out, whether programming in general or a new language or framework, you don’t know how you will like to work. So instead of obsessing over finding the ideal development environment and toolchain, just pick tools that are good enough:

• Popular: So you can easily find help.
• Easy to get going: Your goal is ship useful code, and as a beginner time spent fiddling with your dev environment won’t help with that.

Once you have enough experience, you will start developing opinions. You might become choosy about which tools you use, or end up customizing them to your needs. You might even write an article about your particular dev environment and favorite tools.

But however strong your preferences are, chances are that given tools you don’t quite like as much, you will still do just fine. If you know what you’re doing, you can chop vegetables with any sharp knife, even if it’s not your favorite.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

(I have shown this technique in my mailing list. If this kind of thing seems interesting, why not subscribe?)

Imagine you want to log something that is, potentially, expensive to calculate. For example, in DEBUG mode, you would like to count the classes of the objects in gc.get_objects() and log those counts: this is often a useful technique for diagnosing reference leaks. This is pretty heavy to calculate, and logging it always sounds wasteful.

The logging module has lazy interpolation.

A line like

logging.debug("total number: %r", 2500)

will not bother calculating the string interpolation total number: 2500 unless the logging level is set to output it.

However, calling logging.debug("total number: %r", get_object_counts()) will still call get_object_counts() regardless. It is not the cost of calculating the interpolation, but that of calculating the counts themselves that you would like to avoid.

One way to piggy-back on the lazy interpolation to do lazy evaluation is to write a custom class with a lazy __repr__():

class OnDemand:
    def __init__(self, callable):
        self.callable = callable
    def __repr__(self):
        return repr(self.callable())

This defines a class, OnDemand, which will only call the function when it needs its repr.

This allows us to write code like:

logging.debug("total number: %r", OnDemand(get_object_counts))`

Now, get_object_counts() will not be called at all: notice that the logging line does not call it. This makes the lazy evaluation explicit: we explicitly delay evaluation with OnDemand.

Here is a simple example of how it would work:

>>> def get_object_counts():
...     print("doing something expensive")
...     return 5
...
>>> debug_logging.debug("result is %r", OnDemand(get_object_counts))
doing something expensive
DEBUG:debug:result is 5
>>> warnonly_logging.debug("result is %r", OnDemand(get_object_counts))
>>> warnonly_logging.debug("result is %r", get_object_counts())
doing something expensive

Note that when the warnonly_logging is called with a .debug() method, the expensive calculation is not done: not only is the log message ignored, but the function that calculates the value is not done.

The last line shows that without the careful usage of OnDemand, the expensive calculation is done.

This should make it easy to sprinkle heavy calculations in logging.debug statements. Coupled with easy ways to trigger logging level changes (which are beyond the scope of this article) this gives a powerful way to get insight into your program's innards.

(Thanks to Adi Stav, Avy Faingezicht, Chris Withers, Dave Briccetti, and Lucas Wiman on their feedback on an early draft of this article. Any mistakes or issues that remain are my responsibility.)

When you’re looking for a new programming job, how do you explain your value? The usual approach is a long list of technologies, but this leaves out a critical skill: your ability to solve problems.

If you can convey your level of skill at problem solving, you can get:

• More job offers.
• Jobs with technologies you don’t know.
• A higher salary by getting slotted into a higher pay grade.

Often just a few extra words can make a big difference in demonstrating your skills. Let’s see how you can do it.

Three levels of problem solving skills

As I discussed elsewhere in more detail, problem solving comes in three stages, each approximately corresponding to a particular career stage (these names are due to Randall Koutnik):

1. Staff or principal software engineers are Finders: they find new problems.
2. Senior software engineers are Solvers: they solve already-identified problems.
3. Junior software engineers are Implementers: they implement already-identified solutions.

The earlier you are in the problem-solving process, the more productive you are, and therefore the more valuable as an employee.

As a result, you need to communicate how advanced your skill is across these three levels to demonstrate your productivity. Everything from your resume to the stories you tell in interviews should communicate your level of skill.

Explaining your skill level

Explaining your skill level involves telling stories that use the correct words and sufficient information to demonstrate your skill. I’m going to use resumes as an example here, but you should ensure you do this in interviews as well—if you’re practicing with a friend, make sure they’re checking for this, it’s easy to leave the information out.

Consider the following entry from a resume:

Moved deployment from manually-managed hosts to a new Kubernetes cluster.

This experience entry uses an implementation-level verb: “moved”. Similarly, “coded”, “tested”, “wrote”, “fixed”, “optimized"—these are all about implementation. And maybe it’s implementation was tricky and difficult, and it’s good to convey that, but if you also solved the problem or identified the problem it’s impossible to tell from this phrasing.

Any task you write about in your resume and talk about in interviews was the result of someone identifying a problem, and someone coming up with the solution. If it was you, make sure you say that.

Let’s say in our example above you were the one tasked with figuring out an alternative to manually-managed hosts. If so, you need to add additional context and verbs that convey that:

Investigated alternatives to manually-managed hosts, decided on Kubernetes, and moved the deployment to a new cluster.

Now we can clearly see you solved the problem.

If you were the one who identified the problem, again you need to make sure you explicitly call that out:

Identified manually-managed hosts as an operational problem, and got management buy-in to change to a better system. Subsequently investigated alternatives, decided on Kubernetes, and moved the deployment to a new cluster.

Now we can see all the value you provided.

If you only identified a problem, that’s fine too, just say so:

Noticed a critical customer-facing bug that was impacting many users; after the team responsible for that area fixed it, they reported a $300,000 increase in revenue as a result.

Communicating value at different career stages

Now that you’ve seen how to phrase your skill level, let’s see how this works at different career stages.

Implementers

When you’re at the start of your career you will mostly be implementing other people’s solutions. However, in one or two cases you might be starting to solve problems, or even find problems. Make sure your resume highlights those instances, however small.

Sometimes this will happen in non-programming contexts. For example, I knew one early stage software engineer who made very insightful suggestions about hiring. Mention it anyway.

If you had another career before switching to programming, you’ll likely have plenty of examples. Make sure to highlight those even if they’re unrelated to coding; those problem-finding and solving skills will at least partially transfer.

Solvers

If you can solve problems on your own, you want to both:

1. Communicate this fact.
2. Highlight the places where you did identify problems, even if it’s happened in only a few cases.

Review your resume and make sure all the relevant entries explicitly talk about the ways in which you came up with the solution. If you already have a suitable job title, like senior software engineer, then getting the phrasing right isn’t quite as important—but it’s still worth doing.

If you still have a junior job title but your skills have progressed, it’s doubly important to ensure you’re highlighting your problems solving skills.

Once you’ve done that, try to expand on any places where you were involved in identifying problems.

Finders

Your goal as a Finder is to ensure you’re not confused with a Solver. It’s very easy to phrase things in a way that doesn’t make clear you identified the problem—after all, identifying the problem may only have a taken a few minutes.

But those initial few minutes where you noticed something needs to be done are quite often the most value parts of the whole process, so make sure you explicitly talk about finding the problem. This is even more important if your current job title doesn’t reflect your actual level of skill.

What to do next

Even if submitting resumes isn’t the best way to find a job, you still need one, and writing it is a good way to rehearse for an interview.

So get your resume out, and for each experience entry make sure it’s clear whether you implemented the solution, solved the problem, and/or found the problem. This will take you an hour, no more, and at the end of this process you’ll have a much easier time communicating some of your most valuable non-technological skills.

And if you’d like to learn some more of those skills, check out my new book, The Secret Skills of Productive Programmers.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

Another week has passed, and another 3 million people in the US have filed for unemployment. While the current situation hasn’t impacted programming jobs quite as much, it’s just a matter of time before the economic damage hits most everywhere. There will be layoffs, and plenty of them, and occasionally whole companies shutting down.

So even if your job is secure now, you might still lose it in the future. How can you prepare? What can you do to reduce your future risks?

The first thing you need to do is come up with a plan, which is what this article is all about. In particular, you will want to:

• Try to make sure you have the necessary financial resources.
• Make your future job hunt easier, by building a network, making sure your skills are up-to-date, and making sure you have visible public proof of your skills.
• Come up with a series of fallback plans if things don’t go well.

Let’s go over these one-by-one.

Money in the bank

If you lose your job, you lose your paycheck—but you still have to pay your bills. And after the dot-com bust, the last big tech recession, it took years for all the jobs to come back

If you have at least six months of living expenses in cash, that’s a good start. If not, it’s best to think about how to get there.

There are two sides to this:

1. If possible, you need to cut your expenses, which will both allow you to save and reduce how much money you need for each unsalaried month. See this more detailed article.
2. Ensuring your financial assets, if you have any, aren’t correlated with your job.
   1. If you own stock in your own company, you are making a double bet: if the company goes down, you will lose money and your job.
   2. If you work for a startup that needs to raise money soon, a crashing stock market will also greatly reduce the viability of your current job.
   3. More broadly, if you own stocks and to a lesser extent corporate bonds, how correlated are they with your ability to keep a job?
   4. Even more broadly, how much of your net worth is tied to the tech industry, or the economy as a whole?

In short, you want cash on hand, and plenty of it.

Making your future job hunt easier

Searching for a job will be much easier if you:

• Know lots of people.
• Have useful skills.
• Can visibly demonstrate you have those skills.

Let’s cover those one by one.

Knowing lots of people

Applying for a job by sending in your resume is the hardest way to get hired. It’s much easier if you know someone who can vouch for you, can get you past the initial screen, or can fill you in on what the hiring manager really wants.

So the more people you know, the better off you are. Elsewhere I have a guest post about (social) networking, but that can take time and is harder during a pandemic. But there are still a few easy things you can do in the short term:

• Join a public Slack or two for the technology area you specialize in. You can help answer people’s questions, see when people mention they’re hiring, and more broadly get a better sense of the zeitgeist, which is useful for building your skills (see below).
• Keep the contact info for former co-workers. This can be done via LinkedIn, for example, and often there will be an ex-employee Slack. If there isn’t one, you can start it—especially if your company is having initial rounds of layoffs. This too can often be very educational, as former employees might be more forthcoming.
• Find ways to help other people. Can you teach useful skills? Join a local mutual aid organization?

Build useful skills

If you’ve been working at the same job for a while, it’s easy for your technical skills to get a little stale. Unless you’re working at the right place, hang out with the right people, or do the right things, you might not be aware of the latest technology, or you might be using out-of-date practices.

So you’ll want to update your skills a little. As always, doing this extensively outside of your job may not be possible, so try to:

1. Spend an hour a week, ideally during work hours, getting up-to-date on the latest technologies. The goal here is breadth, not depth: sign up for a newsletter for your technology stack (he’s a partial list), skim the topics at a relevant conference, maybe watch a talk or two. I cover learning for breadth here, but the basic idea is that knowing a tool exists and what it does can take very little time, and is quite valuable on its own: both on the job, but also in interviews (“I haven’t used it myself, but I believe tool X is how you would solve this”).
2. Try to learn more technologies on the job, because that is the best place to do so.

Create visible proof of skills

Having skills is one thing, proving you have them is another. It is therefore quite useful during a job hunt to have some visible, public proof you have these skills. For example:

Open source: When I moved to the US in my 20s, my work on an open source project made it much easier for me to get job interviews, and eventually job offers. It wasn’t just that my resume said that I knew computer networking, I could point to a publicly available project used by real people and say “I worked on that”.

Even if you share code that isn’t widely used, it can still be useful as proof of skill.

Conference talks: Speaking publicly about a particular skill, technology, or project is a great way to get public proof of skills. With conferences moving online, speaking at conferences is now much easier. You don’t have to travel or pay for you travel, and you don’t have to get approval from your manager to lose work. If there’s a topic where you know enough to help someone else, look for conferences on the topic and submit a proposal.

Blogging: Have something to share, or learning something new? Write it down and share it publicly. Writing well is an immensely useful skill in general, so this will also count as improving your skills. You can write for your own blog, or you can propose a blog post on your company’s tech blog, if they have one.

Fallback plans

In an ideal world you would lose your job, start a job search, find a new job within a month, and everything will be fine. Sadly we don’t always live in an ideal world.

So if you live in a country like the US that a shitty social net it’s worth coming up with a series of fallback plans, if only for your own peace of mind.

For example, how can you make your money last longer?

1. As soon as you lose your job, apply for unemployment.
2. Cut additional costs.
3. If time stretches on and you still don’t have a job, figure out ways to reduce housing costs. Are you young and have the ability to move back in with your parents? Have more room than you need and the option of adding roommates? All in a pandemic-safe way, of course.
4. Any ways you can make money some other way, if it’s really taking too long?

If you can’t find a job immediately, you will have probably have more time to upgrade your skills.

1. Which skills are worth working on?
2. What’s the best way to improve them?

You’ll also want to meet more people who can help you find a job.

1. Can you go to online meetups?
2. Find more places to interact with people online?

Write this all down, and when you’re worried you’ll at least have the comfort of knowing there will be some things you can do if and when your job goes away.

We’re all in this together

As with most big problems, there is only so much you can do as an individual: to meaningfully improve the situations we need to work together, whether in mutual aid groups or via political organization. On the other hand, you need to ensure that you as an individual are doing OK; you can’t help others if you’re collapsing under your own troubles.

And since this can all be overwhelming, start with a few simple actions:

1. Cut an expense or two.
2. Get in touch with some old co-workers.
3. Sign up for a newsletter.
4. Start writing down your fallback plans.

And then, once you have things under control emotionally, when you have a plan and you know what you’re doing next, start thinking about how you can help others, and work with other people to improve things for everyone.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

Some projects have the policy that all tests must have an explanatory comment – including all of mine. At first, I found that baffling. If that’s you right now, this article is for you.

Numbers in Python come in all shapes and forms. The reason different kind of representations of numbers exist is because they all have different trade-offs. These trade-offs are often surprising!

>>> 1 + 2 - 2 - 1
0
>>> 0.1 + 0.2 - 0.2 - 0.1
2.7755575615628914e-17

>>> a = 2**-53
>>> (a + a) + 1 == a + (a + 1)
False

>>> print(set(type(p) for p in primes))
>>> one = fractions.Fraction(1)
>>> before = datetime.now()
>>> res = sum(one/p for p in primes[:10000])
>>> after = datetime.now()
>>> print("It took", after-before)
>>> print("Size of output", len(str(res)))
>>> print("Approximate value", float(res))
{<class 'int'>}
It took 0:01:16.033260
Size of output 90676
Approximate value 2.7092582487972945

>>> print(set(type(p) for p in primes))
>>> before = datetime.now()
>>> res = sum(1/p for p in primes[:10000])
>>> after = datetime.now()
>>> print("It took", after-before)
>>> print("Size of output", len(str(res)))
>>> print("Approximate value", float(res))
{<class 'int'>}
It took 0:00:00.000480
Size of output 17
Approximate value 2.709258248797317

Approximate value 2.7092582487972945
Approximate value 2.709258248797317
                    1234567891234

>>> getcontext().prec = 6
>>> Decimal(1) / Decimal(7)
Decimal('0.142857')
>>> getcontext().prec = 28
>>> Decimal(1) / Decimal(7)
Decimal('0.1428571428571428571428571429')

>>> getcontext().prec = 6
>>> # 6853 lines elided
... with localcontext() as ctx:
...     ctx.prec = 10
...     Decimal(1) / Decimal(7)
...
Decimal('0.1428571429')

(This article is based on the one I originally published on OpenSource.com.)

In 2010, the My Little Pony franchise was rebooted with the animated show My Little Pony: Friendship is Magic. The combination of accessibility to children with the sophisticated themes the show tackled garnered a following that cut across ages. I was swept up in the wave and discovered there is a lot to learn about DevOps from the show.

The show begins with Twilight Sparkle reading obscure documentation, only to realize that Equestria, where the show is set, is due to suffer a calamity. Though Nightmare Moon has been imprisoned for a thousand years, there is a prophecy she will return.

Lesson 1: Technical debt matters.

Document technical debt. Pay attention to the signs of risk no matter how infrequently they occur. Have a plan to resolve it.

Twilight Sparkle goes to her manager with the news, only to be told that it is not a current priority. She is sent to Ponyville to prepare for the coming celebration, instead.

Lesson 2: Communication with management is key.

Twilight Sparkle communicated her priority but did not convince her management that it was more important than the celebration.

We all need to make clear what the business case is for resolving critical issues. It is also not straightforward to explain technical debt in business terms. If management does not agree on the severity, find new ways to communicate the risk, and team up with others who speak that language.

As the prophecy has foreseen, Nightmare Moon returns and declares eternal night. Twilight quickly understands that she cannot resolve the issue by herself, and she recruits the ponies who will become, with her, the "Mane Six." They each stand for a different element of harmony — Applejack stands for Honesty, Fluttershy for Kindness, Pinkie Pie for Laughter, Rarity for Generosity, Rainbow Dash for Loyalty, and Twilight Sparkle herself for Magic.

Lesson 3: Few are the issues that can be resolved by one person.

When facing an outage, reach out to other people with complementary skills who can help you. It is best if they are different than you: different backgrounds leads to differing perspectives, and that can lead to better problem-solving.

Lesson 4: When resolving an outage, honest communication is key.

Throughout the struggle against the eternal night, the Mane Six have to speak openly and honestly about what's not working. Their blameless communication is part of problem-solving.

Lesson 5: When resolving an outage, kindness to yourself and to others is crucial.

Though tempers flare hot in the land of Equestria, we all benefit from coming back to working together.

Lesson 6: Laughter is important.

Even when everything comes crashing down, remember to take a break, drink a glass of water, and take a deep breath. Stressing out does not help anything.

Lesson 7: Be generous.

Even if you are not on-call right now, if your help is needed to resolve a problem, help out as you hope your colleagues will do for you.

Lesson 8: Be loyal.

An outage is not a time to settle rivalries between teams. Focus on how to collaborate and resolve the outage as a team.

Lesson 9: Though people skills are important, you have to understand the technology on a deep level.

Keep your tech skills sharp. Expertise is not only the ability to learn; it is knowing when that information is needed. Part of being an expert is practice.

After the issue is resolved, Princess Celestia realizes that the Mane Six are crucial to the long-term survival of Equestria, and tells Twilight Sparkle to stay in Ponyville and keep researching the magic of friendship.

Lesson 10: After an outage is resolved, conduct a review, take concrete lessons, and act on them.

I could go on, episode by episode, detailing lessons relevant for DevOps, but I will wrap up with one of my favorite ones.

In the "Winter Wrap-Up" episode, all the ponies in Ponyville help in preparing for the spring. As per tradition, they do not use magic, leaving Twilight Sparkle to wonder how she can contribute. Eventually, she realizes that she can help by making a checklist to make sure everything is done in the right order.

Lesson 11: When automation is impossible or inadvisable, write a solid checklist, and follow it. Do not depend on your memory.

Twilight Sparkle and the Mane Six overcome great obstacles as a team, and now have a system to improve as a team. I hope you, too, can help bring a collaborative DevOps culture to your work.

John H. Conway passed away ten days ago, and I think it's only now I can write a proper eulogy.

I was first introduced to his work, if not his name, when I was at the end of elementary school. I am sure everyone has heard about the Game of Life, but did you know it had a 1D version? The 1D version is significantly simpler, but has the advantage that on a grid paper, you can just play with yourself manually by putting a generation on each line.

This was 12 year old me's "fidget spinner", how I kept myself calm in classes. Starting with an initial configuration and letting it evolve.

Later on, when I went to college, I got to borrow his amazing book, "On Numbers and Games". Now, I am definitely the sort of person who reads math books for fun, but most of them are not fun. They are dry, poorly written, and make leaps all the time. "ONAG" was the exact opposite. It's a short, delightful book, that tries to get across the thinking, the intuition, the methods, and, yes, the joy.

Fast forward a decade or two, and again I found myself enamored with another one of his inventions: the Look-and-Say sequence. My old interview coding question was getting too popular on the interview-question-sites, and I was getting worried. Writing code for the look-and-say sequence is reasonably straightforward, but does require basic skills: looping while keeping a bunch of state variables.

Then I read about his work on the look-and-say sequence, and was utterly amazed and delighted by it. Atoms and decay and asymptotic growth!

Throughtout his career, I think what made his things special is that he embodies the truest mathematician spirit, which is also the truest geek spirit: starting out with something simple, and then nerding out about it until you have built a whole universe.

Whether it is a place where guns shoot spaceships at 3/8 the speed of light, an algebraic field so vast it includes all other ordered fields and also all infinities, or a concept of numbers atomically decaying, he was a master at whipping out mathematicially consistent fictional worlds.

Goodbye John H. Conway, you were taken from us too soon.

This article was written during abnormal circumstances, with much of the planet under lockdown due to the COVID-19 pandemic. Parents with children at home have far less time, and pretty much everyone is feeling stressed and distracted.

Under more normal circumstances there are only so many hours in the day to do your job; now it’s even worse. And yet work needs to get done: code needs to get written, features need to be shipped, bugs need to be fixed.

Faced with an ever growing list of tasks, how do you get everything done?

The short answer is, you can’t. You will never get everything done.

What you can do, though, is choose the right work, the most valuable work, the most useful work, the work with most leverage. Choose the right work and you can gets orders of magnitude improvement in your output.

Let’s see how.

The goal: increased output

Your output as a programmer is based both on your productivity and on how much time you work:

Output = Productivity × Time Worked

The first thing to notice is that there is a hard limit on how much increasing your working hours can help. After all, there are only 168 hours in a week.

If you never slept, ate, or did anything but work—and this will literally kill you—you can work 4.2× as much as a 40-hour workweek, and that’s it. And even with smaller increases in work hours, the gains quickly decline. As you work more hours you’ll become fatigued and make more mistakes; beyond a certain point those extra work hours will decrease your productivity, canceling out any gains.

What is output for a programmer?

Since increasing working hours isn’t really an option, the key to increasing your output is increasing your productivity. Productivity is the output you produce in each fixed unit of time, for example:

Productivity = Output per week

If you’re going to improve your productivity, you need to understand how to measure output.

The obvious measure is how much code you write: the more code, the better. This measure is obvious, popular, and completely wrong.

All other things being equal, is it better to implement the same feature with 10 lines of code, or 10,000 lines of code? If we measure output by code produced, the latter solution is better, but in most cases a 10 line solution is preferable to 10,000 lines. More code means higher maintenance costs, not to mention more opportunities for defects.

Your job as a programmer is not writing code, your job is solving problems: software is a tool, a means to an end. Software becomes valuable because of the problems it solves.

As a rough measure, your output as a programmer can be measured by the problems you solve: the more significant the problems you can solve, the better.

If you work for a business, significance eventually translates directly or indirectly into monetary terms: money made or money saved. In other areas you can come up with domain-specific concrete measures of usefulness: number of people served, carbon emissions reduced, number of scientists using your software, and so on.

Note: If the problems you solve produce negative value you will become anti-productive: the better you are at your job, the more damage you will cause.

If making money hurts people or the environment, your work may be productive for your employer but anti-productive for society as a whole. So make sure you’re carefully considering the ethical consequences of your actions as a worker.

How to increase productivity

Given the above, here’s how you can increase your productivity:

1. Find the most significant problem you can work on.
2. Come up with the most efficient solution to that problem.
3. Implement the solution with minimum wasted time.

Let’s go through these steps one by one, and see why they’re key to productivity.

1. Find the most significant problem

Let’s consider our formula for productivity again:

Productivity = Significant problems solved / Week

There are many problems you could be working on, so first you have to choose one. If you could solve either of these problems, should you be working on:

1. Implementing a particular missing feature; this will increase revenue by $50,000.
2. Fixing a bug that was decreasing customer retention; this will increase revenue by $1,000,000.

All other things being equal, the second problem is obviously the one you should be focusing on. Even if it takes 10× as long to solve and implement that bug fix, it should still be the highest priority:

Productivity of #1 =    $50,000 /  1 Week  =  $50,000 / Week
Productivity of #2 = $1,000,000 / 10 Weeks = $100,000 / Week

Here’s the issue: in order to fix that expensive bug and improve customer retention, you need to know the problem exists. If no one ever notices that customers are leaving, if no one ever finds that bug, if no one realizes the connection between the two—then that problem will never be solved.

And that’s why finding problems is the first and most valuable step in increasing productivity.

2. Come up with an efficient solution

Once you’ve identified the most significant problem—or once your manager assigns you a problem they identified—you need to come up with a solution.

Which solution do you think is better?

1. Takes 1000 lines of code and 4 weeks to implement.
2. Takes 100 lines of code and 3 days to implement.

All other things being equal, the second solution is obviously better. But again, you need to find that solution.

If you only ever find that first solution, then no matter how efficiently you implement it, no matter how focused you are, no matter how much you manage to speed things up—you’re still implementing a much less efficient solution.

And that’s why identifying better solution is the second most valuable step in increasing productivity.

3. Implement the solution without wasting a time

Once you’ve identified a problem and chosen the solution, there is only so much leverage you have to improve productivity. You obviously want to avoid getting stuck and spinning your wheels, because wasted time reduces your productivity.

But given a particular solution, there’s only so much waste you can reduce, only so fast you can go:

Wasted time → $50,000 / 2 weeks = $25,000 / week
No waste    → $50,000 / 1 week  = $50,000 / week

Efficient implementation is the last and least valuable way of increasing productivity.

Technological skills aren’t enough

While you get the most increased productivity from identifying problems and the least from efficient implementation, your career as a programmer progresses in the opposite direction:

1. Junior engineers implement solutions.
2. Senior engineers find solutions and implement them.
3. Principal or staff engineers identify problems, find solutions, and implement them.

So becoming more productive isn’t just about helping your employer’s bottom line, it’s also about learning the skills that will give you more pay and more influence.

Critically, technological skills are necessary but not sufficient to increase your productivity:

• Your JavaScript skills don’t matter if you can never meet deadlines.
• Your testing skills don’t matter if you can’t convince your manager of the value of testing.
• Your software architecture skills don’t matter if no one has ever heard of your product.

Why these skills are “secret”

Most discussions of programming productivity tend to end up focusing purely on technology, coding, and design skills, and skip over these problem-solving skills. Of course, this isn’t a conspiracy of silence, no one is deliberately hiding the existence of the skills.

My guess is that experienced programmers still have to learn new technologies, so they’re more likely to realize the need to explain those particular skills. But having learned them once, they apply skills like timeboxing, or considering multiple different solutions to a problem, without even noticing they’re doing it. And so they end up talking about problem-solving skills rather less, and about technological skills rather more.

How do you learn these skills?

This article is an excerpt from my book, The Secret Skills of Productive Programmers, covering the non-technical skills you need to get better at identifying problems, solving problems, and implementing them on schedule.

Elsewhere on this site you’ll find many free articles on building up your skills.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

The Twisted Requests (treq) package is an HTTP client built on the popular Twisted library that is used for web requests. Async libraries offer the ability to do large amounts of network requests in parallel with relatively little CPU impact. This can be useful in HTTP clients that need to make several requests before they have all the information they need.

This post shows an example of a problem like this, and how to solve it using treq.

I enjoy playing the real-time strategy game Clash Royale. Clash Royale is a mobile strategy player-vs-player game where players play cards in an arena to win. Each card has different strengths and weaknesses, and different players prefer different cards. Clash Royale remembers which card a player plays the most; this is their "favorite" card. Players come together in clans where they can help each other. Supercell, Clash Royale's developer, released an HTTP-based API where different statistics can be queried.

How can we write a program that will output the most popular favorite cards in a clan?

If you want to follow along, you will need to register an account. If you register an account, create an API token via the Clash Royale developer portal. Then choose "Create New Key" under your profile, and enter a name, description, and a valid IP address. (An exact address is required.) Since you should never save an API key in your code, keep it as a separate file in ~/.crtoken:

$ ls ~/.crtoken
/home/moshez/.crtoken

To make it easier to see what is going on, let's start with this introductory program that prints Hello world, and then we'll talk through what it does:

import collections, json, os, sys, urllib.parse
from twisted.internet import task, defer
import treq

with open(os.path.expanduser("~/.crtoken")) as fpin:
    token = fpin.read().strip()

def main(reactor):
    print("Hello world")
    return defer.succeed(None)

task.react(main, sys.argv[1:])

This imports many more modules than we need for the "Hello world" example. We will need these modules for the final version of the program, which will accomplish the more complex task of asynchronously querying an API. After the import, the program reads the token from the file and stores it in the variable token. (We are not going to do anything with the token right now, but it's good to see that code.) Next there is a main function that accepts a Twisted reactor. A reactor is sort of like an interface to the machinery of the Twisted package. In this case the function main is sent as a parameter to task.react, and, which will call main with the reactor and any arguments we give -- the command-line arguments, in this case.

The main function returns a defer.succeed(None). This is how it returns a value of the right type: a deferred value, but one that already has been "fired" or "called." Because of that, the program will exit immediately after printing Hello world, as we need.

Next, we will look at the concepts of async functions and ensureDeferred:

async def get_clan_details(clan):
     print("Hello world", clan)

def main(reactor, clan):
    return defer.ensureDeferred(get_clan_details(clan))

task.react(main, sys.argv[1:])

In this program, which should start with the same imports, we moved all the logic to the async function get_clan_details. Just like a regular function, an async function has an implicit return None at the end. However, async functions, sometimes called co-routines, are a different type than Deferred. In order to let Twisted, which has existed since Python 1.5.2, use this modern feature, we must adapt the co-routine using ensureDeferred.

While we could write all the logic without using co-routines, using the async syntax will allow us to write code that is easier to understand, and we will need to move a lot less of the code into embedded callbacks.

The next concept to introduce is that of await. Later, we will await a network call, but for simplicity, right now, we will await on a timer. Twisted has a special function, task.deferLater, which will call a function with given parameters after some time has passed.

The following program will take five seconds to complete:

async def get_clan_details(clan, reactor):
     out = await task.deferLater(
         reactor,
         5,
         lambda clan: f"Hello world {clan}",
         clan
     )
     print(out)

def main(reactor, clan):
    return defer.ensureDeferred(get_clan_details(clan, reactor))

task.react(main, sys.argv[1:])

A note about types: task.deferLater returns a Deferred, as do most Twisted functions that do not have the value already available. When running the Twisted event loop, we can await on both Deferred values and co-routines.

The function task.deferLater will wait five seconds and then call our lambda, calculating the string to print out.

Now we have all the Twisted building blocks needed to write an efficient clan-analysis program!

Since we will be using the global reactor, we no longer need to accept the reactor as a parameter in the function that calculates these statistics. The way to use the token is as a "bearer" token in the headers:

async def get_clan_details(clan):
    headers={b'Authorization': b'Bearer '+token.encode('ascii')}

We want clan tags to be sent, which will be strings. Clan tags begin with #, so they must be quoted before they're put in URLs. This is because # has the special meaning "URL fragment":

async def get_clan_details(clan):
     # ...
     clan = urllib.parse.quote(clan)

The first step is to get the details of the clan, including the clan members:

async def get_clan_details(clan):
     # ...
     res = await treq.get("https://api.clashroyale.com/v1/clans/" + clan,
                          headers=headers)

Notice that we have to await the treq.get call. We have to be explicit about when to wait and get information since it is an asynchronous network call. Just using the await syntax to call a Deferred function does not let us take full power of asynchronicity (we will see how to do it later).

Next, after getting the headers, we need to get the content. The treq library gives us a helper method that parses the JSON directly:

async def get_clan_details(clan):
     # ...
     content = await res.json()

The content includes some metadata about the clan, which is not interesting for our current purposes, and a memberList field that contains the clan members. Note that while it has some data about the players, the current favorite card is not part of it. It does include the unique "player tag" that we can use to retrieve further data.

We collect all player tags, and, since they also begin with #, we URL-quote them:

async def get_clan_details(clan):
     # ...
     player_tags = [urllib.parse.quote(player['tag'])
                    for player in content['memberList']]

Finally, we come to the real power of treq and Twisted: generating all requests for player data at once! That can really speed up tasks like this one, which queries an API over and over again. In cases of APIs with rate-limiting, this can be problematic.

There are times when we need to be considerate to our API owners and not run up against any rate limits. There are techniques to support rate-limiting explicitly in Twisted, but they are beyond the scope of this post. (One important tool is defer.DeferredSemaphore.)

async def get_clan_details(clan):
     # ...
     requests = [treq.get("https://api.clashroyale.com/v1/players/" + tag,
                          headers=headers)
                 for tag in player_tags]

Remember that requests do not return the JSON body directly. Earlier, we used await so that we did not have to worry about exactly what the requests return. They actually return a Deferred. A Deferred can have an attached callback that will modify the Deferred. If the callback returns a Deferred, the final value of the Deferred will be the value of the returned Deferred.

So, to each deferred, we attach a callback that will retrieve the JSON of the body:

async def get_clan_details(clan):
     # ...
     for request in requests:
         request.addCallback(lambda result: result.json())

Attaching callbacks to Deferreds is a more manual technique, which makes code that is harder to follow but uses the async features more efficiently. Specifically, because we are attaching all the callbacks at the same time, we do not need to wait for the network calls, which potentially can take a long time, to indicate how to post-process the result.

From Deferreds to values

We cannot calculate the most popular favorite cards until all results have been gathered. We have a list of Deferreds, but what we want is a Deferred that gets a list value. This inversion is exactly what the Twisted function defer.gatherResults does:

async def get_clan_details(clan):
     # ...
     all_players = await defer.gatherResults(requests)

This seemingly innocent call is where we use the full power of Twisted. The defer.gatherResults function immediately returns a deferred that will fire only when all the constituent Deferreds have fired and will fire with the result. It even gives us free error-handling: if any of the Deferreds error out, it will immediately return a failed deferred, which will cause the await to raise an exception.

Now that we have all the players' details, we need to munch some data. We get to use one of Python's coolest built-ins, collections.Counter. This class takes a list of things and counts how many times it has seen each thing, which is exactly what we need for vote counting or popularity contests:

async def get_clan_details(clan):
     # ...
     favorite_card = collections.Counter([player["currentFavouriteCard"]["name"]
                                          for player in all_players])

Finally, we print it:

async def get_clan_details(clan):
     # ...
     print(json.dumps(favorite_card.most_common(), indent=4))

So, putting it all together, we have:

import collections, json, os, sys, urllib.parse
from twisted.internet import task, defer
import treq

with open(os.path.expanduser("~/.crtoken")) as fpin:
    token = fpin.read().strip()


async def get_clan_details(clan):
     headers = headers={b'Authorization': b'Bearer '+token.encode('ascii')}
     clan = urllib.parse.quote(clan)
     res = await treq.get("https://api.clashroyale.com/v1/clans/" + clan,
                          headers=headers)
     content = await res.json()
     player_tags = [urllib.parse.quote(player['tag'])
                    for player in content['memberList']]
     requests = [treq.get("https://api.clashroyale.com/v1/players/" + tag,
                          headers=headers)
                 for tag in player_tags]
     for request in requests:
         request.addCallback(lambda result: result.json())
     all_players = await defer.gatherResults(requests)
     favorite_card = collections.Counter([player["currentFavouriteCard"]["name"]
                                          for player in all_players])
     print(json.dumps(favorite_card.most_common(), indent=4))

def main(reactor, clan):
    return defer.ensureDeferred(get_clan_details(clan))

task.react(main, sys.argv[1:])

Thanks to the efficiency and expressive syntax of Twisted and treq, this is all the code we need to make asynchronous calls to an API. If you were wondering about the outcome, my clan's list of favorite cards is Wizard, Mega Knight, Valkyrie, and Royal Giant, in descending order.

(This post is based on the article I wrote for opensource.com)

It has been a long time since I learned how to program, and it is easy to forget some of the hard-won lessons in the beginning. Easy until I try to teach people to program. There is a lot of accidental and inherent complexity in programming, but I am ready for that: I remember to explain how carefully to follow the syntax, and the kind of syntactical gotchas that are easy to fail.

But there is one metalesson that is much easier to forget, and much harder to learn, and to teach. Humans are used to small mistakes having reasonably small consequences. But even in cases with catatrophic consequences, the consequences look related to the mistakes.

However, in programming, small mistakes can lead not just to big consequences, but to weird consequences. A missing comma might mean that things work fine in the testing environment, but in production, every third request gets a slightly wrong result.

This really stumps people. They copy code somewhat imperfectly from the board, or make a small mistake when they edit it to change from "Hello world" to "Goodbye world", and suddenly, a completely unrelated part of the program starts going haywire.

This happens to old hands too. The number of times I have edited code and ran the tests, only to discover a clearly unrelated test failing, is not small. The difference that comes with experience is that I take a deep breath, think "I've got this", and start down the troubleshooting path.

The troubleshooting can include any number of things: I might go in with a debugger, add print statements, do a bisect-diff to figure out what caused the problem, try random things to see what happens, or just trace the execution path carefully.

The troubleshooting process does not matter as much as what comes before it: the deep breath. This is my time to accept the problem has happened, and that I am in for a process which can take two hours, and at the end of which my entire productivity will be "added missing semicolon". Sometimes a breath is not enough, and I need to get up and get some tea. But the most important, and almost invisible step, at the beginning is to step back, remember that this is, weirdly, part of the job, and to become comfortable doing it.

If you want to be any kind of software developer, accept it now. Much of your life will be seeing weird consequences, and tracing it back to a small mistake. Eventually, like everyone, you will succeed. Flush with victory, make a note of your success somewhere: anywhere where the overhead of writing it is low, be it an e-mail to yourself or a note-keeping app.

If you ever decide to teach, or write a blog, this is an unending source of content.

(Thanks to Veronica Hanus for her feedback on an early draft. All issues and mistakes that remain are my responsibility.)

• curve25519-sha256 key exchange algorithm support in Conch.
• "openssh-key-v1" key format support in Conch.
• Security fixes to twisted.web, including preventing request smuggling attacks and rejecting malformed headers. CVE-2020-10108 and CVE-2020-10109 were assigned for these issues, see the NEWS file for full details.
• twist dns --secondary now works on Python 3.
• The deprecation of twisted.news.
• ...and various other fixes, with 28 tickets closed in total. 

There are many wordy articles on configuring your web server’s TLS ciphers. This is not one of them. Instead I will share a configuration which is both compatible enough for today’s needs and scores a straight “A” on Qualys’s SSL Server Test.

This was originally sent to my newsletter. I send one e-mail, always about Python, every other Sunday. If this blog post interests you, consider subscribing.

The underappreciated else keyword in Python has three distinct uses.

if anonymize:
    print("Hello world")
else:
    print("Hello, name")

for x in numbers:
    if x % 2 == 1:
        print("Found", x)
        break
else:
    print("No odd found")

try:
    before, after = things
except ValueError:
    part1 = things[0]
    part2 = 0
    after = 0
else:
    part1, part2 = before

GitHub’s own CI called GitHub Actions has been out of closed beta for a while and offers generous free quotas and a seamless integration with the rest of the site. Let’s have a look on how to use it for an open source Python package.

I’m missing a key part from the public Python discourse and I would like to help to change that.

Since this topic keeps coming up, I’d like to briefly share my thoughts on Python package metadata because it’s – as always – more complex than it seems.

What is a threat? From a game-theoretical perspective, a threat is an attempt to get a better result by saying: "if you do not give me this result, I will do something that is bad for both of us". Note that it has to be bad for both sides: if it is good for the threatening side, they would do it anyway. While if it is good for the threatened side, it is not a threat.

Threats rely on credibility and reputation: the threatening side has to be believed for the threat to be useful. One way to gain that reputation is to follow up on threats, and have that be a matter of public record. This means that the threatening side needs to take into account that they might have to act on the threat, thereby doing something against their own interests. This leads to the concept of a "credible" or "proportionate" threat.

For most of our analysis, we will use the example of a teacher union striking. Similar analysis can be applied to nuclear war, or other cases. People mostly have positive feelings for teachers, and when teacher unions negotiate, they want to take advantage of those feelings. However, the one thing that leads people to be annoyed with teachers is a strike: this causes large amounts of unplanned scheduling crisis in people's lives.

In our example, a teacher union striking over, say, a minor salary raise disagreement is not credible: the potential harm is small, while the strike will significantly harm the teachers' image.

However, strikes are, to a first approximation, the only tool teacher unions have in their arsenal. Again, take the case of a minor salary raise. Threatening with a strike is so disproportional that there is no credibility. We turn to one of the fundamental insights of game theory: rational actors treat utility as linear in probability. So, while starting a strike that is twice as long is not twice as bad, increasing the probability of starting a strike from 0 to 1 is twice as bad (exactly!) as increasing the probability from 0 to 0.5.

(If you are a Bayesian who does not believe in 0 and 1 as probabilities, note that the argument works with approximations too: increasing the probability from a small e to 0.5 is approximately twice as bad as increasing it from e to 1-e.)

All one side has is a strike. Assume the disutility of a strike to that side is -1,000,000. Assume the utility of winning the salary negotiation is 1. They can threaten that if their position is not accepted, they will generate a random number, and if it is below 1/1,000,000, they will start the strike. Now the threat is credible. But to be gain that reputation, this number has to be generated in public, in an uncertain way: otherwise, no reputation is gained for following up on threats.

In practice, usually the randomness is generated by "inflaming the base". The person in charge will give impassioned speeches on how important this negotiation is. With some probability, their base will pressure them to start the strike, without them being able to resist it.

Specifically, note that often a strike is determined by a direct vote of the members, not the union leaders. This means that union leaders can credibly say, "please do not vote for the strike, we are against it". With some probability, that depends on how much they inflamed the base, the membership will ignore the request. The more impassioned the speech, the higher the probability. By limiting their direct control over the decision to strike, union leaders gain the ability to threaten probabilistically.

Nuclear war and union strikes are both well-studied topics in applied game theory. The explanation above is a standard part of many text books: in my case, I summarized the explanation from Games of Strategy, pg. 487.

What is not well studied are the dynamics of open source projects. There, we have a set of owners who can directly influence such decisions as which patches land, and when versions are released. More people will offer patches, or ask for a release to happen. The only credible threat they have is to fork the project if they do not like how it is managed. But forking is often a disproportinate threat: a patch not landing often just means an ugly work-around in user code. There is a cost, but the cost of maintaining a fork is much greater.

But similar to a union strike, or launching a nuclear war, we can consider a "probabilistic fork". Rant on twitter, or appropriate mailing lists. Link to the discussion, especially to places which make the owners not in the best light. Someone might decide to "rage-fork". More rants, or more extreme rants, increase the probability. A fork has to be possible in the first place: this is why the best way to evaluate whether something is open source is to consider "how possible is a fork".

This is why the possibility of a fork changes the dynamics of a project, even if forks are rare: because the main thing that happens are "low-probability maybe-forks".

Never send a human to do a machine’s job.

One of the best things about maintaining open source in the modern era is that there are so many wonderful, free tools to let machines take care of the busy-work associated with collaboration, code-hosting, continuous integration, code quality maintenance, and so on.

There are lots of great resources that explain how to automate various things that make maintenance easier.

Here are some things you can configure your Python project to do:

1. Continuous integration, using any one of a number of providers:
   1. GitHub Actions
   2. CircleCI
   3. Azure Pipelines
   4. Appveyor
   5. GitLab CI&CD
   6. Travis CI
2. Separate multiple test jobs with tox
3. Lint your code with flake8
4. Type-Check your code with Mypy
5. Auto-update your dependencies, with one of:
   1. pyup.io
   2. requires.io, or
   3. Dependabot
6. automatically find common security issues with Bandit
7. check the status of your code coverage, with:
   1. Coveralls, or
   2. Codecov
8. Auto-format your code with:
   1. Black for style
   2. autopep8 to fix common errors
   3. isort to keep your imports tidy
9. Help your developers remember to do all of those steps with pre-commit
10. Automatically release your code to PyPI via your CI provider
    1. including automatically building any C code for multiple platforms as a wheel so your users won’t have to
    2. and checking those build artifacts:
       1. to make sure they include all the files they should, with check-manifest
       2. and also that the binary artifacts have the correct dependencies for Linux
       3. and also for macOS
11. Organize your release notes and versioning with towncrier

All of these tools are wonderful.

But... let’s say you¹ maintain a few dozen Python projects. Being a good maintainer, you’ve started splitting up your big monolithic packages into smaller ones, so your utility modules can be commonly shared as widely as possible rather than re-implemented once for each big frameworks. This is great!

However, every one of those numbered list items above is now a task per project that you have to repeat from scratch. So imagine a matrix with all of those down one side and dozens of projects across the top - the full Cartesian product of these little administrative tasks is a tedious and exhausting pile of work.

If you’re lucky enough to start every project close to perfect already, you can skip some of this work, but that partially just front-loads the tedium; plus, projects tend to start quite simple, then gradually escalate in complexity, so it’s helpful to be able to apply these incremental improvements one at a time, as your project gets bigger.

I really wish there were a tool that could take each of these steps and turn them into a quick command-line operation; like, I type pyautomate pypi-upload and the tool notices which CI provider I use, whether I use tox or not, and adds the appropriate configuration entries to both my CI and tox configuration to allow me to do that, possibly prompting me for a secret. Same for pyautomate code-coverage or what have you. All of these automations are fairly straightforward; almost all of the files you need to edit are easily parse-able either as yaml, toml, or ConfigParser² files.

A few years ago, I asked for this to be added to CookieCutter, but I think the task is just too big and complicated to reasonably expect the existing maintainers to ever get around to it.

If you have a bunch of spare time, and really wanted to turbo-charge the Python open source community, eliminating tons of drag on already-over-committed maintainers, such a tool would be amazing.

The Python standard library is full of underappreciated gems. One of them allows for simple and elegant function dispatching based on argument types. This makes it perfect for serialization of arbitrary objects – for example to JSON in web APIs and structured logs.

(This is based on the series published in opensource.com as 9 articles: 1, 2, 3, 4, 5, 6, 7, 8, 9)

Python contributor Tim Peters introduced us to the Zen of Python in 1999. Twenty years later, its 19 guiding principles continue to be relevant within the community.

The Zen of Python is not "the rules of Python" or "guidelines of Python". It is full of contradiction and allusion. It is not intended to be followed: it is intended to be meditated upon.

In this spirit, I offer this series of meditations on the Zen of Python.

a(function(resultsFromA) {
  b(resultsFromA, function(resultsfromB) {
    c(resultsFromC, function(resultsFromC) {
      console.log(resultsFromC)
   }
  }
}

a(function(resultsFromA) {
b(resultsFromA,
  function(resultsfromB) {
c(resultsFromC,
  function(resultsFromC) {
    console.log(resultsFromC)
}}}

future_value = future_result()
future_value.addCallback(a)
future_value.addCallback(b)
future_value.addCallback(c)

(This is based on my article on opensource.com)

Unit test suites help maintain high-quality products by signaling problems early in the development process. An effective unit test catches bugs before the code has left the developer machine, or at least in a continuous integration environment on a dedicated branch. This marks the difference between good and bad unit tests: good tests increase developer productivity by catching bugs early and making testing faster. Bad tests decrease developer productivity.

Productivity decreases when testing incidental features. The test fails when the code changes, even if it is still correct. This happens because the output is different, but in a way that is not part of the function's contract.

A good unit test, therefore, is one that helps enforce the contract to which the function is committed.

If a good unit test breaks, the contract is violated and should be either explicitly amended (by changing the documentation and tests), or fixed (by fixing the code and leaving the tests as is).

A good unit test is also strict. It does its best to ensure the output is valid. This helps it catch more bugs.

While limiting tests to enforce only the public contract is a complicated skill to learn, there are tools that can help.

One of these tools is Hamcrest, a framework for writing assertions. Originally invented for Java-based unit tests, today the Hamcrest framework supports several languages, including Python.

Hamcrest is designed to make test assertions easier to write and more precise.

def add(a, b):
    return a + b

from hamcrest import assert_that, equal_to

def test_add():
    assert_that(add(2, 2), equal_to(4))

This is a simple assertion, for simple functionality. What if we wanted to assert something more complicated?

def test_set_removal():
    my_set = {1, 2, 3, 4}
    my_set.remove(3)
    assert_that(my_set, contains_inanyorder([1, 2, 4]))
    assert_that(my_set, is_not(has_item(3)))

Note that we can succinctly assert that the result has 1, 2, and 4 in any order since sets do not guarantee order.

We also easily negate assertions with is_not. This helps us write precise assertions, which allow us to limit ourselves to enforcing public contracts of functions.

Sometimes, however, none of the built-in functionality is precisely what we need. In those cases, Hamcrest allows us to write our own matchers.

Imagine the following function:

def scale_one(a, b):
    scale = random.randint(0, 5)
    pick = random.choice([a,b])
    return scale * pick

We can confidently assert that the result divides into at least one of the inputs evenly.

A matcher inherits from hamcrest.core.base_matcher.BaseMatcher, and overrides two methods:

class DivisibleBy(hamcrest.core.base_matcher.BaseMatcher):

    def __init__(self, factor):
        self.factor = factor

    def _matches(self, item):
        return (item % self.factor) == 0

    def describe_to(self, description):
        description.append_text('number divisible by')
        description.append_text(repr(self.factor))

Writing high-quality describe_to methods is important, since this is part of the message that will show up if the test fails.

def divisible_by(num):
    return DivisibleBy(num)

By convention, we wrap matchers in a function. Sometimes this gives us a chance to further process the inputs, but in this case, no further processing is needed.

def test_scale():
    result = scale_one(3, 7)
    assert_that(result,
                any_of(divisible_by(3),
                       divisible_by(7)))

Note that we combined our divisible_by matcher with the built-in any_of matcher to ensure that we test only what the contract commits to.

While editing the article, I heard a rumor that the name "Hamcrest" was chosen as an anagram for "matches". Hrm...

>>> assert_that("matches", contains_inanyorder(*"hamcrest")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/moshez/src/devops-python/build/devops/lib/python3.6/site-packages/hamcrest/core/assert_that.py", line 43, in assert_that
    _assert_match(actual=arg1, matcher=arg2, reason=arg3)
  File "/home/moshez/src/devops-python/build/devops/lib/python3.6/site-packages/hamcrest/core/assert_that.py", line 57, in _assert_match
    raise AssertionError(description)
AssertionError:
Expected: a sequence over ['h', 'a', 'm', 'c', 'r', 'e', 's', 't'] in any order
      but: no item matches: 'r' in ['m', 'a', 't', 'c', 'h', 'e', 's']

Researching more, I found the source of the rumor: it is an anagram for "matchers".

>>> assert_that("matchers", contains_inanyorder(*"hamcrest"))
>>>

If you are not yet writing unit tests for your Python code, now is a good time to start. If you are writing unit tests for your Python code, using Hamcrest will allow you to make your assertion precise—neither more nor less than what you intend to test. This will lead to fewer false negatives when modifying code and less time spent modifying tests for working code.

You need to negotiate at a new job: for your salary, or benefits, or my personal favorite, a shorter workweek. You’re not sure what to do, or how to approach it, or what to say when the company says “how much do you want?” or “here’s our offer—what do you say?”

Here’s the thing: that final conversation about salary might be the most nerve-wracking part, but the negotiation process starts much much earlier. Which means you can enter that final conversation having positioned yourself for success—and feeling less stressed about it too.

The way you can do that is following certain basic principles, which I’ll be covering in this article. I’m going to be focusing on salary negotiation as an example, but the same principles will apply when negotiating for a shorter workweek.

In particular, I’ll be talking about:

1. An example from early in my career when I negotiated very very badly.
2. The right way to negotiate, based on four principles:
   1. Employment is a negotiated relationship.
   2. Knowledge is power.
   3. Negotiate from a position of strength.
   4. Use the right tactics.

The wrong way to negotiate

Before moving on to the principles of negotiation, let me share a story of how I negotiated badly.

During my first real job search I interviewed at a company in New York City that was building a financial trading platform. They were pretty excited about some specific technologies I’d learned while working on Twisted, an open source networking framework. They offered me a job, I accepted, and my job search was over.

But then they sent me their intellectual property agreement, and I actually read legal documents; you should read them too. The agreement would have given the company ownership over any open source work I did, including work on Twisted. I wanted to ensure I could keep doing open source development, especially given that was their reason for hiring me in the first place. I asked for an exemption covering Twisted, they wouldn’t agree, and so we went back and forth trying to reach an agreement.

Eventually they came back with a new offer: in return for not working on Twisted I’d get a 20% salary increase over their initial offer. I thought about it briefly, then said no and walked away from the job. Since I had neither a CS degree—I’d dropped out—nor much of an employment history, open source contribution was important to my career. It was how I’d gotten contracting work, and it was the reason they’d offered me this job. And I enjoyed doing it, too, so I wasn’t willing to give it up.

I posted about this experience online, and an employee of ITA Software, which was based in the Boston area, suggested they were happy to support contributions to open source projects. It seemed worth a try, so I applied for the position. And when eventually I got a job offer from ITA and they asked me for my salary requirements, I asked for the second offer I’d gotten, the one that was 20% higher than my original offer. They accepted, and I’ve lived in the Boston area ever since.

As we go through the principles below, I’ll come back to this story and point out how they were (mis)applied in my two negotiations.

The four principles of negotiation

You can think of the negotiation process as building on four principles:

1. Employment is a negotiated relationship.
2. Knowledge is power.
3. Negotiate from a position of strength.
4. Use the right tactics.

Let’s go through them one by one.

Principle #1: Employment is a negotiated relationship

If you’re an employee, your employment relationship was negotiated. When you got a job offer and accepted it, that was a negotiation, even if you didn’t push back at all. Your choice isn’t between negotiating and not negotiating: it’s between negotiating badly, or negotiating well.

Negotiate actively

If you don’t actively try to negotiate, if you don’t ask for what you want, if you don’t ask for what you’re worth—you’re unlikely to get it. Salaries, for example, are a place where your interests and your employer’s are very much at odds. All things being equal, if you’re doing the exact same work and have the same likelihood of leaving, would your employer prefer to pay you less or more? Most employers will pay you less if they can, and I almost had to learn that the hard way.

Applying the principle: In my story above, I never proactively negotiated. Instead, I accepted a job offer from the financial company without any sort of additional demands. If they were happy to offer me a 20% raise just to quit open source, I probably could have gotten an even higher salary if I’d just asked in the first place.

Negotiation starts early, and never ends

Not only do you need to negotiate actively, you also need to realize that negotiation starts much earlier than you think, and ends only when you leave to a different job:

• The minute you start thinking about applying to a company, you’ve started the negotiation process; as you’ll see, you’ll want to do research before you even talk to them.
• Your interview is part of your negotiation, and you can in fact negotiate the interview process itself (e.g. suggest sharing a code sample instead of doing a whiteboard puzzle).
• As an employee you will continue to negotiate: if you always say “yes” when your boss asks you to work long hours, your contract for a 3-day weekend will mean nothing.

In short, your whole relationship as an employee is based on negotiation.

Distinguish between friend and foe

A negotiation involves two sides: yours, and the company’s. When you’re negotiating it’s important to remember that anyone who works for the company is on the company’s side. Not yours.

I once had to negotiate the intellectual property agreement at a new job. My new employer was based in the UK, and it had a US subsidiary organized by a specialist company. These subsidiary specialists had provided the contract I was signing.

When I explained the changes I wanted to make, the manager at the subsidiary specialist told me that my complaint had no merit, because the contract had been written by the “best lawyers in Silicon Valley.” But the contract had been written by lawyers working for the company, not for me. If his claim had been true (spoiler: they were not in fact the best lawyers in Silicon Valley), that would have just made my argument stronger. The better the company’s lawyers, the more carefully I ought to have read the contract, and the more I ought to have pushed back.

The contracts the company wants you to sign? They were written by lawyers working for the company.

Human Resources works for the company, as does the in-house recruiter. However friendly they may seem, they are not working for you. And third-party recruiters are paid by the company, not you. It’s true that sometimes their commission is tied to your salary, which means they would rather you get paid more. But since they get paid only once per candidate, volume is more important than individual transactions: it’s in their best interest to get you hired as quickly as possible so they can move on to placing the next candidate.

Since all these people aren’t working for you, during a negotiation they’re working against you.

The only potential exception to this rule are friends who also work for the company, and aren’t directly involved in the negotiation process: even if they are constrained in some ways, they’re probably still on your side. They can serve as a backchannel for feedback and other information that the company can’t or won’t share.

Principle #2: Knowledge is power

The more you know about the situation, the better you’ll do as a negotiator. More knowledge gives more power: to you, but also to the company.

Know what you want

The first thing you need to do when negotiating is understand what you want.

• What is your ideal outcome?
• What can you compromise on, and what can’t you compromise on?
• What is the worst outcome you’re willing to accept?

Do your research

You also want to understand where the other side is coming from:

• What is the company’s goal, and the negotiator’s goal? For example, if you discover their goal is minimizing hassle, you might be able to get what you want by making the process a little smoother.
• What resources are available to them? An unfunded startup has different resources than a large company, for example.
• Has the company done something similar in the past, or will your request be unprecedented? For example, what hours do other employees in similar positions work? How much are other employees paid?
• What do other companies in the area or industry provide?
• How is this particular business segment doing: are they losing money, or doing great?

The more you understand going in, the better you’ll do, and that means doing your research before negotiation starts.

Applying the principle: In my story above I never did any research about salaries, either in NY or in Boston. As a result, I had no idea I was being offered a salary far below market rates.

As a comparison, here’s a real example of how research can help your negotiation, from an engineer named Adam:

Adam: “Being informed on salaries really helped my negotiating position. When my latest employer made me an offer I asked them why it was lower than their average salary on Glassdoor.com. The real reason was likely ‘we offer as little as possible to get you on board.’ They couldn’t come up with a convincing reason and so the salary was boosted 10%.”

Glassdoor is a site that allows employees to anonymously share salaries and job reviews. Five minutes of research got Adam a 10% raise: not bad at all!

Listen and empathize

If you only had to make yourself happy this wouldn’t be a negotiation: you need to understand the other side’s needs and wants, what they’re worrying about, what they’re feeling. That means you need to listen, not just talk: if you do, you will often gather useful information that can help you make yourself more valuable, or address a particular worry. And you need to feel empathy towards the person you’re talking to: you don’t need to agree or subordinate yourself to their goals, but you do need to understand how they’re feeling.

Share information carefully

Sharing information at the wrong time during a negotiation can significantly weaken your position. For example, sharing your previous salary will often anchor what the company is willing to offer you:

Adam: “I graduated from university and started working at the end of 2012. At my first job I worked for way under my market rate. I knew this and was OK with it because they were a good company.

Then I switched jobs in 2013. What I hadn’t accounted for was that my salary at my first job was going to limit my future salary prospects. I had to fight hard for raises at my next job before I was in line with people straight out of school, because they didn’t want to double my salary at my previous company.”

In general, when interviewing for a job you shouldn’t share your previous salary, or your specific salary demands—except of course when it is helpful to do so. For example, let’s say you’re moving from Google to a tiny bootstrapped startup, and you know you won’t be able to get the same level of salary. Sharing your current salary can help push your offer higher, or used as leverage to get shorter hours: “I know you can’t offer me my previous salary of $$$, but here’s something you could do—”. Just make sure not to share it too early, or they might decide you’d never accept any offer at all and stop the interview process too early.

Most of the time, however, you shouldn’t share either your previous salary or specific salary requirements. If the company insists on getting your previous salary, you can:

• If you work somewhere with relevant laws (e.g. California and Massachusetts), point out that this question is illegal. Asking about salary expectations is not illegal in these jurisdictions, so be careful about the distinction.
• Ask for the company’s salary range for the position, as well as the next level up in the salary tree. Chances are they will refuse to share, in which case you can correspondingly refuse to share your information.
• Say something like “I expect to be paid industry-standard pay for my experience.”

Applying the principle: I shouldn’t have told ITA Software my salary requirement. Instead, I should have gotten them to make the first offer, which would have given me more information about what they were willing to pay.

Principle #3: Negotiate from a position of strength

The stronger your negotiation position, the more likely you are to get what you want. And this is especially important when you’re asking for something abnormal, like a 3-day weekend.

Have a good fallback (BATNA)

If negotiation fails, what will you do? Whatever it is, that is your fallback, sometimes known as the “Best Alternative to a Negotiated Agreement” (BATNA). The better your fallback, the better your alternative, the stronger your negotiating position is. Always figure out your fallback in advance, before you start negotiating.

For example, imagine you’re applying for a new job:

• If you’re unemployed and have an empty bank account, your fallback might be moving in with your parents. This does not give you a strong negotiating position.
• If you’re employed, and more or less content with your current job, your fallback is staying where you are. That makes your position much stronger.

If you have a strong fallback, you can choose to walk away at any time, and this will make asking for more much easier.

Provide and demonstrate value

The more an organization wants you as an employee, the more they’ll be willing to offer you. The people you’re negotiating with don’t necessarily know your value: you need to make sure they understand why you’re worth what you’re asking.

For example, when you’re interviewing for a job, you need to use at least part of the interview to explain your value to your prospective employer: your accomplishments and skills. Once you’ve established the value of your skills, asking for more—more money, unusual terms—can actually make you seem more valuable. And having another job offer—or an existing job—can also help, by showing you are in demand.

Finally, remember that your goal is to make sure the other side’s needs are met—not at your own expense, but if they don’t think hiring you is worth it, you aren’t going to get anything. Here’s how Alex, another programmer I talked to, explains how he learned this:

Alex: “Think about the other person and how they’re going to react, how you can try to manage that proactively. You need to treat your negotiating partner as a person, not a program.

Initially I had been approaching it adversarially, 'I need to extract value from you, I have to wrestle you for it’ but it’s more productive to negotiate with an attitude of 'we both need to get our needs met.’ The person you’re talking to is looking to hire someone productive who can create value, so figure out how can you couch what you want in a way that proactively addresses the other person’s concerns.”

Principle #4: Use the right tactics

Once you’ve realized you’re negotiating, have done your research, and are negotiating from a position of strength, applying the right negotiation tactics will increase your chances of success even more.

Ask for more than you want

Obviously you don’t want to ask for less than what you want. But why not ask for exactly what you want?

First, it might turn out that the company is willing to give you far more than you expected or thought possible.

Second, if you ask for exactly what you want there’s no way for you to compromise without getting less than what you want. By asking for more, you can compromise while still getting what you wanted.

Applying the principle: If I’d wanted a $72,000 salary, and research suggested that was a fair salary, I should have asked for $80,000. If I was lucky the company would have said yes; if they wanted to negotiate me down, I would have no problems agreeing to a lower number so long as it was above $72,000.

Negotiate multiple things at once

Your goal when negotiating is not to “win.” Rather, your goal is to reach an agreement that passes your minimal bar, and gets you as much as is feasible. Feasibility means you also need to take into account what the other side wants as well. If you’ve reached an impasse, and you still think you can make a deal that you like, try to come up with creative ways to work out a solution that they will like.

If you only negotiate one thing at once, every negotiation has a winner and a loser. For example, if all you’re negotiating is salary, either you’re making more money, or the company is saving money: it’s a zero-sum negotiation. This limits your ability to come up with a solution that maximizes value for you while still meeting the other side’s needs.

Applying the principle: In my story above, the financial company wanted intellectual property protection, I wanted to be able to write open source, and we were at an impasse. So they expanded the scope of the negotiation to include my salary, which allowed them to make tradeoffs between the two—more money for me in return for what they wanted. If I’d cared less about working on open source I might have accepted that offer.

Never give an answer immediately

During the actual negotiation you should never decide on the spot, nor are you required to. If you get a job offer you can explain that you need a little time to think about it: say something like “I have to run this by my spouse/significant other/resident expert.” This will give you the time to consider your options in a calmer state of mind, and not just blurt out “yes” at the first semi-decent offer.

Having someone else review the offer is a good idea in general; a friend of mine ran her job offers by her sister, who had an MBA. But it’s also useful to mention that other person as someone who has to sign off on the offer. That gives you the ability to say you’d like to accept an offer, but your spouse/expert thinks you can do better.

Notice that the employer almost always has this benefit already. Unless you’re negotiating with the owner of the business, you’re negotiating with an agent: someone in HR, say. When you make a demand, the HR person might say “I have go to check with the hiring manager”, and when they come back with less than you wanted it’s not their fault, they’re just passing on the bad news. The implication is that the low offer is just the way it is, and there’s nothing they can do about.

Don’t fall for this trick: they often can change the offer.

Beyond negotiating for salary

You can negotiate for a higher salary—or rather, you should negotiate for a higher salary. The Adam I interviewed in this article is now a partner in DangoorMendel, who can help you negotiate a higher salary.

But salary isn’t the only thing you can negotiate for. You can also negotiate for a shorter workweek.

And yes, this is harder, but it’s definitely possible.

In fact, this article is an excerpt from a book I wrote to help you do just that: You Can Negotiate a 3-Day Weekend.

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

There is a lot of Python code in the wild which does something like:

raise ValueError("Could not fraz the buzz:"
                 f"{foo} is less than {quux}")

This is, in general, a bad idea. It does not matter if the exception is fairly generic, like ValueError or specific like CustomFormatParsingException.

Exceptions are not program panics. Program panics are things which should "never happen", and usually abort either the entire program, or at least an execution thread.

While exceptions sometimes do terminate the program, or the execution thread, with a traceback, they are different in that they can be caught.

The code that catches the exception will sometimes have a way to recover: for example, maybe it’s not that important for the application to fraz the buzz if foo is 0. In that case, the code would look like:

try:
    some_function()
except ValueError as exc:
    if ???

Oh, right. We do not have direct access to foo. If we formatted better, using repr, at least we could tell the difference between 0 and "0": but we still would have to start parsing the representation out of the string.

Because of this, in general, it is better to raise exceptions like this:

raise ValueError("Could not fraz the buzz: foo is too small", foo, quux)

Note that all the exceptions defined in core Python already allow any number of arguments. Those arguments are available as exc.args, if exc is the exception object. If you do end up defining your custom exceptions, the easiest thing is to avoid overriding the __init__: this keeps this behavior.

Raising exceptions this way gives exception handling a lot of power: it can introspect foo, introspect quux and introspect the string. If by some reason the exception class is raised and we want to verify the reason, checking string equality, while not ideal, is still better than trying to match string parts or regular expression matching.

When the exception is presented to the user interface, in that case, it will not look as nice. Exceptions, in general, should reach the UI only in extreme circumstances. In those cases, having something that has as much information is useful for root cause analysis.

This is an update of an older blog post. Thanks to Mark Rice and Ben Nuttall for their improvement suggestions. All mistakes that are left are my responsibility.

• Security fixes for HTTP/2 -- CVE-2019-9512 (Ping Flood), CVE-2019-9514 (Reset Flood), and CVE-2019-9515 (Settings Flood).  Thanks to Jonathan Looney and Piotr Sikora.
• HTTP/2 fixes regarding timeouts.
• trial's assertResultOf, failureResultOf, and successResultOf, now accept Deferred-awaiting coroutines.
• Various other bug fixes for POP3, conch.ssh.keys, and twisted.web.client.FileBodyProducer.

We have more ways to manage dependencies in Python applications than ever. But how do they fare in production? Unfortunately this topic turned out to be quite polarizing and was at the center of a lot of heated debates. This is my attempt at an opinionated review through a DevOps lens.

This has previously been published on opensource.com.

The Zen of Python is loose enough and contradicts itself enough that you can prove anything from it. Let's meditate upon one of its most famous principles: "Explicit is better than implicit."

One thing that traditionally has been implicit in Python is the expected interface. Functions have been documented to expect a "file-like object" or a "sequence." But what is a file-like object? Does it support .writelines? What about .seek? What is a "sequence"? Does it support step-slicing, such as a[1:10:2]?

Originally, Python's answer was the so-called "duck-typing," taken from the phrase "if it walks like a duck and quacks like a duck, it's probably a duck." In other words, "try it and see," which is possibly the most implicit you could possibly get.

In order to make those things explicit, you need a way to express expected interfaces. One of the first big systems written in Python was the Zope web framework, and it needed those things desperately to make it obvious what rendering code, for example, expected from a "user-like object."

Enter zope.interface, which was part of Zope but published as a separate Python package. The zope.interface package helps declare what interfaces exist, which objects provide them, and how to query for that information.

Imagine writing a simple 2D game that needs various things to support a "sprite" interface; e.g., indicate a bounding box, but also indicate when the object intersects with a box. Unlike some other languages, in Python, attribute access as part of the public interface is a common practice, instead of implementing getters and setters. The bounding box should be an attribute, not a method.

A method that renders the list of sprites might look like:

def render_sprites(render_surface, sprites):
    """
    sprites should be a list of objects complying with the Sprite interface:
    * An attribute "bounding_box", containing the bounding box.
    * A method called "intersects", that accepts a box and returns
      True or False
    """
    pass # some code that would actually render

The game will have many functions that deal with sprites. In each of them, you would have to specify the expected contract in a docstring.

Additionally, some functions might expect a more sophisticated sprite object, maybe one that has a Z-order. We would have to keep track of which methods expect a Sprite object, and which expect a SpriteWithZ object.

Wouldn't it be nice to be able to make what a sprite is explicit and obvious so that methods could declare "I need a sprite" and have that interface strictly defined? Enter zope.interface.

from zope import interface

class ISprite(interface.Interface):

    bounding_box = interface.Attribute(
        "The bounding box"
    )

    def intersects(box):
        "Does this intersect with a box"

This code looks a bit strange at first glance. The methods do not include a self, which is a common practice, and it has an Attribute thing. This is the way to declare interfaces in zope.interface. It looks strange because most people are not used to strictly declaring interfaces.

The reason for this practice is that the interface shows how the method will be called, not how it is defined. Because interfaces are not superclasses, they can be used to declare data attributes.

One possible implementation of the interface can be with a circular sprite:

@implementer(ISprite)
@attr.s(auto_attribs=True)
class CircleSprite:
    x: float
    y: float
    radius: float

    @property
    def bounding_box(self):
        return (
            self.x - self.radius,
            self.y - self.radius,
            self.x + self.radius,
            self.y + self.radius,
        )

    def intersects(self, box):
        # A box intersects a circle if and only if
        # at least one corner is inside the circle.
        top_left, bottom_right = box[:2], box[2:]
        for choose_x_from (top_left, bottom_right):
            for choose_y_from (top_left, bottom_right):
                x = choose_x_from[0]
                y = choose_y_from[1]
                if (((x - self.x) ** 2 + (y - self.y) ** 2) <=
                    self.radius ** 2):
                     return True
        return False

This explicitly declares that the CircleSprite class implements the interface. It even enables us to verify that the class implements it properly:

from zope.interface import verify

def test_implementation():
    sprite = CircleSprite(x=0, y=0, radius=1)
    verify.verifyObject(ISprite, sprite)

This is something that can be run by pytest, nose, or another test runner, and it will verify that the sprite created complies with the interface. The test is often partial: it will not test anything only mentioned in the documentation, and it will not even test that the methods can be called without exceptions! However, it does check that the right methods and attributes exist. This is a nice addition to the unit test suite and -- at a minimum -- prevents simple misspellings from passing the tests.

If you have some implicit interfaces in your code, why not document them clearly with zope.interface?

A completely incomplete guide to packaging a Python module and sharing it with the world on PyPI.

I previously wrote a post about shipping a PyGame app to users on macOS. It’s now substantially updated for the new Notarization requirements in Catalina. I hope it’s useful to somebody!

It’s October, and we’re all getting ready for Halloween, so allow me to me tell you a horror story, in Python:

1
2

>>> 0.1 + 0.2 - 0.3
5.551115123125783e-17

Some of you might already be familiar with this chilling tale, but for those who might not have experienced it directly, let me briefly recap.

In Python, the default representation of a number with a decimal point in it is something called an “IEEE 754 double precision binary floating-point number”. This standard achieves a generally useful trade-off between performance, correctness, and is widely implemented in hardware, making it a popular choice for numbers in many programming language.

However, as our spooky story above indicates, it’s not perfect. 0.1 + 0.2 is very slightly less than 0.3 in this representation, because it is a floating-point representation in base 2.

If you’ve worked professionally with software that manipulates money¹, you typically learn this lesson early; it’s quite easy to smash head-first into the problem with binary floating-point the first time you have an item that costs 30 cents and for some reason three dimes doesn’t suffice to cover it.

There are a few different approaches to the problem; one is using integers for everything, and denominating your transactions in cents rather than dollars. A strategy which requires less weird unit-conversion², is to use the built-in decimal module, which provides a floating-point base 10 representation, rather than the standard base-2, which doesn’t have any of these weird glitches surrounding numbers like 0.1.

This is often where a working programmer’s numerical education ends; don’t use floats, they’re bad, use decimals, they’re good. Indeed, this advice will work well up to a pretty high degree of application complexity. But the story doesn’t end there. Once division gets involved, things can still get weird really fast:

1
2
3

>>> from decimal import Decimal
>>> (Decimal("1") / 7) * 14
Decimal('2.000000000000000000000000001')

The problem is the same: before, we were working with 1/10, a value that doesn’t have a finite (non-repeating) representation in base 2; now we’re working with 1/7, which has the same problem in base 10.

Any time you have a representation of a number which uses digits and a decimal point, no matter the base, you’re going to run in to some rational values which do not have an exact representation with a finite number of digits; thus, you’ll drop some digits off the (necessarily finite) end, and end up with a slightly inaccurate representation.

But Python does have a way to maintain symbolic accuracy for arbitrary rational numbers -- the fractions module!

1
2
3
4
5

>>> from fractions import Fraction
>>> Fraction(1)/3 + Fraction(2)/3 == 1
True
>>> (Fraction(1)/7) * 14 == 2
True

You can multiply and divide and add and subtract to your heart’s content, and still compare against zero and it’ll always work exactly, giving you the right answers.

So if Python has a “correct” representation, which doesn’t screw up our results under a basic arithmetic operation such as division, why isn’t it the default? We don’t care all that much about performance, right? Python certainly trades off correctness and safety in plenty of other areas.

First of all, while Python’s willing to trade off some storage or CPU efficiency for correctness, precise fractions rapidly consume huge amounts of storage even under very basic algorithms, like consuming gigabytes while just trying to maintain a simple running average over a stream of incoming numbers.

But even more importantly, you’ll notice that I said we could maintain symbolic accuracy for arbitrary rational numbers; but, as it turns out, a whole lot of interesting math you might want to do with a computer involves numbers which are irrational: like π. If you want to use a computer to do it, pretty much all trigonometry³ involves a slightly inaccurate approximation unless you have a literally infinite amount of storage.

As Morpheus put it, “welcome to the desert of the ℝ”.

I’m a little annoyed at my Apple devices right now.

Time to complain.

“Trust us!” says Apple.

“We’re not like the big, bad Google! We don’t just want to advertise to you all the time! We’re not like Amazon, just trying to sell you stuff! We care about your experience. Magical. Revolutionary. Courageous!”

But I can’t hear them over the sound of my freshly-updated Apple TV — the appliance which exists solely to play Daniel Tiger for our toddler — playing the John Wick 3 trailer at full volume automatically as soon as it turns on.

For the aforementioned toddler.

I should mention that it is playing this trailer while specifically logged in to a profile that knows their birth date¹ and also their play history².

I’m aware of the preferences which control autoplay on the home screen; it’s disabled now. I’m aware that I can put an app other than “TV” in the default spot, so that I can see ads for other stuff, instead of the stuff “TV” shows me ads for.

But the whole point of all this video-on-demand junk was supposed to be that I can watch what I want, when I want — and buying stuff on the iTunes store included the implicit promise of no advertisements.

At least Google lets me search the web without any full-screen magazine-style ads popping up.

Launch the app store to check for new versions?

I can’t install my software updates without accidentally seeing HUGE ads for new apps.

Launch iTunes to play my own music?

I can’t play my own, purchased music without accidentally seeing ads for other music — and also Apple’s increasingly thirsty, desperate plea for me to remember that they have a streaming service now. I don’t want it! I know where Spotify is if I wanted such a thing, the whole reason I’m launching iTunes is that I want to buy and own the music!

On my iPhone, I can’t even launch the Settings app to turn off my WiFi without seeing an ad for AppleCare+, right there at the top of the UI, above everything but my iCloud account. I already have AppleCare+; I bought it with the phone! Worse, at some point the ad glitched itself out, and now it’s blank, and when I tap the blank spot where the ad used to be, it just shows me this:

I just want to use my device, I don’t need ad detritus littering every blank pixel of screen real estate.

Knock it off, Apple.

The following post was originally published on OpenSource.com as part of a series on seven libraries that help solve common problems.

Imagine you have a "shapes" library. We have a Circle class, a Square class, etc.

A Circle has a radius, a Square has a side, and maybe Rectangle has height and width. The library already exists: we do not want to change it.

However, we do want to add an area calculation. If this was our library, we would just add an area method, so that we can call shape.area(), and not worry about what the shape is.

While it is possible to reach into a class and add a method, this is a bad idea: nobody expects their class to grow new methods, and things might break in weird ways.

Instead, the singledispatch function in functools can come to our rescue:

@singledispatch
def get_area(shape):
    raise NotImplementedError("cannot calculate area for unknown shape",
                              shape)

The "base" implementation for the get_area function just fails. This makes sure that if we get a new shape, we will cleanly fail instead of returning a nonsense result.

@get_area.register(Square)
def _get_area_square(shape):
    return shape.side ** 2
@get_area.register(Circle)
def _get_area_circle(shape):
    return math.pi * (shape.radius ** 2)

One nice thing about doing things this way is that if someone else writes a new shape that is intended to play well with our code, they can implement the get_area themselves:

from area_calculator import get_area

@attr.s(auto_attribs=True, frozen=True)
class Ellipse:
    horizontal_axis: float
    vertical_axis: float

@get_area.register(Ellipse)
def _get_area_ellipse(shape):
    return math.pi * shape.horizontal_axis * shape.vertical_axis

Calling get_area is straightforward:

print(get_area(shape))

This means we can change a function that has a long if isintance()/elif isinstance() chain to work this way, without changing the interface. The next time you are tempted to check if isinstance, try using singledispatch!

Pines and firs are dying across the Pacific Northwest, fires rage across the Amazon, it’s the hottest it’s ever been in Paris—climate change is impacting the whole planet, and things are not getting any better. You want to do something about climate change, but you’re not sure what.

If you do some research you might encounter an essay by Bret Victor—What can a technologist do about climate change? There’s a whole pile of good ideas in there, and it’s worth reading, but the short version is that you can use technology to “create options for policy-makers.”

Thing is, policy-makers aren’t doing very much.

So this essay isn’t about technology, because technology isn’t the bottleneck right now, it’s about policy and politics what you can do about it. It’s still written for software developers, because that’s who I write for, but also because software developers often have access to two critical catalysts for political change. And it’s written for software developers in the US, because that’s where I live, and because the US is a big part of the problem.

But before I go into what you can do, let me tell you the story of a small success I happened to be involved in, a small step towards a better future.

Infrastructure and the status quo

About a year ago I spent some of my mornings handing out pamphlets to bicycle riders. I looked like an idiot: in order to show I was one of them I wore my bike helmet, which is weirdly shaped and the color of fluorescent yellow snot.

After finding an intersection with plenty of bicycle riders and a long red light that forces them to stop, I would do the following:

1. When the light turns red, step into the street and hand out the pamphlet.
2. Keep an eye out for the light changing to green so that I didn’t get run over by moving cars.
3. Twiddle my thumbs waiting for the next light cycle.

It was boring, and not very glamorous.

I was one of just many volunteers, and besides gathering signatures we also held rallies, had conversations with city councilors and staff, wrote emails, talked at city council meetings—it was a process. The total effort took a couple of years (and I only joined in towards the end)—but in the end we succeeded.

We succeeded in having the council pass a short ordinance, a city-level law in the city of Cambridge, Massachusetts. The ordinance states that whenever a road that was supposed to have protected bike lanes (per the city’s Bike Plan) was rebuilt from scratch, it would have those lanes built by default.

Now, clearly this ordinance isn’t going to solve climate change. In fact, nothing Cambridge does as a city will solve climate change, because there’s only so much impact 100,000 people can have on greenhouse gas emissions.

But while in some ways this ordinance was a tiny victory in a massive war, if we take a step back it’s actually more important than it seems. In particular, this ordinance has three effects:

1. Locally, safer bike infrastructure means more bicycle riders, and fewer car drivers. That reduces emissions—a little.
2. Over time, more bicycle riders can kick off a positive feedback cycle, reducing emissions even more.
3. Most significantly, local initiatives spread to other cities—kicking off these three effects in those other cities.

Let’s examine these effects one by one.

Effect #1: Fewer cars, less emissions

About 43% of the greenhouse gas emissions in Massachusetts are due to transportation; for the US overall it’s 29% (ref). And that means cars.

The reason people in the US mostly drive cars is because all the transportation infrastructure is built for cars. No bike lanes, infrequent, slow and non-existent buses, no trains… Even in cities, where other means of transportation are feasible, the whole built infrastructure sends the very strong message that cars are the only reasonable way to get around.

If we focus on bicycles, our example at hand, the problem is that riding a bicycle can be dangerous—mostly because of all those cars! But if you get rid of the danger and build good infrastructure—dedicated protected bike lanes that separate bicycle riders from those dangerous cars—then bicycle use goes up.

Consider what Copenhagen achieved between 2008 and 2017 (ref):

With safer infrastructure for bicycles, perception of safety goes up, and people bike more and drive less. Similarly, if you have frequent, fast, and reliable buses and trains, people drive less. And that means less carbon emissions.

In Copenhagen the number of kilometers driven by cars was flat or slightly down over those 10 years—whereas in the US, it’s up 6-7% (ref).

Effect #2: A positive feedback loop

The changes in Copenhagen are a result of a plan the city government there adopted in 2011 (ref): they’re the result of a policy action. And the political will was there in part because there were already a huge number of bicycle riders. So it’s a positive feedback loop, and a good one.

Let’s see how this is happening in Cambridge:

• Cambridge has a slowly growing number of bicycle rider. This means more political support for bike infrastructure—if there’s a group that can mobilize that support!
• With the ordinance, more roads will have safe infrastructure. For example, one neighborhood previously had a safe route only in one direction; the other direction will be rebuilt with a protected bike lane in 2020.
• With safer infrastructure, there will be more bicycle riders, and therefore more support by residents for safer infrastructure. Merely having support isn’t enough, of course, and I’ll get back to that later on.

If Copenhagen can reach 50% of residents with a bicycle commute, so can Cambridge—and the ordinance is a good step in that direction.

Effect #3: The idea spreads

The Cambridge ordinance passed in April 2019—and the idea is spreading elsewhere:

• The California State Assembly is voting on a law with similar provisions (ref), through a parallel push by Calbike.
• In May 2019 a Washington DC Council member introduced a bill which among other points has the same rebuild requirements as the Cambridge ordinance (ref).
• The Seattle City Council passed an ordinance, parts of which were literally copy/pasted from the Cambridge ordinance (ref).

All of this is the result of local advocacy—but I’ve no doubt Cambridge’s example helped. It’s always easier to be the second adopter. And the examples from these larger localities will no doubt inspire other groups and cities, spreading the idea even more.

Change requires politics

Bike infrastructure is just an example, not a solution—but there are three takeaways from this story that I’d like to emphasize:

• If you want to change policy, you need to engage in politics.
• Politics are easier to impact on the local level.
• Local policy changes have a cumulative, larger-scale impact.

By politics I don’t just mean having an opinion or voting for a candidate, but rather engaging in the process of how policy decisions are made.

Merely having an opinion doesn’t change anything. For example, two-thirds of Cambridge residents support building more protected bike lanes (ref). But that doesn’t mean that many protected lanes are getting built—the neighboring much smaller city of Somerville is building far more than Cambridge.

The only reason the city polled residents about bike lanes is because, one suspects, all the fuss we’d been making—emails, rallies, meetings, city council policy orders—made the city staff wonder if bike infrastructure really had a lot of public support or not.

Voting results in some change, but not enough. Elected officials and government staff have lots and lots of things to worry about—if they’re not being pressured to focus on a particular issue, it’s likely to fall behind.

What’s more, the candidates you get to vote for have to get on the ballot, and to do that they need money (for advertising, hiring staff, buying supplies). Lacking money, they need volunteer time.

And it’s much easier for a small group of rich people to provide that support to the candidates they want—so by the time you’re voting, you only get to choose between candidates that have been pre-vetted (I highly recommend reading The Golden Rule to understand how this works on a national level).

What you can do: Become an activist

In the end power is social. Power comes from people showing up to meetings, people showing up for rallies, people going door-to-door convincing other people to vote for the right person or support the right initiative, people blocking roads and making a fuss.

And that takes time and money.

So if you want to change policy, you need to engage in politics, with time and money:

• You can volunteer for candidates’ political campaigns, as early as possible in the process. Too many good candidates get filtered out before they even make the ballot. That doesn’t mean you can just go home after the election—that’s when the real work of legislation starts, which means activism is just as important.
• You can volunteer with groups either acting on a particular issue (transportation, housing policy) or more broadly on climate change.
• Also useful is donating money to political campaigns, both candidates and issue-based organizations.

Here are some policies you might be interested in:

• Transportation policy determines what infrastructure is built—and the current infrastructure favors privately-owned cars over public transportation and bicycles.
• Zoning laws determine what gets built and where. Denser construction would reduce the need for long trips, and more efficient buildings (ideally net zero carbon) would reduce emissions from heating and cooling.
• Moving utilities from private to public ownership, so they can focus on the public good and not on profit.
• Bulk municipal contracts for electricity: this allows for cheaper electricity for all residents, and to have green energy as the default.
• State-level carbon restrictions or taxes.

Where you should do it: Start local

If you are going to become an activist, the local level is a good starting point.

• An easier first step: Cambridge has 100,000 residents—city councilors are routinely elected with just 2500 votes. That means impacting policies here is much easier than at a larger scale. Not only does this mean faster results, it also means you’re less likely to get discouraged and give up—you can see the change happening.
• Direct impact: A significant amount of greenhouse gas emissions in the US are due to causes that are under control of local governments.
• Wider impact: As in the case of Cambridge’s ordinance, local changes can be adopted elsewhere.

Of course, local organizing is just the starting point for creating change on the global level. But you have to start somewhere. And global change is a lot easier if you have thousands of local organizations supporting it.

It’s a good to be a software developer

Let’s get back to our starting point—you’re paid to write software, you want to do something about climate change. As a software developer you likely have access to the inputs needed to make political campaigns succeed—both candidate-based and issue-based:

• Money: Software developers tend to get paid pretty well, certainly better than most Americans. Chances are you have some money to spare for political donations.
• Time: This one is a bit more controversial, but in my experience many programmers can get more free time if they want to.

If you don’t have children or other responsibilities, you can work a 40-hour workweek, leaving you time for other things. Before I got married I worked full-time and went to a local adult education college half-time in the evenings: it was a lot of work, but it was totally doable. Set boundaries at your job, and you’ll have at least some free time for activism.

You can also negotiate a shorter workweek, which is possible in part because software developers are in such demand. I’ve done this, I’ve interviewed people who have done it, I’ve found many random people on the Internet who have done it—it is possible.

If you need help doing it yourself, I’ve written a book to help you negotiate a shorter workweek. If you want to negotiate a shorter workweek so you have time for political activism, you can use the code FIGHTCLIMATECHANGE to get the book for 60% off.

Some common responses

“There will never be the political will to make this happen”

Things do change, for better and for worse, and sometimes unexpectedly. To give a couple of examples:

• In Ireland, the Catholic Church went from all-powerful to losing badly, most recently with Ireland legalizing abortion.
• The anti-gay-marriage Defense of Marriage Act was passed by veto-proof majorities of Congress in 1996—and eight years later in 2004 the first legal gay marriage took place right here in Cambridge, MA.

The timelines for gay marriage and cannabis legalization in the US are illuminating: these things didn’t just happen, it was the result of long, sustained activist efforts, much of it at the local level.

Local changes do make a difference.

“Politics is awful and broken”

So are all our software tools, and somehow we manage to get things done!

“I don’t like your policy suggestions, we should do X instead”

No problem, find the local groups that promote your favorite policies and join them.

“The necessary policies will never work because of problem Y”

Same answer: join and help the local groups working on Y.

“It’s too late, the planet is doomed no matter what we do”

Perhaps, but it’s very hard to say. So we’re in Pascal’s Wager territory here: given even a tiny chance there is something we can do, we had better do our best to make it happen.

And even if humanity really is doomed, there’s always the hope that someday a hyperintelligent species of cockroach will inherit the Earth. And when cockroach archaeologists try to reconstruct our history, I would like them to be able to say, loosely translated from their complex pheromone-and-dancing system of communication: “These meatsacks may not have been as good at surviving as us cockroaches—but at least they tried!”

Time to get started

If you find this argument compelling—that policy is driven by power, and that power requires social mobilization—then it’s up to you to take the next step. Find a local group or candidate pushing for a policy you care about, and show up for the next meeting.

And the meeting after that.

And then go to the rally.

And knock on doors.

And make some friends, and make some changes happen.

Some of the work is fun, some of it is boring, but there’s plenty to do—time to get started!

Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you’d produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

Services like Twitter and Slack have functionality that attempts to interpret parts of the plain text of tweets or message as entered by the user. Pieces of the text that look like links, mentions of another user, hash tags, or stock symbols, cause additional meta data to be added to the object representing the message, so that receiving clients can mark up those pieces of text in a special way. Twitter calls this meta data Tweet Entities and for each piece of interpreted text, it includes indices for the start and end of along with additional information depending on the type of entity. A client can then do in-line replacements at the exact character indices, e.g. by making it into a hyperlink. Twitter Entities served as inspiration for XEP-0372: References.

References can be used in two ways: including a reference as a sibling to the body element of a message. The begin and end attributes then point to the indices of the plain text in the body. This would typically be used if the interpretation of the message is done by the sending client.

Alternatively, a service (e.g. a MUC service) could parse incoming messages and send a separate stanza to mark up the original stanza. In this case you need a mechanism for pointing to that other message. There have been two proposals for this, with slightly differing approaches, and in the examples below, I'll use the proto-XEP Message Fastening. While pointing to the stanza ID of the other message, it embeds a reference element in the apply-to element.

Mentioning another user

Let's start out with the example of mentioning another user.

<message from="room@muc.this.example/Kev" type="groupchat">
  <stanza-id id="2019-09-02-1" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>Some rubbish @ralphm</body>
</message>

A client might render this as:

The MUC service then parses the plain-text message, and finds a reference to my nickname prefixed with an @-sign, and sends a stanza to the room that marks up the message Kev sent to me.

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-2" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-1">
    <reference begin="13" end="19" xmlns="urn:example:reference:0">
      <mention jid="room@muc.this.example/ralphm"/>
    </reference>
  </apply-to>
</message>

This stanza declares that it is attached to the previous message by the stanza ID that was included with the original stanza. In its payload, it includes a reference, referring to the characters 13 through 19. It has a mention child pointing to my occupant JID. Alternatively, the room might have linked to my real JID. A client can then alter the presentation of the original message to use the attached mention reference:

The characters referencing @ralphm are now highlighted, hovering the mention shows a tooltip with my full name, and clicking on it brings you to a page describing me. This information was not present in the stanza, but a client can use the XMPP URI as a key to present additional information. E.g. from the user's contact list, by doing a vCard lookup, etc.

Referring to previous messages

<message from="room@muc.this.example/Ge0rG"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-3" by="room@muc.this.example"/>
  <reference begin="0" end="6" xmlns="urn:example:reference:0">
    <mention jid="room@muc.this.example/ralphm"/>
  </reference>
  <reference begin="26" end="32" xmlns="urn:example:reference:0">
    <message id="2019-09-02-1"/>
  </reference>
  <body>@ralphm did you see Kev's message earlier?</body>
</message>

Unlike before, this example does not point to another stanza with apply-to. Instead, Ge0rG's client added references to go along with the plain-text body: one for the mention of me, and one for a reference to an earlier message.

Emoji Reactions

Instead of reacting with a full message, Slack, like online forum software much earlier, has the ability to attach emoji reactions to messages.

<message from="room@muc.this.example/Kev"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
            id="2019-09-02-4" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-3">
    <reactions xmlns="urn:example:reactions:0">
      <reaction label=":+1:">👍</reaction>
    </reactions>
  </apply-to>
</message>

<message from="room@muc.this.example/ralphm"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-6" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-3">
    <reactions xmlns="urn:example:reactions:0">
      <reaction label=":parrot:"
                img="cid:b729aec3f521694a35c3fc94d7477b32bc6444ca@bob.xmpp.org"/>
    </reactions>
  </apply-to>
</message>

These two examples show two separate instances of a person reacting to the previous message by Ge0rG. It uses the protocol from Message Reactions, another Proto-XEP. However, I expanded on it by introducing two new attributes. The label allows for a textual shorthand, that might be typed by a user. Custom emoji can be represented with the img attribute, that points to a XEP-0231: Bits of Binary object.

Ge0rG 13:16

@ralphm did you see Kev's message earlier?

👍 2  1

The attached emoji are rendered below the original message, and hovering over them reveals who were the respondents. Here my own reaction is highlighted by a squircle border.

Including a link

<message from="room@muc.this.example/ralphm" type="groupchat">
  <stanza-id id="2019-09-02-7" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>Have you seen https://ralphm.net/blog/2013/10/10/logitech_t630?</body>
</message>

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-8" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-7">
    <reference begin="14" end="61" xmlns="urn:example:reference:0">
      <link url="https://ralphm.net/blog/2013/10/10/logitech_t630"/>
    </reference>
  </apply-to>
</message>

Here the MUC service marks up the original messages with an explicit link reference. Possibly, the protocol might be extended so that a service can include shortened versions of the URL for display purposes.

ralphm 13:16

Have you seen https://ralphm.net/blog/2013/10/10/logitech_t630?

Logitech Ultrathin Touch Mouse

Logitech input devices are my favorite. This tiny bluetooth mouse is a nice portable device for every day use or while traveling.

The client has used the markup to fetch meta data on the URL and presents a summary card below the original message. Alternatively, the MUC service could have done this using XEP-0385: Stateless Inline Media Sharing (SIMS):

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-8" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-7">
    <reference begin="14" end="61" xmlns="urn:example:reference:0">
      <link url="https://ralphm.net/blog/2013/10/10/logitech_t630"/>
      <card xmlns="urn:example:card:0">
        <title>Logitech Ultrathin Touch Mouse</ulink></title>
        <description>Logitech input devices are my favorite. This tiny bluetooth mouse is a nice portable device for every day use or while traveling.</description>
      </card>
      <media-sharing xmlns='urn:xmpp:sims:1'>
        <file xmlns='urn:xmpp:jingle:apps:file-transfer:5'>
          <media-type>image/jpeg</media-type>
          <name>ultrathin-touch-mouse-t630.jpg</name>
          <size>23458</size>
          <hash xmlns='urn:xmpp:hashes:2' algo='sha3-256'>5TOeoNI9z6rN5f+cQagnCgxitQE0VUgzCMeQ9JqbhWJT/FzPpDTTFCbbo1jWwOsIoo9u0hQk6CPxH4t/dvTN0Q==</hash>
          <thumbnail xmlns='urn:xmpp:thumbs:1'uri='cid:sha1+21ed723481c24efed81f256c8ed11854a8d47eff@bob.xmpp.org' media-type='image/jpeg' width='116' height='128'/>
        </file>
        <sources>
          <reference xmlns='urn:xmpp:reference:0' type='data' uri='https://test.ralphm.net/images/blog/ultrathin-touch-mouse-t630.jpg' />
        </sources>
      </media-sharing>
    </reference>
  </apply-to>
</message>

Editing a previous message

<message from="room@muc.this.example/ralphm" type="groupchat">
  <stanza-id id="2019-09-02-9" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>Some thoughtful reply</body>
</message>

After sending that message, I want to add a bit more information:

<message from="room@muc.this.example/ralphm" type="groupchat">
  <stanza-id id="2019-09-02-10" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-9">
    <external name='body'/>
    <replace xmlns='urn:example:message-correct:1'/>
  </apply-to>
  <body>Some more thoughtful reply</body>
</message>

Unlike XEP-0308: Last Message Correction, this example uses Fastening to refer to the original message. I would also lift the restriction on correcting just the last message, but allow any previous message to be edited.

Upon receiving the correction, the client indicates that the message has been edited. Hovering over the marker reveals when the message was changed.

Editing a previous message that had fastened references

<message from="room@muc.this.example/Kev" type="groupchat">
  <stanza-id id="2019-09-02-11" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>A witty response mentioning @ralphm</body>
</message>

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-12" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-11">
    <reference begin="28" end="34" xmlns="urn:example:reference:0">
      <mention jid="room@muc.this.example/ralphm"/>
    </reference>
  </apply-to>
</message>

After a bit of consideration, Kev edits his response:

<message from="room@muc.this.example/Kev" type="groupchat">
  <stanza-id id="2019-09-02-13" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-11">
    <external name='body'/>
    <replace xmlns='urn:example:message-correct:1'/>
  </apply-to>
  <body>A slighty wittier response mentioning @ralphm</body>
</message>

Upon receiving the correction, the client discards all fastened references. The body text was changed, so the reference indices are stale. The room can then send a new stanza marking up the new text:

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-14" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-11">
    <reference begin="40" end="46" xmlns="urn:example:reference:0">
      <mention jid="room@muc.this.example/ralphm"/>
    </reference>
  </apply-to>
</message>

Closing notes

• Fastening should also gain a way to unfasten explicitly. I think that should use the stanza ID of the stanza that included the earlier fastening. This allows for undoing individual emoji reactions.

• Unfastening should probably not use the proto-XEP on Message Retraction. That is for retracting the entire original message plus all its fastened items, and invalidating all message references pointing to it.

• It might make sense to have a separate document describing how to handle stanza IDs, so that all specifications could point to it instead of each having their own algorithm. In different contexts, different IDs might be used. The other proposal for attachments, XEP-0367: Message Attaching, has a section (4.1) on this that might be taken as a start.

• In the discussion leading up to this post, a large part was about how to handle all these things attached/fastened to messages in message archives. This is not trivial, as you likely don't want to store a sequence of stanzas, but of (original) messages. Each of those message then might have one or more things fastened to it, and upon retrieval, you want these to come along when retrieving a message. Some of these might be collated, like edits. Some might cause summary counts (emoji, simple polls) with the message itself, and require an explicit retrieval of all the reactions, e.g. when hovering the reaction counts.

  Details on message archive handling is food for a later post. I do think that having a single way of attaching/fastening things to messages makes it much easier to come up with a good solution for archive handling.

• I didn't provide examples for stanza encryption, but discussions on this suggested that stanzas with fastened items would have an empty apply-to, including the id attribute, so that message archives can do rudimentary grouping of fastened items with the original message.

• I didn't include examples on Chat Markers, as its current semantics are that a marker sent by a recipient applies to a message and all prior messages. This means the marker isn't really tied to a single message. I think this doesn't match the model for Message Fastening.