A common issue programmers have when they try to test real-world software is how to deal with third-party dependencies. Let’s examine an old, but counter-intuitive principle.
A common issue programmers have when they try to test real-world software is how to deal with third-party dependencies. Let’s examine an old, but counter-intuitive principle.
Python’s standard
datetime
module is very powerful. However, it has a couple of annoying flaws.
Firstly, datetime
s are considered a kind of
date
1, which causes problems.
Although datetime
is a literal subclass of date
so Mypy and isinstance
believe a datetime “is” a date, you cannot substitute a datetime
for a
date
in a program without provoking errors at runtime.
To put it more precisely, here are two programs which define a function with
type annotations, that mypy finds no issues with. The first of which even
takes care to type-check its arguments at run-time. But both raise
TypeError
s at runtime:
datetime
to date
:1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
1 2 3 4 5 6 |
|
datetime
:1 2 3 4 5 6 |
|
1 2 3 4 5 6 |
|
In some sense, the whole point of using Mypy - or, indeed, of runtime
isinstance
checks - is to avoid TypeError
getting raised. You specify
all the types, the type-checker yells at you, you fix it, and then you can know
your code is not going to blow up in unexpected ways.
Of course, it’s still possible to avoid these TypeError
s with runtime
checks, but it’s tedious and annoying to need to put a check for .tzinfo is
not None
or not isinstance(..., datetime)
before every use of -
or >
.
The problem here is that datetime
is trying to represent too many things with
too few types. datetime
should not be inheriting from date
, because it
isn’t a date, which is why >
raises an exception when you compare the two.
Naive datetime
s represent an abstract representation of a hypothetical civil
time which are not necessarily
tethered to specific moments in physical time. You can’t know exactly what
time “today at 2:30 AM” is, unless you know where on earth you are and what the
rules are for daylight savings time in that place. However, you can still
talk about “2:30 AM” without reference to a time zone, and you can even say
that “3:30 AM” is “60 minutes after” that time, even if, given potential
changes to wall clock time, that may not be strictly true in one specific
place during a DST transition. Indeed, one of those times may refer to
multiple points in civil time at a particular location, when attached to
different sides of a DST
boundary.
By contrast, Aware datetimes
represent actual moments in time, as they
combine civil time with a timezone that has a defined UTC offset to interpret
them in.
These are very similar types of objects, but they are not in fact the same, given that all of their operators have slightly different (albeit closely related) semantics.
datetype
I created a small library, datetype
,
which is (almost) entirely type-time behavior. At runtime, despite
appearances, there are no instances of new types, not even wrappers.
Concretely, everything is a date
, time
, or datetime
from the standard
library. However, when type-checking with Mypy, you will now get errors
reported from the above scenarios if you use the types from datetype
.
Consider this example, quite similar to our first problematic example:
AwareDateTime
or NaiveDateTime
to date
:1 2 3 4 5 6 7 8 9 10 |
|
Now, instead of type-checking cleanly, it produces this error, letting you know
that this call to is_after
will give you a TypeError
.
1 2 |
|
Similarly, attempting to compare naive and aware objects results in errors now.
We can even use the included AnyDateTime
type variable to include a bound
similar to
AnyStr
from
the standard library to make functions that can take either aware or naive
datetimes, as long as you don’t mix them up:
AwareDateTime
to NaiveDateTime
:1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
|
1 2 3 4 5 6 |
|
Although the types in datetype
are Protocol
s,
there’s a bit of included magic so that you can use them as type guards with
isinstance
like regular types. For example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
|
This library is very much alpha-quality; in the process of writing this blog
post, I made a half a dozen backwards-incompatible changes, and there are still
probably a few more left as I get feedback. But if this is a problem you’ve
had within your own codebases - ensuring that date
s and datetime
s don’t get
mixed up, or requiring that all datetime
s crossing some API boundary are
definitely aware and not naive, give it a try with pip install datetype
and
let me know if it catches any bugs!
But, in typical
fashion, not
a kind of time
... ↩
The Python standard library is full of underappreciated gems. One of them allows for simple and elegant function dispatching based on argument types. This makes it perfect for serialization of arbitrary objects – for example to JSON in web APIs and structured logs.
This was originally a thread on Twitter; you can read the original here, but this one has been lightly edited for grammar and clarity, plus I added a pretty rad picture of a frog to it.
Update 2022-05-16: Thanks to some reader feedback I have updated the conclusion to note an example where this advice can productively apply to some ADHDers.
I’m in the midst of trying to unlearn a few things about neurotypical productivity advice but this is one I’ve been thinking about a lot:
Most productivity advice is toxic for ADHDers because it was written by a neurotypical brain for a neurotypical reader.
— Jesse J. Anderson • ADHD Creative (@jessejanderson) February 3, 2022
One of the best things you can do is unlearn axioms like "eat the frog first" and find what actually works for you.@adhddesignerhttps://t.co/lcd74rIDwb
“Eat the frog first” is particularly toxic advice for ADHDers.
Photo by Stephanie LeBlanc on Unsplash
First, for anyone who happens not to know already: “eat the frog first” is a technique which involves finding the task you’re most likely to ignore or put off, and doing it first in your day to ensure that you don’t avoid it.
For a neurotypical person, eating the frog first makes sense, which is of course why this advice exists in the first place. If you’ve been avoiding a task, put it first in your day when you’re going to have the most energy, and use the allure of the more fun tasks later to push through it.
This makes intuitive sense.
The premise of this advice is that you rely on the promise of delayed gratification—and the anticipated inherent satisfaction of having completed the boring and unpleasant thing—in order to motivate you to do it.
Here’s the problem for ADHDers: ADHD is literally the condition of not generating enough dopamine, which means delayed gratification is inherently more difficult for us. The anticipated inherent satisfaction is less motivating because it’s less intense, on a physical level.
An ADHD brain powering through tasks needs momentum. You need to be in a sufficiently excited state to begin doing things. A bored, dopamine-starved ADHD brain is going to be clawing at the walls looking for ANY dopamine-generating distraction to avoid thinking about the frog.
Of course where dopamine won’t do, there’s always adrenaline. Panic can trigger sufficient states of activity as well, although the stress is unhealthy and it’s less reliable in the absence of a real, immediate threats that you can’t ignore.
So what frog-first ADHD days often look like (particularly for adult ADHDers) is a slow slog of not really doing anything useful, while stewing in increasingly negative self-talk, attempting to generate the necessary anger and self-loathing required to truly panic about the frog.
Unfortunately this type of attempt at internal motivation is more likely to result in depression than motivation, which creates a spiral that makes the problem worse.
The neurotypical’s metaphorical frog is just sitting there, waiting to be eaten. Maybe they’ve been avoiding it because it’s a little gross, but fine, they can exert a little willpower and just do it, and move on to more pleasant activities. But the ADHD frog is running away.
Trying to use the same technique just results in the ADHDer sitting in the swamp where the frog used to be, chugging ever-increasing volumes of toxic mud in the hopes that we’ll find a frog in there. Sometimes we even find one! But that’s not success.
At the end of the day, the metaphorical frog does need eating; that’s what makes it a frog. What is the conscientious ADHDer to do?
Unfortunately, there is no singular, snappy answer; difficulty with this type of task is the impenetrable core of the “disorder” part of ADHD. It’ll always be difficult. But there are definitely strategies which can make it relatively easier.
None of these are guaranteed to work, but I am at least reasonably sure that they won’t build a spiral into guilt and depression:
It might literally be better to start the day with something actively unproductive, but fun, like a video game, although this can obviously be risky. For this to work, you need to have very good systems in place.
Start the frog at the end of the day and deliberately interrupt yourself when you stop work. Leave it lingering so some aspect of it annoys you and it distracts you at night. Start the next day pissed off at and obsessing over murdering that piece of shit frog as soon as you can get your hands on it.
This technique is also good because at the end of the day you only need to push yourself just hard enough to load the task into your brain, not all the way through it.
Remember that while “stimulated” doesn’t have to mean “panicked”, it also doesn’t need to mean “happy”. Sometimes, annoyance or irritation is the best way to ensure that you go do something. Consider, for example, the compelling motivation of reading a comment on the Internet that you disagree with.
Overall the distinguishing characteristic of toxic productivity advice is that it makes you spend more time feeling bad than doing stuff. It substitutes panic for healthy motivation, and low self-esteem for a feeling of accomplishment.
The most important point I am trying to make is this: when you take productivity advice — even, or perhaps especially, from me – try to measure its impact on your work and your mental health.
To that point, one piece of feedback I received on an earlier iteration of this article was that, for some ADHDers on stimulant medication2, eating the frog first can work: if you take your medication early in the morning and experience a big, but temporary, increase to executive-function 30 minutes later, being prepared to do your frog-eating at that specific moment can have similar results as for someone more neurotypical. This very much depends on how you specifically react to your medication, however.
So, if eating the frog first is working for you, by all means keep doing it, but you have to ask yourself: are you actually getting more done?
One consistent bit of feedback that I’ve received on my earlier writing about email workflow is that I didn’t include a concrete enough set of instructions for getting started with task-management workflow, particularly with low-friction options that are available for people who don’t necessarily have $100 per year to drop on the cadillac of task-management applications.
Given that the piece seems to be enjoying a small resurgence of attention, I’ve significantly expanded the “Make A Place For Tasks” section of that article, with:
It was nice to be doing this update now, because in the years since that piece was published, almost every major email application has added task-management features, or upgraded them into practical usability; gone are the times when properly filing your emails into clearly-described tasks was an esoteric feature that you needed expensive custom software for.
In this post I’d like to convince you that you should be running Mypyc over your code1 — especially if your code is a library you upload to PyPI — for both your own benefit and that of the Python ecosystem at large.
But first, let me give you some background.
A common narrative about Python’s value proposition, from the very earliest
days of the language2, often recited in response to a teammate saying
“shouldn’t we just write this in $HIGHER_PERFORMANCE_LANGUAGE
instead?” goes
something like this:
Sure, Python is slow.
But that’s okay, because it saves you so much time over implementing your code in
$HIGHER_PERFORMANCE_LANGUAGE
that you’ll have so much more time for optimizing those critical hot-spots where performance is really critical.And if the language’s primitives are too slow to micro-optimize those hot-spots enough, that’s okay too, because you can always re-write just those small portions of the program as a C extension module.
Python’s got you covered!
There is some truth to this narrative, and I’ve quoted from it myself on many occasions. When I did so, I was not quoting it as some facile, abstract hypothetical, either. I had a few projects, particularly very early in my Python career, where I replaced performance-critical C++ code with a one tenth the number of lines of Python, and improved performance by orders of magnitude in the process3.
When you have algorithmically interesting, performance-sensitive code that can benefit from a high-level expressive language, and the resources to invest in making it fast, this process can be counterintuitively more efficient than other, “faster” tools. If you’re working on massively multiplayer online games4 or something equally technically challenging, Python can be a surprisingly good idea.
This little nugget of folk wisdom does sound a bit defensive, doesn’t it? If Python were just fast, you could just use it, you wouldn’t need this litany of rationalizations. Surely if we believed that performance is important in our own Python code, we wouldn’t try to wave away the performance of Python itself.
Most projects are not massively multiplayer online games. On many straightforward business automation projects, this sort of staged approach to performance is impractical.
Not all performance problems are hot spots. Some programs have to be fast all the way through. This is true of some complex problems, like compilers and type checkers, but is also often the case in many kinds of batch processing; there are just a lot of numbers, and you have to add them all up.
More saliently for the vast majority of average software projects, optimization just isn’t in the budget. You do your best on your first try and hope that none of those hot spots get too hot, because as long as the system works within a painfully generous time budget, the business doesn’t care if it’s slow.
The progression from “idiomatic Python” to “optimized Python” to “C” is a one-way process that gradually loses the advantages that brought us to Python in the first place.
The difficult-to-reverse nature of each step means that once you have prototyped out a reasonably optimized data structure or algorithm, you need to quasi-permanently commit to it in order to squeeze out more straight-line performance of the implementation.
Plus, the process of optimizing Python often destroys its readability, for a few reasons:
array
module instead of lists”, and “use %
instead of
.format
”.Maintaining good performance is part of your software’s development lifecycle, not just a thing you do once and stop. So by moving into this increasingly arcane dialect of “fast” python, and then into another programming language entirely with a C rewrite, you end up having to maintain C code anyway. Not to mention the fact that rewriting large amounts of code in C is both ludicrously difficult (particularly if your team primarily knows Python) and also catastrophically dangerous. In recent years, safer tools such as PyO3 have become available, but they still involve switching programming languages and rewriting all your code as soon as you care about speed5.
So, for Python to be a truly general-purpose language, we need some way to just write Python, and have it be fast.
It would benefit every user of Python for there to be an easy, widely-used way to make idiomatic, simple Python that just does stuff like adding numbers, calling methods, and formatting strings in a straight line go really fast — exactly the sorts of things that are the slowest in Python, but are also the most common, particularly before you’ve had an opportunity to cleverly optimize.
There are also a number of tools that have long been in use for addressing this problem: PyPy, Pyrex, Cython, Numba, and Numpy to name a few. Their maintainers all deserve tremendous amounts of credit, and I want to be very clear that this post is not intended to be critical of anyone’s work here. These tools have drawbacks, but many of those drawbacks make them much better suited to specialized uses beyond the more general 80% case I’m talking about in this post, for which Mypyc would not be suitable.
Each one of these tools impose limitations on either the way that you write code or where you can deploy it.
Cython and Numba aren’t really “Python” any more, because they require
special-purpose performance-oriented annotations. Cython has long supported
pure-Python type annotations, but you won’t get any benefit from telling it
that your variable is an int
, only a cython.int
. It can’t optimize a
@dataclass
, only a @cython.cclass
. And so on.
PyPy gets the closest — it’s definitely regular Python — but its strategy has
important limitations. Primarily, despite the phenomenal and heroic effort
that went into
cpyext
,
it seems like there’s always just one PyPy-incompatible
library
in every large, existing project’s dependency list which makes it impossible to
just drop in PyPy without doing a bunch of arcane debugging first.
PyPy might make your program magically much faster, but if it doesn’t work, you have to read the tea leaves on the JIT’s behavior in a profiler which practically requires an online component that doesn’t even work any more. So mostly you just simplify your code to use more straightforward data structures and remove CPython-specific tricks that might trip up the JIT, and hope for the best.
PyPy also introduces platform limitations. It’s always — understandably, since they have to catch up after the fact — lagging a bit behind the most recently released version of CPython, so there’s always some nifty language feature that you have to refrain from using for at least one more release cycle.
It also has architectural limitations. For example, it performs quite poorly on an M1 Mac since it still runs under x86_64 emulation on that platform. And due to iOS forbidding 3rd-party JITs, it won’t ever be able to provide better performance in one of the more constrained environments that needs it more that other places. So you might need to rely on CPython on those platforms anyway… and you just removed all your CPython-specific hacks to try to please the JIT on the other platforms you support.
So while I would encourage everyone to at least try their code on PyPy — if you’re running a web-based backend, it might save you half your hardware budget6 — it’s not going to solve “python is slow” in the general case.
This all sounds pretty negative, so I would be remiss if I did not also point out that the core team is well aware that Python’s default performance needs to be better, and Guido van Rossum literally came out of retirement for one last job to fix it, and we’ve already seen a bunch of benefits from that effort.
But there are some fundamental limitations on the long-term strategy for these optimizations; one of the big upcoming improvements is a JIT, which suffers from some (but not all) of the same limitations as PyPy, and the late-bound, freewheeling nature of Python inherently comes with some performance tradeoffs.
So it would still behoove us to have a strategy for production-ized code that gives good, portable, ahead-of-time performance.
Mypyc takes the annotations meant for Mypy and generates C with them, potentially turning your code into a much more efficient extension module. As part of Mypy itself, it does this with your existing Python type-hints, the kind you’d already use Mypy with to check for correctness, so it doesn’t entail much in the way of additional work.
I’d been curious about this since it was initially released, but I still haven’t had a hard real-world performance problem to really put it through its paces.
So when I learned about the High Throughput Fizzbuzz Challenge via its impressive assembler implementation that achieves 56GiB/s, and I saw even heavily-optimized Python implementations sitting well below the performance of a totally naïve C reference implementation, I thought this would be an interesting miniature experiment to use to at least approximate practical usage.
The dizzying heights of cycle-counting hand-tuned assembler implementations of this benchmark are squarely out of our reach, but I wanted to see if I could beat the performance of this very naïve C implementation with Python that was optimized, but at least, somewhat idiomatic and readable.
I am about to compare a totally naïve C implementation with a fairly optimized hand-tuned Python one, which might seem like an unfair fight. But what I’m trying to approximate here is a micro-instance of the real-world development-team choice that looks like this:
Since Python is more productive, but slower, the effort to deliver each of the following is similar:
- a basic, straightforward implementation of our solution in C
- a moderately optimized Python implementation of our solution
and we need to choose between them.
This is why I’ll just be showing naïve C and not unrolling any loops; I’ll use
-O3
because any team moderately concerned with performance would at least
turn on the most basic options, but nothing further.
Furthermore, our hypothetical team also has this constraint, which really every reasonable team should:
We can trade off some readability for efficiency, but it’s important that our team be able to maintain this code going forward.
This is why I’m doing a bit of optimizing in Python but not going all out by
calling mmap
or pulling in numpy
or attempting to use something super
esoteric like a SIMD library to emulate what the assembler implementations do.
The goal is that this is normal Python code with a reasonable level of
systems-level understanding (i.e. accounting for the fact that pipes have
buffers in the kernel and approximately matching their size maximizes
throughput).
If you want to see FizzBuzz pushed to its limit, you can go check out the challenge itself. Although I think I do coincidentally beat the performance of the Python versions they currently have on there, that’s not what I’m setting out to do.
So with that elaborate framing of this slightly odd experiment out of the way, here’s our naïve C version:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
|
First, let’s do a quick head-to-head comparison with a naïve Python implementation of the algorithm:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
Running both of these on my M1 Max MacBook, the naïve C implementation yields
127 MiB/s of Fizzbuzz output. But, as I said, although we’re not going to have
time for testing a more complex optimized C version, we would want to at
least build it with the performance benefits we get for free with the -O3
compiler option. It turns out that yields us a 27 MiB/s speedup. So 154 MiB/s
is the number we have to beat.7
The naïve Python version achieves a dismal 24.3 MiB/s, due to a few
issues. First of all, although it’s idiomatic, print()
is doing a lot of
unnecessary work here. Among other things, we are encoding Unicode, which the
C version isn’t. Still, our equivalent of adding the -O3
option for C is
running mypyc
without changing anything, and that yields us a 6.8MiB/s
speedup immediately. We still aren’t achieving comparable performance, but a
roughly 25% performance improvement for no work at all is a promising start!8
In keeping with the “some optimizations, but not so much that it’s illegible” constraint described above, the specific optimizations I’ve chosen to pursue here are:
bytes
objects and sys.stdout.buffer
to avoid encoding overheadHopefully, with that explanation, this isn’t too bad:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
|
Running this optimized version actually gets us within the ballpark of the
naïve C version, even beating it by a hair; my measurement was 159 MiB/s, a
small improvement even over -O3
. So, per the “litany against C” from the
beginning of this post, algorithmic optimization of Python really does help a
lot; it’s not just a rationalization. This is a much bigger boost than our
original no-effort Mypyc run, giving us more like an 85% speedup; definitely
bigger than 25%.
But clearly we’re still being slowed down by Python’s function call overhead, object allocations for small integers, and so on, so Mypyc should help us out here: and indeed it does. On my machine, it nets a whopping 233 MiB/s. Now that we are accounting for performance and optimizing a bit, Mypyc’s relative advantage has doubled to a 50% improvement in performance on both the optimized-but-interpreted Python and naïve C versions.
It’s worth noting that the technique I used to produce the extension modules to
test was literally pip install mypy; mypyc .../module.py
, then python -c
“import module”
. I did already have a C compiler installed, but other than
that, there was no setup.
I just wrote Python, and it just worked.
Here’s what I want you to take away from all this:
Unfortunately, due to the limitations and caveats of existing powerful performance tools like Cython and PyPy, over the last few years in the Python community a passive consensus has emerged. For most projects, in most cases, it’s just not worth it to bother to focus on performance. Everyone just uses the standard interpreter, and only fixes the worst performance regressions.
We should, of course, be glad that the standard interpreter is reliably getting faster all the time now, but we shouldn’t be basing our individual libraries’ and applications’ performance strategies on that alone.
The projects that care the most about performance have made the effort to use some of these tools, and they have often invested huge amounts of effort to good effect, but often they care about performance too much. They make the problem look even harder for everyone else, by essentially stipulating that step 1 is to do something extreme like give up and use Fortran for all the interesting stuff.
My goal with this post is to challenge that status quo, spark interest in revisiting the package ecosystem’s baseline performance expectations, and to get more projects — particularly libraries on PyPI — to pick up Mypyc and start giving Python a deserved reputation for being surprisingly fast.
One immediate objection you might be thinking of is the fact that, under the hood, Mypyc is emitting some C code and building it, and so this might create a problem for deployment: if you’ve got a Linux machine but 30% of your users are on Windows, moving from pure-Python to this hybrid workflow might create installation difficulties for them, or at least they won’t see the benefits.
Luckily a separate tool should make that a non-issue:
cibuildwheel
. “CI Build
Wheel”, as its name suggests, lets you build your wheels in your continuous
integration system, and upload those builds automatically upon tagging a
release.
Often, the bulk of the work in using it is dealing with the additional complexities involved in setting up your build environment in CI to make sure you’re appropriately bundling in any native libraries you depend upon, and linking to them in the correct way. Mypyc’s limitation relative to Cython is a huge advantage here: it doesn’t let you link to other native libraries, so you can always skip the worst step here.
So, for maintainers, you don’t need to maintain a pile of janky VMs on your personal development machine in order to serve your users. For users, nobody needs to deal with the nightmare of setting up the right C compiler on their windows machine, because the wheels are prebuilt. Even users without a compiler who want to contribute new code or debug it can run it with the interpreter locally, and let the cloud handle the complicated compilation steps later. Once again, the fact that you can’t require additional, external C libraries here is a big advantage; it prevents you from making the user’s experience inadvertently worse.
cibuildwheel
supports all major operating systems and architectures, and
supported versions of Python, and even lets you build wheels for PyPy while
you’re at it.9
Using Mypyc and cibuildwheel
, we, as PyPI package maintainers, can
potentially produce an ecosystem of much faster out-of-the-box experiences via
prebuilt extension modules, written entirely in Python, which would make the
average big Python application with plenty of dependencies feel snappier than
expected. This doesn’t have to come with the pain that we have unfortunately
come to
expect
from C extensions, either as maintainers or users.
Another nice thing is that this is not an all-or-nothing proposition. If you try PyPy and it blows up in some obscure way on your code, you have to give up on it unless you want to fully investigate what’s happening. But if you trip over a bug in Mypyc, you can report the bug, drop the module where you’re having the problem from the list of things you’re trying to compile, and move on. You don’t even have to start out by trying to jam your whole project through it; just pick a few key modules to get started, and gradually expand that list over time, as it makes sense for your project.
In a future post, I’ll try to put all of this together myself, and hopefully it’s not going to be embarrassingly difficult and make me eat my words.
Despite not having done that yet, I wanted to put this suggestion out now, to
get other folks thinking about getting started with it. For older
projects10, retrofitting all the existing infrastructure to put Mypyc in
place might be a bit of a challenge. But for new projects starting today,
putting this in place when there’s very little code might be as simple as
adding a couple of lines to pyproject.toml
and copy-pasting some
YAML into
a Github workflow.
If you’re thinking about making some new open source Python, give Mypyc a try, and see if you can delight some users with lightning speed right out of the box. If you do, let me know how it turns out.
Thanks to Donald Stufft, Moshe Zadka, Nelson Elhage, Itamar Turner-Trauring, and David Reid for extensive feedback on this post. As always, any errors or inaccuracies remain my own.
Despite the fact that it is self-described “alpha” software; it’s clearly production-quality enough for Mypy itself to rely upon it, and to have thorough documentation, so if it has bugs that need fixing then it would be good to start discovering them. However, this whole post assumes that you do have good test coverage and you’ll be able to run it over your Mypyc-built artifacts; if you don’t, then this might be too risky. ↩
I’d love to offer an attribution here, but I have no idea where it came from originally. It’s nearly impossible to search the web these days for things that people were saying in 2005... but as I recall, it grew up as a sort of oral tradition of call-and-response about performance complaints on forums and Usenet. ↩
At least in the most favorable cases, of course. You can’t do this for everything, but in any sufficiently large C++ system you can always find some fun oversights. ↩
as I was, at the time. ↩
This is not to write off PyO3, which is an excellent tool. It has many uses beyond speed. Beyond the obvious one of “access to libraries in the excellent Rust ecosystem”, one of its important applications is in creating a safer lingua franca among high-level programming languages. If you have a large, complex, polyglot environment with some Ruby, some Java, some Python and some TypeScript, all of which need to share data structures, Rust is a much better option than C for the common libraries that you might need to bind in all of those languages. ↩
It did for Twisted! We ran our website on absolutely ancient hardware for the better part of a decade and PyPy made it fast enough that nobody really noticed. When we were still using CPython, the site had become barely usable. ↩
You can get this implementation and a table of the resgults here, on github. ↩
I’m not sure that this is a meaningful comparison, but C’s no-cost
optimization option of -O3
is a 20% improvement, so we’re in the same
ballpark, which is interesting. ↩
Interestingly, on PyPy, it might actually be faster to upload a pure-Python wheel anyway, since the higher cost of calling into a C module on that platform might negate any benefits of compiling it. But you’ll have to measure it and see. ↩
not to put too fine a point on it, “like the ones that I maintain” ↩
"With everyone and their dog shifting to containers, and away from virtual machines (VMs), we realized that running vendor-provided software on VMs at Google was slowing us down. So we moved."
Bikram co-authored this blog post last year about DASInfra's experience moving workloads from Corp to Anthos. The group I run at work is going down a similar path by migrating VMs to Anthos on bare metal for on-prem.
Taken from The Playlist - a curated perspective on the intersection of form and content (subscribe, discuss)
"Without a next action, there remains a potentially infinite gap between current reality and what you need to do."
David Allen's Getting Things Done is the non-fiction book I've reread the most in my life. I reread it every couple of years and still pick up on new ideas that I missed before, or parts that resonate better now and I'm excited to implement. Before Google, I used to give this book to new employees as a welcome gift.
The book got an update in 2015, and I haven't read the new version yet, so I'm planning an extended GTD book club at work in Q2, spreading the book out over multiple sessions. (In fact, I did just that for the young adult version of the book with my 16 year old godson back home in Belgium) If you've run a GTD book club, drop me a line!
Find out more at Getting Things Done® - David Allen's GTD® Methodology
"Too many meetings end with a vague feeling among the players that something ought to happen, and the hope that it’s not their personal job to make it so. [...] ask “So what’s the next action on this?” at the end of each discussion point in your next staff meeting"
Taken from The Playlist - a curated perspective on the intersection of form and content (subscribe, discuss)
"Most women fight wars on two fronts, one for whatever the putative topic is and one simply for the right to speak, to have ideas, to be acknowledged to be in possession of facts and truths, to have value, to be a human being."
In honor of International Women's Day 2022 (this past March 8th), some quotes from the 2008 article that inspired the term "mansplaining": to comment on or explain something to a woman in a condescending, overconfident, and often inaccurate or oversimplified manner.
I've certainly been (and probably still am) guilty of this behavior, and this is a standing invitation to let me know when I'm doing it to you.
Read the original article with a new introduction at Men Explain Things to Me – Guernica
"None was more astonishing than the one from the Indianapolis man who wrote in to tell me that he had “never personally or professionally shortchanged a woman” and went on to berate me for not hanging out with “more regular guys or at least do a little homework first,” gave me some advice about how to run my life, and then commented on my “feelings of inferiority.” He thought that being patronized was an experience a woman chooses to, or could choose not to have–and so the fault was all mine. Life is short; I didn’t write back."
Taken from The Playlist - a curated perspective on the intersection of form and content (subscribe, discuss)
In the ever more vertical company that Google is becoming, it is even more important to collaborate on some of your communication - more people want to contribute to the message and get it right, and more thought needs to be given to the ever wider audience you're sending mails to.
A while back I copied over AppScript code from an internal Google project to send meeting notes to make a different tool which makes it easy to go from Google Docs draft to a mail in GMail and avoid embarrassing copy/paste errors. I'm happy to be able to retire that little side project in favor of a recently released built-in feature of Google Docs: Draft emails from Google Docs - Docs Editors Help
Taken from The Playlist - a curated perspective on the intersection of form and content (subscribe, discuss)
"The global COVID-19 pandemic has had countless impacts on society. One interesting effect is that it has created an environment in which many people have been able to explore their gender identity and, in many cases, undergo a gender transition. As organizations return to in-person work, be it full-time or hybrid, there is a greater chance of “out” transgender, non-binary, or gender non-conforming employees in the workforce." (From the "5 Ally Actions Newsletter - Mar 25, 2022")
March 31 is the Transgender Day of Visibility. The COVID Cocoon is a nickname given for the phenomenon of people discovering their gender diversity during the pandemic environment.
The full report is an interesting read; one recommendation that we can all contribute to is on Culture and Communication: Proactively communicating that gender diversity is accepted, asking staff for their input, and being open and ready to listen helps create a culture where employees can feel safe, welcome, and valued.
Taken from The Playlist - a curated perspective on the intersection of form and content (subscribe, discuss)
"Your Second Brain is for preserving raw information over time until it's ready to be used, because information is perishable. Your Second Brain is the brain that doesn't forget." - Tiago Forte
Personal Knowledge Management is going through a wave of innovation with new tools like Roam, Logseq, Obsidian, Notion, RemNote, and others gaining traction over Evernote, OneNote and the like. It's a great time to get curious or reacquaint yourself with the tools and processes that strengthen learning, processing, and expressing your knowledge work.
The expression "Second Brain" has been popularized by Tiago Forte, who's been running an online cohort-based class called Building a Second Brain I took the class last year and found it a powerful distillation of an approach to PKM and note-taking. If you want to learn more, they just wrapped up the Second Brain Summit and posted all videos online: Second Brain Summit 2022 - Full Session Recordings - YouTube
The next class cohort is open for enrollment until March 30th midnight ET, at Building a Second Brain: Live 5-Week Online Course, and runs from April 12th to May 10th, 2022.
"Taking notes is the closest thing we have to time travel." - Kendrick Lamar
Taken from The Playlist - a curated perspective on the intersection of form and content (subscribe, discuss)
On behalf of the Twisted contributors I announce the final release of
Twisted 22.2.0
This is a bugfix release.
The main bug is:
- CVE-2022-21716 twisted.conch.ssh.transport.SSHTransportBase now
disconnects the remote peer if the
SSH version string is not sent in the first 4096 bytes.
No new features were introduced in this release.
Release documentation is available at
https://docs.twistedmatrix.com/en/twisted-22.2.0/
Wheels for the release candidate are available on PyPI
https://pypi.org/project/Twisted/22.2.0/
python -m pip install Twisted==22.2.0
Please use it and report any issues.
Many thanks to everyone who had a part in Twisted development,
the supporters of the Twisted Software Foundation,
the developers, and all the people testing and building great things
with Twisted!
Slava Ukraini!
by Adi Roiban (noreply@blogger.com) at March 05, 2022 10:18 AM
Meetings are both necessary and useful, but they fragment your week, your opportunity for flow, and you need non-meeting time for your output as a knowledge worker.
"Those of us on the maker's schedule are willing to compromise. We know we have to have some number of meetings. All we ask from those on the manager's schedule is that they understand the cost." - Paul Graham, Maker's Schedule, Manager's Schedule
"Sometimes the gift interprets JNDI strings in my log messages and executes random code from my LDAP server. This is the nature of gifts."
An interesting musing on the nature of gifts, big companies and startups, and free software, from apenwarr@
Taken from The Playlist - a curated perspective on the intersection of form and content (subscribe, discuss)
"Like other old houses, [...] has an unseen skeleton, a caste system that is as central to its operation as are the studs and joists that we cannot see in the physical buildings we call home. Caste is the infrastructure of our divisions. It is the architecture of human hierarchy, the subconscious code of instructions for maintaining [...] a [...] social order."
Caste has taken the lead in my library as the most highlighted book, and is a deep exploration of Caste as the lens through which to see discrimination, drawing parallels between Europe, the United States, and India, providing a universal framing.
“Young people,” he said, “I would like to present to you a fellow untouchable from the United States of America.” King was floored. He had not expected that term to be applied to him. He was, in fact, put off by it at first.
Taken from The Playlist - a curated perspective on the intersection of form and content (subscribe, discuss)
"Saboteurs are the voices in your head that generate negative emotions in the way you handle life’s everyday challenges. They represent automated patterns in your mind for how to think, feel, and respond. They cause all of your stress, anxiety, self-doubt, frustration, restlessness, and unhappiness. They sabotage your performance, wellbeing, and relationships."
Positive Intelligence is a mental fitness framework and, among other concepts, taught me helpful practical ways to deal with stress, both professionally and personally.
Take the test or read more on How we self-sabotage
Taken from The Playlist - a curated perspective on the intersection of form and content (subscribe, discuss)
Yesterday, 1Password made the following announcement:
Cryptocurrency? We got you. 💸
— 1Password (@1Password) February 23, 2022
We’ve partnered with @phantom to create a simpler, more secure way to manage cryptocurrencies, tokens and NFTs on the @solana blockchain. Get started: https://t.co/yzcDqrWo06
I am very unhappy about this.
As of this writing, the replies to this announcement are, by my count, roughly 95% paying customers who are furious with them for doing this, 3% scammers who are jubilant that this is popularizing their scamming tool of choice, and about 2% blockchain-enthusiasts expressing confusion as to why everyone is so mad.
Scanning through that 2%’s twitter bios and timelines, I could see content other than memes and shilling, so it seemed at least plausible to me that these people are scam victims who haven’t gotten to the blow-off yet, and their confusion is genuine. Given that “why is everyone so mad” is a much less intense reaction than fury or jubilation, I assume that many others read through some of the vitriol and had this reaction, but then didn’t post anything themselves.
This post is for two audiences: that 2%, genuinely wondering what the big deal is, and also those who have a vague feeling that cryptocurrency is bad, but don’t see the point of making much of a fuss about it.
This is why we should make a fuss about it.
The objection most often raised in the comments went something like this:
This is just a feature that you don’t like; if it’s not for you, just don’t use it. Why yell at 1Password just for making a feature that makes someone else happy?
To begin with, the actual technical feature appears to be something related to auto-filling in browser-extension UI, which is fine. I don’t object to the feature. I don’t even object to features which explicitly help people store cryptocurrency more securely, as a harm reduction measure.
Also, to get this out of the way now: cryptocurrency is a scam. I’m not going to argue the case for that here. Others have made the argument far more exhaustively, and you can read literally hundreds of pages and watch hours of video explaining why by clicking here.
The issue is with the co-marketing effort: the fact that 1Password is using their well-respected brand to help advertise and legitimize scam-facilitation technology like Solana and Phantom.
Even if we were to accept all this, it’s a scam, 1Password is marketing it, etc, my hypothetical blockchain-curious interlocutor here might further object:
What’s the big deal about legitimizing these things, even if they are fraud? Surely you can just not get defrauded, and ignore the marketing?
That’s true, but it also misses the point: legitimizing and promoting these things does various kinds of harm.
More broadly, although I’m writing about 1Password’s specific announcement here, and a small amount of the reasoning will be specific to password management tools, most of the concerns I’ll describe are fairly general to any company promoting or co-marketing with cryptocurrency, and thus hopefully this post will serve for future instances where we should tell some other company to stop supporting blockchains as well.
So with all that out of the way, here are some of the harms that one might be concerned about, from the least selfish concern to the most.
I don’t know how to explain to you that you should care about other people, but if you do care about other people, this could hurt them.
First and foremost, the entire scam of cryptocurrency rests upon making people believe that the assets are worth something. Most people are not steeped in the technical minutiae of blockchains, and tend to trust things based on institutional reputation. 1Password has a strong brand, related to information security, and they’re saying that cryptocurrencies are good, so it’s likely to convince a nonzero number of people to put their money into this technology that has enormous non-obvious risks. They could easily lose everything.
Advertising 1Password in this way additionally encourages users to maintain custody of their own blockchain assets on their own devices. Doing so with 1Password is considerably less risky than it might be otherwise, so if this were to only reach people who were already planning to store their wallets on their own computers, then great.
However, this might encourage users who had not previously thought to look at cryptocurrency at all to do so, and if they found it via 1Password they might start using 1Password to store their first few secrets. Storing them in this way, although less risky, is still unreasonably risky, given the lack of any kind of safety mechanisms on blockchain-backed transactions. Even if they’re savvy enough not to get scammed, nobody is savvy enough not to get hacked, particularly by sophisticated technical attacks which are worth leveraging against high-value targets like people with expensive crypto wallets on their computers.
To be clear, crypto exchanges are, on average, extremely bad at the job of not getting their users money stolen, but individual users are likely to be even worse at that job.
If you don’t care about other people much, but you still care about living in a functioning society, then the promotion of blockchain based financial instruments is a huge destabilization risk. As Dan Olson explains in the devastating video essay / documentary Line Goes Up, blockchain-based financial instruments share a lot of extremely concerning properties that made mortgage-backed securities and collateralized debt obligations so financially toxic in the 2008 crash. Large-scale adoption of these things could lead to a similar crisis, or even worse, a global deflationary spiral in the style of the one that caused the great depression, setting off the kind of economic damage that could result in mass famine and mass death.
Of course, any individual company or celebrity advertising crypto is not going to trigger an immediate economic collapse. Each of these is a snowflake in an avalanche. I have no illusions that convincing just 1Password to stop this is going to turn the tide of the entire blockchain catastrophe that is unfolding all around us, or indeed that my one little post here is going to make the decisive difference between, 1Password stopping vs. not.
But that’s exactly why I’m trying to persuade you, dear reader, that this is a big deal and we should all try to work together to stop it.
While this specific blockchain is “greener” than others, but given the huge proportion of cryptocurrency generally that is backed by electrical waste, and the cultural and technical incentives that make trading one blockchain asset for another more common than cashing out to dollars, it’s still a legitimate concern that promoting blockchain in general will promote environmental destruction indirectly.
Furthermore, the way that Solana is less energy-intensive than other blockchains is by using proof-of-stake, so there’s a sliding scale here between economic and environmental damage, given that proof-of-stake is designed to accelerate wealth accumulation among non-productive participants, and thereby encourages hoarding. So the reduction in environmental damage just makes the previous point even worse.
Even if you’re a full blown sociopath with no concern for others and an iron-clad confidence that you can navigate the collapse of the financial system without any harm to you personally, there is still a pretty big negative here: increased risk from threat actors. Even if you like and use blockchain, and want to use this feature, this risk still affects you.
If 1Password happened to have some features that blockchain nerds could use to store their secrets, then attackers might have some interest in breaking in to 1Password, and could possibly work on tools to do so. That’s the risk of existing on the Internet at all. But if 1Password loudly advertises, repeatedly, that they are will be integrating with a variety of cryptocurrency providers, then this will let attackers know that 1Password is the preferred cryptocurrency storage mechanism.
This further means that attackers will start trying to figure out ways to target 1Password users, on the assumption that we’re more likely to have crypto assets lying around on our filesystems; not only developing tools to break in to 1Password but developing tools to fingerprint users who have the extension installed, who have accounts on the service, whose emails show up on the forum, etc.
Now, of course, 1Password users keep plenty of high-value information inside 1Password already; that’s the whole point. But cryptocurrency is special because of the irreversible nature of transactions, and the immediacy of the benefit to cybercriminals specifically.
If you steal all of someone’s bank passwords, you could potentially get a bunch of their money, but it is expensive and risky for the criminals. The transactions can be traced directly to actual human account holders immediately; anti-money-laundering regulations mean that this can usually be accomplished even across international borders. Transfers can be reversed.
This discrepancy between real money and cryptocurrency is exactly why ransomware was created by cryptocurrency. It makes cryptocurrency attractive specifically to the kinds of people who have expertise and resources to mount wide-spectrum digital attacks against whole populations.
Of course, if they develop tools to fingerprint and hack 1Password users, but they don’t luck out and find easy-to-steal crypto on your computer, they might as well try to steal other things of value, like your identity, credit information, and so on. These are higher-risk, but now that they’ve built all that infrastructure and hacked all these machines, there’s a big sunk cost that makes it more worthwhile.
I really hope that 1Password abandons this destructive scheme. Even if they fully walk this back, I will still find it much harder to recommend their product in the future; there will need to be some active effort to repair trust with their user community. If I’ve convinced you of the problems here, please let them know as a reply to the tweet, the email linked from their blog post, their community forum, or the Reddit post of the announcement, so that they can get a clear signal that this is unacceptable.
This post recommends calling pygame.display.flip
from a thread, which I
tested extensively on mac, windows, and linux before posting, but after some
feedback from readers, I realize that this strategy is not in fact
cross-platform; specifically, the nvidia drivers on linux appear to either
crash or display a black window if you try to do this. The SDL
FAQ does say that you can’t call
“video functions” from multiple threads, and flip
does do that under the
hood. I do plan to update this post again, either with a method to make it
safe, or a method to use slightly more complex timing heuristics to accomplish
the same thing. In the meanwhile, please be aware that this may cause
portability problems for your code.
I’ve written about this
before, but in that
context I was writing mainly about frame-rate independence, and only gave a
brief mention of vertical sync; the title also mentioned Twisted, and upon
re-reading it I realized that many folks who might get a lot of use out of its
technique would not have bothered to read it, just because I made it sound like
an aside in the context of an animation technique in a game that already
wanted to use Twisted for some reason, rather than a comprehensive best
practice. Now that Pygame 2.0 is out, though, and the vsync=1
flag is more
reliably available to everyone, I thought it would be worth revisiting.
Per the many tutorials out there, including the official one, most Pygame mainloops look like this:
1 2 3 4 5 6 7 8 |
|
Obviously that works okay, or folks wouldn’t do it, but it can give an impression of a certain lack of polish for most beginner Pygame games.
The thing that’s always bothered me personally about this idiom is: where does the networking go? After spending many years trying to popularize event loops in Python, I’m sad to see people implementing loops over and over again that have no way to get networking, or threads, or timers scheduled in a standard way so that libraries could be written without the application needing to manually call them every frame.
But, who cares how I feel about it? Lots of games don’t have networking1. There are more general problems with it. Specifically, it is likely to:
Why should anyone care about power when they’re making a video game? Aren’t games supposed to just gobble up CPUs and GPUs for breakfast, burning up as much power as they need for the most gamer experience possible?
Chances are, if you’re making a game that you expect anyone that you don’t personally know to play, they’re going to be playing it on a laptop2. Pygame might have a reputation for being “slow”, but for a simple 2D game with only a few sprites, Python can easily render several thousand frames per second. Even the fastest display in the world can only refresh at 360Hz3. That’s less than one thousand frames per second. The average laptop display is going to be more like 60Hz, or — if you’re lucky — maybe 120. By rendering thousands of frames that the user never even sees, you warm up their CPU uncomfortably4, and you waste 10x (or more) of their battery doing useless work.
At some point your game might have enough stuff going on that it will run the CPU at full tilt, and if it does, that’s probably fine; at least then you’ll be using up that heat and battery life in order to make their computer do something useful. But even if it is, it’s probably not doing that all of the time, and battery is definitely a use-over-time sort of problem.
If you’re rendering directly to the screen without regard for vsync, your players are going to experience Screen Tearing, where the screen is in the middle of updating while you’re in the middle of drawing to it. This looks especially bad if your game is panning over a background, which is a very likely scenario for the usual genre of 2D Pygame game.
Pygame lets you turn on
VSync,
and in Pygame 2, you can do this simply by passing the pygame.SCALED
flag and
the vsync=1
argument to
set_mode()
.
Now your game will have silky smooth animations and scrolling5! Solved!
But... if the fix is so simple, why doesn’t everybody — including, notably, the official documentation — recommend doing this?
The solution creates another problem: pygame.display.flip
may now block
until the next display refresh, which may be many milliseconds.
Even worse: note the word “may”. Unfortunately, behavior of vsync is quite
inconsistent between platforms and
drivers,
so for a properly cross-platform game it may be necessary to allow the user to
select a frame rate and wait on an asyncio.sleep
than running flip
in a
thread. Using the techniques from the answers to this stack overflow
answer
you can establish a reasonable heuristic for the refresh rate of the relevant
display, but if adding those libraries and writing that code is too complex,
“60” is probably a good enough value to start with, even if the user’s monitor
can go a little faster. This might save a little power even in the case where
you can rely on flip
to tell you when the monitor is actually ready again;
if your game can only reliably render 60FPS anyway because there’s too much
Python game logic going on to consistently go faster, it’s better to achieve
a consistent but lower framerate than to be faster but inconsistent.
The potential for blocking needs to be dealt with though, and it has several knock-on effects.
For one thing, it makes my “where do you put the networking” problem even worse: most networking frameworks expect to be able to send more than one packet every 16 milliseconds.
More pressingly for most Pygame users, however, it creates a minor performance
headache. You now spend a bunch of time blocked in the now-blocking flip
call, wasting precious milliseconds that you could be using to do stuff
unrelated to drawing, like handling user input, updating animations, running
AI, and so on.
The problem is that your Pygame mainloop has 3 jobs:
What you want to do to ensure the smoothest possible frame rate is to draw
everything as fast as you possibly can at the beginning of the frame and then
call flip
immediately to be sure that the graphics have been delivered to the
screen and they don’t have to wait until the next screen-refresh. However,
this is at odds with the need to get as much done as possible before you call
flip
and possibly block for 1/60th of a second.
So either you put off calling flip
, potentially risking a dropped frame if
your AI is a little slow, or you call flip
too eagerly and waste a bunch of
time waiting around for the display to refresh. This is especially true of
things like animations, which you can’t update before drawing, because you
have to draw this frame before you worry about the next one, but waiting until
after flip
wastes valuable time; by the time you are starting your next
frame draw, you possibly have other code which now needs to run, and you’re
racing to get it done before that next flip
call.
Now, if your Python game logic is actually saturating your CPU — which is not hard to do — you’ll drop frames no matter what. But there are a lot of marginal cases where you’ve mostly got enough CPU to do what you need to without dropping frames, and it can be a lot of overhead to constantly check the clock to see if you have enough frame budget left to do one more work item before the frame deadline - or, for that matter, to maintain a workable heuristic for exactly when that frame deadline will be.
The technique to avoid these problems is deceptively simple, and in fact it was
covered with the deferToThread
trick presented in my earlier
post. But again,
we’re not here to talk about Twisted. So let’s do this the
no-additional-dependencies, stdlib-only way, with asyncio:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
|
At some point I will probably release my own wrapper library6 which does something similar to this, but I really wanted to present this as a technique rather than as some packaged-up code to use, since do-it-yourself mainloops, and keeping dependencies to a minimum, are such staples of Pygame community culture.
As you can see, this technique is only a few lines longer than the standard recipe for a Pygame main loop, but you now have access to a ton of additional functionality:
I really hope that this sees broader adoption so that the description “indie game made in Python” will no longer imply “runs hot and tears a lot when the screen is panning”. I’m also definitely curious to hear from readers, so please let me know if you end up using this technique to good effect!7
And, honestly, a few fewer could stand to have it, given how much unnecessary always-online stuff there is in single-player experiences these days. But I digress. That’s why I’m in a footnote, this is a good place for digressing. ↩
“Worldwide sales of laptops have eclipsed desktops for more than a decade. In 2019, desktop sales totaled 88.4 million units compared to 166 million laptops. That gap is expected to grow to 79 million versus 171 million by 2023.” ↩
At least, Nvidia says that “the world’s fastest esports displays” are both 360Hz and also support G-Sync, and who am I to disagree? ↩
They’re playing on a laptop, remember? So they’re literally uncomfortable. ↩
Assuming you’ve made everything frame-rate independent, as mentioned in the aforementioned post. ↩
because of course I will ↩
And also, like, if there are horrible bugs in this code, so I can update it. It is super brief and abstract to show how general it is, but that also means it’s not really possible to test it as-is; my full-working-code examples are much longer and it’s definitely possible something got lost in translation. ↩
Popular products often have a "Getting Started" tutorial in order to guide you in using them for the first timees. Those guides are easy to follow, pleasant to use, and lead you do make bad design choices.
Follow the "Getting Started" guide to get a feel for the product. Then, skip to the end: in the "Advanced" section, you will find how the people who really use it in production use it.
What do flask.Blueprint, requests.Session, and ansible.plugins have in common?
They all share two curious properties:
All of these technologies are popular for their niche. All of them have friendly "Getting started" guides.
Those guides do not cover the relevant piece, the piece that makes this technology usable in most realistic cases. Most people who use these technology start with "Getting Started", and follow the path it suggests. The path it suggests eventually breaks down.
When this path breaks down, often people will think to themselves "I don't need the advanced stuff. My use case is pretty vanilla." They will shim and work around the limitations, until either giving up on the whole enterprise, or asking an expert.
At the point where they asked an expert, the expert will be exasperated. "Just use flask.Blueprint/requests.Session/ansible.plugins for anything beyond a toy example. This is the way to really use the technology."
The expert is exasperated because this is the fifth (or the hundred and fifth!) time this has come. Someone is about to give up in disgust, only to learn that the solution was simple all along: the first three paragraphs in the "Advanced" section cover how to do this.
I chose these three examples because I know them well. I can give similar examples from almost every popular technology: following the "Quickstart" guide leads to a huge mess, while a fairly simple way to do it in a better way is presented in a section titled "Advanced" or "Developer" or something similar.
These things happen so often that this cannot be a coincidence. Just like the common shape of whales and fish, despite separate evolutionary origins, the explanation is the shape of the dynamic systems at play.
Whales and fish look similar because they both have to solve a similar problem: moving through water efficiently. Having an appropriate aquadynamic shape is the right solution.
Technologies, open source or commercial, live and die by their adoption. A technology that is hard to adopt is a useless relic.
There are many things that can impact the ease of adoption of a technology. One of the easiest levers to pull, by the core development team, is the "Quickstart" guide. This guide tries to get a toy example up and running as fast as possible.
At this point, any extra complexity is a big turn off. Any extra step will have a large fraction of the users dropping off. This exponential decay means that to survive, the Quickstart guide needs to be heavily optimized.
It does not matter whether it's "design" (developers explicitly optimizing the guide) or "evolution" (technologies which don't do that do not end up surviving long enough to cause problems). It is probably a combination of both factors.
As can be predicted from general optimization problems, you can optimize more when you drop constraints. Removing the constraint "must lead to a proper production solution" from the requirements for the Quickstart guide leads to a guide more optimized to win hearts and minds.
At the point the project is popular, this will end up being the shape of the documentation:
Being upset with Quickstart guides, as I have for many years, is like being upset with gravity, which I also have for many years. Those phenomena are annyoing, but they are the results of immutable laws of nature.
In order to fly, or use technologies, you need to first understand the physics involved and then figure out how to use them to accomplish your goals. Now that the shape is understandable, how can it make adopting technologies easier?
The quickstart guide is still useful. Follow it. Generate a toy example. Play around with it. Get a feel for the technology. See if it's at all a good fit.
Put all of these attempts in a source-controlled repository. Now move all of the code written so far into a subdirectory, experiments. Go to the advanced section, and figure out what are the 1-3 things that you need to do in order to properly use the technology. Alternatively, consult someone with experience.
Use the notes from the experiments, plus the understanding from the "advanced" section, to build your first "real" proof of concept. From this point on, this is the "toy example" you iterate on.
The experiments serve as research notes for how to do the specific things you needed. Never use the experiments directly.
Good technologies will gently nudge you to using them badly. This is not good or bad: this is a law of nature. Understand it, and start using platforms better!
Thanks to Alex Scammon for his feedback on an earlier draft of this post. Any mistakes that remain are my responsibility.
Last night I rode our bike home from Brooklyn, with my daughter crying loudly "You are not my papa!" most of the way.
We were a few minutes late picking her up from her class, and she was the last one there, crying in the arms of the teacher, and yelling something loudly, too loud to understand.
I picked her up, hugged her, asked what's wrong and tried to calm her down, but she wasn't having it. I put her in the back of our bike, strapping her in, checking with my son what she could be saying. We finally started making out that she was saying "not my papa".
I tried to convince her that I am, in fact, her papa, but she just kept repeating the same thing. We started our ride back home, and at the first red light I was acutely aware of her still yelling the same thing while standing still in traffic next to other bikes. What would I do if I was stuck in traffic next to a vehicle with a crying child yelling "You are not my papa?", I wondered. I started asking her questions like, "what hair color does your papa have?" to get her to stop and think, and I would respond, "that's interesting, just like me". I'd ask a few questions like that until the lights turned green.
I was hoping this would work for all the stops on our 25 minute ride home, and I was hoping we'd not run into any police cars along the way, just in case. Of course, two minutes later, I was parallel with a string of five police cars, all with their lights flashing. I kept repeating the questions at every stop, until she fell asleep as she usually does on the bike.
She slept all the way through dinner, and the next morning at breakfast I asked her, "who's your papa?" And she beamed at me and yelled, "you are my papa!"
My best guess at what happened is that at pickup she saw a string of papas pick up their kids, but didn't see me, and started saying "you are not my papa" at every other papa, until I was the last one to show up. I'll never show up last again.
by glyph (noreply@blogger.com) at February 09, 2022 07:51 PM
How to seamlessly support typing.Protocol
on Python versions older and newer than 3.8. At the same time.
Obsidian's Gems of the Year 2021 nomination has been a great source of cool ideas to add tweaks to my Obsidian setup.
In particular, Quick Capture (mac/iOS) and Inbox Processing was a great gem to uncover as I try and implement the weekly review stage of my Second Brain/PARA setup!
I noticed that the archive/move script was a little slow, taking several seconds to open up the dialog for selecting a folder, breaking my flow. I checked the code and noticed it built a set of folders recursively.
I simplified the code for my use case, removing the archive folder path, and using the file explorer's built in move dialog (which is much faster) and a callback to advance.
The resulting gist is Obsidian: Archive current file and then open next file in folder (Templater script) · GitHub
I'm sure it could be improved further if I understood the execution, variable scope, and callback model better, but this is good enough for me!
I get very little coding time these days, and I hate working in an environment I haven't had a chance to really master yet. It's all trial and error through editing a javascript file in a markdown editor with no syntax highlighting. But it's still a nice feeling when you can go in and out of a code base in a few hours and scratch the itch you had.
An attempt at catharsis. This is a deeply personal blog post about the most influential project I’ve ever created: attrs, the progenitor of modern Python class utilities. I’m retelling its history from my perspective, how it begot dataclasses, and how I’m leading it into the future.
One of my favorite Podcasts, Podcast.__init__, always makes sure to ask one question towards the end of every episode: when should you not use the tool or technique introduced in this episode? This is a great question. When people have a poor answer I assume that the tool is not good, or at least that it is not mature enough.
If you use a tool for long enough, and enough circumstances, you will have some cases where it is a poor fit. Not just a poor fit: a painful mistake that you will regret.
This does not mean the tool is bad. This is true for the some of the best tools I have used, and some of the ones I have the most experience with.
If a tool is not fundamentally useful, it will not get to the point where it is causing problems. It will be dropped long before it is used in a place or in a way where it is a mistake that makes people regret.
A good rule of thumb for when you really understand how a tool works is that you can point out at least three big problems with it. Try it now with some of your favorite tools and tecniques. List three big problems with them.
For example, it is no secret that I like, and have plenty of experience in, Python. There are more than three problems, and sorting by how big they are is subjective.
Regardless, here are three big problems in Python:
setuptools
, venv
, pip
, and
pip-tools
.
These are all separate projects,
the documentation is in different places,
and understanding how they interact takes a while to master.sys
,
useful things,
like heapq
,
useless things,
like urllib.request
,
and poor substitues for third-party packages,
like dataclasses
.
Knowing which is which is mostly folklore,All of this is not to say I think Python is bad: I use it every day! It is to show how harsh you should be when talking about the problems with something you like. Until you can do that, avoid using the tool anywhere important, unless someone else, who can articulate the tool's problems, is guiding the project.
Thanks to benny Vasquez and Alex Scammon for their useful comments on an earlier draft of this post. Any mistakes that remain are my responsibility alone.
This is a bit of a rant, and it's about a topic that I’m not an expert on, but I do feel strongly about. So, despite the forceful language, please read this knowing that there’s still a fair amount of epistemic humility behind what I’m saying and I’m definitely open to updating my opinion if an expert on journalism or public policy were to have some compelling reason for the Chestertonian fence of the structure of journalistic institutions. Comments sections are the devil’s playground so I don’t have one, but feel free to reach out and if we have a fruitful discussion I’m happy to publish it here.
One of the things that COVID has taught me is that the concept of a “story” in the news media is a relic that needs to be completely re-thought. It is not suited to the challenges of media communication today.
Specifically, there are challenging and complex public-policy questions which require robust engagement from an informed electorate1. These questions are open-ended and their answers are unclear. What’s an appropriate strategy for public safety, for example? Should policing be part of it? I have my preferred snappy slogans in these areas but if we want to step away from propaganda for a moment and focus on governance, this is actually a really difficult question that hinges on a ton of difficult-to-source data.
For most of history, facts were scarce. It was the journalist’s job to find facts, to write them down, and to circulate them to as many people as possible, so that the public discourse could at least be fact-based; to have some basis in objective reality.
In the era of the Internet, though, we are drowning in facts. We don't just have facts, we have data. We don't just have data, we have metadata; we have databases and data warehouses and data lakes and all manner of data containers in between. These data do not coalesce into information on their own, however. They need to be collected, collated, synthesized, and interpreted.
Thus was born the concept of Data Journalism. No longer is it the function of the journalist simply to report the facts; in order for the discussion to be usefully grounded, they must also aggregate the facts, and present their aggregation in a way that can be comprehended.
Data journalism is definitely a step up, and there are many excellent data-journalism projects that have been done. But the problem with these projects is that they are often individual data-journalism stories that give a temporal snapshot of one journalist's interpretation of an issue. Just a tidy little pile of motivated reasoning with a few cherry-picked citations, and then we move on to the next story.
And that's when we even get data journalism. Most journalism is still just isolated stories, presented as prose. But this sort of story-after-story presentation in most publications provides a misleading picture of the world. Beyond even the sample bias of what kinds of stories get clicks and can move ad inventory, this sequential chain of disconnected facts is extremely prone to cherry-picking by bad-faith propagandists, and even much less malicious problems like recency bias and the availability heuristic.
Trying to develop a robust understanding of complex public policy issues by looking at individual news stories is like trying to map a continent's coastline by examining individual grains of sand one at a time.
What we need from journalism for the 21st century is a curated set of ongoing collections of consensus. What the best strategy is to combat COVID might change over time. Do mask mandates work? You can't possibly answer that question by scrounging around on pubmed by yourself, or worse yet reading a jumbled stream of op-ed thinkpieces in the New York Times and the Washington Post.
During COVID, some major press institutions started caving to the fairly desperate need for this sort of structure by setting up "trackers" for COVID vaccinations, case counts, and so on. But these trackers are still fit awkwardly within the "story" narrative. This one from the Washington post is a “story” from 2020, but has data from December 16th, 2021.
These trackers monitor only a few stats though, and don’t provide much in the way of meta-commentary on pressing questions: do masks work? Do lockdowns work? How much do we know about the efficacy of various ventilation improvements?
Each journalistic institution should maintain a “tracker” for every issue of public concern, and ideally they’d be in conversation with each other, constantly curating their list of sources in real time, updating conclusions as new data arrives, and recording an ongoing tally of what we can really be certain about and what is still a legitimate controversy.2
At least, they do if you want the human race to survive into the next century, but let’s take it as read that at least my readers want that. ↩
Now that I write this out, maybe scientific research institutions would be having a better time of it if they had something similar as well... model-consensus trackers would be a place that people could actually be rewarded for publishing replication failures, since you could be incentivized to materially update the consensus rather than being rewarded just for having something that gets cited a lot and never replicated. Hmm. ↩
This weekend a catastrophic bug in log4j2
was
disclosed,
leading to the potential for remote code execution on a huge number of
unpatched endpoints.
In this specific case, it turns out there was not really any safe way to use the API. Initially it might appear that the issue was the treatment of an apparently fixed format string as a place to put variable user-specified data, but as it turns out it just recursively expands the log data forever, looking for code to execute. So perhaps the lesson here is nothing technical, just that we should remain ready to patch, or that we should pay the maintainers.
Still, it’s worth considering that injection vulnerabilities of this type exist pretty much everywhere, usually in places where the supposed defense against getting catastrophically RCE’d is to carefully remember that the string that you pass in isn’t that kind of string.
While not containing anything nearly so pernicious as a place to put a URL that
lets you execute arbitrary attacker-controlled code, Python’s
logging
module does contain a fair amount of confusing indirection around its log
message. Sometimes — if you’re passing a non-zero number of *args
— the
parts of the logging module will interpret msg
as a format string; other
times it will interpret it as a static string. This is in some sense a
reasonable compromise; you can have format strings and defer formatting if you
want, but also log.warning(f"hi, {attacker_controlled_data}")
is fairly
safe by default. It’s still a somewhat muddled and difficult to document
situation.
Similarly, Twisted’s logging system does always treat its string argument as a format string, which is more consistent. However, it does let attackers put garbage into the log wherever the developer might not have understood the documentation.1
This is to say nothing of the elephant in the room here: SQL. Almost every SQL
API takes a bunch of strings, and the ones that make you declare an object in
advance (i.e. Java’s PreparedStatement
) don’t mind at all if you create one
at runtime.
In the interest of advancing the state of the art just a little here, I’d like to propose a pattern to encourage the idiomatic separation of user-entered data (i.e. attacker-controlled payloads) from pre-registration of static, sensitive data, whether it’s SQL queries, format strings, static HTML or something else. One where copying and pasting examples won’t instantly subvert the intended protection. What I suggest would look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
|
The idea here is that sql_statements.declarations()
detects which module it’s
in, and only lets you write those declarations once. Attempting to stick
that inside your function and create some ad-hoc formatted string should
immediately fail with a loud exception; copying this into the wrong part of
your code just won’t work, so you won’t have a chance to create an injection
vulnerability.
If this idea appeals to you, I’ve written an extremely basic prototype here on github and uploaded it to PyPI here.
I’m not dropping a 0day on you, there’s not a clear vulnerability here; it only lets you draw data from explicitly-specified parameters into the log. If you use it wrong, you just might get an "Unable to format event" type error, which we'll go out of our way to not raise back to you as an exception. It just makes some ugly log messages. ↩
Codecov’s unreliability breaking CI on my open source projects has been a constant source of frustration for me for years. I have found a way to enforce coverage over a whole GitHub Actions build matrix that doesn’t rely on third-party services.
The essence of software engineering is solving problems.
The first impression of this insight will almost certainly be that it seems like a good thing. If you have a problem, then solving it is great!
But software engineers are more likely to have mental health problems1 than those who perform mechanical labor, and I think our problem-oriented world-view has something to do with that.
So, how could solving problems be a problem?
As an example, let’s consider the idea of a bug tracker.
For many years, in the field of software, any system used to track work has been commonly referred to as a “bug tracker”. In recent years, the labels have become more euphemistic and general, and we might now call them “issue trackers”. We have Sapir-Whorfed2 our way into the default assumption that any work that might need performing is a degenerate case of a problem.
We can contrast this with other fields. Any industry will need to track work that must be done. For example, in doing some light research for this post, I discovered that the relevant term of art in construction3 is typically “Project Management” or “Task Management” software. “Projects” and “Tasks” are no less hard work, but the terms do have a different valence than “Bugs” and “Issues”.
I don’t think we can start to fix this ... problem ... by attempting to change the terminology. Firstly, the domain inherently lends itself to this sort of language, which is why it emerged in the first place.
Secondly, Atlassian has desperately been trying to get everybody to call their bug tracker a “software development tool” where you write “stories” for years, and nobody does. It’s an issue tracker where you file bugs, and that’s what everyone calls it and describes what they do with it. Even they have to protest, perhaps a bit too much, that it’s “way more than a bug and issue tracker”4.
This pervasive orientation towards “problems” as the atom of work does extend to any knowledge work, and thereby to any “productivity system”. Any to-do list is, at its core, a list of problems. You wouldn’t put an item on the list if you were happy with the way the world was. Therefore every unfinished item in any to-do list is a little pebble of worry.
As of this writing, I have almost 1000 unfinished tasks on my personal to-do list.
This is to say nothing of any tasks I have to perform at work, not to mention the implicit א0 of additional unfinished tasks once one considers open source issue trackers for projects I work on.
It’s not really reasonable to opt out of this habit of problematizing everything. This monument to human folly that I’ve meticulously constructed out of the records of aspirations which exceed my capacity is, in fact, also an excellent prioritization tool. If you’re a good engineer, or even just good at making to-do lists, you’ll inevitably make huge lists of problems. On some level, this is what it means to set an intention to make the world — or at least your world — better.
On a different level though, this is how you set out to systematically give yourself anxiety, depression, or both. It’s clear from a wealth of neurological research that repeated experiences and thoughts change neural structures5. Thinking the same thought over and over literally re-wires your brain. Thinking the thought “here is another problem” over and over again forever is bound to cause some problems of its own.
The structure of to-do apps, bug trackers and the like is such that when an item is completed — when a problem is solved — it is subsequently removed from both physical view and our mind’s eye. What would be the point of simply lingering on a completed task? All the useful work is, after all, problems that haven’t been solved yet. Therefore the vast majority of our time is spent contemplating nothing but problems, prompting the continuous potentiation6 of neural pathways which lead to despair.
I don’t want to pretend that I have a cure for this self-inflicted ailment. I do, however, have a humble suggestion for one way to push back just a little bit against the relentless, unending tide of problems slowly eroding the shores of our souls: a positivity journal.
By “journal”, I do mean a private journal. Public expressions of positivity7 can help; indeed, some social and cultural support for expressing positivity is an important tool for maintaining a positive mind-set. However, it may not be the best starting point.
Unfortunately, any public expression becomes a discourse, and any discourse inevitably becomes a dialectic. Any expression of a view in public is seen by some as an invitation to express its opposite8. Therefore one either becomes invested in defending the boundaries of a positive community space — a psychically exhausting task in its own right — or one must constantly entertain the possibility that things are, in fact, bad, when one is trying to condition one’s brain to maintain the ability to recognize when things are actually good.
Thus my suggestion to write something for yourself, and only for yourself.
Personally, I use a template that I fill out every day, with four sections:
“Summary”. Summarize the day in one sentence that encapsulates its positive vibes. Honestly I put this in there because the Notes app (which is what I’m using to maintain this) shows a little summary of the contents of the note, and I was getting annoyed by just seeing “Proud:” as the sole content of that summary. But once I did so, I found that it helps to try to synthesize a positive narrative, as your brain may be constantly trying to assemble a negative one. It can help to write this last, even if it’s up at the top of your note, once you’ve already filled out some of the following sections.
“I’m proud of:”. First, focus on what you personally have achieved through your skill and hard work. This can be very difficult, if you are someone who has a habit of putting yourself down. Force yourself to acknowledge that you did something useful, even if you didn’t finish anything, you almost certainly made progress and that progress deserves celebration.
“I’m grateful to:”. Who are you grateful to? Why? What did they do for you? Once you’ve made the habit of allowing yourself to acknowledge your own accomplishments, it’s easy to see those; pay attention to the ways in which others support and help you. Thank them by name.
“I’m lucky because:”. Particularly in post-2020 hell-world it’s easy to feel like every random happenstance is an aggravating tragedy. But good things happen randomly all the time, and it’s easy to fail to notice them. Take a moment to notice things that went well for no good reason, because you’re definitely going to feel attacked by the universe when bad things happen for no good reason; and they will.
Although such a journal is private, it’s helpful to actually write out the answers, to focus on them, to force yourself to get really specific.
I hope this tool is useful to someone out there. It’s not going to solve any problems, but perhaps it will make the world seem just a little brighter.
“Maintaining Mental health on Software Development Teams”, Lena Kozar and Vova Vovk, in InfoQ ↩
“Construction Task and Project Tracking”, from Raptor Project Management Software ↩
Jira Features List, Atlassian Software ↩
“Culture Wires the Brain: A Cognitive Neuroscience Perspective”, Denise C. Park and Chih-Mao Huang, Perspect Psychol Sci. 2010 Jul 1; 5(4): 391–400. ↩
Long-term potentiation and learning, J L Martinez Jr, B E Derrick ↩
The #PositivePython hashtag on Twitter was a lovely experiment and despite my cautions here about public solutions to this problem, it’s generally pleasant to participate in. ↩
Trying out something new: today I’m launching my own Today I Learned section. In this essay I will sum up what my plans and hopes are.
As I mentioned previously, I’ve recently been medicated for ADHD.
Everyone’s experience with medication, even the same medication, is different, but my particular experience — while hugely positive — has involved not so much a decrease in symptoms, but rather a shifting of my symptom profile. Some of my executive functions (particularly task initiation) have significantly improved, but other symptoms, such as time blindness have gotten significantly worse. This means, for example, I can now easily decide to perform a task, and actually maintain focus on that task for hours1, but it’s harder to notice that it’s time to stop, and still somewhat difficult to tear myself away from it.
I’ve tried pomodoro timers before and I’ve had mixed success with them. While I tend to get more done if I set a pomodoro, it’s hard to remember to set the timers in the first place, and it’s hard to do the requisite time-keeping to remember how many pomodoros I’ve already set, how many more I’ll have the opportunity to set, etc. Physical timers have no associated automation and data recording, and apps can be so unobtrusive that I can easily forget about them entirely. I’ve long had an aspiration to eventually write my own custom-tailored app that addresses some of these issues.
As part of a renewed interest in ADHD management techniques, I watched this video about ADHD treatments from Dr. Russell Barkley, wherein he said (I’m paraphrasing) “if I don’t put an intervention into your visual field it might as well not exist”.
I imagined timer that:
So, last weekend, leveraging my newly enhanced task-initiation and concentration-maintenance abilities, I wrote it, and I’ve been using it all week. Introducing Pomodouroboros, the pomodoro timer that reminds you that the uncaring void marches on regardless of your plans or intentions.
I’ve been using it all week and preliminary results are extremely positive.
This thing is in an extremely rough state. It has no tests, no docs, and an extremely inscrutable UI that you need to memorize in order to use effectively. I need plenty of help with it. I contemplated keeping it private and just shipping a binary, but a realistic assessment of my limited time resources forced me to admit that it already kind of does what I need, and if I want to enhance it to the point where it can help other people, I’ll need plenty of help.
If this idea resonates with you, and you’re on macOS, check out the repo, make
a virtualenv somehow, install its dependencies, I don’t know how you make
virtualenvs or install dependencies, I’m not your dad2, and run ./runme
.
If you’re on another platform, check out the code, ask me some questions, and
maybe try to write a
port to
one of
them.
I cannot express how alien the sensation is to have conscious control over initiating this process; I’ve certainly experienced hyperfocus before but it’s always been something that happens to me and not something that I do ↩
If I am your dad, come talk to me, based on your family history it’s quite likely that you do have ADHD and I’m happy to talk about how to get this installed for you offline. ↩
Originally published on Enable Architect.
Modern computer systems supply business-critical services everywhere -- from Amazon providing shopping services to Healthcare.gov providing enrollment in health insurance plan. We all rely on such systems. But, unfortunately, these systems are complex and can fail in surprising ways.
By now, it is a well-understood best practice that when failure happens, it's an opportunity to learn and improve. Thus, blameless retrospectives (sometimes called "post-mortems") are by now a development-cycle staple.
However, the processes by which organizations conduct the failure analysis, and make improvement recommendations, are still based on shaky foundations. It is time to do better.
It is possible to do Root Cause Analysis (RCA) as originally defined. This means looking for the initial action that started the problem (i.e. the "root") and then figuring out how to prevent it in the future. However, in recent years this method is seen to be of limited value. The root cause is hard to define in increasingly complex systems and not necessarily the right thing to change.
Most organizations that conduct RCA do not follow the original definition. Instead, they do ad-hoc modifications. They look for all contributing causes, starting with the root cause, and then offer mitigation.
In acknowledgment of the limitations of RCA, there is a new emphasis on service reliability. Reliability often focuses on the need to have services resilient to upstream failure.
Acknowledging the complexity of modern systems and formalizing it, the Causal Analysis based on System Theory (CAST) process does precisely that: a way to improve service reliability. Instead of ad-hoc modifications to a fundamentally broken analysis process, CAST offers an alternative from-the-ground-up analysis method based on Professor Levenson's research into system safety theory.
CAST is a modern approach to analyze failure, as described in Professor Levenson's book. As written, it assumes a physical system. However, this process is adaptable to investigating software, and especially for service outages. It is an alternative to the so-called RCA.
CAST contains five steps. Although it sometimes makes sense to go back to a previous stage as you uncover more information, in general, the analysis should follow the steps in order:
When assembling basic information, the first part is to define the system involved. This indicates what the boundaries of the analysis are. This part is essential: it should be clear what part is the system and the environment.
Next, describe the loss: the undesirable behavior. Explain the hazard (the original change) that led to it
From the hazard, identify the system-level safety constraints required to prevent it. Those are the system safety requirements and constraints.
The next part is to construct a timeline. Describe what happened. Avoid any conclusions, and especially avoid assigning blame. This part will usually include open questions, especially about why things happened.
Analyze the loss in terms of the system requirements and controls in place. This includes any mechanisms that were put in place to prevent such problems. Indicate what interactions happened between different parts that led to the problem. Note any contextual factors that influenced the events.
The model of underlying causality CAST treats safety as a control problem, not a failure problem. Thus, the cause is always that the control structure and controls constructed to prevent the hazard.
If a control structure for the system does not already exist, it might be helpful to start with an abstract high-level control structure.
Examine the components of the control structure to determine why they were not effective in preventing the loss.
Start at the bottom of the control structure. Explain each component's role in the accident and analyze its behavior and why it did what it did. As context, add the details from the original design for why these controls were deemed adequate.
Identify general systemic factors that contributed to the loss. These factors cut across the different control structure components. Thus, it is important to add this step explicitly to account for such cross-cutting concerns.
Create recommendations for changes to the control structure to prevent a similar loss in the future. These might include a continuous improvement program as part of an overall risk management program.
The CAST process is a modern theory-inspired method that is tested by practice, improving safety and reliability. Professor Levenson has many of her books, including the CAST handbook, available from the MIT website, where you can learn more about the background, the theory, and the practice.
Now go forth, and conduct better retrospectives!
On behalf of the Twisted contributors I announce the final release of Twisted 21.7.0
This article was originally published on Enable Architect
Many people have had the insight that DevOps is about people. Often, they will summarize it as "DevOps is about empathy". I have found, however, that idealizing empathy is just as bad as thinking that DevOps is about a single technology.
I remember when I first heard Paul Bloom talking on Rationally Speaking. Julia Galef introduced him by saying:
"I'm writing a book on empathy," psychologist Paul Bloom tells people. They respond warmly, until he follows up with, "I'm against it."
Many of the people who are titled, at times, Site Reliability Engineers (SRE) -- or DevOps Engineers, or Production Engineers, or Platform Engineers, or other terms that indicate the same responsibilities connected to DevOps practices -- are fundamentally and intentionally different than many of the people they have to work with. In order to be good at this job, you need a reasonably solid ability to program and a reasonably solid ability to handle operational issues.
This is an uncommon skill set. It often is acquired by starting as a software developer or an IT administrator and slowly gaining complementary skills. This means there are few truly "junior" people in that role. Gaining the necessary skills and experience takes time.
Being more senior than someone, and having skills they lack, makes it difficult to empathize. It is difficult and inaccurate to guess what someone might be struggling with or what they need help with. Site Reliability Engineers who try to use empathy imagine themselves in the other person's role and will build tools and processes that would be good for themselves if they were in that role.
In contrast, sympathy begins with trusting that people have unique insight into their own lived experiences. A Site Reliability Engineer focusing on sympathy will start by talking to people, understanding their problems, and believing them when describing pain points. A Site Reliability Engineers focusing on sympathy will involve others in the decision process to solve those problems. A Site Reliability Engineer focusing on sympathy will release partial solutions to focus groups to see how they fail when used by people different from them.
Sympathy. Compassion. Trust. These are the main tools a Site Reliability Engineer uses daily to make DevOps possible. Developing those is easy, as long as you care about people. Caring about people is the only thing that can't be taught.
I cannot teach anyone to care about people. If you do care about people, you already have the most important skill needed to succeed as an SRE.
With in-person conferences starting to open up, I need to clear the dust off of some skills that have not been used in a while. One of those is how to pack for travel.
This list works for me. It will probably not work for you as-is. Among other things, I have very specific guidelines.
I don't count things I usually carry in my pockets: phone, wallet, house keys. This is because I do not need to pack them.
I also do not like checking in luggage, so I have optimized the list for avoiding that. Among other things, I intentionally minimized the list.
My goal is to be able to pack, from scratch, in under 15 minutes. I pack things into my travel backpack, but I also pack a small walking-around backpack. This way, the big backpack can stay in the hotel room, and I can work around with a bare-bones backpack (just a laptop and maybe a battery).
The conflict between subclassing and composition is as old as object-oriented programming. The latest crop of languages like Go or Rust prove that you don’t need subclassing to successfully write code. But what’s a pragmatic approach to subclassing in Python, specifically?
Sometimes you will be working on hairy and complicated feature in a shared repository. Maybe it's for work. Maybe it's an open source project.
As a responsible person, you are working on a branch. The usual way of working involves a lot of "intermediate" check-ins. Those serve, if nothing else, as a form of backup.
If you really enjoy backing up your code, you are probably already pushing to a remote branch on your source control server.
This is a great workflow. Responsible, safe, and resistant to local failures.
What could be better?
Well, it is often the case that people hesitate to open the pull request (or merge request, as known in some systems) before they are "ready". Time to post the PR, already!
If the PR is not ready, you can mark it as not ready for merging. Most modern systems allow an explicit flag to make pull requests as "draft". If nothing else, you can use a dedicated label like "do not merge".
There are a few benefits. One is that when your Continuous Integration system runs, this gives an obvious place to keep the results. This avoids the need to dig in the guts of the CI system to find the latest tests.
Speaking of digging through guts, most of these systems allow an easy search of "all my open PRs". This means that to find out the branches you have been working on, for example when getting back to the office from the weekend, you can just open the handy-dandy link and immediately see the list. (This list is also useful as "what do I need to clean up because it has become irrelevant.")
For some teams, this requires a culture adjustment. People need to allow for the code's state in their review, if they review at all. Ideally, this encourages the team to have a serious conversation on when code is reviewed, by whom, and according to what criteria.
After this conversation happens, and assuming people open PRs early in the process, magic starts happening. Because now, when needing localized feedback (for example, "am I using this function correctly") you can link to the right place in the PR and ask for specific feedback.
This feedback, given constructively and helpfully, allows the entire team to learn. Working in public, at least among your team, is helpful to everyone.
So don't delay, post the PR, and start working better with your colleagues.
I recently came across an excellent tool called KeyCombiner that helps you practice keyboard shortcuts (3 sets for free, $29/6 months for more sets). I spent some time to create a set for Amazing Marvin, my current todo manager of choice.
The shareable URL to use in KeyCombiner is https://keycombiner.com/collecting/collections/shared/f1f78977-0920-4888-a86d-d00a7201502e
I generated it from the printed PDF version of Marvin's keyboard guide and a bunch of manual editing, in a google sheet.
Keyboard shortcuts are great timesavers and help reduce friction, but it's getting harder to learn them properly, and this tool has been a great help for some other apps, and for figuring out common shortcuts across apps, and for picking custom shortcuts (in other apps) that don't conflict. If this is a problem you recognize, give KeyCombiner a try.
It is possible to work with Python quite a bit and not be aware of some of the subtler details of package management. Since Python is a popular “glue” language, one of its core strengths is integrating with libraries written in other languages: from database drivers written in C, numerical algorithms written in Fortran, to cryptographic algorithms written in Rust. In all these cases, one way to avoid error-prone and frustrating installation errors in the target environment is to distribute pre-built code. However, while source code can be made portable, making the build output portable is a lot more complicated.
Note: This post focuses specifically about binary wheels on Linux. Binary wheels exist for other platforms, but those are beyond the current scope.
The Python
manylinux
project,
composed of three PEPs,
two software repositories,
and support in pip,
addresses how to accomplish that.
These problems are hard,
and few other ecosystems solve them as well as Python.
The solution has many moving parts,
developed over the course of ten years.
Unfortunately,
this means that understanding all of those is not easy.
While this post cannot make it easy, it can at least make it easier, by making sure all the details are in one place.
Python packages come in two main forms:
Wheels are "pre-built" packages that are easier and faster to install. The name comes originally from a bad joke: the Monty Python Cheese Shop sketch, since PyPI used to be called "Cheese Shop" and cheese is sometimes sold in wheels. The name has been retconned for another bad joke, as a reference to the phrase "reinventing the wheel", allowing Python packaging talks to make cheap puns. For the kind of people who give packaging talks, or write explainers about packaging formats, these cheap jokes fill the void in what would otherwise be their soul.
Even for packages that include no native code, only pure Python, wheels have some advantages. They do not execute any potentially-fragile code on installation, and querying their dependencies can be done without a Python interpreter.
However, when packages do include native code the story is more complicated.
Let's start with the relatively straightforward part:
portable binary wheels for Linux are called
manylinux
,
not
alllinux
.
This is because it relies on the GNU C library,
and specific features of it.
There is another popular libc for Linux:
musl.
There is absolutely no attempt to be compatible with
musl-based Linux distributions[#],
the most famous among them is Alpine Linux.
[#] For now. See PEP-656
However, most other distributions derive from either Debian (for example, Ubuntu) or from Fedora (CentOS, RHEL, and more). Those all use the GNU C library.
GNU libc has an official
"infinite backwards compatibility"
policy:
libc6
version
X.Y
is compatible with
W.Z
if
X>=W
or
X=W
and
Y>=Z
.
Aside: the 6 in libc6 does not refer to the version of the GNU C Library: Linux only moved to adopt the GNU C Library in libc6. The libc4 library was written from scratch, while libc5 combined code from GNU C Library version 1 and some bits from BSD C library. In libc6, Linux moved to rely on GNU C Library version 2.x, first released in January 1997. The GNU C Library is still, over twenty years later, on major version 2. We will ignore some nuances, and just treat all GNU C Library versions as 2.X.
The infinite compatibility policy means that binaries built against libc6 version 2.17, for example, are compatible with libc6 version 2.32.
The relevant PEP is dense but worth reading. "Portable" is a loaded word, and unpacking it is important. The specific meaning of "portable" is encoded in the auditwheel policy file. This file concedes the main point: portability is a spectrum.
When the manylinux project started, in 2016, the oldest security-supported open source distribution was CentOS: specifically, CentOS 5.11. It was released in 2014. However, because CentOS tracks RHEL, and RHEL is conservative, the GNU C library (glibc, from now on) it used was 2.5: a version released in 2006.
Even then,
it was clear that the
"minimum"
compatibility level will be a moving target.
Because of that,
that compatibility level was named
manylinux1
.
In 2018,
the manylinux project moved to a more transparent naming scheme:
the date in which the relevant compatible CentOS release was first released.
Thus,
instead of
manylinux1
,
the next compatibility target
(defined in 2018)
was called
manylinux2010
,
referencing CentOS 6.
In April 2019,
manylinux2014
was defined as a compatibility tag,
referencing CentOS 7.
In the beginning of 2021, Red Hat, in a controversial move, changed the way CentOS works, effectively nullifying the value any future releases have as a way of specifying a minimum glibc version support.
The Python community decided to switch to a new scheme:
directly naming the version of glibc supported.
The first such tag,
manylinux_2_24
,
was added in November 2020.
The next release of
auditwheel
,
4.0,
moves all releases to glibc-based tags,
while keeping the original names as
"aliases".
It also adds a compatibility level
manylinux_2_27
.
The compatibility level of a manylinux wheel is defined by the glibc symbols it links against. However, this is not the only compatibility manylinux wheels care about: this just puts them on a serial line from "most compatible" to "least compatible".
Each compatibility level also includes A list of allowed libraries to dynamically link against. Specific symbol versions and ABI flags that depend on both glibc and gcc.
However,
many Python extensions include native code precisely because they need to link
against a C library.
As a concrete example,
the
mysqlclient
wheel would not compile if the
libmysql
headers are not installed,
and would not run if the
libmysql
shared library
(of a version that matches the one the package was compiled against)
is not installed.
It would seem that portable binary wheels are only of limited utility if they
do not support the main use case.
However, the
auditwheel
tool includes one more twist:
patching ELF.
Elves predate Tolkien's Middle-Earth. They appear in many Germanic and Nordic mythologies: sometimes as do-gooders, sometimes as evil-doers, but always associated with having powerful magic.
Our context is no less magical, but more modern. ELF ("Executable and Loader Format") is the format of executable and shared libraries in Linux, since libc5 (before that, Linux used the so-called a.out format).
When auditwheel is asked to repair a wheel for a specific platform version,
it checks for any shared libraries it links against that are not part of the
pre-approved list.
If it finds any,
it patches them directly into the module.
This means that post repair
,
the new ("repaired") wheel will not depend on any libraries outside the
approved list.
These repaired binary wheels will include the requested manylinux tag and the patched modules. They can be uploaded to PyPI or other Python packaging repositories (such as DevPI).
For pip to install the correct wheels it needs to be up-to-date in order to self-check the OS and decide which manylinux tags are compatible.
Because wheels tagged as
linux_<cpu architecture>
(for example,
linux_x86_64
)
cannot be assumed on any platform other than the one they have been compiled for,
PyPI rejects those.
In order to upload a binary wheel for Linux to PyPI,
it has to be tagged with a manylinux tag.
It is possible to upload multiple manylinux wheels for a single package,
each with a different compatibility target.
When installing packages, pip will prefer to use a wheel, if available, instead of a source distribution. When pip checks the availability of a wheel, it will introspect the platform it is running it, and map it to the list of compatible manylinux distributions. Since the list is changing, it is possible that a newer pip will recognize more compatibilities than an older pip.
Once pip finds the list of manylinux tags compatible with its platform,
it will install the least-compatible wheel that is still compatible with the
platform:
for example,
it will prefer
manylinux2014
to
manylinux2010
if both are compatible.
If there are no binary wheels available,
pip will fall back to installing from a source distribution
(sdist
).
As mentioned before,
installing from
sdist
,
at the very least,
requires a functional compiler and Python header files.
It might also have specific build-time dependencies, depending on the package.
Thanks to SurveyMonkey for encouraging me to do the research this post is based on.
Thanks to Nathaniel J. Smith and Glyph for their feedback on this blog, post-publication. I have added some corrections and clarifications based on their feedback.
All mistakes that remain are my responsibility.