Planet Twisted

Your project deadline is getting closer and closer, and you’re stuck: you don’t know what to do. Your manager won’t help, they just push you to work evenings and weekends–and when it looks like you’re going to fail, they hand the project over to another programmer.

You’ve failed your manager, and there’s a little voice in the back of your head telling you that maybe you’re missing what it takes to be a programmer, maybe you just don’t have the requisite natural ability.

That little voice is lying to you.

It’s the other way around: your manager has failed you, and is compounding the failure by conveying a destructive mindset, what’s known as a fixed mindset. To understand what I’m talking about, let’s take a quick detour into the psychology of education, and then return to those long hours you’ve been working.

Growth mindset vs fixed mindset

Carol Dweck, a professor of psychology at Stanford University, contrasts two mindsets when learning:

• If you have a growth mindset you assume your abilities can change over time, and that your skills can improve by learning and practice.
• If you have a fixed mindset you assume your abilities are fixed: some people are naturally more talented than others, and there is no way to change your level of abilities.

According to Dweck’s research, students with a growth mindset do better than students with a fixed mindset. A fixed mindset is self-defeating: it keeps you from learning, and it keeps you from even trying to learn.

In the context of programming this suggests that starting with the attitude that programming is a set of skills, skills that you can learn, will result in a much better learning experience.

But what does this have to do with long working hours?

You can’t work smarter, so you gotta work longer

In an article that was one of the inspirations for this post, Carol Dweck points out that a common failure mode among educators is to praise effort, working harder, instead of praising learning. While they may claim to be encouraging a growth mindset, they are simply perpetuating a fixed mindset.

This failure mode appears in the software world as well. Let’s assume for the moment that programming is a natural ability: just before we’re born, the Angel of Software injects between 0 and 100 milliliters of Programming Juice into our soul. If you’re really lucky, you might even get 110ml!

Now, given that each and every one of us only has a limited amount of Programming Juice, how can you maximize our output? You can’t learn more, so there’s no way to do things more efficiently. You can’t improve your skills, so there’s no way to become more productive.

So what’s left?

All together now: WORK LONGER HOURS!

Working longer ain’t smart

The truth, of course, is that there is no Angel of Software, there is no Programming Juice. Programming is just a bunch of skills. You can learn those skills, and so can most anyone else, given motivation, time, support, and some good teachers. And you can become more and more productive by continuing to learn.

If you believe in fixed talent, if you believe you can’t improve, you won’t try to learn. Long hours will be the only tool left to you.

When faced with a problem: just work longer hours.

When faced with another problem: work even longer.

You’ll work and work and work, and you’ll produce far less than you would have if you’d spent all that time improving your skills. And eventually you’ll hit a problem you can’t solve, and that you will never solve by working longer hours.

A growth mindset will serve you far better. You need to believe that skills can grow, and then you need to actually do the work to learn more and grow your skills.

And when you fail–and you will fail, because we all fail on occasion–take this as another opportunity to learn: look for the patterns and cues you should have have spotted. Having learned your lesson, next time you’ll do better.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

(Thanks to Glyph Lefkowitz for some of the inspiration for this port, and to Mahmoud Hashemi for helpful comments and suggestions. All mistakes and issues that remain are mine alone.)

The Python REPL has always been touted as one of Python's greatest strengths. With Jupyter, Jupyter Lab in its latest incarnation, the REPL has been lifted into the 21st century. It has become the IDE of the future: interactive, great history and logging -- and best of all, you can use it right from your browser, regardless of your platform!

However, we still have 20th century practices for developing web applications. Indeed, the only development is that instead of "CTRL-c, up-arrow, return", we now have "development servers" which are not "production ready" support auto-reloading -- the equivalent of a robot doing "CTRL-c, up-arrow, return".

Using the REPL to develop web applications is pure bliss. Just like using it to develop more linear code: we write a function, test it ad-hocly, see the results, and tweak.

When we are sufficiently pleased, we can then edit the resulting notebook file into a Python module -- which we import from the next version of the notebook, in order to continue the incremental development. Is such a thing possible?

Let's start by initializing Twisted, since this has to happen early on.

from tornado.platform.twisted import install
reactor = install()

Whew! Can't forget that! Now with this out of the way, let's do the most boring part of every Python program: the imports.

from twisted.web import server
from twisted.internet import endpoints, defer
import klein
import treq

Now, let's start Klein app. There are several steps involved here.

root = klein.app.resource()

We take the Klein resource object...

site = server.Site(root)

...make a wrapping site...

ep = endpoints.serverFromString(reactor, "tcp:8000")

...create an endpoint...

ep.listen(site)

<Deferred at 0x7f54c5702080 current result: <<class 'twisted.internet.tcp.Port'> of <class 'twisted.web.server.Site'> on 8000>>

and like "Harry met Sally", eventually bring the two together for Klein to respond on port 8000. We have not written any application code, so Klein is currently "empty".

What does that mean?

async def test_slash():
    response = await treq.get('http://localhost:8000')
    content = await response.content()
    return content

This function uses Python 3's async/await features, to use treq (Twisted Requests) and return the result. We can use it as our ad-hoc debugger (but we could also use a web browser -- this is naturally hard to show in a post, though).

defer.ensureDeferred(test_slash()).addBoth(print)

<Deferred at 0x7f54c5532630>

b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">n<title>404 Not Found</title>n<h1>Not Found</h1>n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>n'

Ah, yeah. Even / gives a 404 error -- we have literally defined nothing. OK, this is easy to fix:

@klein.route("/")
def something_useful(request):
    return 'Hello world'

Wait, did this work?

defer.ensureDeferred(test_slash()).addBoth(print)

<Deferred at 0x7f54c53d8d30>

b'Hello world'

Yep. But it's not a proper sentence...woops.

@klein.route("/")
def something_useful(request):
    return 'Hello, world!'

Nice. Punctuation. Force. Determination. Other nouns.

Did it change anything?

defer.ensureDeferred(test_slash()).addBoth(print)

<Deferred at 0x7f54c4b9e240>

b'Hello, world!'

Yes.

Incremental web application development. Without an "auto-loading" "not production grade" server.

We took advantage of several incidental issues. The Jupyter kernel is Tornado based. Twisted has both a production-grade web development framework, and the ability to run on top of the tornado loop. The combination is powerful.

Do you think of yourself as a Python programmer, or a Ruby programmer? Are you a front-end programmer, a back-end programmer? Emacs, vim, Sublime, or Visual Studio? Linux or macOS?

If you think of yourself as a Python programmer, if you identify yourself as an Emacs user, if you know you’re better than those vim-loving Ruby programmers: you’re doing yourself a disservice. You’re a worse programmer for it, and you’re harming your career.

Why? Because you are not your tools, and your tools shouldn’t define your skillset.

Your tools are not your skills

Ask a programmer to list their skills and more often than not you’ll get a list of technologies. But technologies are just a small set of the skills you need as a programmer.

You need to gather requirements.

You need to debug code.

You need to design user experiences.

You need to build abstractions.

You need to avoid security problems.

And so on and so forth.

None of these skills are technologies. And if you think your only skills are technologies you won’t notice the skills you don’t have. And if you don’t know what you’re missing, you won’t take the time to learn the skills that can make you truly productive and valuable.

Your tools are not your job

If you define yourself by your tools, you are limiting yourself to what jobs you can get.

First, because you won’t apply to those jobs.

Second, because you will market yourself based on tools, instead of all the other skills that might get you that job anyway.

(I’ve written elsewhere about how you can get a job with technologies you don’t know).

Your tools are not you

If your identity is tied up with your tools, you won’t listen to people who use different technologies. Some tools are better than others at certain tasks. Some tools are interchangeable. But an expert using a bad tool can often do more than a novice with a bad tool.

Spending your time fighting over which tool is better is a waste of your time. Instead, spend your time listening and learning from everyone, whatever tools they use: most skills will transfer just fine.

The technologies you use, the tools you build with, are just that: tools. Learn to use them, and learn to use them well. But always remember that those tools are there to serve you, you are not there to serve your tools.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

(Thanks to Paul Ganssle for his suggestions and improvements. All mistakes that remain are mine.)

When exposing a Python program as a command-line application, there are several ways to get the Python code to run. The oldest way, and the one people usually learn in tutorials, is to run python some_file.py.

If the file is intended to be usable as both a module and as a command-line parameter, it will often have

if __name__ == '__main__':
    actually_run_main()

or similar code.

This sometimes has surprising corner-case behavior, but even worse -- some_file.py is not looked for in either $PATH or sys.path, it must be explicitly handed. It also changes the default Python path from including the current directory, to including the location of some_file.py.

The new recommended modern way, of course, is to set entry_points in the setup.py file. When the distribution is installed, a console script is auto-generated and added to the same place the Python interpreter is found. This means that we need to think carefully about the other things that might have the same name on our $PATH to avoid collisions.

There is a third way, which is subtle. When Python sees the -m <some name> option, it will look for a module or a package by that name. If it finds a module, it will run it with __name__ being "__main__" in order to trigger the path that actually does something -- again leading to some, if not all, issues discussed earlier.

However if it finds a package it will run its __main__.py module (still setting __name__ to "__main__") -- not its __init__.py.

This means that at the top of __main__.py we can invert the usual logic:

if __name__ != '__main__':
    raise ImportError("Module intended to be main, not imported",
                      __name__)

from . import main_function
main_function()

This allows running python -m <some package>, but anyone who tried to accidentally import <some package>.__main__ will get an error -- as well they should!

Among other things, this means we only care about our sys.path, not our $PATH. For example, this will work the same whether the package is installed to the global Python or --user installed.

Finally, if an entrypoint is desired, one can easily be made to run __main__:

entrypoint = toolz.compose(lambda _dummy: None,
    functools.partial(runpy.run_module,
                      "<some package>",
                      run_name='__main__'))

Using the builtin module runpy, the builtin module functools and the third party module toolz.

In today's edition of Pi day post, we will imagine we have a pie. (If you lack imagination, go out and get a pie.) (Even if you do not lack imagination, go out and get a pie.)

As is traditional, we got a round pie. Since pies are important, we will base our unit of measure on this pie -- the diameter of the pie will be 1.

Since we had to carry it home, we put it in a box. We are all ecologically minded, of course, so we put it in a box which is square -- with its length size 1.

We note something interesting -- the box's bottom's area is 1x1 -- or 1. The radius of the pie is 1/2, so its area is pi * 0.25.

As we are driving home, the pie on our passenger seat, we start wondering how we can estimate Pi. Luckily, we got some sugar. What if we sprinkled some sugar, and took notes for each grain, whether it was on the pie, or not?

Let's use Python to simulate:

import random
import attr

First, we need some randomness generator. Then, we also want to use the attrs library, because it makes everything more fun.

We make a Point class. Other than the basics, we give it a class method -- a named constructor which will generate a random point on the unit square -- this is where our sugar grain falls.

We also give it a way to calculate distance from another point, using the Pythagorean theorem.

@attr.s(frozen=True)
class Point:
    x = attr.ib()
    y = attr.ib()

    def distance(self, pt):
        return ((self.x - pt.x) ** 2 + (self.y - pt.y) ** 2) ** 0.5

    @classmethod
    def unit_square_random(cls):
        return cls(x=random.random(), y=random.random())

The center of the pie is at 0.5 by 0.5.

center = Point(0.5, 0.5)

A point is inside the pie if it is less than 0.5 away from the center.

def is_in_circle(pt):
    return center.distance(pt) < 0.5

Now we are ready. Even with just 100,000 grains of sugar, we get 2-digit accuracy.

inside = total = 0
for _ in range(10 ** 5):
    total += 1
    inside += int(is_in_circle(Point.unit_square_random()))
print((inside / total) * 4)

3.14052

Every day you go to work and you have to fight. Your boss wants to deliver features as fast as possible, and so you have to either argue for best practice, or work extra hours to do things the right way.

You need to fight for readable and maintainable code. And you need to fight for reusable code. And you need to fight for tests.

And you fight and you fight and you fight, and you keep on losing.

It doesn’t have to be this way. Software development doesn’t have to be a fight. Strange as it may seem, in this game the only winning move is not to play.

Instead of playing “Let’s Write High-Quality Software”, there’s a different and better game you can play: “Let’s Solve The Problem”.

Avoid false dichotomies

The problem with the game of High-Quality Software is that it’s based on a dichotomy, with only two options: low-quality and high-quality. I started this article with another dichotomy: arguing with your boss vs. working longer. And a lot of technical discussions quickly devolves into dichotomies, e.g.:

• Shipping features vs. fixing technical debt.
• Testing your code vs. coding quickly.

Dichotomies are tempting, and perhaps even built-in to the way we understand the world: there’s a left hand and a right hand, a wrong way and a right way. But there’s nothing inherently superior in just having two choices. In fact, all it’s doing is making you a worse engineer, because you’re focusing on arguing instead of focusing on solving the problem.

To combat this tendency, Gerald Weinberg suggests the Rule of Three: always consider at least three solutions to any problem. Let’s start with our first dichotomy: arguing with your boss vs. working longer hours to do things the right way. If there’s a third choice, what might it be?

Stop arguing, start listening

When your boss or colleagues argue for a specific design, instead of telling them why they’re wrong, listen to their reasons. Behind every design proposal is a set of goals, motivations, presumed constraints: those are the things you need to address. If you criticize their proposal based on goals they don’t care about, you’re not going to get anywhere.

So first, listen.

Once you understand why they want what they want:

1. Consider more than the initial two choices, their proposal and your initial reaction.
2. Try to find a solution that addresses both your goals.
3. Explain you thinking in ways that are relevant to their goals, not just yours.

Example scenario: testing

Your boss proposes writing code without unit tests. Why? Because they want customers to be happy. Since customers have been complaining about how long it takes for new features to be delivered, your boss believes this is one way to speed up delivery.

You want to write unit tests. Why? You want code to be maintainable over time.

Merely arguing for unit tests isn’t going to address your boss’ concerns. But if you look past the initial false dichotomy, there are other solutions to consider. For example:

• Investigate why customers aren’t getting features quickly. Perhaps the bottleneck isn’t due to how fast you’re coding, but elsewhere in the delivery process (you don’t do frequent releases, customers need to upgrade manually… there could be many reasons). Fix that, and you will have time to write unit tests and ship quickly.
• Figure out places where bugs would be costly to customers, explain those costs to your boss, and propose unit testing only that part of the code.
• Investigate customer needs in more detail. Perhaps existing customers are complaining about feature delivery, but you’re also losing many customers due to bugs.
• Suggest using tools that will speed up test writing enough that the additional time won’t bother your boss.
• Suggest limiting unit test writing to a predetermined amount of time: “this will only add 4 hours to a one week project”.

No doubt there are many more potential solutions to the standoff.

There’s always another solution

There is almost never a single correct solution to any problem, nor a single best solution. You can solve problems by relaxing unnecessary constraints, by focusing on different levels of the situation (organization, process, code), by redefining the problem, and more. Worst comes to worst, you can address many problems by switching jobs; different people like different environments, after all.

So don’t try to win technical arguments, and in fact, don’t treat them as arguments at all. When you disagree with someone, take the time to listen, and then try to come up with a solution that address their concerns and yours. And if your first idea doesn’t do that… it’s time to come up with a third, and fourth, and fifth solution.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

At the end of the day you’re done with work, you go home—and you don’t spend any of your free time coding. And that’s fine, you have other things going on in your life. But your coworker does spend another 20 hours a week coding, and all that practice means they’ll end up better programmers than you, and so they’ll get promoted faster, and they’ll get paid more. And that’s not fine.

It’s also not true.

That inadequacy you’re feeling is based on false assumptions, a misunderstanding of the skills it takes to be a good programmer. Enthusiasts who love coding as a hobby are valuable, yes, but so are you: even if extra hours of coding result in better skills—often a very doubtful assumption—their approach is not inherently better.

All but the smallest of programming projects are team efforts, requiring a variety of skills, knowledge, and mindsets. The mindset of someone who view programming as a tool to do their jobs—let’s call them pragmatists—is just as useful as the mindset of enthusiasts.

To understand why, let’s go through the lifecycle of the enthusiast programmer, see how their skills evolve, and compare them to the pragmatists’ approach. As you’ll see, enthusiasts and pragmatists start out with different strengths, and software teams can benefit from both.

The enthusiast’s path

The enthusiast’s starting point is a love of programming: they love to code. They take joy in writing software, and understanding the intricacies of the technologies they use.

Over time, as they become more skilled and experienced, they expand their point of view. Instead of just wanting to write code, they want to write good code, high-quality code: they want to be proud of their work. So now they focus not just on the joy of coding, but also on less enjoyable but important disciplines like testing.

If they continue to grow in skill and experience, they will eventually hit a point where they see code as a tool: a means to an end. Software is still fun, yes, but it also needs to be written to achieve certain goals. At this point quality becomes less of an obvious objective criteria: every situation is different, and what works in one situation doesn’t work in another.

Strengths and weaknesses: Enthusiasts often have a strong grasp of the tools and technologies they use. On the other hand, they are sometimes prone to holy wars and zealotry (they feel it’s important to choose the very best tool), and until they finally realize software is a means, not an end, they have a harder time working towards goals that aren’t about writing software.

The pragmatist’s path

The enthusiast’s weaknesses are the pragmatist’s strength. Pragmatists start out seeing software as a means to end: they will tell you “I like writing software, but mostly I care about what the software lets me do.” So what can take the enthusiast years to learn—a focus on goals—comes naturally to the pragmatist.

Of course, if they’re not deliberately focusing on learning, pragmatist can sometimes suffer from less understanding of tools and techniques. This can be mitigated by learning on the job, and making sure to build awareness of available technologies.

Software is not a competition

With time and practice, the strengths of enthusiasts and pragmatists can converge. Even so, different people will typically end up with different skillsets, different approaches, and different ways of building software.

This is how it should be. Software projects are too big to be built by one person, too complex to be encompassed by one view point.

When you compare yourself to your peers, you shouldn’t ignore your weakness, but you should also value your own strengths. You can and should learn from your colleagues, but chances are there’s something you can teach them.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

I skipped work today because I’m sick: nothing terrible, nothing life threatening, I’m just—exhausted. I definitely can’t focus on code, I can’t even focus on television; so far I’ve been re-reading a favorite old novel, and looking at Twitter when I get distracted.

The world will not come to an end because I’m not working. My company won’t collapse because I took the day off. My project will not fail.

I’m sick and I’m taking the day off, and that’s OK. And if you can’t take a day off—whether because you’re sick, or just because you feel like it—then something is very very wrong.

And now… I think I’m going to lie down on the sofa.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

Working long hours as a programmer can be both comforting and comfortable. After all, you have an easy solution to every problem: a few more hours at your desk, a few more hours staring at a screen in the dark. And working long hours is a natural attitude to take, since at the root of it, working long hours comes from valuing work, your craft as a programmer, and your obligations as a professional.

There is a different approach you can take. Instead of valuing work, you can value outcomes: outcomes at your job, outcomes in your life, outcomes in the world. Valuing outcomes is not comfortable. It forces you to reconsider and doubt your every step, and if you fail—and we all fail occasionally—your hard work becomes worthless.

But while valuing outcomes is uncomfortable and difficult, if you do it long enough you will find yourself—every once in the while—with the ability to grasp the moment and change the world. Just a little, in whatever corner you live in, but change it nonetheless.

The most direct way to learn this attitude is to reduce your working hours. If you had to do your work in 32 hours a week, instead of 45, or 50, or 60—what would need to change?

The experience of working fewer hours

Testing and quality

If you work long hours, you can pick some level of quality and stick to it:

• If you prioritize shipping features as quickly as possible, you can simply work longer to compensate for the buggy features you’re shipping.
• If you believe in quality above all else, you can spend the extra time to reach 100% test coverage of every little piece of code.

But if you work fewer hours, you will need to make decisions and tradeoffs, because you won’t have those extra hours. Some code will need automated tests for stability and maintainability, some code will need manual tests every time you change it, some code will be used once and thrown away. And you will need to make that choice, every single time. Which is to say, you will be forced to think about why you’re writing this code, and what it will be used for.

Planning

If you work long hours, the direction you go in matters less: if it’s the wrong direction, you can just work a few more hours when you eventually find out after you finish your task.

But if you work fewer hours, you will find yourself constantly plagued by doubt: is this the right direction? Are you doing the right thing? You may end up changing direction half-way, abandoning your work—however much it pains you—when you realize there’s a better way to your destination. You will have to spend much more time thinking and planning up front, to make sure you arrive at the right place on time.

Speaking of time, deadlines cause less worry when you work long hours. If you hit your deadline, great. If you didn’t, well, you worked long and hard, and who can blame you if you failed?

If you work fewer hours, deadlines are a constant gnawing presence, always getting closer. You will need to plan for them well in advance, always asking questions, on the lookout for unstated requirements and forgotten deliverables. And sometimes you’ll be forced to find new solutions so you can hit those deadlines in time—and, of course, sometimes you will fail to find those solutions. And then you’ll look bad, because you didn’t work on the weekend, so whose fault is it the deadline was missed?

Your value

When you work long hours, it’s easier to demonstrate how valuable you are to your boss: you were there on the weekend, after all. If you work fewer hours, well, who can say you’re really getting much done? Did you deliver anything of value? How will you prove it?

Even worse than your boss is your own self-image. When you work long hours, your self-worth is about your work: you work hard, and that at least is something to be proud of, even if the project failed. If you work fewer hours you won’t be able to feel proud of all the hours you worked. You’ll be forced to consider the results, the outcomes: if you failed to deliver, you failed.

From comfort to effectiveness

Go through all this and— you’ll become effective.

• You will think more, and find the simple solution that takes a day, where previously you would just start coding—and spend a week coding, at that.
• You will learn to build what is needed, where previously you would build what you knew.
• You will learn to prioritize, and prioritize well.

Valuing outcomes is not comfortable, but it is empowering. And good day or bad, you get to go home at a reasonable hour and do whatever you feel like. Which feels pretty damn good too.

You don’t have to take my word for it, either: it’s easy to try out. If you’re working long hours, try spending a month working 40 hours a week, no more. Once you can no longer rely on working longer hours to solve all your problems, you’ll soon find yourself approaching your work very differently.

And if you want a head start on learning these skills, read my book, The Programmer’s Guide to a Sane Workweek.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

I have written before about Python tooling. However, as all software, things have changed -- and I wanted to write a new post, with my current understanding of best practices.

There is so much to learn as a programmer it’s hard to even know where to begin. And even if you do begin, there’s always something newer and shinier to distract you, and so you end up making these giant lists of things to learn and they get longer and longer and longer and you feel like shit about it but really you don’t have the time to learn all of it, now do you?

So, why not take a few minutes to stop worrying about the things you should be learning, and instead let’s talk a little about all the things you don’t need to learn. And maybe when we’re done you’ll feel a little better.

But first, let’s talk about—

DEATH.

We’re all gonna die (eventually)

Someday you’re going to die. So will I. So will everyone else, our friends and enemies both.

Typically one measures one’s lifespan in years or days, but I once went and measured it in books. I did the math and figured out how many books I could read before I reached my presumptive death from old age. At the time I was reading about two books a week, but even so the number didn’t seem anywhere near sufficient.

And having started down this morbid path, I arrived at an even worse place. Every time I read a book I’d get to thinking: “Is this book worth reading before I die? Shouldn’t I be reading something more edifying than this entertaining yet trashy novel? Isn’t re-reading a book a complete waste of my time?” Instead of enjoying the book I was reading, I was worrying about some other better book I could have been reading instead.

Eventually I got over it. I won’t be able to read every book I want to before I die. Neither will you.

Sorry.

It’s not so bad, though. When you’re lying there dying you’ll probably be thinking about the friends and family you’ll miss. Or maybe you’ll just be tired, and looking forward to an end to your pain and sorrow. Or, if you’re having a really bad day, you’ll be thinking that you shouldn’t have had that last drink before driving home—don’t drink and drive, kids. Better yet, don’t drive at all; commuting is a terrible way to spend your life.

In any case, when it’s time for you to die you probably won’t be worrying about the books you haven’t read. And as you lie on your deathbed, looking back at your life, you definitely won’t be worrying about that new JavaScript framework you didn’t get to play with.

Some skills you don’t need to learn

There are many skills I do think you should learn (I have a whole pile of posts on this here website, in fact), but honestly—it’s just software. If you don’t learn these skills before you die, that’s really not a big deal.

Software is a tool: tools are useful, and important, and you need them to build many things. But our tools are there to serve us, we should not be serving our tools.

Not to mention all the skills you really don’t need to learn.

You don’t need to learn every blindingly shiny new technology that will End Poverty and Bring Peace to Humanity. It probably won’t do either, and quite possibly it won’t turn out to be good for much at all. I started my career programming building multimedia CD-ROMs, which were really hot for about 6 months in the mid ‘90s, and somewhere in a landfill there’s still boxes of old unusable CDs I worked on that no one cares about.

You don’t need to learn everything programming language, certainly not in your spare time. You can and should learn those on the job, as and when they become useful.

You don’t need to learn how to use every new library, tool, or framework. Just knowing they exist is usually more than enough: when and if you need them, you’ll know they exist and go learn them then.

Some things to do before you die that are more important than learning another programming skill

Spend more time with your friends.

Spend more time with your family (unless you don’t get along, sorry).

Eat good food.

Visit UNESCO World Heritage sites.

If you haven’t seen one, a full solar eclipse is amazing.

Make the world a better place, even if it’s just a little.

Whatever you think is important and worthwhile.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

So your manager stops by and says “hey, we’ve got some new employees starting in a month, and you know SomeTechnology best, I’d like you to get them up to speed.” And you nod and smile and feel the impostor syndrome kicking in: you don’t know how to do this.

Software is complicated. Sometimes even when you’ve been using a tool or a language for a year or two, you might only know a small part of the functionality. It’s already tough and intimidating to teach as it is, it’s even worse when you’re not confident in what you know. How can you teach when you’re not an expert?

But as it turns out, being an expert can impede your ability to teach. You can become a better teacher by taking advantage your lack of expertise.

The problem with being an expert

Experts can actually make worse teachers. Specifically, experts often suffer from “expert blind spot”, where their own knowledge blinds them to the way their students are thinking (I first learned of this concept from the book How Learning Works).

Experts differ from beginners in a number of ways that impact learning.

1. Experts have much denser and interconnected conceptual models. Things that are obviously related to them are not at all obvious to beginners, who have sparsely connected conceptual models.
2. Experts will skip steps, doing certain things so automatically they don’t even realize they’re doing them. From the beginners’ point of view this results in unexplained jumps.
3. Expertise is often unconscious. When asked, experts may not be able to explain why they’re doing what they’re doing, even if they know it’s the right thing.

As a result, being an intermediate learner, where you know how to do a task but still need to consciously walk through the steps, can actually make you a better teacher.

Teaching when you’re not an expert

Since you’re not an expert, you don’t need to pretend to be one. Instead you can teach the valuable knowledge you do have: the combination of the understanding of the technology, and the understanding or recent memory of how beginners think.

• Instead of demonstrating a task you know perfectly, you can demonstrate a task you don’t quite know how to do, e.g. by doing a live coding exercise. You can then show the process you go through for finding solutions—how you debug the problem, how and where you search for solutions and documentation. (Thanks to Noa Bendit-Shtull for the idea.)
• Similarly, you can cover common mistakes and how to recover from them. Where an expert might never make these mistakes, you might still make them occasionally, or at least have the recent memory of what those mistakes were.

By showing how to deal with mistakes, and by admitting mistakes to your students, you’re also making the topic less intimidating. Unlike some experts, who get confused and perhaps annoyed when mistakes happen—they never make those mistakes after all—you are giving your students the confidence that they too can work through problems. (Thanks to reader Jake for this point.)

Go teach!

Teaching doesn’t need to be scary. It can just be you sitting down at a computer and saying “this is tricky, but I’ve figured out some ways to get things done, and so can you. Let me show you how.”

This also applies to teaching more broadly. Every once in a while you’ll see a conference call for proposals, and you’ll say to yourself “I’m not an expert, I can’t teach anything.” But the truth is you can: your perspective as an intermediate learner is often much more valuable to beginners than that of an expert. Submit a talk proposal even if you’re not an expert: you too have a valuable perspective.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

Have you ever read an amazing-sounding job posting, and then felt that sinking feeling in your stomach when you reached the list of required technologies? Maybe you’re a Python programmer, and they use Ruby on Rails; maybe you’re mostly a front-end developer, and they want back-end skills. Whatever the missing technologies are, this is an awesome job you don’t have a chance of getting.

Or do you?

In fact, it is possible, albeit not always easy. Years ago I got a job writing C++, when all I knew was Python and Java. And for the past few months I’ve been doing scientific computing for the first time, with a new toolchain, and math skills I haven’t used in almost 20 years.

In this post I’ll cover:

• Why you should apply anyway.
• Which companies to apply to.
• How to pitch yourself so you can get the job.

Apply anyway

Even if you feel you’re not fully qualified for a job, you should still apply:

First, the list of technologies may be irrelevant. Sometimes the technologies listed are things the company might want to use someday, not what they actually use today. Sometimes they are perfectly happy hiring candidates with different technologies, and they put the list of technologies down because they aren’t very thoughtful about how they write job ads.

But what if they do actually want those technologies? The list of technologies and skills is what the company would like in the ideal world, not necessary what they can get in practice. In the ideal world many companies would probably also like to pay you half as much and have you be twice as experienced, with the ability to create gold out of lead and summon unicorns at will. In practice, companies hire the candidates they can get, not the magical and often non-existent candidates they dream of.

Finally, technologies aren’t everything. There are many other skills you have as a programmer, and some of them may trump the particular technologies you lack. We’ll revisit this in the section on pitching yourself.

Which companies to apply to

It’s hard to say as an outsider which companies will be more flexible, but here are some things to look for:

• Companies that use less popular technologies: they’ll have a harder time hiring, which means they are probably more willing to train people. I got that job writing C++ with a company whose main product was written in Common Lisp, which is not a common skill.
• Companies that explicitly talk about their commitment to training. For example, companies that talk about how they fund conference visits or classes.
• Companies where you have other relevant skills: perhaps you’re an expert in this business domain, or you have adjacent skills.
• Companies you’re really excited about. If you can convey that excitement in your cover letter, that may be enough to get you over that initial hump.

How to pitch yourself

Once you’ve picked the companies to apply to, you want to customize your cover letter, resume, and if you get there your presentation at the job interview. Here are some ways to do that:

• Talk about your ability to learn quickly, with concrete examples. If you can show that at your last job you learned a new technology stack in a couple months you will take away the fear of a long slow ramp-up to productivity.
• Stress adjacent skills. You’ll have hard time going straight from front-end development to a data science project. But, in practice much of data science is cleaning up dirty data, which is experience you might have in a different domain.
• Talk about skills beyond just technologies.
  Do they do consulting work? Stress your ability to work with non-engineers and gather requirements.
  Are they growing rapidly? Talk about the time you helped a team grow its engineering processes.
  Is failure costly? Stress your experience in testing and building robust software.

The key is to research the company, try to understand their needs, and then demonstrate you can have value to them beyond just the list of technologies you know.

Next time you’re looking for a job, don’t limit yourself only to jobs with a technology stack you know. Look for jobs that excite you, for jobs you think you can do with just a little ramp-up time. And then apply for those jobs, knowing that most of them will ignore you—but it only takes one yes to get an exciting new job, and an exciting new opportunity to learn new skills.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

Are you having a hard time getting responses to your resume? Do you know you have the skills for a job, but have a hard time convincing the company to hire you?

If your resume starts out with a long list of technologies you are selling yourself short. Knowing and using programming languages, libraries, and so on is a critical job skill for a software developer, but it’s only part of what makes a good programmer. And the skills you aren’t mentioning may well be keeping you from getting hired, even if you already have them.

In this post I’ll discuss:

• The most important part of your resume: the opening paragraphs.
• Why starting with technologies is a mistake.
• What to lead with instead.

The most important part of your resume

Recruiters spend very little time reading resumes, and more broadly people tend to skim when reading. That means the first thing a reader see when they read your resume will have the most impact. The lower on the page, the less likely it is to be reached.

You should imagine the hiring manager as looking for some hint or phrase that will make them think “aha, this sounds promising, I should keep reading.” And you should also imagine them giving up looking after less than 10 seconds.

That means you need put the most important pieces of information at the very top of your resume. The first sentence, the first paragraph, no lower. Your goal isn’t to convey everything, you just want to make yourself sound interesting and relevant enough that the recruiter keeps reading.

For example, let’s say you worked in bioinformatics two jobs ago, and you’re applying to a bioinformatics job again. Your opening paragraph should mention your biotechnology work; that way the reader will be motivated to keep reading. Having given that motivation, you can still do the normal reverse chronological job history in the second part of your resume—but perhaps the paragraph for your current job should be a little shorter if it’s less relevant.

Don’t lead with a list of technologies

Since you need to make yourself sound interesting in the first few seconds, a list of technologies is a bad way to start. Merely claiming to know a technology doesn’t tell the hiring manager how useful of an employee you’ll be.

In some jobs knowing particular technologies will give you a head start, which is why some recruiters foolishly use technology keywords as a filter, but it’s never really enough to do your job as a programmer. So spending precious moments of attention listing the technologies you know is a bad way to convey your value:

• Many of the technologies you know won’t be relevant to the company. If a company only uses Ruby, the fact you know Idris is unlikely to be relevant.
• Even if you’re listing a technology the company does use, mentioning it isn’t very compelling proof you can do anything with it. Can you read existing code and fix small bugs, or can you create a completely new project from scratch with it? Those require different levels of skills.
• Even if you’re the world’s foremost expert at a technology, that doesn’t mean you’ll be productive. Will your boss have to tell you exactly what to do, or will you be able to work independently?
• Most companies use many technologies, and add more technologies over time. If you know Python but refuse to learn other programming languages, you are a far less useful employee than someone who doesn’t know Python but is able and willing to quickly pick up whatever is needed.

What to do instead

Instead of opening with a list of technologies, open with a paragraph demonstrating the ways in which you’re a valuable employee, ideally tailored to the company you’re applying for. Specifically, you want to convince the hiring manager you can do the job they’re hiring for, which is either solving problems, or at more experienced levels identifying and solving problems. Writing software is merely a means to that end.

So you want a starting paragraph that emphasizes:

• Your ability to solve real, relevant projects with your skills.
• Your ability to work independently.
• Your ability to learn quickly.
• Your relevant domain knowledge, if any.
• Your ability to work with others.

And you want to include just enough real-world examples that you will convince them to keep reading. Once you’ve got them reading the main job-listing section of your resume you can go into much more detail.

You can still have a list of technologies if you want, but don’t waste the precious first half page of your resume on it. Personally I don’t bother: I just mention names of relevant technologies at the jobs where I used them.

If you’re currently looking for a job, go look at your resume, set a timer for 10 seconds and read it from the start. What will the recruiter learn about you? And is that all you want them to know? If all they’ve learned is that you know tool X or language Y, it’s time to make some changes.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

For some people, a 40-hour workweek is something to aspire to; for others, it’s still too much time taken up by a job. If you fall into that second category, if you want more time for hobbies, family and friends, or working on your own software projects, you too might dream of working less than full time.

But how do you get there? Almost no one advertises part-time programming jobs–believe me, I’ve me looked.

The answer: negotiation. I’ve negotiated a shorter workweek a few times myself, and I’ve met other programmers who have done so as well, some with just a few years of experience. And of all the programmers I’ve met who’ve negotiated part-time work, Mike’s record is the most impressive.

Mike has spent pretty much all his career working part-time: he’s been working part-time for more than 15 years. To help you get to a shorter, saner workweek, I sat down to interview Mike about how he does it.

❧

Q. What does a sane workweek mean to you?

MIKE: Well, I only ever worked full time for about 1 year, and I learned pretty quickly that a sane workweek for me was less than 40 hours a week. I guess it’s up to each individual, but I think most of us are forced to work more than we’d like, and at least for me 30-32 hours is better.

I want to work on average less, but I make it clear [when starting a job] that [I understand] things happen, stuff needs to get done. I would definitely work longer hours here and there.

Q. How did you realize you wanted to work less hours?

MIKE: Part of the reason I decided to demand this so early was, I started that first job when I was in university, and I stayed on for several years part-time while I was school. My contract was 10 hours a week, not very much but I was also going to classes and had to do homework and shit. I suppose that got me used to being able to go cycling or climbing or hiking on fairly short notice.

They were terrible at planning at this company, when a contract deadline was coming up everyone was working 60 to 70-hour weeks, having dinner at the office all the time. I wasn’t forced to get caught up in that, since I had my excuse of going to school. I worked a lot more than 10 hours I was contracted to, though.

I saw this cycle there a lot, where managers would pull numbers out of their ass and promise them to clients, and then all the programmers were panicking for the last month. The panic mode of getting this done. [My feeling is] I’m not committed to those numbers you pulled out of your ass to tell your client, so partly it was reaction to that, seeing these people spending their entire lives at the office. I wanted to carve out my time early on.

Q. So how did you end up part-time after that first year?

MIKE: At that I job I tried to quit.

But they basically offered me more money, so I was like, “what if you gave me 75% of that, and I worked less?” And I guess they wanted to keep me around and they went for that.

At that job I was on payroll as a full time employee, but I got a bunch more holidays. So I got a quarter of my time in holidays; 72 days off a year? I was constantly booking time off, which was interesting.

That was how I landed my first part time thing. And then I did eventually quit that company and worked elsewhere, but I had found a new job while I was still in a part time situation, so it was a lot easier to demand a similar deal going forward.

Q. How many companies have you done this with? How many part time jobs?

MIKE: A bunch. All of my jobs. 7, no, 8.

(Mike goes off to look at his resume)

If you count the two I have now, two part time contracts, then 9.

Q. How exactly did you negotiate for shorter hours at later jobs?

I have several interviews here and there. And sometimes—if it’s a company where I’m meh, not that serious about or interested in—I’ll send them an email early in the process saying “hey I want to work part time.” And then if they won’t go for it I won’t bother.

If I definitely wanted to work there, I’d go for the interview without telling them [I wanted a shorter workweek]. And if I got back for subsequent interviews or I got an offer I’d say “now I’m working 75%, I want to work part time somehow, I’m open to various different arrangements, I’m interested in working with you, lets work something out.” Mostly if the companies were in the job offer stage they’d consider it very seriously. One was “no, no way,” they didn’t want to talk to me anymore. Mostly they’re interested in negotiating somehow.

Q. What kinds of counter-offers did you get?

MIKE: Sometimes it was stuff like “I want to hire you, how do I sell this management? Help me sell it to management.” That job took me 6 months to get from interview to legitimate part time job offer. That was the longest, and also unbeknownst to that company I’d been laid off: I’d started negotiating when I had a job, and was subsequently laid off. They wanted me to sell it to management. I said, “I don’t want to work 5 days, how much happens Fridays? Not much? Then I’ll work Monday to Thursday.”

Usually I negotiated the full-time salary first, and then the details of pro-rating later: mostly it was about the details, once you’d convinced someone you were worth having. The easiest ones to negotiate are the ones which are 80% time, especially one day a week off. Usually I’d say Monday or Friday, and get a longer weekend, go to the mountains. But one of the people I inspired got himself an 80% contract, and he said “you’ve got to take Wednesday off.” With Wednesdays, you only ever work two days in a row. It was awesome.

I always set expectations: “you’re getting the best 4/5ths of my time, but only 4/5ths. I’m not going to pound out as much code.” Realistically you kinda do, but the expectation should be if you’re not there Wednesdays you shouldn’t be doing as much work. That’s part of the argument, that you’re doing almost as much useful work.

Q. Did you find asking for a shorter workweek got easier over time?

Yeah, for sure, it definitely got easier. After the first couple times I felt more confident asking in the first place. It got easier in that sense.

Q. When you tell people you don’t work full time, how do they react?

It’s still considered weird by a lot of my friends, but I also get the other reaction, “I wish I could do that.” Some people when you start new jobs, once you’d explained why you weren’t there Wednesday, they’d say “I totally want that” but then I’d say I only get paid 80%. Once they’d realize they’d get paid less, [they’d say] “I’m not doing that.”

Other people would say “cool, I want that.” Definitely people I’ve worked with over the years have been inspired to do that, which is pretty nice.

Q. Why was the lower salary not an issue for you?

MIKE: I don’t know, I guess I decided early on free time was more important than more money. Programmers are paid pretty well in my experience, 80% of a programmer job is plenty of money to get by on. Typically it’s a lot more than what a lot of people are getting.

For me it was putting a premium on my time, and wanting to have that time to do other things. I’ve got hobbies, always had hobbies, different sports over the years which takes time. But even just stuff like having time to work on your own programming project, which I think is really valuable. Valuable enough to me that I’ll actually pay money for it in the form of taking less salary.

I definitely don’t have any regrets for doing that.

Q. Any final advice?

MIKE: My biggest piece of advice is that you have to be willing to quit your job to get a sane workweek, or you won’t sound convincing when you ask for it. If you’re looking for a new job, just ask, and if you’re still at a job, then you’re in a better position to ask.

Just go for it!

❧

Itamar here: while I agree that working less than full time is wonderful, it also isn’t for everyone. Perhaps to you a sane workweek means working for yourself, or working remotely, or just getting a job where you can go home at 5PM.

But whatever a sane workweek means to you, the basics are the same. As in Mike’s case, you need the skills to be productive and valuable, you need to be able to negotiate, and you need enough financial security to have a strong negotiating position.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

I’ve written and spoken at some length about shipping software in the abstract. Sometimes I’ve even had the occasional concrete tidbit, but that advice wasn’t really complete.

In honor of Eevee’s delightful Games Made Quick???, I’d like to help you package your games even quicker than you made them.

Who is this for?

About ten years ago I made a prototype of a little PyGame thing which I wanted to share with a few friends. Building said prototype was quick and fun, and very different from the usual sort of work I do. But then, the project got just big enough that I started to wonder if it would be possible to share the result, and thus began the long winter of my discontent with packaging tools.

I might be the only one, but... I don’t think so. The history of PyWeek, for example, looks to be a history of games distributed as Github repositories, or, at best, apps which don’t launch. It seems like people who participate in game jams with Unity push a button and publish their games to Steam; people who participate in game jams with Python wander away once the build toolchain defeats them.

So: perhaps you’re also a Python programmer, and you’ve built something with PyGame, and you want to put it on your website so your friends can download it. Perhaps many or most of your friends and family are Mac users. Perhaps you tried to make a thing with py2app once, and got nothing but inscrutable tracebacks or corrupt app bundles for your trouble.

If so, read on and enjoy.

What changed?

If things didn’t work for me when I first tried to do this, what’s different now?

• the packaging ecosystem in general is far less buggy, and py2app’s dependencies, like setuptools, have become far more reliable as well. Many thanks to Donald Stufft and the whole PyPA for that.
• Binary wheels exist, and the community has been getting better and better at building self-contained wheels which include any necessary C libraries, relieving the burden on application authors to figure out gnarly C toolchain issues.
• The PyGame project now ships just such wheels for a variety of Python versions on Mac, Windows, and Linux, which removes a whole huge pile of complexity both in generally understanding the C toolchain and specifically understanding the SDL build process.
• py2app has been actively maintained and many bugs have been fixed - many thanks to Ronald Oussoren et. al. for that.
• I finally broke down and gave Apple a hundred dollars so I can produce an app that normal humans might actually be able to run.

There are still weird little corner cases you have to work around — hence this post – but mostly this is the story of how years of effort by the Python packaging community have resulted in tools that are pretty close to working out of the box now.

Step 0: Development Setup

Get a good Python. Use Homebrew, and brew install python3. If you need python 2, brew install python2. Don’t use the System python. Probably nothing will work.

You probably also want to use a virtualenv for development. This post is about how to build a for-real thing that other people can download, but part of the magic of Python is the interactive, real-time dynamic nature of everything. Running the full build pipeline every time you change a file or an asset is slow and annoying. However, there’s a weird thing where certain parts of the macOS GUI won’t work right (in PyGame’s case, mostly keyboard focus) unless your code appears to be in an application bundle.

I made this dumb little thing which lets you fake out enough of this that the OS won’t hassle you: you just need to pip install venvdotapp; venvdotapp inside the virtualenv where you’re making your pygame app.

Finally: pip install all your requirements into your virtualenv, including PyGame itself.

Step 1: Make an icon

All good apps need an icon, right?

When I was young, you just popped over into ResEdit Resorcerer MPW CodeWarrior Project Builder Icon Composer Xcode and created a new ICON resource cicn resource .tiff file .icns file. Nowadays there’s some weird opaque stuff with xcassets files and Contents.json and “Copy Bundle Resources” in the default Swift and Objective C project templates and honestly I can’t be bothered to keep track of what’s going on with this nonsense any more.

Luckily the OS ships with the macOS-specific “scriptable image processing system”, which can helpfully convert an icon for you. Make yourself a 512x512 PNG file in your favorite image editor (with an alpha channel!) that you want to use as your icon, then run it something like this:

1

$ sips -s format icns Icon.png --out Icon.icns

somewhere in your build process, to produce an icon in the appropriate format.

There’s also one additional wrinkle with PyGame: once you’ve launched the game, PyGame helpfully assigns the cute, but ugly, default PyGame icon to your running process. To avoid this, you’ll need these two lines somewhere in your initialization code, somewhere before pygame.display.init (or, for that matter, pygame.display.<anything>):

1
2

from pygame.sdlmain_osx import InstallNSApplication
InstallNSApplication()

Obviously this is pretty Mac-specific so you probably want this under some kind of platform-detection conditional, perhaps this one.

Step 2: Just Include All The Dang Files, I Don’t Care About Performance

Unfortunately py2app still tries really hard to jam all your code into a .zip file, which breaks the world in various hilarious ways. Your app will probably have some resources you want to load, as will PyGame itself.

Supposedly, packages=["your_package"] in your setup.py should address this, and it comes with a “pygame” recipe, but neither of these things worked for me. Instead, I convinced py2app to just splat out all the files by using the not-quite-public “recipe” plugin API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

import py2app.recipes
import py2app.build_app

from setuptools import find_packages, setup

pkgs = find_packages(".")

class recipe_plugin(object):
    @staticmethod
    def check(py2app_cmd, modulegraph):
        local_packages = pkgs[:]
        local_packages += ['pygame']
        return {
            "packages": local_packages,
        }

py2app.recipes.my_recipe = recipe_plugin

APP = ['my_main_file.py']
DATA_FILES = []
OPTIONS = {}
OPTIONS.update(
    iconfile="Icon.icns",
    plist=dict(CFBundleIdentifier='com.example.yourdomain.notmine')
)

setup(
    name="Your Game",
    app=APP,
    data_files=DATA_FILES,
    include_package_data=True,
    options={'py2app': OPTIONS},
    setup_requires=['py2app'],
    packages=pkgs,
    package_data={
        "": ["*.gal" , "*.gif" , "*.html" , "*.jar" , "*.js" , "*.mid" ,
             "*.png" , "*.py" , "*.pyc" , "*.sh" , "*.tmx" , "*.ttf" ,
             # "*.xcf"
        ]
    },
)

This is definitely somewhat less efficient than py2app’s default of stuffing the code into a single zip file, but, as a counterpoint to that: it actually works.

Step 3: Build it

Hopefully, at this point you can just do python setup.py py2app and get a shiny new app bundle in dist/$NAME.app. We haven’t had to go through the hell of quarantine just yet, so it should launch at this point. If it doesn’t, sorry :-(.

You can often debug more obvious fail-to-launch issues by running the executable in the command line, by running ./dist/$NAME.app/Contents/MacOS/$NAME. Although this will run in a slightly different environment than double clicking (it will have all your shell’s env vars, for example, so if your app needs an env var to work it might mysteriously work there) it will also print out any tracebacks to your terminal, where they’ll be slightly easier to find than in Console.app.

Once your app at least runs locally, it’s time to...

Step 4: Code sign it

All the tutorials that I’ve found on how to do this involve doing Xcode project goop where it’s not clear what’s happening underneath. But despite the fact that the introductory docs aren’t quite there, the underlying model for codesigning stuff is totally common across GUI and command-line cases. However, actually getting your cert requires Xcode, an apple ID, and a credit card.

After paying your hundred dollars, go into Xcode, go to Accounts, hit “+”, “Apple ID”, then log in. Then, in your shiny new account, go to “Manage Certificates”, hit the little “+”, and (assuming, like me, you want to put something up on your own website, and not submit to the Mac App Store), and choose Developer ID Application. You probably think you want “mac app distribution” because you are wanting to distribute a mac app! But you don’t.

Next, before you do anything else, make sure you have backups of your certificate and private key. You really don’t want to lose the private key associated with that cert.

Now quit Xcode; you’re done with the GUI.

You will need to know the identifier of your signing key though, which should be output from the command:

1

$ security find-identity -v -p codesigning | grep 'Developer ID' | sed -e 's/.*"\(.*\)"/\1/'

You probably want to put that in your build script, since you want to sign with the same identity every time. The command to do the signing is:

1

$ codesign -fs "${YOUR_DEVELOPER_ID_IDENTITY}" --deep "dist/${NAME}.app"

Step 5: Archive it

The right way to do this is probably to use dmgbuild or something like it, but what I promised here was quick and dirty, not beautiful and best practices.

You have to make an archive that preserves symbolic links. There are a few options for this:

• open dist/, then in the Finder window that comes up, right click on the app and “compress” it
• cd dist; tar czf $NAME.app.tgz $NAME.app
• cd dist; zip -yr $NAME.app.zip $NAME.app

Most importantly, if you use the zip command line tool, you must use the -y option. Without it, your downloadable app bundle will be somewhat mysteriously broken even though the one before you zipped it will be fine.

Step 6: Download it

Ideally, at this point, everything should be working. But to make sure that code-signing and archiving went correctly, you should have either a pristine virtual machine with no dev tools and no Python installed, or a non-programmer friend’s machine that can serve the same purpose. They probably need a relatively recent macOS - I know that apps made using the above technique will definitely work on High Sierra and will definitely break on Yosemite; they probably start working at some OS version between those.

There’s no tooling that I know of that can clearly tell you whether your mac app depends on some detail of your local machine. Even for your dependencies, there’s no auditwheel for macOS. So it’s always a good idea to check your final app build on a fresh computer before you announce it.

Coda

If you were expecting to get to the end and download my cool game, sorry to disappoint! It really is a half-broken prototype that is in no way ready for public consumption, and given my current load of personal and professional responsibilities, you definitely shouldn’t expect anything from me in this area any time soon, or, you know, ever.

But, from years of experience, I know that it’s nearly impossible to summon any motivation to work on small projects like this without the knowledge that the end result will be usable in some way, so I hope that this helps someone else set up their Python game-dev pipeline.

I’d really like to turn this into a 3-part series, with a part for Linux (perhaps using flatpak? is that a good thing?) and a part for Windows. However, given my aforementioned time constraints, I don’t think I’m going to have the time or energy to do that research, so if you’ve got the appropriate knowledge, I’d love to host a guest post on this blog, or even just a link to yours.

If this post helped you, if you have questions or corrections, or if you’d like to write the Linux or Windows version of this post, let me know.

Jupyter is a tool that came out of the data science community. In science, being able to replicate experiments is of the utmost importance -- so a tool where you can "show your work" is helpful. However, being able to show your work -- have colleagues validate what you have done, repeat it if needs be, and learn new techniques -- is also useful in the world of Site Reliability Engineering and DevOps.

The Jupyter console allows us to experiment (carefullly!) with APIs, running one function at a time, and validating the results. It allows building the needed automation from simple atoms, all the while learning how to use them.

The Jupyter Python kernel is popular in the data science community because so many data science libraries are available for Python. Luckily, the same is true in the SRE/DevOps community.

import github3
import os

The GitHub API has several client libraries in Python. My personal favorite is github3 -- I find its interface allowing remarkably idiomatic Python.

fname=os.path.expanduser("~/.git-token")
with open(fname) as fin:
    token = fin.read().strip()
gh = github3.login(token=token)

I have prepared a GitHub authentication token in a file. This allows the NoteBook to be published widely, without leaking access credentials. Never put usernames and passwords in Jupyter notebooks!

repositories = list(gh.organization('twisted').iter_repos())
repositories[:3], len(repositories)

([<Repository [twisted/txmongo]>,
  <Repository [twisted/twisted]>,
  <Repository [twisted/txaws]>],
 26)

This is a list of the repositories in the Twisted GitHub organization. It is nice to be able to validate we got a reasonable value by checking the first three. In previous versions of the notebook, my usage of github3 had an error -- and this was a list of all repositories I had access to, including private ones! Being able to inspect values interactively meant I could fix the bug easily.

repositories[1].branch('trunk').commit.sha

'51d23b6cb004e60ed699b3b100b4c476f5aebc4e'

As an example of why this might be useful, we are checking the commit hash of the trunk branch. This can be used in validating which version we are running somewhere, or checking if there have been new commits.

The GitHub API is big, and this is not meant to be an exhaustive guide to it. However, this approach is powerful -- it can be used, for example, to automatically create pull requests for a list of repositories. This comes in handy when needing to change the build structure, for example.

from fabric import api as fabpi

The Fabric library, (here used in its fabric3 fork) is an all-purpose ad-hoc library for operations.

Again, a full tutorial is beyond the scope of this article.

fabpi.local("uname -a", capture=True)

However, the advantages of running Fabric from Jupyter notebook are big. Because Fabric is specifically designed for ad-hoc operations, there is often no way to know exactly what someone did. Even with a fabfile, the logging is often lacking.

Running the functions from a notebook means an official log of what was done -- that can easily be attached to the relevant ticket, or to the post-mortem. (Either the post-mortem that the operations were meant to fix, or the ones they inevitably caused).

import docker
client = docker.DockerClient(base_url='unix://var/run/docker.sock')

The Docker client is also available as a Python library. Once again, the possibilities are endless.

[im for im in client.images.list() if (im.tags or [''])[0].startswith('debian')]

[<Image: 'debian:latest'>,
 <Image: 'debian:stable-slim', 'moshez/debian:2017-10-26T10-58-56-455022'>]

import boto3

The boto3 library is an API for the Amazon Web Services -- which includes everything from load balancers, through orchestrating containers, to sending e-mail.

The Jupyter console is a great adjunct to the AWS web console -- while results can often be inspected in the web console, any actions done from the notebook can be repeated, tweaked, and automated.

ec2 = boto3.client('ec2', region_name='us-west-2')

Hank Green recently shared the embarrassing secret to his productivity, which he summarises as:

Everything creative I do, I do my best to get it 80% of the way to as good as I can make it and go no further. I just don’t try to get it to 100%.

I recommend watching the video or reading the transcript, as he does a great job of explaining why and how this works.

I want to try to adopt this attitude in 2018. I have prided myself—perhaps without justification—on producing high-quality work but I expend way too much time fine-tuning and end up writing, speaking, and coding less than I would like.

That said, there might be a difference between creative projects that get released and then abandoned and software engineering projects that get deployed and then maintained forever.

With software projects like that, there’s an amazing power to continuously applying effort to the same code-base to make it incrementally better. Slow, steady, 80% efforts that acculumate over time to make something beautiful, maintainable and useful.

But you never know when the budget is going to be yanked from under you. In the worst case, you’ll be on the hook for running something in production without having the time or opportunity to fix the problems you find.

It’s the fear of this that sometimes drives me to reach for 100% as good as I can do. I guess the only thing I can do about this is ignore the fear.

Have you ever made conscious efforts to improve your productivity by lowering your ambitions around quality? How did it go?

Since the inception of wheels that install Python packages without executing arbitrary code, we need a static way to encode conditional dependencies for our packages. Thanks to PEP 508 we do have a blessed way but sadly the prevalence of old setuptools and pip versions make it a minefield to use.

Ten tricks to level up your Python.

(Thanks to Tom Prince and Nelson Elhage for suggestions for improvement.)

Nowadays, almost all text will be encoded in UTF-8 -- for good reasons, it is a well thought out encoding. Some of it will be in Latin 1, AKA ISO-8859-1, which is popular in the western world. Less of it will be in other members of the ISO-8859 family (-2 or higher). Some text from Japan will occasionally still be in Shift-JIS. These encodings are all reasonable -- too reasonable.

What about more interesting encodings?

>>> "hello".encode('cp1140')
b'\x88\x85\x93\x93\x96'

MASK = (1 << 8) - 1
with open('input', 'rb') as fin, open('output', 'wb') as fout:
    while True:
        c = fin.read(1)
        if not c:
            break
        c = c & MASK # <--- this right here
        fout.write(c)

>>> source = b'I\xe2\x9d\xa4U'
>>> declaration = source.decode('utf-8')
>>> declaration.encode('punycode')
b'IU-ony'

>>> source = b'I\xe2\x9d\xa4me'
>>> source.decode('utf-8').encode('punycode')
b'Ime-4r6a'

"It's ok to mess up your own room."

Sometime there is a problem where the design is obvious -- at least to you. Maybe it's simple. Maybe you've solved one like that many times. In those cases, just go ahead -- use Test-Driven-Development, lint your code as you're writing, and push a branch full of beautiful code, ready to be merged.

This is the exception, rather than the rule, though. In real life, half of the time, we have no idea what we are doing. Is a recursive or iterative solution better? How exactly does this SaaS work? What does Product Management actually want? What is, exactly, the answer to life, the universe and everything?

A lot of the time, solving a problem becomes with exploratory programming. Writing little snippets, testing them out, writing more snippets, throwing some away when they seem to be going down a bad path, saving some from earlier now that we understand the structure better. Poking and prodding at the problem, until the problem's boundaries become clearer.

This is, after all, why dyamic languages became popular -- Python became popular in web development and scientific computing precisely because in both places, "exploratory programming" is important.

In those cases, every single rule about "proper software development" goes straight out the window. Massive functions are fine, when you don't know how to break them. Code with one letter variables is fine, when you are likely to throw it away. Code with bad formatting is fine, when you are likely to refactor it. Code with no tests is fine, if it's doing the wrong thing anyway. Code with big commented out sections is fine, if those are likely to prove useful in an hour.

In short, every single rule of "proper software development" goes out the window when we are exploring a problem, testing its boundaries. All but one -- work on a branch, and keep your work backed up. Luckily, all modern version control systems have good branch isolation and easy branch pushing, so there is no problem. No problem, except the social one -- people are embarrassed at writing "bad code". Please don't be. Everyone does it. Just don't merge it into the production code -- that will legitimately annoy other people.

But as long as the mess is in your own room, don't worry about cleaning it up.

When you’re just getting started as a programmer and you need help frequently, asking for help can feel daunting, like you’ll lose either way. Ask too soon and you’ll end up feeling stupid for not having figured out the answer on your own. And if you don’t ask for help, your manager can get annoyed that you’re taking too long to solve the problem.

Asking for help is a skill, and a skill you can learn. Once you’ve mastered this skill you will be able ask questions at the right time, and in the right way. In this post I’ll cover:

• Some ways you shouldn’t ask for help.
• When to ask for help, so that it’s neither too soon nor too late, by planning your task in advance.
• How to ask for help in a way that will maximize your learning and make you look better to your manager.

The wrong way to ask for help

There are two main failures when asking for help, asking too much and asking too little.

“Help help help help help help help help”: You will of course have many questions when you’re learning a new codebase or a new technology. But if you’re asking your lead developer a question every 10 minutes, you’re going to annoy them. A lot. You’re impeding their ability to work, and you’re probably not spending enough time learning on your own.

Instead of asking your questions one by one as they occur, write them all down. Then, when your local expert seems to have a free moment, or if it’s been a few hours since you last asked a question, go and ask them all your questions at once. This will be less intrusive, and chances are you will have figured out some of the answers on your own in the interim.

“I don’t want to ask for help!”: Asking for help can be embarrassing, it’s true. And trying to figure stuff out on your own can help you learn. But if you wait too long, or never ask for help, you’ll both learn less and annoy your manager, because inevitably you’ll end up spinning your wheels and wasting time.

Instead, wait until you’ve given it a reasonable try, and then ask. You’ll learn how to do that in the next section.

Knowing when to ask for help with timeboxing

So how exactly do you know when to ask for help? Advance planning. By knowing how long you have to spend on the task, and then setting a timebox, a limited amount of time to work on it on your own, you can have an alert (metaphorical or real) telling you “it’s been too long, time to ask for help.”

Here’s how the process works:

1. When your lead developer gives you a task, ask how much time you should spend on it. They might say something like “we need that ready in a couple of days, but really it should only take you a day.” So now you know you want to aim to finish the task within a day. (Over time you’ll learn how to do this yourself, and also whether your manager is overly optimistic or pessimistic about their estimates.)
2. Now that you know your deadline, set a timebox, a limited amount of time that is less than your deadline. If your deadline is a day, you might set it to three hours.
3. Now start your task. After you hit your timebox (e.g. three hours), see where you’re at: are you making good progress? Great, set another timebox and keep working. Not making progress? It’s time to ask for help.

If your deadline is one day, and you ask for help after three hours, you’ve not asked too late: there’s still time to finish the task. And you haven’t asked too soon, either, you’ve at least tried on your own.

Learning more (and looking good) when you’re asking for help

You’ve hit your timebox, and you’re asking for help: how do you get the most value out of your questions?

Don’t ask yes/no questions: “is this how I do this?”

If your lead developer actually answers with “yes” or “no”, you’re only gaining 1 bit of information, the smallest amount of information possible. Instead, ask open ended questions: “what should the result be like?” “can you walk me through how this works?”, etc.

Always present a potential answer the question.

It doesn’t have to be the best answer, or the correct answer (if it were, you probably wouldn’t be asking for help, after all). But you should always say something like “my best guess is this works like this, because of X and Y, but I’m still a little confused - could you explain this?”

Providing an answer serves multiple purposes:

1. It forces you to try to come up with an answer and learn more. Sometimes you’ll figure it out on your own!
2. It demonstrates to your manager that you made an effort, making you look good.
3. It helps your manager understand what you know and what you don’t, which means they’ll have an easier time helping you.

How to ask for help: a recap

Here’s the short version:

1. Do ask for help.
2. Batch up your questions.
3. Set a timebox on tasks, and ask for help if you hit the timebox and you’re still stuck.
4. Ask open-ended questions, and always provide a potential answer.

Next time your manager gives you a task, apply these guidelines: they’ll be happier, and you’ll learn more.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

Speed, reproducibility, easy rollbacks, and predictability is what we strive for when deploying our diverse Python applications. And that’s what we achieved by leveraging virtual environments and Linux system packages.

How do you find a job with work/life balance? Most companies won’t tell you “we want you to work long hours” on their careers page, it’s hard to ask, and it’s not like you can go to a job board and search for work/life balance. Until now, that is.

Key Values is a newly launched site that lets you filter jobs by values. Instead of the standard boring “at X we’re passionate about doing Y with technology stack Z” (more on that below), you can search by the things that make a job work or not work for you. That means you can search for programming jobs with work/life balance (just click “WORK/LIFE BALANCE” to filter to those jobs), but you can also search for companies that are good for junior devs, or have a flat organization. Different people have different values, and Key Values reflects that.

It’s still early days, so there aren’t a huge number of jobs yet, but I love the concept and wanted to hear more. So I got in touch with Lynne Tye, the creator of Key Values, to hear how she ended up creating such a different, and useful, approach to hiring.

Q. Could you share your background with our readers, how you became a programmer?

LYNNE: I studied brain and cognitive sciences at MIT, then went into a PhD program in neuroscience at UCSF. Two years in, I realized it wasn’t for me, and I dropped out. Then I had a couple of odd jobs while I was soul searching, and a few months later I started working at Homejoy, as an operations manager for the Bay Area, a people manager.

While I was working at Homejoy, I noticed how powerful the engineers were. They could make so much impact with just one line of code, and I always felt frustrated when I needed them to fix a small bug that was making my life a nightmare. What I was doing just wasn’t as scalable, like having lots of 1-on-1 meetings. So after Homejoy, I decided I wanted to learn how to code.

Q. What did you learn from your experience before starting Key Values?

LYNNE: Scientific academia is one of the few industries where there’s a master/apprentice relationship, a very clear structure of mentorship. I think that the way you view relationships, the way you make decisions about joining labs is based on the idea of working relationships that need to be as compatible and symbiotic as possible. A lot of times these mentors stay with you your whole life [like family], your mentor’s mentor is your “grandfather”. I noticed this was lacking when I started doing web development.

After grad school, I was feeling pretty lost and really wasn’t sure what I wanted to do with my life. I had basically been laser focused on becoming an academic professor and research scientist for the last 6 years and hadn’t once looked up to consider a different career. One of the main frustrations I had with research was how slow it was, and slow it was to get feedback.

The environment at Homejoy gave me all of that. It was intense, exciting, fast-paced, and there was constantly feedback from all directions. At the time, it was my dream job. And I think it just also made me realize that it wasn’t for everyone, but it was perfect for me. It made me realize that everyone has their own set of personal values/goals and it’s so important to find work that aligns with those.

Q. Doesn’t sound like there was much work/life balance at Homejoy.

LYNNE: Hahaha, there definitely wasn’t. But, I didn’t want work/life balance! When I left grad school, I genuinely thought work/life balance was a proxy for laziness, or a lack of passion. Of course, after grinding it out at Homejoy for a year and a half, I burned out quite a bit. Afterwards, I wanted work/life balance for the first time. And I found it in the lifestyle I had as a web freelancer.

Ironically, after a couple of years of having so much work/life balance, I started to miss the excitement and sense of urgency of working a lot. That’s where I am now: I’d feel a little sad if I was on a team and the only one working past 6pm or 7pm.

All of this sharpened my views on finding a new career: you need to know what you want, and you should be picky and demanding when you’re evaluating your options.

Q. As an operations manager at Homejoy you did some of the hiring. What did you look for?

LYNNE: It’s funny to look back at it. I didn’t have the language at the time, I didn’t have the framework or language to say what we were. And hiring was so new to me, I didn’t have any experience with it, I hadn’t really articulated the actual values we had.

It was very [intense], we were not shy about it. Everyone worked really late, really early, and on weekends.it felt really exciting, and I don’t think anyone felt like it was work. We all enjoyed spending lots of time together and had all decided we were willing to make that commitment. At the time, I think I was always looking to hire people similar to the existing team.

Q. It seems many companies can’t articulate what they want?

LYNNE: They can’t. Many employers and job seekers have not taken the time to evaluate who they are, where they’re trying to go in terms of culture, and how that impacts hiring. I view my job as helping teams articulate values. And not only helping them articulate them and write them, but also to challenge them, asking whether they’re translating them into actions, or whether they’re things they just write on their website or on their walls.

Q. One my pet peeves in hiring is the focus on particular technologies. Why do you think hiring managers focus on this so much?

LYNNE: In general I think that job descriptions and the way that people are approaching recruiting and sourcing is outdated, given how much information we all have access to now. Previously it was harder to get information about different employment opportunities, so the biggest differentiators were salary, and do you have experience with hard skills we need today. As time has gone by these things are still important, but people have the ability to compare more teams and have more information to compare them by, and job descriptions haven’t reflected this change.

Software development has changed over the years. I can’t speak from experience, but it’s easier to build things today than it was 20 years. The ability to learn technologies, it wasn’t the same conversation it was 20 years ago. I don’t think it makes sense anymore to talk about experience with a particular technology.

Some companies are happy to have people learn on the job, but people just follow the [job posting] template everyone uses:

• Generic part about what the company does.
• Generic part about how much you’ll learn, how much fun it is, how much impact you’ll make.
• Bullet point with requirements, experience with X, Y, Z technology.
• And then another set of bullet points about benefits and perks, and not-so-compelling reasons to join the company.

Q. Which brings us to Key Values, where you’re trying to do things differently. What exactly does Key Values do?

LYNNE: I try to help job seekers find teams that share their values.

Q. How did you come up with these values?

LYNNE: I interviewed dozens and dozens of engineers. I noticed it’s challenging for people to articulate or identify what they care about most. And I noticed that as people were telling me what they were looking for, it came with a story about a previous experience they had where their job didn’t have that value, and that brought to light why that was so important them.

After interviewing lots of engineers, I spent time thinking about values, and phrasing them in ways where they would apply to many teams, but not every team. For example, had I had “Mission driven” every single team would have selected it, and it wouldn’t help people differentiate between different teams. And I didn’t want to include values that were specific to one, or even zero teams. It was about striking the balance between those two extremes.

Q. How do you figure out what values the companies have?

LYNNE: Initially, I thought it would be more like research, I wanted to interview every engineer on the team, provide statistics. But I realized it’s not scalable, and I didn’t want to force teams to share information they weren’t comfortable sharing. You’ll never find a team that says “we never eat lunch together, we’re not friends, we’re really not social here” or “we have terrible code quality here.”

By limiting how many values [a team can choose] it tells you what they prioritize. Being limited and being forced to rank [the values they choose] is very informative, it discloses a lot of information implicitly.

Q. On your website you have job listings with these values, and you share with them with world. What can tell you from your data about what engineers care about?

LYNNE: The two things visitors pick most are work/life balance and high quality code base. This is both surprising and not surprising at all. [Next is] “remote ok”, although that is is a property, not a value, and I think that makes sense since I still don’t have that many team profiles on Key Values yet. I also think developers are more and more interested in remote opportunities. Close to that are “flexible work arrangements”, “team is diverse”.

To me, these are an indication of what engineers are not getting at their current jobs.

Q. Why is Key Values the only job board that lets you search based on work/life balance?

LYNNE: I don’t think previously there was a way for teams to truthfully tell whether the team really cared. By having a limited list [of values], and priorities, it lets you see who doesn’t prioritize it, otherwise I think most companies wouldn’t volunteer that information. How would you ask? If you poll companies, I can’t imagine any of them wanting to publicly state that they don’t.

At the end of the day, how you define work/life balance has implications, it’s difficult to categorize these things. Anyone who is reading about it, or talking about, it’s pretty divisive and polarizing. Some people think if you work more than 40 hours a week you don’t have work/life balance, but I would disagree. My goal is to give companies a chance to tell us how they interpret work/life balance, and expose people to different definitions of that term.

Q. What does a sane workweek mean to you?

LYNNE: A sane workweek to me wouldn’t be a good description, I’d say i’m looking for a sane work month. I love working, I consider myself pretty industrious, but the flexibility to decide when I work is more important. Sometimes I want to work a ridiculous amount one week, and then take a few days off, maybe have a long weekend. And that’s just in terms of when I’m working, and how much.

In general I don’t believe in 40 hours a week, because I don’t operate that way. I don’t have as regular of a schedule, and would 100% rather work 60 hours a week if I could decide when and where I can work, as opposed to a 9-5 at the same physical place with no flexibility. I’d feel much more suffocated with the latter.

In terms of a relationship with an employer, I think the most important thing to me is working someplace where they genuinely support and show interest in other aspects of my life. And that they share some of their priorities in life with me. [The means] having a network of people around you who understand who you are as a whole and support all of you, For me, it means a lot to not just talk about work at work, but to really interact with one another as friends too. I know for sure that this isn’t true for everyone, but I prefer to blur the boundary between professional and personal. I don’t like having complete work/life separation.

❧

OK, back to Itamar here: that was my interview, and now I’d like to ask for your help. Key Values is as far as I know the only place where you can search for jobs with work/life balance, or other values you may care about. That’s hugely valuable, and so I want to see Lynne’s project succeed. If you agree, here’s what you can do:

• Is your company hiring? Get in touch with Lynne and get your company listed.
• Are you looking for a job, or plan to look for one in the future? Go visit Key Values and sign up for the newsletter: the more people use the site, the easier it’ll be for Lynne to get more companies on board.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

Most Python programmers don’t spend a lot of time thinking about how equality and hashing works. It usually just works. However there’s quite a bit of gotchas and edge cases that can lead to subtle and frustrating bugs once one starts to customize their behavior – especially if the rules on how they interact aren’t understood.

If you’re looking for a job with work/life balance, you’ll obviously want to avoid a company where everyone works long hours. But what about companies where only some people work long hours? If you ask for an explanation of what’s going on at these companies, one common answer you’ll hear is something like “oh, they’re young, they don’t have families, they enjoy their work so they work longer hours.”

As you venture into the unknown waters of a potential employer, is this something you should worry about? Those younger programmers, you’re told, are having a wonderful time. They’re waving their arms, true, and there’s a tangle of tentacles wrapped around them, but that’s just the company mascot, a convivial squid. Really, they’re having fun.

Or are they?

To be fair, there really are companies where programmers stay later at the office because they enjoy hanging out with people they like, and working on interesting problems while they do it. More experienced programmers are older, are therefore more likely to have children or other responsibilities, and that’s why they’re working shorter hours. On the other hand, it may be that the pile of tentacles wrapped around these programmers is not so much convivial as voracious and hostile: the kraken of overwork.

The kraken of overwork

Another potential reason less experienced programmers are working longer hours is that they don’t know how to get their work done in a reasonable amount of time. Why? Because no one taught them the necessary skills.

I’m not talking about technical skills here, but rather things like:

• Prioritizing based on your goals.
• Timeboxing your work, and therefore knowing when to ask for help.
• Figuring out user requirements.

Many managers don’t quite realize these skills exist, or can’t articulate what they are, or don’t know how to teach them. So even if the inexperienced programmers are given reasonable amounts of work, they are never taught the skills to finish their work on time. In this situation having children or a family is not causal, it’s merely correlated with experience. Experienced programmers know how to get their work done in a reasonable amount of time, but inexperienced programmers don’t.

In which case the young programmers’ lack of kids, and “oh, they just enjoy their work”, is just a rationalization: an excuse for skills that aren’t being taught, an excuse for pointless and wasted effort. Those inexperienced programmers waving their hands around aren’t having fun, they’re being eaten by a squid—and no one is helping save them.

Avoiding the kraken

So when you’re interviewing for a job, how do you tell the difference between these two reasons for long hours? Make sure you ask about training and career development.

• This excellent list of questions to ask during an interview, by Elena Nikoaeleva, suggests asking “how does the company help junior people to grow?”
• If you’re being interviewed by a manager, ask them for an example of mentoring and teaching they’ve done.
• If you’re being interviewed by a junior developer, ask them about a time they went off track and took to long to finish a track, to see if they got any help.

Hopefully the answers will demonstrate the less-experienced programmers are taught the skills they need, and helped when they’re floundering. If no, you may wish to avoid this company, especially if you are less experienced. Good luck interviewing, and watch out for krakens!

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

(This is an adaptation of part of the talk Kurt Rose and I gave at PyBay 2017)

An abstraction cascade is a common anti-pattern in legacy system. It is useful to understand how to recognize it, how it tends to come about, how to fix it -- and most importantly, what kind of things will not fix it. The last one is important, in general, for anti-patterns in legacy systems: if the obvious fix worked, it would have been already dealt with, and would not be a common anti-pattern in legacy systems.

def get_address(name):
    if name in services:
        if services[name].ip:
            return service[name].ip, service[name].port
        elif services[name].address:
            # Added for issue #2321
            if ':' in services[name].address:
               return service[name].address.split(':')
            else:
               # Fixes issues #6985
               # TODO: Hotfix, clean-up later
               return service[name].address, DEFAULT_PORT
    return dns_lookup(name), DEFAULT_PORT

def get_address(name):
    return dns_lookup(name), DEFAULT_PORT

def get_address(name):
    if name in services:
        # Fixes issues #6985
        # TODO: Hotfix, clean-up later
        return service[name].address, DEFAULT_PORT
    return dns_lookup(name), DEFAULT_PORT

def get_address(name):
    if name in services:
        # Added for issue #2321
        if ':' in services[name].address:
            return service[name].address.split(':')
        else:
            # Fixes issues #6985
            # TODO: Hotfix, clean-up later
            return service[name].address, DEFAULT_PORT
    return dns_lookup(name), DEFAULT_PORT

def get_address(name):
    if name in services:
        if services[name].ip:
            return service[name].ip, service[name].port
        elif services[name].address:
            # Added for issue #2321
            if ':' in services[name].address:
               return service[name].address.split(':')
            else:
               # Fixes issues #6985
               # TODO: Hotfix, clean-up later
               return service[name].address, DEFAULT_PORT
    return dns_lookup(name), DEFAULT_PORT

class AbstractNameFinder(object):
    def matches(self, name):
        raise NotImplementedError()
    def get_address(self, name):
        raise NotImplementedError()
class DNS(AbstractNameFinder):
    def matches(self, name):
        return True
    def get_address(self, name):
        return dns_lookup(name), DEFAULT_PORT
class Local(AbstractNameFinder):
    def matches(self, name):
        return hasattr(services.get(name), 'ip')
    def get_address(self, name):
        return services[name].ip, services[name].port
finders = [Local(), DNS()]
def get_address(name):
    for finder in finders:
        if finder.match(name):
            return finder.get_address(name)

Gather is a plugin framework -- and it now has its own blog.

Use it! If you like it, tell us about it, and if there is a problem, tell us about that.

Are you worried that you’re writing bad code? Code that doesn’t follow best practices, code without tests, code that violates coding standards, code you simply don’t want to think about because it’s so very very embarrassing?

In fact, there is no such thing as inherently bad code, or for that matter inherently good code. This doesn’t mean you shouldn’t be judging your code, it’s just that if you’re worrying about whether your code is “good” or “bad” then you’re worrying about the wrong thing.

In this post I will:

• Demonstrate that “bad” code can be “good” code under different circumstances, and that “best practices” aren’t.
• Suggest a handy mental exercise to help you move away from thinking in terms of “bad vs good”.

“Bad” code, “good” code

Let’s look at a couple of examples of “bad” code and see that, under some circumstances, this “badness” is irrelevant.

Hard-to-read code

As everyone knows, “good” code is easy to read and follow. You need to choose readable variables, clear function names, and so on and so forth. But then again—

• If you’re writing code for fun, for your own use, who cares? You are the only one who will ever read this code, so you can choose whatever system makes sense to you so long as you can it understand.
• If you’re entering the The International Obfuscated C Code Contest, you have a completely different set of criteria. There are some truly amazing entries in past contests; calling them “bad code” is just wrong.
• If the value your code provided to your organization is sufficiently higher than the cost of maintenance, it can’t really be said to be “bad” code. The world is full of spreadsheets and one-off scripts that are hard-to-read, but nonetheless work correctly and produce huge value.

Unit tests

As everyone knows, “good” code has unit tests, and “bad” code does not. But then again—

• You’re writing a packaging utility for your product. You have end-to-end tests that make sure the resulting packages install correctly. Unit tests are a waste of time: they prove no additional correctness, and your packaging tool is not your product, it’s just a utility whose code will never be shared elsewhere.
• You’re building an exploratory prototype. Once you’ve figured out if this idea works you’ll be throwing the code away away and starting from scratch. Why write unit tests?

There’s no such thing as “best practices”

At this point you might be getting a little annoyed. “Yes,” you might say, “these are some exceptions, but for the most part there are standard best practices that everyone can and should follow.” Consider:

• Formal verification is a practical and useful technique for finding bugs in your code. See for example how AWS uses formal verification to validate their services.
• NASA’s software development practices are amazing, both effective and expensive. One quote to give you a flavor: “The specs for the [shuttle] program fill 30 volumes and run 40,000 pages.”

Both NASA’s techniques and formal verification lead to far less defects. So should they be best practices? It depends: if you’re building the website for a local chain of restaurants, using these techniques is ridiculous. You’ll ship a year late and 10× over budget. On the other hand, if you’re writing the software for a heart monitor, or a spacecraft that’s going to Pluto, “I wrote some unit tests!” is very definitely not best practices.

A sanity check for your code

Instead of feeling embarrassed about your “bad” code, or proud of your “good” code, you should judge your code by how well it succeeds at achieving your goals. Whether you’re competing in the International Obfuscated C Code Contest or working on NASA’s latest mission, you need to use techniques suitable to your particular situation and goal. Judging one’s code can be a little tricky, of course: it’s easy to miss the ways in which you’ve failed, easy to underestimate the ways in which you’ve succeeded.

So try this exercise to give yourself some perspective: every time you write some code figure out the tradeoffs you’re making. That is, identify the circumstances and goals for which your current practices are insufficient, and those for which your current practices are overkill. If you can’t come up with an answer, if your code seems suitable for all situations, that is a danger sign: there’s always a tradeoff, even if you can’t see it.

Finally, when you encounter someone else’s code, be kind: don’t tell them their code is “bad”. Instead, go through the same exercise with them. Figure out their goals, and then walk through the tradeoffs involved in how they’ve written their code. This is a far more useful way of improving their code, and can help you understand why you make the decisions you do.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

Too Many Secrets

A wise man once said, “you shouldn’t use ENV variables for secret data”. In large part, he was right, for all the reasons he gives (and you should read them). Filesystem locations are usually a better operating system interface to communicate secrets than environment variables; fewer things can intercept an open() than can read your process’s command-line or calling environment.

One might say that files are “more secure” than environment variables. To his credit, Diogo doesn’t, for good reason: one shouldn’t refer to the superiority of such a mechanism as being “more secure” in general, but rather, as better for a specific reason in some specific circumstance.

Supplying your PyPI password to tools you run on your personal machine is a very different case than providing a cryptographic key to a containerized application in a remote datacenter. In this case, based on the constraints of the software presently available, I believe an environment variable provides better security, if you use it correctly.

Popping A Shell By Any Other Name

If you upload packages to the python package index, and people use those packages, your PyPI password is an extremely high-privilege credential: effectively, it grants a time-delayed arbitrary code execution privilege on all of the systems where anyone might pip install your packages.

Unfortunately, the suggested mechanism to manage this crucial, potentially world-destroying credential is to just stick it in an unencrypted file.

The authors of this documentation know this is a problem; the authors of the tooling know too (and, given that these tools are all open source and we all could have fixed them to be better about this, we should all feel bad).

Leaving the secret lying around on the filesystem is a form of ambient authority; a permission you always have, but only sometimes want. One of the worst things about this is that you can easily forget it’s there if you don’t use these credentials very often.

The keyring is a much better place, but even it can be a slightly scary place to put such a thing, because it’s still easy to put it into a state where some random command could upload a PyPI release without prompting you. PyPI is forever, so we want to measure twice and cut once.

Luckily, even more secure places exist: password managers. If you use https://1password.com or https://www.lastpass.com, both offer command-line interfaces that integrate nicely with PyPI. If you use 1password, you’ll really want https://stedolan.github.io/jq/ (apt-get install jq, brew install jq) to slice & dice its command-line.

The way that I manage my PyPI credentials is that I never put them on my filesystem, or even into my keyring; instead, I leave them in my password manager, and very briefly toss them into the tools that need them via an environment variable.

First, I have the following shell function, to prevent any mistakes:

1
2
3
4

function twine () {
    echo "Use dev.twine or prod.twine depending on where you want to upload.";
    return 1;
}

For dev.twine, I configure twine to always only talk to my local DevPI instance:

1
2
3
4
5
6

function dev.twine () {
    env TWINE_USERNAME=root \
        TWINE_PASSWORD= \
        TWINE_REPOSITORY_URL=http://127.0.0.1:3141/root/plus/ \
        twine "$@";
}

This way I can debug Twine, my setup.py, and various test-upload things without ever needing real credentials at all.

But, OK. Eventually, I need to actually get the credentials and do the thing. How does that work?

1Password

1password’s command line is a little tricky to log in to (you have to eval its output, it’s not just a command), so here’s a handy shell function that will do it.

1
2
3
4
5
6

function opme () {
    # Log this shell in to 1password.
    if ! env | grep -q OP_SESSION; then
        eval "$(op signin "$(jq -r '.latest_signin' ~/.op/config)")";
    fi;
}

Then, I have this little helper for slicing out a particular field from the OP JSON structure:

1
2
3

function _op_field () {
    jq -r '.details.fields[] | select(.name == "'"${1}"'") | .value';
}

And finally, I use this to grab the item I want (named, memorably enough, “PyPI”) and invoke Twine:

1
2
3
4
5
6
7

function prod.twine () {
    opme;
    local pypi_item="$(op get item PyPI)";
    env TWINE_USERNAME="$(echo ${pypi_item} | _op_field username)" \
        TWINE_PASSWORD="$(echo "${pypi_item}" | _op_field password)" \
        twine "$@";
}

LastPass

For lastpass, you can just log in (for all shells; it’s a little less secure) via lpass login; if you’ve logged in before you often don’t even have to do that, and it will just prompt you when running command that require you to be logged in; so we don’t need the preamble that 1password’s command line did.

Its version of prod.twine looks quite similar, but its plaintext output obviates the need for jq:

1
2
3
4
5

function prod.twine () {
    env TWINE_USERNAME="$(lpass show PyPI --username)" \
        TWINE_PASSWORD="$(lpass show PyPI --password)" \
        twine "$@";
}

In Conclusion

“Keep secrets out of your environment” is generally a good idea, and you should always do it when you can. But, better a moment in your process environment than an eternity on your filesystem. Environment-based configuration can be a very useful stopgap for limiting the lifetimes of credentials when your tools don’t support more sophisticated approaches to secret storage.¹

Post Script

If you are interested in secure secret storage, my micro-project secretly might be of interest. Right now it doesn’t do a whole lot; it’s just a small wrapper around the excellent keyring module and the pinentry / pinentry-mac password prompt tools. secretly presents an interface both for prompting users for their credentials without requiring the command-line or env vars, and for saving them away in keychain, for tools that need to pull in an API key and don’t want to make the user manually edit a config file first.

One day you go to work and discover your technical skills are obsolete:

• The programming language you know best has been declining in popularity for a decade.
• The web framework you know best has been completely changed in v2, and rewritten in another language for good measure.
• Job postings are stomach-churning lists of tools you’ve never used, or even heard of.
• You’re asked to compare two technologies, and you don’t know where to start.

You feel like your growth has been stunted: there are all these skills you should have been learning, but you never did because you didn’t need them at work. Your coworkers seem to know all about the latest tools and you don’t, and eventually, maybe soon, you’ll just be left behind.

What should you do? How can you get out of this mess and salvage your career?

I’m not going to say “code in your spare time”, because that’s not possible for many people. And while I believe it’s completely possible to keep your skills up-to-date as part of your job (e.g. I’ve written about having a broad grasp of available technology and practicing on the job), the assumption at this point is that you haven’t done so.

Here are your goals, then:

1. Get your technical skills up to speed, and quickly.
2. Do it all during work hours.
3. End up looking good to your manager.

In this post I’ll explain one way to do so, which involves:

• Understanding the organizational dynamic that causes organizations to use out-of-date technology.
• Actively seeking out these problematic technologies, and then improving the situation both for your organization and for you.

Why you’re using old technology

Most programmers, probably including you, work on existing software projects. Existing software projects tend to use older technology. The result: you are likely to be using older, out-of-date technology, rather than the latest and (speculatively) greatest.

When a project gets started the programmers in charge pick the best technology they know of, and hope for the best. After that initial set of choices most projects stick to their current technology set, only slowly upgrading over time. Of course, there’s always new technologies coming out that claim to be better than existing ones.

Updating an existing project to new technologies is difficult, which means changes are often put off until it’s truly necessary. It takes effort, ranging from a small effort to upgrade to a newer version of a library version, to a large effort for changing infrastructure like version control, to an enormous effort if you want to switch to a new programming language. So even when a clearly superior and popular technology becomes available, the cost of switching may not be worth it. Time spent switching technologies is time that could be spent shipping features, after all.

Two real-world examples of this dynamic:

• The Twisted open source project was started in 2000, and used the CVS version control system. Subversion, a superior version control system, was released the same year, and soon new open source projects defaulted to using it. Eventually Twisted switched to Subversion. And then Git was released, and eventually when it was clear Git was winning Twisted switched to Git. In both cases Twisted’s adoption lagged behind new projects, which could easily start off using a better VCS since they didn’t have to pay the cost of upgrading existing infrastructure.
• A large company, founded in 1997, built the initial version of their software in Perl. At the time Perl was a very popular programming language. Over the past 20 years Perl’s mindshare has shrunk: it’s harder to hire people with Perl knowledge, and there’s less 3rd party libraries being developed. So the company is stuck with a massive working application written in an unpopular language; the cost of switching to a new language is still too high.

The switch to new technology

Eventually the cost of sticking with an old technology becomes too high, and management starts putting the resources into upgrading. In a well-run company this happens on a regular, on-going basis, and management will have spent the resources to keep programmers’ skills up-to-date. In these companies learning about new technologies, and then deciding which are worth the cost of adopting, will be an ongoing activity.

In many companies, however, the cost of keeping programmer skills up-to-date is dumped on to you. You are expected to spend your spare time researching new programming languages, tools, and techniques. If you enjoy doing that, great. If you don’t, your technical knowledge will stagnate, at some cost to the company, but even more so to you.

Helping your project, upgrading your skills

If you find yourself in this situation then you can turn your project’s out-of-date technology into a learning opportunity for you. Technology’s purpose is to solve business problems: you need identify business problems where your current technology isn’t working well, and try to solve those problems. This will allow you to research and learn new technologies while helping your project improve.

Specifically, you should:

1. Identify obsolete and problematic technologies.
2. Identify potential replacements.
3. Convince your manager that this is a problem that merits further resources. Your goal is to get the time to build a proof-of-concept or pilot project where you can expand your understanding of a relevant, useful new technology.

If all goes well you’ll have both demonstrated your value to your manager, and been given the time to learn a new technology at work. But even if you fail to convince your manager, you’ll have an easier time when it comes to interviewing at other jobs, and some sense of which technologies are worth learning.

Let’s go through these steps one by one.

1. Identify obsolete and problematic technologies

Your project is likely using many out-of-date technologies: you want to find one that is both expensive to your project, and not too expensive to replace. Since you’re going to have to convince your manager to put some resources into the project, you want to have some clear evidence that an obsolete technology is costing the company.

Look for things like:

• Technologies that are shrinking in popularity; some Google searches for “$YOURTECH popularity” can help you determine this, as can Google Trends.
• Repetitive work, where you have to manually do the same task over and over again.
• Trouble hiring people with pre-existing knowledge.
• The system everyone knows is broken: it crashes all the time, say, or corrupts data.

You can do this while you work: just look for signs of trouble as you go about your normal business.

2. Identify potential replacements

Once you’ve identified a problem technology, you need to find a plausible replacement. It’s likely that any problem you have is not a rare problem: someone else has had this issue, so someone else has probably come up with a solution. In fact, chances are there are multiple solutions. You just need to find a reasonable one.

You should:

1. Figure out the keywords that describe this technology. For example, if you need to line up images automatically the term of art is “image registration”. If you want to run a series of long-running processes in a row the terms of art are “workflow management”, “batch processing”, “data pipelines”, and other terms. You can do this by reading the documentation for some known solution, talking to a colleague with broader technical knowledge, or some search engine iteration.
2. Once you have the keywords, you can start finding solutions. The documentation for a tool will often provide more keywords with which to extend your search, and will often mention competitors.
3. Search engine queries can also find more alternatives, e.g. search for “$SOMETECH alternatives” or look at the Google search auto-completes for “$SOMETECH vs”.
4. Once you have found a number of alternatives, get a sense of what the top contender or contenders are. Criteria might include complexity, maturity, risk (is it developed only by a startup?), features, popularity, and so on.

The goal is become aware of the range of technologies available, and get a superficial understanding of theirs strengths (“React has a bigger community, but skimming the Vue tutorial was easier”, for example). This process can therefore be done over the course of a few hours, at most a day or two, during your work day in-between scheduled tasks.

Remember, you can always rope in a colleague with broader technical knowledge to help out: the goal is to improve things for your project, after all.

3. Getting management buy-in

At this point you should have:

1. Identified a problem area.
2. Identified a technology or three that might solve this problem.

Next you need to convince your manager that it’s worth the cost of trying out a new technology. In particular you need to:

• Demonstrate there’s a real problem.
• Suggest a potential solution that will solve the problem.
• Give some evidence this solution is reasonable, and not too expensive.
• Suggest a next step that will allow you to investigate further (and learn more!). Ideally, some sort of pilot project where you can try the technology out on a small scale and see what it’s like using it in practice.

Your pitch might go something like this:

Demonstrate the problem: “Hey, you know how we have all these problems with Foobar, where we spend all our time redoing the formatting instead of working on actual features?”

Suggest a solution: “I’ve been looking into it and I think the problem is that we’re using a pretty old library; it turns out there’s some newer tools that could make things easier.”

Evidence for solution: “For example the Format.js library, it’s got a huge userbase, and from the docs it’s pretty easy to use, and see this example, that’s exactly what we waste all our time on doing manually!”

Next step: “So, how about for that small project we’re doing next month I try this out instead of our usual Foobar setup, and see how it goes?”

If your manager agrees: success! You now have time to learn a new technology in depth, on the job.

If your manager doesn’t agree, all is not lost. You’ve gained awareness of what newer technologies are available; you might spend a little spare time here and there at work learning it more in depth. And when you next interview for a job, you’ll have some sense of technologies to either brush up on, or at least to mention during the interview: “Formatting? Oh, we used Foobar, which is pretty bad because X, Y and Z. But I did some research and found Format.js and it seemed a lot better because A, B and C. So that’s what I’d use in the future.”

Don’t just be a problem solver

The process I describe above is just one approach; no doubt there are others. The key skill involved, however, can’t be replaced: learning how to identify problems is critical to your success as a programmer.

As a junior programmer you get handed a solution, and then you go off and implement it. When you’re more experienced you get handed problems, and come up with solutions on your own: you become a problem solver. In many ways this is an improvement, both in your skills and in your value as an employee, but it’s all a dangerous place to be.

If you’re a junior programmer no one expects much of you, but once you’re past that point expectations rise. And if you’re only a problem solver, then you’re at the mercy of whoever has the job of identifying problems. If they fail to identify an important problem, like the use of an old technology, or decide your career isn’t a problem worth worrying about, then you might find yourself in trouble: working on a failed project, or lacking the skills you need.

Don’t just be a problem solver: learn how to identify problems on your own. Every day when you go to work, every time you look at some code, every time you see a bug report or a feature request, every time you feel bored, every time someone complains, ask yourself: “What is the problem here?” As you learn to identify problems, you’ll start recognizing obsolete technology. As you learn to identify problems, you’ll start noticing the limits of your own skills and your current career choices. You’ll become a more valuable employee, and you’ll become more effective at achieving your own goals.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

A completely incomplete guide to packaging a Python module and sharing it with the world on PyPI.

I was going to write a post about how knowing some abstract algebra can help you write clearer programs.

Then I saw Eugenia Cheng’s excellent talk, Category Theory in Everyday Life, which was a keynote at Haskell Exchange 2017.

It’s excellent. She says what I wanted to say much better than I could, and says many more things that I wouldn’t have thought to say at all. You should watch it.

The talk assumes very little technical or mathematical knowledge, and certainly no knowledge of Haskell.

Porting SPAKE2 from Python to Haskell helped me understand how SPAKE2 worked, and a large part of that is due to specific features of Haskell.

-- | Whether or not an extended point is a member of Ed25519.
data GroupMembership = Unknown | Member

-- | A point that might be a member of Ed25519.
data ExtendedPoint (groupMembership :: GroupMembership)
  = ExtendedPoint
  { x :: !Integer
  , y :: !Integer
  , z :: !Integer
  , t :: !Integer
  } deriving (Show)

isExtendedZero :: ExtendedPoint irrelevant -> Bool

doubleExtendedPoint
  :: ExtendedPoint preserving
  -> ExtendedPoint preserving

affineToExtended
  :: AffinePoint
  -> ExtendedPoint 'Unknown

ensureInGroup
  :: ExtendedPoint 'Unknown
  -> Either Error (ExtendedPoint 'Member)

inbound_elem = g.bytes_to_element(self.inbound_message)
if inbound_elem.to_bytes() == self.outbound_message:
   raise ReflectionThwarted
pw_unblinding = self.my_unblinding().scalarmult(-self.pw_scalar)
K_elem = inbound_elem.add(pw_unblinding).scalarmult(self.xy_scalar)

generateKeyMaterial
  :: AbelianGroup group
  => Spake2Exchange group  -- ^ An initiated SPAKE2 exchange
  -> Element group  -- ^ The outbound message from the other side (i.e. inbound to us)
  -> Element group -- ^ The final piece of key material to generate the session key.

-- | Initiate the SPAKE2 exchange. Generates a secret (@xy@) that will be held
-- by this side, and transmitted to the other side in "blinded" form.
startSpake2
  :: (AbelianGroup group, MonadRandom randomly)
  => Spake2 group
  -> randomly (Spake2Exchange group)

-- | Determine the element (either \(X^{\star}\) or \(Y^{\star}\)) to send to the other side.
computeOutboundMessage
  :: AbelianGroup group
  => Spake2Exchange group
  -> Element group

There’s that moment late at night when you can’t sleep, and you’re so tired you can’t even muster the energy to check the time. So you stare blindly at the ceiling and look back over your life, and you think: “Did I really accomplish anything? Was my work worth anything at all?”

I live in a 140-year-old house, a house which has outlasted its architect and builders, and quite possibly will outlast me. But having spent the last twenty years of my life building software, I can’t really hope to have my own work live on. In those late night moments I sometimes believe that my resume, like that of most programmers, should open with a quote from Shelley’s mocking poem:

My name is Ozymandias, King of Kings;
Look on my Works, ye Mighty, and despair!
Nothing beside remains. Round the decay
Of that colossal Wreck, boundless and bare
The lone and level sands stretch far away.

Who among us has not had projects canceled, rewritten from scratch, obsoleted, abandoned or discarded? Was that code worth writing, or was all that effort just a futile waste?

Decay, boundless and bare

Consider some of the projects I’ve worked on. I’ve been writing software for 20+ years at this point, which means I’ve accumulated many decayed wrecks:

• The multimedia CD-ROMs I created long ago no longer run on modern operating systems, not so much because of Microsoft but because of my own design mistake.
• The dot-com I worked for turned out to be a dot-bomb.
• An offline educational platform turned out, on reflection by the customer, not to require offline capabilities. It was rewritten (by someone else) as a simpler web app.
• The airline reservation project I was small part of, a massive and rarely undertaken project, finally went live on a small airline. Google, which had acquired the company that built it, shut the project down a couple of years later. Some parts were used elsewhere and lived on, but I’m told that they have since been rewritten; by now the legacy software has probably been decommissioned.
• Projects done for startups… gone down with the company, or abandoned by a pivot, or surviving in zombie form as unmaintained open source.

I could go on, but that would just make me sadder. This is not to say none of my software lives on: there are open source projects, mostly, that have survived quite a whole, and will hopefully continue for many more. But I’ve spent years of my life working on software that is dead and gone.

How about you? How much of your work has survived?

Which yet survive

So what do you have left, after all these years of effort? You get paid for your work, of course, and getting paid has its benefits. And if you’re lucky your software proved valuable to someone, for a while at least, before it was replaced or shut down. For me at least that’s worth even more than the money.

But there’s something else you gain, something you get to take with you when the money is spent and your users have moved on: knowledge, skills, and mistakes you’ll know how to avoid next time. Every failure I’ve listed above, every mistake I’ve made, every preventable rewrite, is something I hope to avoid the next time around.

And while software mostly dies quickly, the ideas live on, and if we pay attention it’ll be the good ideas that survive. I’ve borrowed ideas for my own logging library from software that is now dead. If my library dies one day, and no doubt it will, I can only hope its own contributions will be revived by one of my users, or even someone who just half-remembers a better way of doing things.

Dead but not forgotten

Since the ultimate benefit of most software projects is what you learned from them, it’s important to make sure you’re actually learning. It’s easy to just do your work and move on. If you’re not careful you’ll forget to look for the mistakes to avoid next time, and you won’t notice the ideas that are the only thing that can truly survive in the long run.

• Every month or two, take a look at what you’ve been working on, and ask yourself: “Am I learning something new?” If you aren’t, it’s time for a change: perhaps just a bit of introspection to see what there is to be learned, perhaps a new project, maybe even a new job.
• If you have learned something, ask yourself if you’ve ensured that this knowledge is passed on to others, so they can gain something from it.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

When you’re just starting out in your career as a programmer, the variety and number of skills you think you need to learn can be overwhelming. And working with colleagues who can produce far more than you can be intimidating, demoralizing, and confusing: how do they do it? How can some programmers create so much more?

The obvious answer is that these productive programmers have technical skills. They know more programming languages, more design patterns, more architectural styles, more testing techniques. And all these do help: they’ll help you find a bug faster, or implement a solution that is more elegant and efficient.

But the obvious answer is insufficient: technical skills are necessary, but they’re not enough, and they often don’t matter as much as you’d think. Productivity comes from avoiding unnecessary work, and unnecessary work is a temptation you’ll encounter long before you reach the point of writing code.

In this post I’m going to cover some of the ways you can be unproductive, from most to least unproductive. As you’ll see, technical programming skills do help, but only much further along in the process of software development.

How to be unproductive

1. Destructive work

The most wasteful and unproductive thing you can do is work on something that hurts others, or that you personally think is wrong. Instead of creating, you’re actively destroying. Instead of making the world a better place, you’re making the world a worse place. The better you are at your job, the less productive you are.

Being productive, then, starts with avoiding destructive work.

2. Work that doesn’t further your goals

You go to work every day, and you’re bored. You’re not learning anything, you’re not paid well, you don’t care one way or another about the results of your work… why bother at all?

Productivity can only be defined against a goal: you’re trying to produce some end result. If you’re working on something that doesn’t further your own goals—making money, learning, making the world a better place—then this work isn’t productive for you.

To be productive, figure out your own goals, and then find work that will align your goals with those of your employer.

3. Building something no one wants

You’re working for a startup, and it’s exciting, hard work, churning out code like there’s no tomorrow. Finally, the big day comes: you launch your product to great fanfare. And then no one shows up. Six months later the company shuts down, and you’re looking for a new job.

This failure happens at big companies too, and it happens to individuals building their side project: building a product that the world doesn’t need. It doesn’t matter how good a programmer you are: if you’re working on solving a problem that no one has, you’re doing unnecessary work.

Personally, I’ve learned a lot from Stacking the Bricks about how to avoid this form of unproductive work.

4. Running out of time

Even if you’re working on a real problem, on a problem you understand well, your work is for naught if you fail to solve the problem before you run out of time or money. Technical skills will help you come up with a faster, simpler solution, but they’re not enough. You also need to avoid digressions, unnecessary work that will slow you down.

The additional skills you need here are project planning skills. For example:

• Prioritization, figuring out what is most important.
• Timeboxing, setting timeouts for your work after which you stop and reassess your situation, e.g. by asking for help.
• Planning, working out the critical path from where you want to be to where you are now.

5. Solving the symptoms of a problem, instead of the root cause

Finally, you’ve gotten to the point of solving a problem! Unfortunately, you haven’t solved the root cause because you haven’t figured out why you’re doing your work. You’ve added a workaround, instead of discovering the bug, or you’ve made a codepath more efficient, when you could have just ripped out an unused feature altogether.

Whenever you’re given a task, ask why you’re doing it, what success means, and keep digging until you’ve found the real problem.

6. Solving a problem inefficiently

You’ve solved the right problem, on time and on budget! Unfortunately, your design wasn’t as clean and efficient as it could have been. Here, finally, technical skills are the most important skills.

Beyond technical skills

If you learn the skills you need to be productive—starting with goals, prioritizing, avoiding digressions, and so on—your technical skills will also benefit. Learning technical skills is just another problem to solve: you need to learn the most important skills, with a limited amount of time. When you’re thinking about which skills to learn next, take some time to consider which skills you’re missing that aren’t a programming language or a web framework.

Here’s one suggestion: during my 20+ years as a programmer I’ve made all but the first of the mistakes I’ve listed above. You can hear these stories, and learn how to avoid my mistakes, by signing up for my weekly Software Clown email.

You shouldn't have to work evenings or weekends to succeed as a software engineer. Take control of your time and your career by reading The Programmer's Guide to a Sane Workweek.

Thanks to Paul Kehrer for reviewing! Any mistakes or oversights that are left are my responsibility.

AES's maximum key size is 256 bits (there are also 128 and 192 bit versions available). Is that enough? Well, if there is a cryptographic flaw in AES (i.e., a way to recover some bits of the key by some manipulation that takes less than 2**256 operations), then it depends on how big the flaw is. All algorithms come with the probablistic "flaw" that, on average, only 50% of the keys need to be tested -- since the right key is just as easily in the first half as the second half. This means, on average, just 2**255 operations are needed to check "all" keys.

If there is an implementation flaw in your AES implementation, then it depends on the flaw -- most implementation flaws are "game over". For example, if the radio leakage from the CPU is enough to detect key bits, the entire key can be recovered -- but that would be true (with only minor additional hardship) if the key was 4K bit long. Another example is a related subkey attack, where many messages are encrypted with keys that have a certain relationship to each other (e.g., sharing a prefix). This implementation flaw (in a different encryption algorithm) defeated the WEP WiFi standard.

What if there is none? What if actually recovering a key requires checking all possibilities? Can someone do it, if they have a "really big" computer? Or a $10B data-center?

How much is 256-bit security really worth?

Let's see!

We'll be doing a lot of unit conversions, so we bring in the pint library, and create a new unit registry.

import pint
REGISTRY = pint.UnitRegistry()

Assume we have a really fast computer. How fast? As fast as theoretically possible, or so. The time it takes a photon to cross the nucleus of the hydrogen atom (a single proton) is called a "jiffy". (If someone tells you they'll be back in a jiffy, they're probably lying -- unless they're really fast, and going a very short distance!)

REGISTRY.define('jiffy = 5.4*10**-44 seconds')

Some secrets are temporary. Your birthday surprise party is no longer a secret after your friends yell "surprise!". Some secrets are long-lived. The British kept the secret of the broken Enigma until none were in use -- long after WWII was done.

Even the Long Now Foundation, though, does not have concrete plans post-dating the death of our sun. No worries, unless the Twisted gets more efficient, the cursed orb has got a few years on it.

sun_life = 10**10 * REGISTRY.years

With our super-fast computer, how many ticks do we get until the light of the sun shines no longer...

ticks = sun_life.to('jiffy').magnitude

...and how many do we need to brute-force AES?

brute_force_aes = 2**256

Luckily, brute-force parallelises really well: just have each computer check a different part of the key-space. We have fast computer technology, and quite a while, so how many do we need?

parallel = brute_force_aes / ticks

No worries! Let's just take over the US, and use its entire Federal budget to finance our computers.

US_budget = 4 * 10**12

Assume our technology is cheap -- maintaining each computer, for the entire lifetime of the sun, costs a mere $1.

Do we have enough money?

parallel/US_budget

4953.566155198452

Oh, we are only off by a factor of about 5000. We just need the budget of 5000 more countries, about as wealthy as the US, in order to fund our brute-force project.

Again, to be clear, none of this is a cryptographic analysis of AES -- but AES is the target of much analysis, and thus far, no theoretical flaw has been found that gives more than a bit or two. Assuming AES is secure, and assuming the implementation has no flaws, brute-forcing AES is impossible -- even with alien technology, plenty of time and access to quite a bit of the world's wealth.

Some while ago I got a new wifi-capable camera. Of course, it has some awful proprietary system for actually transferring images to a real computer. Fortunately, it's all based on a needlessly complex HTTP interface which can fairly easily be driven by any moderately capable HTTP client. I played around with FlashAero a bit first but it doesn't do quite what I want out of the box and the code is a country mile from anything I'd like to hack on. It did serve as a decent resource for the HTTP interface to go alongside the official reference which I didn't find until later.

Fast forward a bit and I've got txflashair doing basically what I want - essentially, synchronizing the contents of the camera to a local directory. Great. Now I just need to deploy this such that it will run all the time and I can stop thinking about this mundane task forever. Time to bust out Docker, right? It is 2017 after all.

This afternoon I took the Dockerfile I'd managed to cobble together in the last hack session:

FROM python:2-alpine

COPY . /src
RUN apk add --no-cache python-dev
RUN apk add --no-cache openssl-dev
RUN apk add --no-cache libffi-dev
RUN apk add --no-cache build-base

RUN pip install /src

VOLUME /data

ENTRYPOINT ["txflashair-sync"]
CMD ["--device-root", "/DCIM", "--local-root", "/data", "--include", "IMG_*.JPG"]

FROM python:2-alpine

RUN apk add --no-cache python-dev
RUN apk add --no-cache openssl-dev
RUN apk add --no-cache libffi-dev
RUN apk add --no-cache build-base
RUN apk add --no-cache py-virtualenv
RUN apk add --no-cache linux-headers

RUN virtualenv /app/env

COPY requirements.txt /src/requirements.txt
RUN /app/env/bin/pip install -r /src/requirements.txt

COPY . /src

RUN /app/env/bin/pip install /src

FROM python:2-alpine

RUN apk add --no-cache py-virtualenv

COPY --from=0 /app/env /app/env

VOLUME /data

ENTRYPOINT ["/app/env/bin/txflashair-sync"]
CMD ["--device-root", "/DCIM", "--local-root", "/data", "--include", "IMG_*.JPG"]

Hello, the Internet. If you don’t mind, I’d like to ask you a question about photographs.

My spouse and I both take pictures. We both anticipate taking more pictures in the near future. No reason, just a total coincidence.

We both have iPhones, and we both have medium-nice cameras that are still nicer than iPhones. We would both like to curate and touch up these photos and actually do something with them; ideally we would do this curation collaboratively, whenever either of us has time.

This means that there are three things we want to preserve:

1. The raw, untouched photographs, in their original resolution,
2. The edits that have been made to them, and
3. The “workflow” categorization that has been done to them (minimally, “this photo has not been looked at”, “this photo has been looked at and it’s not good enough to bother sharing”, “this photo has been looked at and it’s good enough to be shared if it’s touched up”, and “this has been/should be shared in its current state”). Generally speaking this is a “which album is it in” categorization.

I like Photos. I have a huge photo library with years of various annotations in it, including faces (the only tool I know of that lets you do offline facial recognition so you can automatically identify pictures of your relatives without giving the police state a database to do the same thing).

However, iCloud Photo Sharing has a pretty major issue; it downscales photographs to “up to 2048 pixels on the long edge”, which is far smaller even than the 12 megapixels that the iPhone 7 sports; more importantly it’s lower resolution than our television, so the image degradation is visible. This is fine for sharing a pic or two on a small phone screen, but not good for a long-term archival solution.

To complicate matters, we also already have an enormous pile of disks in a home server that I have put way too much energy into making robust; a similarly-sized volume of storage would cost about $1300 a year with iCloud (and would not fit onto one account, anyway). I’m not totally averse to paying for some service if it’s turnkey, but something that uses our existing pile of storage would definitely get bonus points.

Right now, my plan is to dump all of our photos into a shared photo library on a network drive, only ever edit them at home, try to communicate carefully about when one or the other of us is editing it so we don’t run into weird filesystem concurrency issues, and hope for the best. This does not strike me as a maximally robust solution. Among other problems, it means the library isn’t accessible to our mobile devices. But I can’t think of anything better.

Can you? Email me. If I get a really great answer I’ll post it in a followup.