Planet Twisted

If you want to limit your working hours as a programmer you need to keep your boss happy. But what can you do if your boss seems to care more about your time in the office than how much you produce?

Not to mention the comparison to your other coworkers. If they fix ten bugs a week, but you only fix two, your boss might not be happy—even if the bugs you fixed had far more impact, and were much harder to address.

If you’re stuck in this situation it may seem impossible to reduce your working hours. But before you give up and start looking for another job, it’s worth seeing if you can improve the situation where you are.

Two reasons your boss might like long hours

If you’re going to be judged by hours you need a manager who cares about the organization’s or team’s goals. There are two possibilities about how your boss is thinking:

Hours as proxy: Your boss cares about achieving goals, and is using hours as a mental shorthand for value produced. If you actually are a valuable worker, and make sure they know it, they won’t notice your working hours, as in this real occurrence:

A programmer I know was having a conversation with their manager when the manager mentioned, in an offhand manner, that the company expected people to work 50 hours a week.

This programmer had always worked 40-45 hours a week, and the manager had never complained or noticed, because the programmer did good work. So the programmer kept their mouth shut, didn’t comment, and kept on working their usual hours.

Hours as goal: Your boss may truly only care about hours in the office number of bugs fixed, or some other irrelevant measure. Which is to say, they’re incompetent. In this case the suggestion that follows won’t work, unless perhaps you can bypass your boss and reach someone who does care about organizational goals. Usually a job with different team or organization will serve you better.

Assuming your boss only uses hours as a proxy measure, let’s see what you can do.

Starting with goals

It’s 3PM on a Wednesday, and your boss swings by your desk and asks how things are going. You explain you’re upgrading one of your JavaScript dependencies.

Your boss nods and wanders off, wondering if you’re actually doing anything worthwhile. You have just wasted an opportunity to demonstrate your value.

What’s the alternative? Starting with goals in mind.

Elsewhere I’ve talked about how starting with goals in mind will keep you focused, and is key to making you more productive. Starting with your organizational and team goals in mind can also help you both choose valuable work and explain its value to your boss.

For every task you work on, you should have a clear logical path from the big picture organizational goals, down to your team’s goals, down to your project’s goals, down to why this particular task at this particular time is a good way to advance those goals. If you can’t make that connection, if you can’t explain why you’re doing what you’re doing:

1. You may not actually be doing anything valuable.
2. Even if you are, you can’t prove it.

Let’s get back to that JavaScript dependency. If you started with goals in mind you might have decided this wasn’t a particularly useful task to begin with, and worked on something else. Or, perhaps you know exactly why you’re doing it.

In that case, the conversation might go something like this:

“You know how we’ve decided we wanted to increase user retention? Well, it looks like one of the problems is that our site is rendering way too slowly, so half our users bounce before the page finishes loading.

Turns out that font loading is the problem, and this library has a feature to fix that in its latest release. Once I’ve upgraded I should be able to get pages to render in a quarter of the time, and I’m hoping that’ll increase user retention. And I have some other ideas in case that isn’t sufficient.”

Your boss goes away understanding that what you’re doing is valuable. And if they’re anywhere near competent they won’t be thinking about the bug queue, or how many hours they’ve seen you in the office. They’ll be thinking about the good work you’re doing, and how they can tell their boss that the retention problem is being addressed.

Communicating your value based on goals

You can explain your work in this way in this way when asked, but there’s no reason not to do so proactively as well. Once a week, or once a month, you can take stock of what you’ve achieved and send an email to your boss. And you can also keep a copy of this email to update your resume when the time comes to look for a new job.

To recap:

1. Understand why you’re doing your work.
2. Choose work that addresses those goals.
3. Communicate to your boss why your work is helping those goals.

This will shift many managers from a hour mindset—driven by an assumption that your work isn’t producing that much value—to a value mindset. Your boss will know your output is valuable, and as a result won’t require the proxy measure of hours worked.

Learning how to work towards goals is, of course, easier said than done. It took me many years and many mistakes along the way. If you’d like to accelerate your learning, and take advantage of everything I’ve learned working as a programmer over the past 20 years, you can sign up for my Software Clown weekly newsletter. Every week I share a mistake and what you can learn from it.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

You’re tired of working yourself to death. You’ve had enough of the pressure of long hours, and you want not more money but more time: you want a shorter workweek. You want to work 35 hours a week, or 32, or even less.

But in an industry where some companies brag about their 70-hour workweeks, can you find a programming job with a short workweek?

The short answer is that yes, you can work 35 hours or less. I’ve done it at multiple jobs, and I know other programmers who do so as well.

The longer answer is that you have multiple different options, different ways of achieving this goal. Let’s see what they are.

Option #1: Find a job that offers shorter hours

While they are few and far between, some organizations do offer shorter workweeks. For example, over on the excellent Key Values job board you can learn about Monograph, a company that provides a 32-hour workweek by default.

Option #2: Negotiate a custom deal

Just like you can negotiate a higher salary, you can also negotiate a shorter workweek. The best place to do it is at your current job, because you’ve likely got expensive-to-replace knowledge of business logic, organizational procedures, local tech stack, and so on. But you can also negotiate for a shorter workweek at a new job, if you do it right.

If you’re interested in seeing how this is done, read my interview with a programmer who has been working part-time for 15 years.

Option #3: Become a consultant (the right way)

If you’re a consultant and you do it right, you can raise your rates high enough that you don’t need to work full time to make a living. Doing it right is important, though: if your hourly rate is low enough you’re going to have to work long hours.

To learn about some of what it takes, Jonathan Stark’s Value Pricing Bootcamp is one place to start.

Option #4: Start a product business (the right way)

If you’re selling a product that you’ve created, the hours you work don’t map one-to-one to your income. You have the upfront time for creating the product, and ongoing time for marketing and maintaining the product, but at that point you can sell the same product over and over with much smaller investment of time.

Consider for example Amy Hoy’s explanation of how bootstrapping a business allowed her to make a living even with a chronic illness.

Option #5: Early retirement

Living below your means is a good idea in general: the more money you have in the bank the easier it’ll be for you to find a better job, for example. But if you don’t want to work at all, over the long term cutting your expenses can help you stop working altogether.

Liberate yourself from the office

One of the pernicious side-effects of the culture of long hours in tech is that even a 40-hour workweek seems impossible. But long hours aren’t necessary: they’re a crutch for bad management. Working shorter hours can actually make you more productive, productive enough that your total output goes up even with shorter hours.

In the short run, if you want to work fewer hours you have to do something about it.

In the long run, there’s no reason why a 32-hour workweek couldn’t be the standard—if we all push for it hard enough.

If you want to get your time back for yourself, the time to start is now. And if you want some extra guidance on how to do it, check out The Programmer’s Guide to a Sane Workweek.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

A common way to expose an API in Python is as inheritance. Though many projects do that, there is a better way.

But first, let's see. How popular is inheritance-as-an-API, anyway?

Let's go to the Twisted website. Right at the center of the screen, at prime real-estate, we see:

What's there? The following is abridged:

class Echo(protocol.Protocol):
    def dataReceived(self, data):
        self.transport.write(data)
class EchoFactory(protocol.Factory):
    def buildProtocol(self, addr):
        return Echo()

(This is part of an example on building an echo-server protocol.)

If you are wondering who came up with this amazing API, it is the same person who is writing the words you are reading. I certainly thought it was an amazing API!

Look at how many smart people agreed with me.

Django takes a page of tutorial to get there, but sure enough:

class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')
class Choice(models.Model):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    choice_text = models.CharField(max_length=200)
    votes = models.IntegerField(default=0)

Jupyter's echo kernel starts:

class EchoKernel(Kernel):
    implementation = 'Echo'
    implementation_version = '1.0'
    language = 'no-op'

Everyone is doing it. A project I have been a developer on for ~16 years. The most popular Python web library, responsible for who-knows-how-many requests per second in Instagram. A project that won the ACM award (and well deserved, at that).

However, popularity is not everything. This is not a good idea.

When exposing class inheritance as a public interface, that means committing to a level of backwards compatibility that is unheard of. Even adding private methods or attributes becomes dangerous.

Let's give a toy example:

class Writer:

    _write = lambda x: None

    def set_output(self, output):
        self._write = output.write

    def write(self, message):
        formatted = self.format(message)
        self._write(message)

    def format(self, message):
        raise NotImplementedError("format")

This is a simple writer, that, while initially sending everything down a black hole, can be set to write the output to a file-like object. It needs to format the messages, so the proper usage is to subclass and override format (while taking care not to define methods called set_output or _write.)

class BufferWriter(MultiWriter):

    _buffer = False

    def format(self, message):
        if self._buffer:
            return 'Buffer: ' + message
        else:
            return 'Message: ' + message

    def switch_buffer(self):
        self._buffer = not self._buffer

The simplest formatting would return the message as is. However, this formatter is slightly less trivial -- it prefixes the message with the word Buffer or Message, depending on an internal variable that can be switched.

Now we can do things like:

>>> bp = BufferWriter()
>>> bp.set_output(sys.stdout)
>>> bp.write("hello")
Message: hello
>>> bp.switch_buffer()
>>> bp.write("hello")
Buffer: hello

This looks good, so far. Of course, things are never so simple in real life. The writer library, naturally, gets thousands of stars on GitHub. It becomes popular. There's a development community, complete with a discord channel and a mailing list. So naturally, important features get added.

class Writer:

    _buffer = ""

    _write = lambda x: None

    def set_output(self, output):
        self._write = output.write

    def write(self, message):
        self._buffer += self.format(message)
        if len(self._buffer) > 10:
            self._write(self._buffer)
            self._buffer = ""

    def format(self, message):
        raise NotImplementedError("format")

Turns out people needed to buffer some of the shorter messages. This was a crucial performance improvement, that all users were clamoring for, so version 2018.6.1 is highly anticipated.

It breaks, though, the BufferWriter. The symptoms are weird: TypeError s and other such fun. All because both the superclass and the subclass are competing to access self._buffer.

With enough care, these problems can be avoided. A library which exposes classes for inheritance must add all new private methods or attributes as __ and, naturally, never ever add any public methods or attributes. Sadly, nobody does that.

So what's the alternative?

from zope import interface

class IFormatter(interface.Interface):

    def format(message):
        """show stuff"""

We define an abstract interface. This interface [1] has only one method -- format.

@attr.s
class Writer:

    _buffer = ""

    _write = lambda x: None

    _formatter = attr.ib()

    def set_output(self, output):
        self._write = output.write

    def write(self, message):
        self._buffer += self._formatter.format(message)
        if len(self._buffer) > 10:
            self._write(self._buffer)
            self._buffer = ""

We use the attrs library [#] to define our main functionality: a class that wraps other objects, which we expect to be IFormatter.

We can automatically verify, by instead having the _formatter line say:

_formatter = attr.ib(validator=lambda instance, attribute, value:
                               verify.verifyObject(IFormatter, value))

Note that this separates the concerns: the "fake method" format has moved to a "fake class" (an interface).

@interface.implementer(IFormatter)
class BufferFormatter:

    _buffer = False

    def format(self, message):
        if self._buffer:
            return 'All Channels: ' + message
        else:
            return 'Limited Channels: ' + message

    def switch_buffer(self):
        self._buffer = not self._buffer

Note that now, if we only have the Writer object, there is no way to switch prefixes. Correctly switching prefixes means keeping access to the original object.

If there is a need to "call back" to the original methods, the original object can be passed in to the wrapped object. One advantage is that, being a distinct object, it is obvious one should only call into public methods and only access public variables.

Passing ourselves to a method is, in general, not an ideal practice. What we really should do, is to pass specific methods or variables directly into the method. But this is funny: when using inheritance, we always effectively pass ourselves to every method. So even this refactoring is a net improvement. When the biggest criticism of a refactoring is "this could now be improved even more", it usually means it is a good idea.

Credits:

• Thanks to Tom Goren for his feedback -- the original version was more aggressive.
• Thanks to Glyph Lefkowitz for pushing me to make the example better.
• Thanks to Augie Fackler and Nathaniel Manista for much of the inspiration.

You want to work 40 hours a week, you want to head home at 5PM, but—there’s this bug. And you’re stuck, and you really need to fix it, and you’re in the middle so you work just a little longer. Next thing you know you’re leaving work at 6PM.

And before long you’re working 50 hours a week, and then 60 hours a week, and if you stop working overtime it’ll hit your output, and then your manager will have a talk with you but how you really need to put in more effort. So now you’re burning out, and you’re not sure what you can do about it.

But what if you were more productive?

What if you knew how to get your work done on company time, and could spent your own time on whatever you wanted?

You can—with a little time management.

Some caveats

Before we get to the actual techniques you’ll be using, some preliminaries.

First, these techniques will only work if you have a manager who judges you based on results, not hours in the office. Keep in mind that there are many managers who claim they want a 50-hour workweek, but in practice will be happy if you do a good job in just 40. I’m also assuming your company is not in constant crisis mode. If these assumptions are wrong, better time management won’t help: it’s time to find another job.

Second, these techniques are here to help you in day-to-day time management. If production is down, you may need to work longer hours. (And again, if production is down every week, it’s time to find another job.)

Finally, for simplicity’s sake I’m assuming you get in at 9:00AM and want to leave at 5:PM. Adjust the times below accordingly if you start later in the day.

Taking control over your time

Since your problem is time creep, the solution is hard limits on when you can start new work—together with time allocated to planning so future work is more productive.

Here’s the short version of a schedule that will help you do more in less time:

1. When you get in to work you read your checkpoint from the previous workday (I’ll explain this in a bit).
2. Until 3:30PM you work as you normally would.
3. After 3:30PM you continue on any existing task you’re already working on. If you finish that task you can start new tasks only if you know they will take 15 minutes or less. If you don’t have any suitable tasks you should spend this time planning future work.
4. At 4:45PM you stop what you’re doing and checkpoint your work.
5. At 5:00PM you go home.

Let’s delve deeper so you can understand what to do, and why this will help you.

End of day → start of next day: checkpointing

In the last 15 minutes of your day you stop working and checkpoint your work. That is, you write down everything you need to know to get started quickly the next morning when you come to work.

If you’re in the middle of a task, for example, you can check in “XXX” comments into your code with notes on the next changes you were planning to make. If you’re doing planning, you can assign yourself a task and write down as much as possible about how you should implement it.

This has two benefits:

1. Next morning when you get to work, and even more so after a weekend or vacation, you’ll spend much less time context swapping and trying to remember where you were. Instead, you’ll have clear notes about what to do next.
2. By planning your work for the next day, you’re setting up your brain to work out the problem in the background, while you’re enjoying your free time. You’re more likely to wake up in the morning with a solution to a hard problem, or have an insight in the shower. For more about this see Rich Hickey’s talk on Hammock Driven-Development.

No new large tasks after 3:30PM

By the time the afternoon rolls by you’ve been working for quite a few hours, and your brain isn’t going to work as well. If you’re in the middle of a task you can keep working on it, but if you finish a task you should stop taking on large new tasks near the end of the day. You’ll do much better starting them the next day, when you’re less tired and have a longer stretch of time to work on them.

How should you spend your time? You can focus on small tasks, like code reviews.

Even more importantly, you can spend your afternoon doing planning:

• Take vague tasks and write down the details and sub-tasks.
• Investigate potential solutions.
• Research new technologies.
• Try to understand the underlying causes of problems you’re seeing come up again and again.
• Think about the big picture of what you’re working on.

In the long run planning will make your implementation work faster. And by limiting planning to only part of your day you’re making sure you don’t spend all of your time planning.

Going home at 5:00PM exactly

There’s nothing inherently wrong with spending a few more minutes finishing something past 5:00PM. The problem is that you’re experiencing hour creep—it’s a problem for you specifically. Having a hard and fast rule about when you leave will force you not to stay until 6:00 or 7:00PM.

Plus, sometimes it’s not just a few minutes, sometimes you’ll need more than that to solve the problem. And a task that will take two hours in the evening might take you only 10 minutes in the morning, when you’re well-rested.

In the long run you’ll be more productive by not working long hours.

A recap

Here’s a recap of how you should be spending your day at work:

• 9:00AM-3:30PM: Start by reading your checkpoint notes from the day before so you can get started immediately, then work normally.
• 3:30PM-4:45PM: Continue on existing task, if you’re finished then transition to small tasks and planning.
• 4:45PM-5:00PM: Checkpoint your work, then leave your office.
• 5:00PM-…: Whatever you want to do.

There’s nothing magic about this particular set of rules, of course. You will likely want change or customize this plan to your own needs and situation.

Nonetheless, since you are suffering from hour creep I suggest following this particular plan for a couple of weeks just so you start getting a sense of the benefits. Once you’ve taken control over your time you can start modifying the rules to suit your needs better.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

You probably already know what “unit testing” means. So do I.

But—what if our definitions are different? Does unit testing mean:

• Testing a self-contained unit of code with only in-memory objects involved.
• Or, does it mean automated testing?

I’ve seen both definitions used quite broadly. For example, the Python standard library has a unittest module intended for generic automated testing.

So we have two different definitions of unit testing: which one is correct?

Not just unit testing

You could argue that your particular definition is the correct one, and that other programmers should just learn to use the right terminology. But this seems to be a broader problem that applies to other forms of testing.

There’s “functional testing”:

• It might mean black box testing of the specification of the system, as per Wikipedia.
• At an old job, in contrast, we used the term differently: testing of interactions with external systems outside the control of our own code.

Or “regression testing”:

• It might mean verifying software continues to perform correctly, again as per Wikipedia.
• But at another job it meant tests that interacted with our external API.

Why is it so hard to have a consistent meaning for testing terms?

Testing as a magic formula

Imagine you’re a web developer trying to test a HTTP-based interaction with very simple underlying logic. Your thought process might go like this:

1. “Unit testing is very important, I should unit test this code—that means I should test each function in isolation.”
2. “But, oh, it’s quite difficult to test each function individually… I’d have to simulate a whole web framework! Not to mention the logic is either framework logic or pretty trivial, and I really want to be testing the external HTTP interaction.”
3. “Oh, I know, I’ll just write a test that sends an HTTP request and make assertions about the HTTP response.”
4. “Hooray! I have unit tested my application.”

You go off and share what you’ve learned—and then get scolded for not doing real unit testing, for failing to use the correct magic formula. “This is not unit testing! Where are your mocks? Why are you running a whole web server?”

The problem here is that the belief that one particular kind of testing is a magic formula for software quality. “Unit testing is the answer!” “The testing pyramid must be followed!”

When a particular formula proves not quite relevant to our particular project, our practical side kicks in and we tweak the formula until it actually does what we need. The terminology stays the same, however, even as the technique changes. But of course whether or not it’s Authentic Unit Testing™ is irrelevant: what really matters is whether it’s useful testing.

A better terminology

There is no universal criteria for code quality; it can only be judged in the context of a particular project’s goals. Rather than starting with your favorite testing technique, your starting point should be your goals. You can then use your goals to determine, and explain, what kind of testing you need.

For example, imagine you are trying to implement realistic looking ocean waves for a video game. What is the goal of your testing?

“My testing should ensure the waves look real.”

How would you do that? Not with automated tests. You’re going to have to look at the rendered graphics, and then ask some other humans to look at it. If you’re going to name this form of testing you might call it “looks-good-to-a-human testing.”

Or consider that simple web application discussed above. You can call that “external HTTP contract testing.”

It’s more cumbersome than “unit testing,” “end-to-end testing,” “automated testing”, or “acceptance testing"—but so much more informative. If you told a colleague about it they would know why you were testing, and they’d have a pretty good idea of how you were doing the testing.

Next time you’re thinking or talking about testing don’t talk about "unit testing” or “end-to-end testing.” Instead, talk about your goals: what the testing is validating or verifying. Eventually you might reach the point of talking about particular testing techniques. But if you start with your goals you are much more likely both to be understood and to reach for the appropriate tools for your task.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

You’ve just started a new job and you’re not productive yet: you get stuck, you need help, you don’t know enough. You need to learn a new codebase, a new set of conventions, a new set of business problems. You might even need to learn a new programming language.

And so now you feel anxious—

Are you doing well enough?

Are you producing enough code?

How is your manager feeling about your progress?

It’s natural to make yourself feel more comfortable by working overtime. You’re showing your manager that you’re trying, at least, and by working long hours you might get a little bit more done. You don’t want to work overtime in the long run, of course, but you can worry about that in the future.

Unfortunately, working long hours is—as you might suspect—the wrong solution: at best it won’t help, and it might even make your situation worse. Let’s see why overtime isn’t helpful, and then move on to a better solution: a solution that will make you more productive and make you look good to your manager.

Long hours won’t solve your problem

Working overtime might make you feel a little better. Unfortunately it’s also a bad solution in the short run, and a big problem in the long run.

In the short run, you’re not actually going to get more done. Long hours will just tire you out, won’t help you learn any faster, and pretty much are never the solution to producing more (here’s some research if you don’t believe me). Even worse, you might end up giving your manager the wrong impression: you’re working long hours and you’re still not productive yet?

In the long run, you’re setting bad expectations about your work hours. If you have a mediocre manager, let alone a bad one, they will often expect you to keep working those long hours. You need to set boundaries from the start: “here are my work hours, I won’t work more outside of emergencies.”

There’s a better solution: focusing on your real goal, which is learning everything you need to know about your new project.

The real solution: learning with feedback

You have two core problems:

1. You need to learn a lot, and you don’t necessarily even know what you need to learn.
2. You can’t demonstrate you’re being productive to your manager the usual way, by fixing bugs or adding features.

You can solve both problems at once with the following process:

1. Every Friday, with your week’s work still fresh in your mind, write down:
   • Everything you’ve learned that week.
   • What you think you need to learn next.
2. First thing Monday morning when you get back to work, send an email to your manager with what you wrote Friday, and an additional question: “What is missing from this list? What else do I need to learn?”
3. Your manager can now provide you with feedback about additional things you need to learn.
4. When you get stuck and don’t want to ask for help just yet, take a break and go learn something on your list.

If you follow this process:

• Your manager will know you’re not slacking off.
• You’ll get feedback about your progress and what to do next.
• You’ll be better focused on learning the right things first, which will make you productive faster.

And of course, no overtime required.

Want more suggestions for getting started on your best foot? Last time I started a new programming job I created a personal checklist: all the things I should be doing on my first few days at work. If you’d like to read it, you can download it here.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

Assume MyClass._dangerous(self) is a private method. We could have implemented the same functionality without a private method as follows:

• Define a class InnerClass with the same __init__ as MyClass
• Define InnerClass.dangerous(self) with the same logic of MyClass._dangerous
• Make MyClass into a wrapper class over InnerClass, where the wrapped attribute is private.
• Proxy all necessary work into InnerClass.

This might seem onerous, but consider that now, dangerous is part of the public interface of a class, and would need to be explicitly documented as to assumptions and guarantees. This documentation would have had to have been in comments around _dangerous anyway -- in order to clarify what its assumptions are, and what invariants it is violating in MyClass -- otherwise, maintaining the code that calls _dangerous would be hard.

Of course, this documentation is almost certain to be missing. The existence of _dangerous itself implies this was an almost mechanical refactoring of similar code into a method, with the excuse of "its private" used to avoid considering the invariants and interface.

Even if the documentation did exist, now it is possible to unit-test that the documentation is correct. Furthermore, if we use best practices when we define MyClass -- in other words, avoid creating an InnerClass object in the initializer, and only creating it in an MyClass.from_parameters, we are also in a good position to unit test MyClass.

This, of course, presented the worst case: the code for _dangerous touches absolutely every data member of MyClass. In real life, the worst case is not often encountered. When we look at a private method as a code smell, and contemplate the best way to refactor it away, it turns out that we often can find a coherent set of attributes that really does make sense as InnerClass on their own merits.

Credit: This is based on an off-handed comment Glyph made in his blog post about attrs. I am also grateful to him for reviewing a draft copy of this post, and making many useful suggestions. All mistakes in interpretation or explanation are mine alone.

Your manager keeps telling you that you’re not getting enough done. So you decide to become more focused, since as everyone knows, to be a productive programmer you need to stay focused. Deep-diving into TV Tropes, chatting with your friends, or reading up on that fancy new web framework might be fun, often even educational, but they won’t get that feature you’re working on out the door.

So you get noise canceling headphones, and only read your email once a day, and use the Pomodoro technique, and became laser-focused on your code—but still, you’re not productive enough. Your colleague across the hall doesn’t write code faster than you, and yet somehow they make more of an impact, they get things more done. You know it, and your manager knows it.

Why?

Because staying focused is not enough to make you productive. In fact, it’s often the other way around: staying focused is a side-effect of what truly makes you productive.

• To understand why staying focused isn’t enough, we’ll take a detour from programming and go visit my past self: a young soldier being escorted into a military jail.
• Then, we’ll apply the lesson we learned and see how understanding your goals is key to becoming more productive, and how your goals can help you stay focused.

A short visit to a military jail

Imagine a yard full of dirty gravel, and mixed in with the gravel are tiny twigs, trash, and the like. How long could you spend crawling around looking for this debris before you’d get bored? How long could you stay focused?

Long ago I lived in Israel, and as a Jewish citizen I was required to serve three years in the military. For a variety of reasons, personal and political, I had no interest in becoming a soldier, and so I attempted to avoid conscription by getting a medical discharge for mental health reasons. While on the base I was part of a transients’ unit on the military base: we would clean bathrooms and the like while awaiting processing.

As our story unfolds, I was having a very bad day. My attempt to get a discharge was failing, as the military psychiatrist had decided there was nothing wrong me. And to make things worse, the sergeant who ran the unit wanted me to go off and do some work on the base, and I couldn’t deal with it.

So I said “no"—which to say, I refused orders, serious business in the military. The sergeant organized a quick trial, and the officer in charge sentenced me to a day in the on-base jail. Perhaps for entertainment, perhaps to enforce the importance of obeying orders, while I was in the jail my guards ordered me to search for little bits of tiny debris that were mixed in the jailyard’s gravel.

And so I spent quite a while, crawling around on my knees in the rain, working hard at a pointless task. The guards were impressed, and eventually they felt bad enough to give me an umbrella to keep the rain off.

The moral of the story

I started this episode by refusing to work, and refusing work that had some purpose (washing dishes, or cleaning a bathroom). I ended by working hard doing something that was a complete waste of time.

Both choices were good ones, because in both cases I was working towards my goals:

1. My broadest goal was getting kicked out of the military. Cooperating was doing me no favors: spending some time in jail for refusing orders demonstrated I was not going to be a good soldier.
2. My secondary goal was minimizing the amount of time I spent in jail. I had met a soldier on base who had spent his time in jail getting in trouble with his guards, so he’d been sentenced to even more time. He ended up spending months on a military prison base. I wanted to be a model prisoner, so I could get out of jail as quickly as possible.

Staying focused and avoiding distractions is all fine and good, so long as the work you’re doing actually helps you achieve your goals. If it’s not, you’re staying focused on the wrong thing. I could have stayed focused by following orders—and that would have been the wrong way to achieve my goal of getting kicked out of the military.

Plus, knowing your goals can help you stay focused. If you don’t care about your task, then you’ll have a hard time focusing. But once you do understand why you’re doing what you do, you’ll have an easier time staying on task, and you’ll have an easier time distinguishing between necessary subtasks and distracting digressions. And that’s why I was able to enthusiastically clean debris from gravel.

This then is the key to achieving your goals, to productivity, and to staying focused: understanding your goals, and then working towards them as best you can.

Applying your goals to staying focused

So how do you use goals to stayed focused?

1. Figure out the goals for your task.
2. Strengthen your motivation.
3. Judge each part of your work based on your goals.

1. Discovering your goals

Start with the big picture: why are you working this job? Your goals might include:

• Money: Getting paid so you can buy food and shelter.
• Social pressure: You want your coworkers and boss to think well of you.
• Organizational goals: You believe in what the company is doing.
• A sense of obligation: You want to help your customers or users.
• Building and playing: Solving a hard problem is fun.
• Curiosity: Learning is fun too.

Then focus down on your particular task: why is this task necessary? It may be that to answer this question you’ll need do more research, talking to the product owner who requested a feature, or the user who reported a bug. This research will, as an added bonus, also help you solve the problem more effectively.

Combine all of these and you will get a list of goals that applies to your particular task. For example, let’s say you’re working on a bug in a flight search engine. Your goals might be:

1. Money: I work to make money.
2. Organizational goal: I work here because I think helping people find cheap, convenient flights is worth doing.
3. Task goal: This bug should be fixed because it prevents users from finding the most convenient flight on certain popular routes.
4. Fun: This bug involves a challenging C++ problem I enjoy debugging.

2. Strengthening your motivation

Keeping your goals in mind will help you avoid distractions, and the more goals you’re meeting, and the more your various goals point in the same direction, the better you’ll do. If you have weak or contradictory goals then you can try different solutions:

• If you work for a company whose goals don’t mean much to you, then you’ll have a harder time focusing: consider finding a new job where you’re doing something you care more about.
• If after enough research you’ve decided your task is pointless, you can either try to push back (mark the bug as WONTFIX, go talk to the product manager), try to add an additional motivation (is this a good opportunity to learn something new?), or just live with the fact that it’ll take you longer to implement.

3. Judging your work

Understanding your goals will not only help you avoid small distractions (noise, TV Tropes), but bigger distractions as well: digressions, seemingly useful tasks that shouldn’t actually be worked on. Specifically, as you go about solving your task you can use your goals to judge whether a new potential subtask is worth doing.

Going back to the example above, imagine you encounter some interesting C++ language feature while working on it can be tempting to dive in. But judged by the four goals it will only serve the fourth goals, having fun, and likely won’t further your other goals. So if the bug is urgent then you should probably wait until it’s fixed to play around.

On the other hand, if you’re working on a pointless feature, your sole goals might be "keep my manager happy so I can keep getting paid.” If you have two days to do the task, and it’ll only take two hours to implement it, spending some time getting “distracted” learning a technical skill might help with a different goal: switching to a more interesting position or job.

Start with your goals

Once you know goals, you can actually know what it takes to be productive, because you’ll know what you’re working towards. Once you know your goals, you can start thinking about how to avoid distractions because you’ll know you’re doing work that’s worth doing.

Before you start a task, ask yourself: what are my goals? And don’t start coding until you have an answer.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

If it’s been months since you had a day where you feel good—

If you hate getting out of bed in the morning because that means you’ll have to go to work—

If your job is tiring you out so much you can’t get through the day without a nap—

It’s time to quit your shitty job. It’s time to quit your shitty job and go someplace better, a job where a good night’s sleep is all you need. A job where you’re valued. A job where people don’t shout at each other, or demean you, or destroy the project you’ve put all your energy into.

But quitting can be difficult: you have a sense of commitment, the fear of change, the indecision about whether your job is really that shitty. So to help you make your decision, and quit in the best possible way, in the rest of this post I will cover:

1. Identifying a shitty job.
2. Whether you should quit (spoiler: yes).
3. Preparations you should make before quitting: legal, bureaucratic, social.
4. When to quit.
5. How you should quit.

(Note that some of this will be US-centric, since that’s where I live and what I know best.)

Identifying a shitty job

Shitty jobs can be surprisingly hard to identify.

Sometimes this is because you don’t have a reasonable baseline, or the shittiness has become normalized through exposure. I’ve heard of companies with the following symptoms, for example, and I would consider either grounds for immediately starting a search for a better job:

• People shouting at each other during meetings on a regular basis.
• Getting paid late. Money for working hours is the basic contract of employment: if you’re paid late more than once you’re being told that contract isn’t important.

Another reason you might not notice you have a shitty job is a subtle shift over time. A good job slowly gets worse, and your existing relationships and loyalty blind you to the symptoms—for a while, anyway. You might be forced to reconsider due to:

• Layoffs, especially while the company still continuing to hire.
• Managers being hired without being interviewed by their future direct reports.

I could go on with other examples, but there are two core themes here:

1. Your company doesn’t value its employees.
2. You don’t trust company management in the aggregate.

Again, this may not always have been the case. You may trust many of your managers, and know that they value you and your coworkers. But things change, and not always for the better: what matters is the way the company is now, and who has power now, not the way it used to be.

Should you quit your shitty job?

Yes.

But you should do so at the right time, and with a little preparation.

When should you quit your shitty job?

Ideally, you should have another job lined up before you quit.

I once had to give notice of quitting unexpectedly, without prior planning. A more observant coworker gave notice the same day, but they had started looking a couple months before, when we had a round of layoffs. So while I spent a couple months not getting paid, they moved straight on to another job. The lesson: it pays to look for early signs of shittiness, so that you can leave in the best possible way.

Once you realize you have a shitty job, you should start interviewing elsewhere. Having an existing job improves your negotiation position, since you always have the implicit alternative offer of staying where you are. Two offers you can play against each other, or “I’m far along in interview process with another company” is better, but lacking that you need to downplay how shitty your current job is.

You’ll want a break to catch your breath and relax in between jobs: you can easily negotiate a couple of weeks time off in between jobs. A month shouldn’t be much harder to get.

In practice, your job may be so awful that it leaves you with no time or energy to look for another job. In this case you might be forced to quit without a new job lined up. You can prepare for this by living below your means and saving some money.

Preparing for quitting

Here are some things you should do before quitting any job:

• Get non-work contact details for all your coworkers.
• Maximize any benefits you can. When I quit a job with a 401k and donation matches, I maxed those out early in the year. Note that in small enough companies HR might notice when you change 401k contributions.
• Try to get continued access to your company’s open source projects that you might want to work on after you leave. Often asking is sufficient: I once asked the VP of engineering after I gave notice, and was told I could keep commit access (presumably because I was effectively offering to do work for free).
• Write down details about your work that can help make your resume look better: specific numbers you improved (sales, performance, costs), and the like. If the company has an overly broad definition of proprietary information you might not be able to put them on your resume—but the company might fold one day, so it’s good to have a reminder of what you did.

At a shitty job you may also need to make copies of some documents: specifically, any emails or other documents where promises are made to you re pay, benefits, and so on. Once you’ve been paid what you’re owed and you’ve left your job, you won’t need those anymore and they should be deleted or shredded. But when it’s your last day at work and you’re trying to get the back pay they owe you, you want to make sure you have documentation.

Speaking of back pay, if you work for a company that has an “unlimited vacation” policy, take some vacation before you quit. You’re not going to get paid for those vacation days you haven’t taken. (In general, if a company has “unlimited vacations” I recommend taking lots of vacation throughout the year, since it’s use or lose it.)

How to quit

It’s a shitty job, and you may be utterly relieved at leaving it, but—you should quit politely. Your management may simply be misguided, or suffering under pressures you don’t understand (VCs in cover-your-ass mode can be quite destructive). Your manager might grow as a person. Your co-workers might end up working with you again.

So just give your notice, with the smallest possible amount of time you have to stay there. You can tell close coworkers why you’re leaving (they probably already know). And on your last day of work just leave, quietly and politely.

For a while you will feel sad: those projects will never get finished. But mostly you will feel relief.

It’s time—

—time to quit your shitty job.

As I mentioned above, I once made the mistake of hanging on when I shouldn’t have, unlike a more clued-in coworker. (You can hear the whole story by signing up for my Software Clown newsletter, where I share 20+ years of my mistakes so that you can avoid them.)

Don’t make my mistake. I had to quit anyway, and without the benefit of advance planning or having a job lined up. Start looking for a new job now, while you’re still able to hold on—your job probably probably won’t be getting any better.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

We use quay.io a fair bit at work—all our internal Docker images are stored there. I like it a lot, but the website makes it really hard to see who can access your repositories.

In particular, if someone ever leaves your organization, you have to click through all of your repositories one at a time to see whether they have been granted access to a repository as an individual, rather than as a member of a team. This might be OK if you have two or three repositories, but not if you have hundreds.

I had some spare time today, so I wrote a tool to help with this. It’s called quay-admin and you can install it now:

$ pip install quayadmin

This will give you a command-line tool called quay-admin that you can run to see which users outside of your organization have access to your repositories.

I originally tried to write it in Go, basing it off my colleague’s excellent quay-exporter project—a tool that turns security vulnerability warnings into Prometheus metrics so you can get alerted. Unfortunately, getting Go to work well with Swagger APIs is a bit fiddly, and I didn’t have that much spare time. So I tried Python, knowing that it has excellent libraries for working with RESTful services.

First cut used requests, which helped me figure out which APIs I needed and how they gave me the data I wanted. Next version used treq, which allowed me to parallelize, which saves precious seconds of my only life.

It’s been an age since I’ve written Twisted code, but it all comes rushing back fairly quickly. I’ve found that I miss certain things from Haskell’s async library, notably mapConcurrently, but they are easy enough to add.

Releasing Python code is way different though. At Glyph‘s recommendation, I tried flit, which seems to work OK.

Thanks to dstufft, glyph, dreid, AlexGaynor, wsanchez, and others who patiently answered my questions while I was writing this, and who in some cases wrote much of the actual software I am building on top of.

Thanks also to quay.io for actually publishing their API docs. It genuinely helps.

We would like Twisted to support contextvars -- this would allow cross-async libraries, like eliot to do fancy things.

Klein is almost ready to be used as-is. Glyph has the good branch which adds

• CSRF protection
• Forms
• Sessions
• Authentication

But it is too big, and we need to break it to reviewable pieces to add it to master.

The other option for a Twisted-native web framework is Cyclone. It is not under heavy development, but this is mostly because it is done and reasonably stable: Duo Security is using it in production.

We are slowly improving the Request object by taking it out of the built-in and reimplementing it externally. Wilfredo is doing it in a side-project.

We talked a little about advanced use cases: How do you use a reactor in a non-main thread? The only marginally documented installSignalHandlers argument does that just fine.

If you want to spread processing between multiple processes, Ampoule does that. Help is greatly appreciated.

If you want to do weird things with resources, Moshe did something on Twitch this one time.

We made sure everyone knows their help would be appreciated, and gameified: Review tickets and participate on the mailing list.

Remeber: the book Expert Twisted is available for pre-orders!

We started out the conversation with talking about writing good Dockerfiles. There is no list of "best practices" yet. Hynek reiterated for us "ship applications, not build environments". Moshe summarized it as "don't put gcc in the deployed image."

We discussed a little bit what we are trying to achieve with better docker files. Shared base? Reproducible builds?

We talked about some of the challenges for building Docker on CI systems, especially from inside containers.

Docker on air-gapped machines is hard. So many parts assume free access to the internet.

We went on to discuss how to use multistage Dockerfiles. One important bit is what "installable artifact" to move. Some suggested wheels. Moshe suggested Pex. Hynek suggested copying a virtual environment, and Moshe showed an example

There was some discussion on making small images. The consensus was that Alpine is usually part of the answer.

There was a lot of discussion on the trade-offs between updating too soon, and too late. Some of the techniques to control update times were mentioned:

• Building everything from source
• Hashing various inputs into the image tag
• Using Red Hat Satellite

We talked about GPU containers, for machine learning. Apparently nvidia-docker is still nascent but works.

We talked about how to keep your registry clean. Unfortunately, the consensus is that you will need to build your own tooling.

We discussed what registries people use.

• GitLab Docker Registry got mixed reviews.
• ECR was a popular option.
• Nexus was mentioned.

We touched lightly on performance. Docker can use either overlayfs vs devicemapper. It's complicated

Would you run your DB in Docker? Docker is just a packaging format. You can run Postgres in Docker just fine, and mount in the data directory. However, usually people are asking about using Orchestration Frameworks for that.

StatefulSets in K8s are sometimes useful for databases.

If you are running your dev DB in Docker, data is not important. In that case, consider using eatmydata to improve performance.

We all agreed you should never use the system Python for your applications. Then how do you get Python in your Docker image?

• Use the python:<something> images on Docker Hub
• Compile it yourself
• Use PyEnv
• Use the deadsnakes PPA on Ubuntu

Finally, we discussed the ultimate heresy: running more than one process inside your container. Or is it? Moshe mentioned that anyone running uwsgi or gunicorn is already running a process manager: just one that happens to be part of the WSGI "binary". We mentioned supervisor and NColony for explicit process management.

The following guest post is by Moshe Zadka, explaining the importance of networking and how you can do it with minimal time outside work.

A good professional network is a long-term asset. When you’re looking for a new job you can talk to people you know, ask them if their company is hiring, and then have them submit your resume directly to the hiring manager. This will allow you to skip the “resume filter”, and often get you past the phone screen as well.

But even if a professional network is useful, how do you find the time to build it? You probably don’t want to have to get out every evening to a social gathering, and spend hours talking, just to plan for a hypothetical future job.

One way to start building your professional network with little time spent is by focusing on your current job. Over time your colleagues will leave for other jobs, and every former colleague is a potential referral to another company. So you should always make sure you have non-work contact details for your colleagues.

Unfortunately, you can only expand your network so much from attrition at work: if you want a larger network, you will have to do some work. But by making judicious use of your time, and going to the right venues, you can grow and maintain a good professional network while still only spending one or two evenings a month at events.

How networking helped me

In one of the San Francisco Python meetups, I met someone working at PayPal– a company I had no interest working for. However, we kept in touch. At some point, he moved to a start-up. At another point, I found myself looking for a job, after a company shutdown. Because I had kept in touch with him, it was easy to reach out.

Even though I was on a tight timeline for getting another job, he made sure I was fast-tracked through the process – a pro-forma resume review, and skipping the phone screen. I still had to go through a half-day’s worth of interview panel, but removing the simple filters from my path probably saved a week or more worth of being unemployed, and also let me put pressure on other prospective employers to fast-track me.

Business cards

Business cards sound like an antiquated thing, something you might see on “Mad Men”. However, even with modern smartphones, there is no faster way to share your contact details with someone you’ve just met. For that, a business card’s most important part is your e-mail address.

In all of the opportunities below, give people a business card when the conversation is done. Making your own cards is free to cheap nowadays, no need to wait for your job to print you one. In any case, you want to make sure you have your personal e-mail on the card, not your work e-mail.

Conferences

If you are already employed, ask your job to send you to relevant conferences. Some places have a budget for “professional development”, others have funds specifically marked “conferences” – or maybe it’s under the recruiting budget. Choose a relevant conference, and remind your manager that sending you to the conference is a form of training: a great investment in employees.

Some companies will only fund your trip if you speak at a conference. Most conferences understand that some people will only come if they speak, and structure their timeline accordingly: you’ll know whether your talk is accepted far enough in advance to get your manager to sign off on sending you. An efficient way to send talk proposals is to recycle – if a talk is declined from one conference, it is fine to send it to another closely related one, although sometimes it will have to be tweaked slightly. If the audiences are sufficiently distinct you can even reuse a talk you’ve already given.

Once you are at a conference, attend birds of feather sessions, and try to sit with new people for conference meals, if those are served. This is a great way to meet more people at the conference. Giving a talk is also a great way to meet more people: you can often meet other speakers, and many people in the audience will want to talk to you afterwards.

Meetups

Many places have tech meetups in the evening. You can probably find time to go to a meetup once a month. If you do go, make sure to make the most of your time – mingle, talk and hand out your business card.

Avoid going to the same meetup month after month – while it is comfortable, it tends to be the same people: your goal should be to expand your network. So once you stop meeting new people, switch to a new meetup.

As with conferences, giving a talk at a meetup is great way to meet new people. You might even be able to work on your talk at work, if you can pitch it as a recruiting event to your manager.

Engineering blog

If your company has an engineering blog, participate. Find something you have done recently which was interesting or surprising, and write about that experience.

If your company does not have an engineering blog, see if you can make one happen. It helps with recruiting, and helps people develop in their career.

Keeping in touch

Keep in touch with your network. If you come across an article relevant to someone, send it to them with a note “thought it might be interesting”. Often they will already have read it, but will be interested in sharing their thoughts.

Summary

Developing and maintaining a professional network does not need a huge time investment – a little bit goes a long way, if properly allocated. And when the day comes that you need a new job, that small investment will pay off. While you usually can’t just get a job just by knowing someone, a network will help skip past companies “resume filters”, and you can have a more streamlined interview process if you have a friend on the inside.

Moshe is a core developer on the Twisted project, and has been using Python for 20 years. He’s just published a book, “from python import better”.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

I am pleased to announce that the recent TLS certificate problems and outages to jml.io have been fully resolved.

Here are some notes on what happened and what I did about it.

Announcment: My book, from python import better, has been published. This post is based on one of the chapters from it.

When Python started out, one of the oft-touted benefits was "batteries included!". Gone were the days of searching for which XML parsing library was the best -- just use the one built-in to the standard library. However, the standard library can only hold so much special purpose stuff. Few now remember, but it used to have SGI Audio specific functionality.

These days, one of the biggest benefits of Python is the extensive third-party repository of stuff. This is the Python Package Index (PyPI), formerly known as the "Cheese Shop" after an obscure Monty Python skit. Of course, what else would be available from the Cheese shop than wheels of cheese? But a second pun was hiding behind the term "wheels": those are the things that need no reinvention!

The new PyPI warehouse launched, with new code hosting unbelievable amounts of content: around 140,000 packages at times of this post (unless I take too long in publishing it, and then who knows how big PyPI will be!)

Nobody can sift through 140K package descriptions, of course. A short-lived attempt to have "Stars" fell victim to allegations of ballot stuffing and moderation, and was quickly removed. Searching on key words would be useful, but searching without sorting rarely is -- and what would you sort on?

PyPI is not the place to find which libraries are useful. It is the place to find objective truths: which version is the latest, when was it released, what is a project's homepage, etc. Recommendations are best found elsewhere.

The first place I like to start is with the Awesome Python list. However, it is important to note that its contribution guidelines are just "submit a link" and there is no official way to remove a library from the list. Thus, the "awesome" in the name means "someone once thought it was awesome, and cared enough to add it". The list should be treated as mild suggestions. Before using a library, check release history, GitHub health, code quality and other metrics you might care about.

Another useful resource is Planet Python. It is a feed aggregator of various blogs. Many of the blog posts will feature either a recommendation of a particular library, a release announcement, or just discussion which involves using a third-party library. Along side the written word is the live performance -- PyVideo links to more Python talks than you can shake a stick at, aggregating talks from conferences around the world: again, many of the talks will feature discussion of a particular third-party library.

Last, but not least, the live, interactive, version of PyVideo: Python meetups and conferences. Those are where I discovered some of my favorite libraries.

We’ll get to the iPad and CAP theorem soon, but first, let’s talk about negotiation:

• Your boss hands you a project with an impossible deadline–so you end up working evenings and weekends.
• You get a job offer that’s lower than you’d like–and you accept it.

And that’s just the way things are, and it’s not like there’s anything you can do, right?

Maybe. But quite possibly there is something you can do.

To explain why, I’d like to share an edifying tale, a story of broken promises and ultimate–albeit minor–triumph. Along the way we’ll take a detour into distributed systems theory, and when we’re done there will even be a moral (hint: it’s about negotiation).

A purchase is made

Once upon a time, at a different job that subsidized such things, my wife purchased an iPad. Years passed, operating systems were upgraded, and over time this iPad became too slow to run some apps, and too old to run others.

It was time to buy a new iPad.

The day Apple released their new 2018 iPad we went to Apple’s online store and purchased a refurbished 2017 model. We got a confirmation email, and looked forward to a tablet that could keep up with many companies’ unwillingness to ship performant code (looking at you, Skype).

The next day Apple sent us another email: our order had been canceled, and our money would be refunded. When we checked the store, the refurbished model was no longer in stock.

Our somewhat cheaper iPad was not coming.

A theory is introduced, with some references to distributed systems

Why did the Apple Store cancel our order? Perhaps it was a bug, but I have another theory: the constraints of the CAP theorem.

Eric Brewer’s CAP theorem states that a distributed data store–a system composed of multiple nodes–can only have two out of three properties:

1. Consistency: all nodes have same view of the data.
2. Availability: the system can respond successfully to clients.
3. Partition-tolerance: if the network between the nodes fails, the system can continue to operate.

Now, the online Apple Store is quite likely to be a distributed system, given the need to scale to many users. And it needs to store data, the size of the inventory of each item in the store. Given a choice between those three properties, the only two reasonable choices are availability and partition-tolerance.

It’s far better to have a store that is available than to have completely consistent tracking of inventory. There is a cost to this choice, though: every once in a while a large rush of orders will cause inconsistent views of the available inventory.

• Node A thinks there is one iPad left, and sells it to customer 1.
• At the very same time, Node B thinks there is one iPad left, and sells it to customer 2.

Because the system can’t enforce consistency, the same iPad is sold to two people. What to do?

One common solution (alas, I can’t find the original paper where I read the idea) is “compensation” or “apology”: out-of-band business processes to repair the mistakes. In this case, a post-processing stage that notices the double-sold iPad and handles it somehow.

How this rare but inevitable mistake is handled is a policy decision, and Apple’s chosen policy is to simply cancel the order–contrary to a guarantee they make on their website.

A complaint is made

If you go to the Apple Store website’s refurbished section, you will see in small letters at the top that “availability is guaranteed once we receive your full payment.” Given that promise, and the fact we’d gotten a confirmation email for our payment, my wife called up customer service and politely asked why our order was canceled.

The representative went off, and after some delay she indicated that she’d talked to her manager and she’d gotten approval to send us a new iPad instead. So the next week we received a 2018 iPad, while only paying the cost of a refurbished 2017 model.

Success, and easy success at that that. My wife didn’t have to complain loudly, point out Apple was in the wrong, or hassle anyone. She just asked.

The topic of negotiation is reintroduced

Apple made a promise (payment == guaranteed delivery), and then violated it. Why? Violating the guarantee on their website was an opening offer in a negotiation.

When you’re negotiating, you need to ask for what you want, or you won’t get it. In practice most people won’t ask and won’t complain, and so it’s in Apple’s interest to start with a low offer: most people will get the email canceling the order, grumble a bit, and re-order something else.

Often how you ask is also important: you need to ask the right way. I once had a $5000 medical expense denied over and over by a health insurance company. Eventually I indicated I wished to file a grievance–a bureaucratic procedure I found on their website–and suddenly they found a “coding error” that had caused the problem and my bill was paid. (My uninformed guess is that grievances get reported to state regulators.)

The same dynamic is part of how many companies take advantage of employees, and the cost you’ll pay for not asking is even higher. When a company gives you a job offer they will have a salary range they are willing to pay, but they will usually not offer their high number. Instead, they will offer you the lowest number possible they think you might accept. Sometimes that number will also be far below market rate.

Many people will simply accept that initial salary offer–I’ve certainly made that mistake–and therefore end up getting paid much less than they could otherwise get. But of course that’s just an initial offer, and simply asking for more will usually result in a higher offer. And you can do even better if you do a little work beforehand.

A summary is presented

Here’s what we’ve learned:

1. Recognize negotiating situations: you’ve already lost if you don’t realize you’re negotiating.
2. Ask for what you want, or you won’t get it.
3. Ask in the right way.

For a useful guide to negotiating your salary and benefits in general, see Valerie Aurora’s negotiating guide. And if it’s your working environment you care about–working shorter hours, working from home, and the like–check out The Programmer’s Guide to a Sane Workweek.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

• The dropping of Python 3.3 support.
• Python 3 fixes (notably to trial -j, asyncioreactor, conch, and mail)
• Python 3 TCP speed improvements (less copying when sending data)
• Better TLS curve selection support for both old and new OpenSSLs
• IPv6 fixes for WSGIResource
• 60+ closed tickets with many fixed bugs!

Your project deadline is getting closer and closer, and you’re stuck: you don’t know what to do. Your manager won’t help, they just push you to work evenings and weekends–and when it looks like you’re going to fail, they hand the project over to another programmer.

You’ve failed your manager, and there’s a little voice in the back of your head telling you that maybe you’re missing what it takes to be a programmer, maybe you just don’t have the requisite natural ability.

That little voice is lying to you.

It’s the other way around: your manager has failed you, and is compounding the failure by conveying a destructive mindset, what’s known as a fixed mindset. To understand what I’m talking about, let’s take a quick detour into the psychology of education, and then return to those long hours you’ve been working.

Growth mindset vs fixed mindset

Carol Dweck, a professor of psychology at Stanford University, contrasts two mindsets when learning:

• If you have a growth mindset you assume your abilities can change over time, and that your skills can improve by learning and practice.
• If you have a fixed mindset you assume your abilities are fixed: some people are naturally more talented than others, and there is no way to change your level of abilities.

According to Dweck’s research, students with a growth mindset do better than students with a fixed mindset. A fixed mindset is self-defeating: it keeps you from learning, and it keeps you from even trying to learn.

In the context of programming this suggests that starting with the attitude that programming is a set of skills, skills that you can learn, will result in a much better learning experience.

But what does this have to do with long working hours?

You can’t work smarter, so you gotta work longer

In an article that was one of the inspirations for this post, Carol Dweck points out that a common failure mode among educators is to praise effort, working harder, instead of praising learning. While they may claim to be encouraging a growth mindset, they are simply perpetuating a fixed mindset.

This failure mode appears in the software world as well. Let’s assume for the moment that programming is a natural ability: just before we’re born, the Angel of Software injects between 0 and 100 milliliters of Programming Juice into our soul. If you’re really lucky, you might even get 110ml!

Now, given that each and every one of us only has a limited amount of Programming Juice, how can you maximize our output? You can’t learn more, so there’s no way to do things more efficiently. You can’t improve your skills, so there’s no way to become more productive.

So what’s left?

All together now: WORK LONGER HOURS!

Working longer ain’t smart

The truth, of course, is that there is no Angel of Software, there is no Programming Juice. Programming is just a bunch of skills. You can learn those skills, and so can most anyone else, given motivation, time, support, and some good teachers. And you can become more and more productive by continuing to learn.

If you believe in fixed talent, if you believe you can’t improve, you won’t try to learn. Long hours will be the only tool left to you.

When faced with a problem: just work longer hours.

When faced with another problem: work even longer.

You’ll work and work and work, and you’ll produce far less than you would have if you’d spent all that time improving your skills. And eventually you’ll hit a problem you can’t solve, and that you will never solve by working longer hours.

A growth mindset will serve you far better. You need to believe that skills can grow, and then you need to actually do the work to learn more and grow your skills.

And when you fail–and you will fail, because we all fail on occasion–take this as another opportunity to learn: look for the patterns and cues you should have have spotted. Having learned your lesson, next time you’ll do better.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

(Thanks to Glyph Lefkowitz for some of the inspiration for this port, and to Mahmoud Hashemi for helpful comments and suggestions. All mistakes and issues that remain are mine alone.)

The Python REPL has always been touted as one of Python's greatest strengths. With Jupyter, Jupyter Lab in its latest incarnation, the REPL has been lifted into the 21st century. It has become the IDE of the future: interactive, great history and logging -- and best of all, you can use it right from your browser, regardless of your platform!

However, we still have 20th century practices for developing web applications. Indeed, the only development is that instead of "CTRL-c, up-arrow, return", we now have "development servers" which are not "production ready" support auto-reloading -- the equivalent of a robot doing "CTRL-c, up-arrow, return".

Using the REPL to develop web applications is pure bliss. Just like using it to develop more linear code: we write a function, test it ad-hocly, see the results, and tweak.

When we are sufficiently pleased, we can then edit the resulting notebook file into a Python module -- which we import from the next version of the notebook, in order to continue the incremental development. Is such a thing possible?

Let's start by initializing Twisted, since this has to happen early on.

from tornado.platform.twisted import install
reactor = install()

Whew! Can't forget that! Now with this out of the way, let's do the most boring part of every Python program: the imports.

from twisted.web import server
from twisted.internet import endpoints, defer
import klein
import treq

Now, let's start Klein app. There are several steps involved here.

root = klein.app.resource()

We take the Klein resource object...

site = server.Site(root)

...make a wrapping site...

ep = endpoints.serverFromString(reactor, "tcp:8000")

...create an endpoint...

ep.listen(site)

<Deferred at 0x7f54c5702080 current result: <<class 'twisted.internet.tcp.Port'> of <class 'twisted.web.server.Site'> on 8000>>

and like "Harry met Sally", eventually bring the two together for Klein to respond on port 8000. We have not written any application code, so Klein is currently "empty".

What does that mean?

async def test_slash():
    response = await treq.get('http://localhost:8000')
    content = await response.content()
    return content

This function uses Python 3's async/await features, to use treq (Twisted Requests) and return the result. We can use it as our ad-hoc debugger (but we could also use a web browser -- this is naturally hard to show in a post, though).

defer.ensureDeferred(test_slash()).addBoth(print)

<Deferred at 0x7f54c5532630>

b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">n<title>404 Not Found</title>n<h1>Not Found</h1>n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>n'

Ah, yeah. Even / gives a 404 error -- we have literally defined nothing. OK, this is easy to fix:

@klein.route("/")
def something_useful(request):
    return 'Hello world'

Wait, did this work?

defer.ensureDeferred(test_slash()).addBoth(print)

<Deferred at 0x7f54c53d8d30>

b'Hello world'

Yep. But it's not a proper sentence...woops.

@klein.route("/")
def something_useful(request):
    return 'Hello, world!'

Nice. Punctuation. Force. Determination. Other nouns.

Did it change anything?

defer.ensureDeferred(test_slash()).addBoth(print)

<Deferred at 0x7f54c4b9e240>

b'Hello, world!'

Yes.

Incremental web application development. Without an "auto-loading" "not production grade" server.

We took advantage of several incidental issues. The Jupyter kernel is Tornado based. Twisted has both a production-grade web development framework, and the ability to run on top of the tornado loop. The combination is powerful.

Do you think of yourself as a Python programmer, or a Ruby programmer? Are you a front-end programmer, a back-end programmer? Emacs, vim, Sublime, or Visual Studio? Linux or macOS?

If you think of yourself as a Python programmer, if you identify yourself as an Emacs user, if you know you’re better than those vim-loving Ruby programmers: you’re doing yourself a disservice. You’re a worse programmer for it, and you’re harming your career.

Why? Because you are not your tools, and your tools shouldn’t define your skillset.

Your tools are not your skills

Ask a programmer to list their skills and more often than not you’ll get a list of technologies. But technologies are just a small set of the skills you need as a programmer.

You need to gather requirements.

You need to debug code.

You need to design user experiences.

You need to build abstractions.

You need to avoid security problems.

And so on and so forth.

None of these skills are technologies. And if you think your only skills are technologies you won’t notice the skills you don’t have. And if you don’t know what you’re missing, you won’t take the time to learn the skills that can make you truly productive and valuable.

Your tools are not your job

If you define yourself by your tools, you are limiting yourself to what jobs you can get.

First, because you won’t apply to those jobs.

Second, because you will market yourself based on tools, instead of all the other skills that might get you that job anyway.

(I’ve written elsewhere about how you can get a job with technologies you don’t know).

Your tools are not you

If your identity is tied up with your tools, you won’t listen to people who use different technologies. Some tools are better than others at certain tasks. Some tools are interchangeable. But an expert using a bad tool can often do more than a novice with a bad tool.

Spending your time fighting over which tool is better is a waste of your time. Instead, spend your time listening and learning from everyone, whatever tools they use: most skills will transfer just fine.

The technologies you use, the tools you build with, are just that: tools. Learn to use them, and learn to use them well. But always remember that those tools are there to serve you, you are not there to serve your tools.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

(Thanks to Paul Ganssle for his suggestions and improvements. All mistakes that remain are mine.)

When exposing a Python program as a command-line application, there are several ways to get the Python code to run. The oldest way, and the one people usually learn in tutorials, is to run python some_file.py.

If the file is intended to be usable as both a module and as a command-line parameter, it will often have

if __name__ == '__main__':
    actually_run_main()

or similar code.

This sometimes has surprising corner-case behavior, but even worse -- some_file.py is not looked for in either $PATH or sys.path, it must be explicitly handed. It also changes the default Python path from including the current directory, to including the location of some_file.py.

The new recommended modern way, of course, is to set entry_points in the setup.py file. When the distribution is installed, a console script is auto-generated and added to the same place the Python interpreter is found. This means that we need to think carefully about the other things that might have the same name on our $PATH to avoid collisions.

There is a third way, which is subtle. When Python sees the -m <some name> option, it will look for a module or a package by that name. If it finds a module, it will run it with __name__ being "__main__" in order to trigger the path that actually does something -- again leading to some, if not all, issues discussed earlier.

However if it finds a package it will run its __main__.py module (still setting __name__ to "__main__") -- not its __init__.py.

This means that at the top of __main__.py we can invert the usual logic:

if __name__ != '__main__':
    raise ImportError("Module intended to be main, not imported",
                      __name__)

from . import main_function
main_function()

This allows running python -m <some package>, but anyone who tried to accidentally import <some package>.__main__ will get an error -- as well they should!

Among other things, this means we only care about our sys.path, not our $PATH. For example, this will work the same whether the package is installed to the global Python or --user installed.

Finally, if an entrypoint is desired, one can easily be made to run __main__:

entrypoint = toolz.compose(lambda _dummy: None,
    functools.partial(runpy.run_module,
                      "<some package>",
                      run_name='__main__'))

Using the builtin module runpy, the builtin module functools and the third party module toolz.

In today's edition of Pi day post, we will imagine we have a pie. (If you lack imagination, go out and get a pie.) (Even if you do not lack imagination, go out and get a pie.)

As is traditional, we got a round pie. Since pies are important, we will base our unit of measure on this pie -- the diameter of the pie will be 1.

Since we had to carry it home, we put it in a box. We are all ecologically minded, of course, so we put it in a box which is square -- with its length size 1.

We note something interesting -- the box's bottom's area is 1x1 -- or 1. The radius of the pie is 1/2, so its area is pi * 0.25.

As we are driving home, the pie on our passenger seat, we start wondering how we can estimate Pi. Luckily, we got some sugar. What if we sprinkled some sugar, and took notes for each grain, whether it was on the pie, or not?

Let's use Python to simulate:

import random
import attr

First, we need some randomness generator. Then, we also want to use the attrs library, because it makes everything more fun.

We make a Point class. Other than the basics, we give it a class method -- a named constructor which will generate a random point on the unit square -- this is where our sugar grain falls.

We also give it a way to calculate distance from another point, using the Pythagorean theorem.

@attr.s(frozen=True)
class Point:
    x = attr.ib()
    y = attr.ib()

    def distance(self, pt):
        return ((self.x - pt.x) ** 2 + (self.y - pt.y) ** 2) ** 0.5

    @classmethod
    def unit_square_random(cls):
        return cls(x=random.random(), y=random.random())

The center of the pie is at 0.5 by 0.5.

center = Point(0.5, 0.5)

A point is inside the pie if it is less than 0.5 away from the center.

def is_in_circle(pt):
    return center.distance(pt) < 0.5

Now we are ready. Even with just 100,000 grains of sugar, we get 2-digit accuracy.

inside = total = 0
for _ in range(10 ** 5):
    total += 1
    inside += int(is_in_circle(Point.unit_square_random()))
print((inside / total) * 4)

3.14052

Every day you go to work and you have to fight. Your boss wants to deliver features as fast as possible, and so you have to either argue for best practice, or work extra hours to do things the right way.

You need to fight for readable and maintainable code. And you need to fight for reusable code. And you need to fight for tests.

And you fight and you fight and you fight, and you keep on losing.

It doesn’t have to be this way. Software development doesn’t have to be a fight. Strange as it may seem, in this game the only winning move is not to play.

Instead of playing “Let’s Write High-Quality Software”, there’s a different and better game you can play: “Let’s Solve The Problem”.

Avoid false dichotomies

The problem with the game of High-Quality Software is that it’s based on a dichotomy, with only two options: low-quality and high-quality. I started this article with another dichotomy: arguing with your boss vs. working longer. And a lot of technical discussions quickly devolves into dichotomies, e.g.:

• Shipping features vs. fixing technical debt.
• Testing your code vs. coding quickly.

Dichotomies are tempting, and perhaps even built-in to the way we understand the world: there’s a left hand and a right hand, a wrong way and a right way. But there’s nothing inherently superior in just having two choices. In fact, all it’s doing is making you a worse engineer, because you’re focusing on arguing instead of focusing on solving the problem.

To combat this tendency, Gerald Weinberg suggests the Rule of Three: always consider at least three solutions to any problem. Let’s start with our first dichotomy: arguing with your boss vs. working longer hours to do things the right way. If there’s a third choice, what might it be?

Stop arguing, start listening

When your boss or colleagues argue for a specific design, instead of telling them why they’re wrong, listen to their reasons. Behind every design proposal is a set of goals, motivations, presumed constraints: those are the things you need to address. If you criticize their proposal based on goals they don’t care about, you’re not going to get anywhere.

So first, listen.

Once you understand why they want what they want:

1. Consider more than the initial two choices, their proposal and your initial reaction.
2. Try to find a solution that addresses both your goals.
3. Explain you thinking in ways that are relevant to their goals, not just yours.

Example scenario: testing

Your boss proposes writing code without unit tests. Why? Because they want customers to be happy. Since customers have been complaining about how long it takes for new features to be delivered, your boss believes this is one way to speed up delivery.

You want to write unit tests. Why? You want code to be maintainable over time.

Merely arguing for unit tests isn’t going to address your boss’ concerns. But if you look past the initial false dichotomy, there are other solutions to consider. For example:

• Investigate why customers aren’t getting features quickly. Perhaps the bottleneck isn’t due to how fast you’re coding, but elsewhere in the delivery process (you don’t do frequent releases, customers need to upgrade manually… there could be many reasons). Fix that, and you will have time to write unit tests and ship quickly.
• Figure out places where bugs would be costly to customers, explain those costs to your boss, and propose unit testing only that part of the code.
• Investigate customer needs in more detail. Perhaps existing customers are complaining about feature delivery, but you’re also losing many customers due to bugs.
• Suggest using tools that will speed up test writing enough that the additional time won’t bother your boss.
• Suggest limiting unit test writing to a predetermined amount of time: “this will only add 4 hours to a one week project”.

No doubt there are many more potential solutions to the standoff.

There’s always another solution

There is almost never a single correct solution to any problem, nor a single best solution. You can solve problems by relaxing unnecessary constraints, by focusing on different levels of the situation (organization, process, code), by redefining the problem, and more. Worst comes to worst, you can address many problems by switching jobs; different people like different environments, after all.

So don’t try to win technical arguments, and in fact, don’t treat them as arguments at all. When you disagree with someone, take the time to listen, and then try to come up with a solution that address their concerns and yours. And if your first idea doesn’t do that… it’s time to come up with a third, and fourth, and fifth solution.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

At the end of the day you’re done with work, you go home—and you don’t spend any of your free time coding. And that’s fine, you have other things going on in your life. But your coworker does spend another 20 hours a week coding, and all that practice means they’ll end up better programmers than you, and so they’ll get promoted faster, and they’ll get paid more. And that’s not fine.

It’s also not true.

That inadequacy you’re feeling is based on false assumptions, a misunderstanding of the skills it takes to be a good programmer. Enthusiasts who love coding as a hobby are valuable, yes, but so are you: even if extra hours of coding result in better skills—often a very doubtful assumption—their approach is not inherently better.

All but the smallest of programming projects are team efforts, requiring a variety of skills, knowledge, and mindsets. The mindset of someone who view programming as a tool to do their jobs—let’s call them pragmatists—is just as useful as the mindset of enthusiasts.

To understand why, let’s go through the lifecycle of the enthusiast programmer, see how their skills evolve, and compare them to the pragmatists’ approach. As you’ll see, enthusiasts and pragmatists start out with different strengths, and software teams can benefit from both.

The enthusiast’s path

The enthusiast’s starting point is a love of programming: they love to code. They take joy in writing software, and understanding the intricacies of the technologies they use.

Over time, as they become more skilled and experienced, they expand their point of view. Instead of just wanting to write code, they want to write good code, high-quality code: they want to be proud of their work. So now they focus not just on the joy of coding, but also on less enjoyable but important disciplines like testing.

If they continue to grow in skill and experience, they will eventually hit a point where they see code as a tool: a means to an end. Software is still fun, yes, but it also needs to be written to achieve certain goals. At this point quality becomes less of an obvious objective criteria: every situation is different, and what works in one situation doesn’t work in another.

Strengths and weaknesses: Enthusiasts often have a strong grasp of the tools and technologies they use. On the other hand, they are sometimes prone to holy wars and zealotry (they feel it’s important to choose the very best tool), and until they finally realize software is a means, not an end, they have a harder time working towards goals that aren’t about writing software.

The pragmatist’s path

The enthusiast’s weaknesses are the pragmatist’s strength. Pragmatists start out seeing software as a means to end: they will tell you “I like writing software, but mostly I care about what the software lets me do.” So what can take the enthusiast years to learn—a focus on goals—comes naturally to the pragmatist.

Of course, if they’re not deliberately focusing on learning, pragmatist can sometimes suffer from less understanding of tools and techniques. This can be mitigated by learning on the job, and making sure to build awareness of available technologies.

Software is not a competition

With time and practice, the strengths of enthusiasts and pragmatists can converge. Even so, different people will typically end up with different skillsets, different approaches, and different ways of building software.

This is how it should be. Software projects are too big to be built by one person, too complex to be encompassed by one view point.

When you compare yourself to your peers, you shouldn’t ignore your weakness, but you should also value your own strengths. You can and should learn from your colleagues, but chances are there’s something you can teach them.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

I skipped work today because I’m sick: nothing terrible, nothing life threatening, I’m just—exhausted. I definitely can’t focus on code, I can’t even focus on television; so far I’ve been re-reading a favorite old novel, and looking at Twitter when I get distracted.

The world will not come to an end because I’m not working. My company won’t collapse because I took the day off. My project will not fail.

I’m sick and I’m taking the day off, and that’s OK. And if you can’t take a day off—whether because you’re sick, or just because you feel like it—then something is very very wrong.

And now… I think I’m going to lie down on the sofa.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

Working long hours as a programmer can be both comforting and comfortable. After all, you have an easy solution to every problem: a few more hours at your desk, a few more hours staring at a screen in the dark. And working long hours is a natural attitude to take, since at the root of it, working long hours comes from valuing work, your craft as a programmer, and your obligations as a professional.

There is a different approach you can take. Instead of valuing work, you can value outcomes: outcomes at your job, outcomes in your life, outcomes in the world. Valuing outcomes is not comfortable. It forces you to reconsider and doubt your every step, and if you fail—and we all fail occasionally—your hard work becomes worthless.

But while valuing outcomes is uncomfortable and difficult, if you do it long enough you will find yourself—every once in the while—with the ability to grasp the moment and change the world. Just a little, in whatever corner you live in, but change it nonetheless.

The most direct way to learn this attitude is to reduce your working hours. If you had to do your work in 32 hours a week, instead of 45, or 50, or 60—what would need to change?

The experience of working fewer hours

Testing and quality

If you work long hours, you can pick some level of quality and stick to it:

• If you prioritize shipping features as quickly as possible, you can simply work longer to compensate for the buggy features you’re shipping.
• If you believe in quality above all else, you can spend the extra time to reach 100% test coverage of every little piece of code.

But if you work fewer hours, you will need to make decisions and tradeoffs, because you won’t have those extra hours. Some code will need automated tests for stability and maintainability, some code will need manual tests every time you change it, some code will be used once and thrown away. And you will need to make that choice, every single time. Which is to say, you will be forced to think about why you’re writing this code, and what it will be used for.

Planning

If you work long hours, the direction you go in matters less: if it’s the wrong direction, you can just work a few more hours when you eventually find out after you finish your task.

But if you work fewer hours, you will find yourself constantly plagued by doubt: is this the right direction? Are you doing the right thing? You may end up changing direction half-way, abandoning your work—however much it pains you—when you realize there’s a better way to your destination. You will have to spend much more time thinking and planning up front, to make sure you arrive at the right place on time.

Speaking of time, deadlines cause less worry when you work long hours. If you hit your deadline, great. If you didn’t, well, you worked long and hard, and who can blame you if you failed?

If you work fewer hours, deadlines are a constant gnawing presence, always getting closer. You will need to plan for them well in advance, always asking questions, on the lookout for unstated requirements and forgotten deliverables. And sometimes you’ll be forced to find new solutions so you can hit those deadlines in time—and, of course, sometimes you will fail to find those solutions. And then you’ll look bad, because you didn’t work on the weekend, so whose fault is it the deadline was missed?

Your value

When you work long hours, it’s easier to demonstrate how valuable you are to your boss: you were there on the weekend, after all. If you work fewer hours, well, who can say you’re really getting much done? Did you deliver anything of value? How will you prove it?

Even worse than your boss is your own self-image. When you work long hours, your self-worth is about your work: you work hard, and that at least is something to be proud of, even if the project failed. If you work fewer hours you won’t be able to feel proud of all the hours you worked. You’ll be forced to consider the results, the outcomes: if you failed to deliver, you failed.

From comfort to effectiveness

Go through all this and— you’ll become effective.

• You will think more, and find the simple solution that takes a day, where previously you would just start coding—and spend a week coding, at that.
• You will learn to build what is needed, where previously you would build what you knew.
• You will learn to prioritize, and prioritize well.

Valuing outcomes is not comfortable, but it is empowering. And good day or bad, you get to go home at a reasonable hour and do whatever you feel like. Which feels pretty damn good too.

You don’t have to take my word for it, either: it’s easy to try out. If you’re working long hours, try spending a month working 40 hours a week, no more. Once you can no longer rely on working longer hours to solve all your problems, you’ll soon find yourself approaching your work very differently.

And if you want a head start on learning these skills, read my book, The Programmer’s Guide to a Sane Workweek.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

I have written before about Python tooling. However, as all software, things have changed -- and I wanted to write a new post, with my current understanding of best practices.

There is so much to learn as a programmer it’s hard to even know where to begin. And even if you do begin, there’s always something newer and shinier to distract you, and so you end up making these giant lists of things to learn and they get longer and longer and longer and you feel like shit about it but really you don’t have the time to learn all of it, now do you?

So, why not take a few minutes to stop worrying about the things you should be learning, and instead let’s talk a little about all the things you don’t need to learn. And maybe when we’re done you’ll feel a little better.

But first, let’s talk about—

DEATH.

We’re all gonna die (eventually)

Someday you’re going to die. So will I. So will everyone else, our friends and enemies both.

Typically one measures one’s lifespan in years or days, but I once went and measured it in books. I did the math and figured out how many books I could read before I reached my presumptive death from old age. At the time I was reading about two books a week, but even so the number didn’t seem anywhere near sufficient.

And having started down this morbid path, I arrived at an even worse place. Every time I read a book I’d get to thinking: “Is this book worth reading before I die? Shouldn’t I be reading something more edifying than this entertaining yet trashy novel? Isn’t re-reading a book a complete waste of my time?” Instead of enjoying the book I was reading, I was worrying about some other better book I could have been reading instead.

Eventually I got over it. I won’t be able to read every book I want to before I die. Neither will you.

Sorry.

It’s not so bad, though. When you’re lying there dying you’ll probably be thinking about the friends and family you’ll miss. Or maybe you’ll just be tired, and looking forward to an end to your pain and sorrow. Or, if you’re having a really bad day, you’ll be thinking that you shouldn’t have had that last drink before driving home—don’t drink and drive, kids. Better yet, don’t drive at all; commuting is a terrible way to spend your life.

In any case, when it’s time for you to die you probably won’t be worrying about the books you haven’t read. And as you lie on your deathbed, looking back at your life, you definitely won’t be worrying about that new JavaScript framework you didn’t get to play with.

Some skills you don’t need to learn

There are many skills I do think you should learn (I have a whole pile of posts on this here website, in fact), but honestly—it’s just software. If you don’t learn these skills before you die, that’s really not a big deal.

Software is a tool: tools are useful, and important, and you need them to build many things. But our tools are there to serve us, we should not be serving our tools.

Not to mention all the skills you really don’t need to learn.

You don’t need to learn every blindingly shiny new technology that will End Poverty and Bring Peace to Humanity. It probably won’t do either, and quite possibly it won’t turn out to be good for much at all. I started my career programming building multimedia CD-ROMs, which were really hot for about 6 months in the mid ‘90s, and somewhere in a landfill there’s still boxes of old unusable CDs I worked on that no one cares about.

You don’t need to learn everything programming language, certainly not in your spare time. You can and should learn those on the job, as and when they become useful.

You don’t need to learn how to use every new library, tool, or framework. Just knowing they exist is usually more than enough: when and if you need them, you’ll know they exist and go learn them then.

Some things to do before you die that are more important than learning another programming skill

Spend more time with your friends.

Spend more time with your family (unless you don’t get along, sorry).

Eat good food.

Visit UNESCO World Heritage sites.

If you haven’t seen one, a full solar eclipse is amazing.

Make the world a better place, even if it’s just a little.

Whatever you think is important and worthwhile.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

So your manager stops by and says “hey, we’ve got some new employees starting in a month, and you know SomeTechnology best, I’d like you to get them up to speed.” And you nod and smile and feel the impostor syndrome kicking in: you don’t know how to do this.

Software is complicated. Sometimes even when you’ve been using a tool or a language for a year or two, you might only know a small part of the functionality. It’s already tough and intimidating to teach as it is, it’s even worse when you’re not confident in what you know. How can you teach when you’re not an expert?

But as it turns out, being an expert can impede your ability to teach. You can become a better teacher by taking advantage your lack of expertise.

The problem with being an expert

Experts can actually make worse teachers. Specifically, experts often suffer from “expert blind spot”, where their own knowledge blinds them to the way their students are thinking (I first learned of this concept from the book How Learning Works).

Experts differ from beginners in a number of ways that impact learning.

1. Experts have much denser and interconnected conceptual models. Things that are obviously related to them are not at all obvious to beginners, who have sparsely connected conceptual models.
2. Experts will skip steps, doing certain things so automatically they don’t even realize they’re doing them. From the beginners’ point of view this results in unexplained jumps.
3. Expertise is often unconscious. When asked, experts may not be able to explain why they’re doing what they’re doing, even if they know it’s the right thing.

As a result, being an intermediate learner, where you know how to do a task but still need to consciously walk through the steps, can actually make you a better teacher.

Teaching when you’re not an expert

Since you’re not an expert, you don’t need to pretend to be one. Instead you can teach the valuable knowledge you do have: the combination of the understanding of the technology, and the understanding or recent memory of how beginners think.

• Instead of demonstrating a task you know perfectly, you can demonstrate a task you don’t quite know how to do, e.g. by doing a live coding exercise. You can then show the process you go through for finding solutions—how you debug the problem, how and where you search for solutions and documentation. (Thanks to Noa Bendit-Shtull for the idea.)
• Similarly, you can cover common mistakes and how to recover from them. Where an expert might never make these mistakes, you might still make them occasionally, or at least have the recent memory of what those mistakes were.

By showing how to deal with mistakes, and by admitting mistakes to your students, you’re also making the topic less intimidating. Unlike some experts, who get confused and perhaps annoyed when mistakes happen—they never make those mistakes after all—you are giving your students the confidence that they too can work through problems. (Thanks to reader Jake for this point.)

Go teach!

Teaching doesn’t need to be scary. It can just be you sitting down at a computer and saying “this is tricky, but I’ve figured out some ways to get things done, and so can you. Let me show you how.”

This also applies to teaching more broadly. Every once in a while you’ll see a conference call for proposals, and you’ll say to yourself “I’m not an expert, I can’t teach anything.” But the truth is you can: your perspective as an intermediate learner is often much more valuable to beginners than that of an expert. Submit a talk proposal even if you’re not an expert: you too have a valuable perspective.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

Have you ever read an amazing-sounding job posting, and then felt that sinking feeling in your stomach when you reached the list of required technologies? Maybe you’re a Python programmer, and they use Ruby on Rails; maybe you’re mostly a front-end developer, and they want back-end skills. Whatever the missing technologies are, this is an awesome job you don’t have a chance of getting.

Or do you?

In fact, it is possible, albeit not always easy. Years ago I got a job writing C++, when all I knew was Python and Java. And for the past few months I’ve been doing scientific computing for the first time, with a new toolchain, and math skills I haven’t used in almost 20 years.

In this post I’ll cover:

• Why you should apply anyway.
• Which companies to apply to.
• How to pitch yourself so you can get the job.

Apply anyway

Even if you feel you’re not fully qualified for a job, you should still apply:

First, the list of technologies may be irrelevant. Sometimes the technologies listed are things the company might want to use someday, not what they actually use today. Sometimes they are perfectly happy hiring candidates with different technologies, and they put the list of technologies down because they aren’t very thoughtful about how they write job ads.

But what if they do actually want those technologies? The list of technologies and skills is what the company would like in the ideal world, not necessary what they can get in practice. In the ideal world many companies would probably also like to pay you half as much and have you be twice as experienced, with the ability to create gold out of lead and summon unicorns at will. In practice, companies hire the candidates they can get, not the magical and often non-existent candidates they dream of.

Finally, technologies aren’t everything. There are many other skills you have as a programmer, and some of them may trump the particular technologies you lack. We’ll revisit this in the section on pitching yourself.

Which companies to apply to

It’s hard to say as an outsider which companies will be more flexible, but here are some things to look for:

• Companies that use less popular technologies: they’ll have a harder time hiring, which means they are probably more willing to train people. I got that job writing C++ with a company whose main product was written in Common Lisp, which is not a common skill.
• Companies that explicitly talk about their commitment to training. For example, companies that talk about how they fund conference visits or classes.
• Companies where you have other relevant skills: perhaps you’re an expert in this business domain, or you have adjacent skills.
• Companies you’re really excited about. If you can convey that excitement in your cover letter, that may be enough to get you over that initial hump.

How to pitch yourself

Once you’ve picked the companies to apply to, you want to customize your cover letter, resume, and if you get there your presentation at the job interview. Here are some ways to do that:

• Talk about your ability to learn quickly, with concrete examples. If you can show that at your last job you learned a new technology stack in a couple months you will take away the fear of a long slow ramp-up to productivity.
• Stress adjacent skills. You’ll have hard time going straight from front-end development to a data science project. But, in practice much of data science is cleaning up dirty data, which is experience you might have in a different domain.
• Talk about skills beyond just technologies.
  Do they do consulting work? Stress your ability to work with non-engineers and gather requirements.
  Are they growing rapidly? Talk about the time you helped a team grow its engineering processes.
  Is failure costly? Stress your experience in testing and building robust software.

The key is to research the company, try to understand their needs, and then demonstrate you can have value to them beyond just the list of technologies you know.

Next time you’re looking for a job, don’t limit yourself only to jobs with a technology stack you know. Look for jobs that excite you, for jobs you think you can do with just a little ramp-up time. And then apply for those jobs, knowing that most of them will ignore you—but it only takes one yes to get an exciting new job, and an exciting new opportunity to learn new skills.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

Are you having a hard time getting responses to your resume? Do you know you have the skills for a job, but have a hard time convincing the company to hire you?

If your resume starts out with a long list of technologies you are selling yourself short. Knowing and using programming languages, libraries, and so on is a critical job skill for a software developer, but it’s only part of what makes a good programmer. And the skills you aren’t mentioning may well be keeping you from getting hired, even if you already have them.

In this post I’ll discuss:

• The most important part of your resume: the opening paragraphs.
• Why starting with technologies is a mistake.
• What to lead with instead.

The most important part of your resume

Recruiters spend very little time reading resumes, and more broadly people tend to skim when reading. That means the first thing a reader see when they read your resume will have the most impact. The lower on the page, the less likely it is to be reached.

You should imagine the hiring manager as looking for some hint or phrase that will make them think “aha, this sounds promising, I should keep reading.” And you should also imagine them giving up looking after less than 10 seconds.

That means you need put the most important pieces of information at the very top of your resume. The first sentence, the first paragraph, no lower. Your goal isn’t to convey everything, you just want to make yourself sound interesting and relevant enough that the recruiter keeps reading.

For example, let’s say you worked in bioinformatics two jobs ago, and you’re applying to a bioinformatics job again. Your opening paragraph should mention your biotechnology work; that way the reader will be motivated to keep reading. Having given that motivation, you can still do the normal reverse chronological job history in the second part of your resume—but perhaps the paragraph for your current job should be a little shorter if it’s less relevant.

Don’t lead with a list of technologies

Since you need to make yourself sound interesting in the first few seconds, a list of technologies is a bad way to start. Merely claiming to know a technology doesn’t tell the hiring manager how useful of an employee you’ll be.

In some jobs knowing particular technologies will give you a head start, which is why some recruiters foolishly use technology keywords as a filter, but it’s never really enough to do your job as a programmer. So spending precious moments of attention listing the technologies you know is a bad way to convey your value:

• Many of the technologies you know won’t be relevant to the company. If a company only uses Ruby, the fact you know Idris is unlikely to be relevant.
• Even if you’re listing a technology the company does use, mentioning it isn’t very compelling proof you can do anything with it. Can you read existing code and fix small bugs, or can you create a completely new project from scratch with it? Those require different levels of skills.
• Even if you’re the world’s foremost expert at a technology, that doesn’t mean you’ll be productive. Will your boss have to tell you exactly what to do, or will you be able to work independently?
• Most companies use many technologies, and add more technologies over time. If you know Python but refuse to learn other programming languages, you are a far less useful employee than someone who doesn’t know Python but is able and willing to quickly pick up whatever is needed.

What to do instead

Instead of opening with a list of technologies, open with a paragraph demonstrating the ways in which you’re a valuable employee, ideally tailored to the company you’re applying for. Specifically, you want to convince the hiring manager you can do the job they’re hiring for, which is either solving problems, or at more experienced levels identifying and solving problems. Writing software is merely a means to that end.

So you want a starting paragraph that emphasizes:

• Your ability to solve real, relevant projects with your skills.
• Your ability to work independently.
• Your ability to learn quickly.
• Your relevant domain knowledge, if any.
• Your ability to work with others.

And you want to include just enough real-world examples that you will convince them to keep reading. Once you’ve got them reading the main job-listing section of your resume you can go into much more detail.

You can still have a list of technologies if you want, but don’t waste the precious first half page of your resume on it. Personally I don’t bother: I just mention names of relevant technologies at the jobs where I used them.

If you’re currently looking for a job, go look at your resume, set a timer for 10 seconds and read it from the start. What will the recruiter learn about you? And is that all you want them to know? If all they’ve learned is that you know tool X or language Y, it’s time to make some changes.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

For some people, a 40-hour workweek is something to aspire to; for others, it’s still too much time taken up by a job. If you fall into that second category, if you want more time for hobbies, family and friends, or working on your own software projects, you too might dream of working less than full time.

But how do you get there? Almost no one advertises part-time programming jobs–believe me, I’ve me looked.

The answer: negotiation. I’ve negotiated a shorter workweek a few times myself, and I’ve met other programmers who have done so as well, some with just a few years of experience. And of all the programmers I’ve met who’ve negotiated part-time work, Mike’s record is the most impressive.

Mike has spent pretty much all his career working part-time: he’s been working part-time for more than 15 years. To help you get to a shorter, saner workweek, I sat down to interview Mike about how he does it.

❧

Q. What does a sane workweek mean to you?

MIKE: Well, I only ever worked full time for about 1 year, and I learned pretty quickly that a sane workweek for me was less than 40 hours a week. I guess it’s up to each individual, but I think most of us are forced to work more than we’d like, and at least for me 30-32 hours is better.

I want to work on average less, but I make it clear [when starting a job] that [I understand] things happen, stuff needs to get done. I would definitely work longer hours here and there.

Q. How did you realize you wanted to work less hours?

MIKE: Part of the reason I decided to demand this so early was, I started that first job when I was in university, and I stayed on for several years part-time while I was school. My contract was 10 hours a week, not very much but I was also going to classes and had to do homework and shit. I suppose that got me used to being able to go cycling or climbing or hiking on fairly short notice.

They were terrible at planning at this company, when a contract deadline was coming up everyone was working 60 to 70-hour weeks, having dinner at the office all the time. I wasn’t forced to get caught up in that, since I had my excuse of going to school. I worked a lot more than 10 hours I was contracted to, though.

I saw this cycle there a lot, where managers would pull numbers out of their ass and promise them to clients, and then all the programmers were panicking for the last month. The panic mode of getting this done. [My feeling is] I’m not committed to those numbers you pulled out of your ass to tell your client, so partly it was reaction to that, seeing these people spending their entire lives at the office. I wanted to carve out my time early on.

Q. So how did you end up part-time after that first year?

MIKE: At that I job I tried to quit.

But they basically offered me more money, so I was like, “what if you gave me 75% of that, and I worked less?” And I guess they wanted to keep me around and they went for that.

At that job I was on payroll as a full time employee, but I got a bunch more holidays. So I got a quarter of my time in holidays; 72 days off a year? I was constantly booking time off, which was interesting.

That was how I landed my first part time thing. And then I did eventually quit that company and worked elsewhere, but I had found a new job while I was still in a part time situation, so it was a lot easier to demand a similar deal going forward.

Q. How many companies have you done this with? How many part time jobs?

MIKE: A bunch. All of my jobs. 7, no, 8.

(Mike goes off to look at his resume)

If you count the two I have now, two part time contracts, then 9.

Q. How exactly did you negotiate for shorter hours at later jobs?

I have several interviews here and there. And sometimes—if it’s a company where I’m meh, not that serious about or interested in—I’ll send them an email early in the process saying “hey I want to work part time.” And then if they won’t go for it I won’t bother.

If I definitely wanted to work there, I’d go for the interview without telling them [I wanted a shorter workweek]. And if I got back for subsequent interviews or I got an offer I’d say “now I’m working 75%, I want to work part time somehow, I’m open to various different arrangements, I’m interested in working with you, lets work something out.” Mostly if the companies were in the job offer stage they’d consider it very seriously. One was “no, no way,” they didn’t want to talk to me anymore. Mostly they’re interested in negotiating somehow.

Q. What kinds of counter-offers did you get?

MIKE: Sometimes it was stuff like “I want to hire you, how do I sell this management? Help me sell it to management.” That job took me 6 months to get from interview to legitimate part time job offer. That was the longest, and also unbeknownst to that company I’d been laid off: I’d started negotiating when I had a job, and was subsequently laid off. They wanted me to sell it to management. I said, “I don’t want to work 5 days, how much happens Fridays? Not much? Then I’ll work Monday to Thursday.”

Usually I negotiated the full-time salary first, and then the details of pro-rating later: mostly it was about the details, once you’d convinced someone you were worth having. The easiest ones to negotiate are the ones which are 80% time, especially one day a week off. Usually I’d say Monday or Friday, and get a longer weekend, go to the mountains. But one of the people I inspired got himself an 80% contract, and he said “you’ve got to take Wednesday off.” With Wednesdays, you only ever work two days in a row. It was awesome.

I always set expectations: “you’re getting the best 4/5ths of my time, but only 4/5ths. I’m not going to pound out as much code.” Realistically you kinda do, but the expectation should be if you’re not there Wednesdays you shouldn’t be doing as much work. That’s part of the argument, that you’re doing almost as much useful work.

Q. Did you find asking for a shorter workweek got easier over time?

Yeah, for sure, it definitely got easier. After the first couple times I felt more confident asking in the first place. It got easier in that sense.

Q. When you tell people you don’t work full time, how do they react?

It’s still considered weird by a lot of my friends, but I also get the other reaction, “I wish I could do that.” Some people when you start new jobs, once you’d explained why you weren’t there Wednesday, they’d say “I totally want that” but then I’d say I only get paid 80%. Once they’d realize they’d get paid less, [they’d say] “I’m not doing that.”

Other people would say “cool, I want that.” Definitely people I’ve worked with over the years have been inspired to do that, which is pretty nice.

Q. Why was the lower salary not an issue for you?

MIKE: I don’t know, I guess I decided early on free time was more important than more money. Programmers are paid pretty well in my experience, 80% of a programmer job is plenty of money to get by on. Typically it’s a lot more than what a lot of people are getting.

For me it was putting a premium on my time, and wanting to have that time to do other things. I’ve got hobbies, always had hobbies, different sports over the years which takes time. But even just stuff like having time to work on your own programming project, which I think is really valuable. Valuable enough to me that I’ll actually pay money for it in the form of taking less salary.

I definitely don’t have any regrets for doing that.

Q. Any final advice?

MIKE: My biggest piece of advice is that you have to be willing to quit your job to get a sane workweek, or you won’t sound convincing when you ask for it. If you’re looking for a new job, just ask, and if you’re still at a job, then you’re in a better position to ask.

Just go for it!

❧

Itamar here: while I agree that working less than full time is wonderful, it also isn’t for everyone. Perhaps to you a sane workweek means working for yourself, or working remotely, or just getting a job where you can go home at 5PM.

But whatever a sane workweek means to you, the basics are the same. As in Mike’s case, you need the skills to be productive and valuable, you need to be able to negotiate, and you need enough financial security to have a strong negotiating position.

Are you ready to get a part-time job?

Do you want even more details? Would you like hear more stories of people who have gotten a part-time job, and how they did it? Want a blueprint for how to make yourself more valuable, and a process for negotiating for a part-time job?

The Programmer’s Guide to a Sane Workweek has chapters on becoming a more valuable employee, exercises to helping you improve your negotiating skills, interviews with other programmers who work part-time, and a process for negotiating part-time hours at a new job.

Is your job taking way all your personal time and freedom? You can succeed as a software engineer without working crazy hours.

I’ve written and spoken at some length about shipping software in the abstract. Sometimes I’ve even had the occasional concrete tidbit, but that advice wasn’t really complete.

In honor of Eevee’s delightful Games Made Quick???, I’d like to help you package your games even quicker than you made them.

Who is this for?

About ten years ago I made a prototype of a little PyGame thing which I wanted to share with a few friends. Building said prototype was quick and fun, and very different from the usual sort of work I do. But then, the project got just big enough that I started to wonder if it would be possible to share the result, and thus began the long winter of my discontent with packaging tools.

I might be the only one, but... I don’t think so. The history of PyWeek, for example, looks to be a history of games distributed as Github repositories, or, at best, apps which don’t launch. It seems like people who participate in game jams with Unity push a button and publish their games to Steam; people who participate in game jams with Python wander away once the build toolchain defeats them.

So: perhaps you’re also a Python programmer, and you’ve built something with PyGame, and you want to put it on your website so your friends can download it. Perhaps many or most of your friends and family are Mac users. Perhaps you tried to make a thing with py2app once, and got nothing but inscrutable tracebacks or corrupt app bundles for your trouble.

If so, read on and enjoy.

What changed?

If things didn’t work for me when I first tried to do this, what’s different now?

• the packaging ecosystem in general is far less buggy, and py2app’s dependencies, like setuptools, have become far more reliable as well. Many thanks to Donald Stufft and the whole PyPA for that.
• Binary wheels exist, and the community has been getting better and better at building self-contained wheels which include any necessary C libraries, relieving the burden on application authors to figure out gnarly C toolchain issues.
• The PyGame project now ships just such wheels for a variety of Python versions on Mac, Windows, and Linux, which removes a whole huge pile of complexity both in generally understanding the C toolchain and specifically understanding the SDL build process.
• py2app has been actively maintained and many bugs have been fixed - many thanks to Ronald Oussoren et. al. for that.
• I finally broke down and gave Apple a hundred dollars so I can produce an app that normal humans might actually be able to run.

There are still weird little corner cases you have to work around — hence this post – but mostly this is the story of how years of effort by the Python packaging community have resulted in tools that are pretty close to working out of the box now.

Step 0: Development Setup

Get a good Python. Use Homebrew, and brew install python3. If you need python 2, brew install python2. Don’t use the System python. Probably nothing will work.

You probably also want to use a virtualenv for development. This post is about how to build a for-real thing that other people can download, but part of the magic of Python is the interactive, real-time dynamic nature of everything. Running the full build pipeline every time you change a file or an asset is slow and annoying. However, there’s a weird thing where certain parts of the macOS GUI won’t work right (in PyGame’s case, mostly keyboard focus) unless your code appears to be in an application bundle.

I made this dumb little thing which lets you fake out enough of this that the OS won’t hassle you: you just need to pip install venvdotapp; venvdotapp inside the virtualenv where you’re making your pygame app.

Finally: pip install all your requirements into your virtualenv, including PyGame itself.

Step 1: Make an icon

All good apps need an icon, right?

When I was young, you just popped over into ResEdit Resorcerer MPW CodeWarrior Project Builder Icon Composer Xcode and created a new ICON resource cicn resource .tiff file .icns file. Nowadays there’s some weird opaque stuff with xcassets files and Contents.json and “Copy Bundle Resources” in the default Swift and Objective C project templates and honestly I can’t be bothered to keep track of what’s going on with this nonsense any more.

Luckily the OS ships with the macOS-specific “scriptable image processing system”, which can helpfully convert an icon for you. Make yourself a 512x512 PNG file in your favorite image editor (with an alpha channel!) that you want to use as your icon, then run it something like this:

1

$ sips -s format icns Icon.png --out Icon.icns

somewhere in your build process, to produce an icon in the appropriate format.

There’s also one additional wrinkle with PyGame: once you’ve launched the game, PyGame helpfully assigns the cute, but ugly, default PyGame icon to your running process. To avoid this, you’ll need these two lines somewhere in your initialization code, somewhere before pygame.display.init (or, for that matter, pygame.display.<anything>):

1
2

from pygame.sdlmain_osx import InstallNSApplication
InstallNSApplication()

Obviously this is pretty Mac-specific so you probably want this under some kind of platform-detection conditional, perhaps this one.

Step 2: Just Include All The Dang Files, I Don’t Care About Performance

Unfortunately py2app still tries really hard to jam all your code into a .zip file, which breaks the world in various hilarious ways. Your app will probably have some resources you want to load, as will PyGame itself.

Supposedly, packages=["your_package"] in your setup.py should address this, and it comes with a “pygame” recipe, but neither of these things worked for me. Instead, I convinced py2app to just splat out all the files by using the not-quite-public “recipe” plugin API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

import py2app.recipes
import py2app.build_app

from setuptools import find_packages, setup

pkgs = find_packages(".")

class recipe_plugin(object):
    @staticmethod
    def check(py2app_cmd, modulegraph):
        local_packages = pkgs[:]
        local_packages += ['pygame']
        return {
            "packages": local_packages,
        }

py2app.recipes.my_recipe = recipe_plugin

APP = ['my_main_file.py']
DATA_FILES = []
OPTIONS = {}
OPTIONS.update(
    iconfile="Icon.icns",
    plist=dict(CFBundleIdentifier='com.example.yourdomain.notmine')
)

setup(
    name="Your Game",
    app=APP,
    data_files=DATA_FILES,
    include_package_data=True,
    options={'py2app': OPTIONS},
    setup_requires=['py2app'],
    packages=pkgs,
    package_data={
        "": ["*.gal" , "*.gif" , "*.html" , "*.jar" , "*.js" , "*.mid" ,
             "*.png" , "*.py" , "*.pyc" , "*.sh" , "*.tmx" , "*.ttf" ,
             # "*.xcf"
        ]
    },
)

This is definitely somewhat less efficient than py2app’s default of stuffing the code into a single zip file, but, as a counterpoint to that: it actually works.

Step 3: Build it

Hopefully, at this point you can just do python setup.py py2app and get a shiny new app bundle in dist/$NAME.app. We haven’t had to go through the hell of quarantine just yet, so it should launch at this point. If it doesn’t, sorry :-(.

You can often debug more obvious fail-to-launch issues by running the executable in the command line, by running ./dist/$NAME.app/Contents/MacOS/$NAME. Although this will run in a slightly different environment than double clicking (it will have all your shell’s env vars, for example, so if your app needs an env var to work it might mysteriously work there) it will also print out any tracebacks to your terminal, where they’ll be slightly easier to find than in Console.app.

Once your app at least runs locally, it’s time to...

Step 4: Code sign it

All the tutorials that I’ve found on how to do this involve doing Xcode project goop where it’s not clear what’s happening underneath. But despite the fact that the introductory docs aren’t quite there, the underlying model for codesigning stuff is totally common across GUI and command-line cases. However, actually getting your cert requires Xcode, an apple ID, and a credit card.

After paying your hundred dollars, go into Xcode, go to Accounts, hit “+”, “Apple ID”, then log in. Then, in your shiny new account, go to “Manage Certificates”, hit the little “+”, and (assuming, like me, you want to put something up on your own website, and not submit to the Mac App Store), and choose Developer ID Application. You probably think you want “mac app distribution” because you are wanting to distribute a mac app! But you don’t.

Next, before you do anything else, make sure you have backups of your certificate and private key. You really don’t want to lose the private key associated with that cert.

Now quit Xcode; you’re done with the GUI.

You will need to know the identifier of your signing key though, which should be output from the command:

1

$ security find-identity -v -p codesigning | grep 'Developer ID' | sed -e 's/.*"\(.*\)"/\1/'

You probably want to put that in your build script, since you want to sign with the same identity every time. The command to do the signing is:

1

$ codesign -fs "${YOUR_DEVELOPER_ID_IDENTITY}" --deep "dist/${NAME}.app"

Step 5: Archive it

The right way to do this is probably to use dmgbuild or something like it, but what I promised here was quick and dirty, not beautiful and best practices.

You have to make an archive that preserves symbolic links. There are a few options for this:

• open dist/, then in the Finder window that comes up, right click on the app and “compress” it
• cd dist; tar czf $NAME.app.tgz $NAME.app
• cd dist; zip -yr $NAME.app.zip $NAME.app

Most importantly, if you use the zip command line tool, you must use the -y option. Without it, your downloadable app bundle will be somewhat mysteriously broken even though the one before you zipped it will be fine.

Step 6: Download it

Ideally, at this point, everything should be working. But to make sure that code-signing and archiving went correctly, you should have either a pristine virtual machine with no dev tools and no Python installed, or a non-programmer friend’s machine that can serve the same purpose. They probably need a relatively recent macOS - I know that apps made using the above technique will definitely work on High Sierra and will definitely break on Yosemite; they probably start working at some OS version between those.

There’s no tooling that I know of that can clearly tell you whether your mac app depends on some detail of your local machine. Even for your dependencies, there’s no auditwheel for macOS. So it’s always a good idea to check your final app build on a fresh computer before you announce it.

Coda

If you were expecting to get to the end and download my cool game, sorry to disappoint! It really is a half-broken prototype that is in no way ready for public consumption, and given my current load of personal and professional responsibilities, you definitely shouldn’t expect anything from me in this area any time soon, or, you know, ever.

But, from years of experience, I know that it’s nearly impossible to summon any motivation to work on small projects like this without the knowledge that the end result will be usable in some way, so I hope that this helps someone else set up their Python game-dev pipeline.

I’d really like to turn this into a 3-part series, with a part for Linux (perhaps using flatpak? is that a good thing?) and a part for Windows. However, given my aforementioned time constraints, I don’t think I’m going to have the time or energy to do that research, so if you’ve got the appropriate knowledge, I’d love to host a guest post on this blog, or even just a link to yours.

If this post helped you, if you have questions or corrections, or if you’d like to write the Linux or Windows version of this post, let me know.

Jupyter is a tool that came out of the data science community. In science, being able to replicate experiments is of the utmost importance -- so a tool where you can "show your work" is helpful. However, being able to show your work -- have colleagues validate what you have done, repeat it if needs be, and learn new techniques -- is also useful in the world of Site Reliability Engineering and DevOps.

The Jupyter console allows us to experiment (carefullly!) with APIs, running one function at a time, and validating the results. It allows building the needed automation from simple atoms, all the while learning how to use them.

The Jupyter Python kernel is popular in the data science community because so many data science libraries are available for Python. Luckily, the same is true in the SRE/DevOps community.

import github3
import os

The GitHub API has several client libraries in Python. My personal favorite is github3 -- I find its interface allowing remarkably idiomatic Python.

fname=os.path.expanduser("~/.git-token")
with open(fname) as fin:
    token = fin.read().strip()
gh = github3.login(token=token)

I have prepared a GitHub authentication token in a file. This allows the NoteBook to be published widely, without leaking access credentials. Never put usernames and passwords in Jupyter notebooks!

repositories = list(gh.organization('twisted').iter_repos())
repositories[:3], len(repositories)

([<Repository [twisted/txmongo]>,
  <Repository [twisted/twisted]>,
  <Repository [twisted/txaws]>],
 26)

This is a list of the repositories in the Twisted GitHub organization. It is nice to be able to validate we got a reasonable value by checking the first three. In previous versions of the notebook, my usage of github3 had an error -- and this was a list of all repositories I had access to, including private ones! Being able to inspect values interactively meant I could fix the bug easily.

repositories[1].branch('trunk').commit.sha

'51d23b6cb004e60ed699b3b100b4c476f5aebc4e'

As an example of why this might be useful, we are checking the commit hash of the trunk branch. This can be used in validating which version we are running somewhere, or checking if there have been new commits.

The GitHub API is big, and this is not meant to be an exhaustive guide to it. However, this approach is powerful -- it can be used, for example, to automatically create pull requests for a list of repositories. This comes in handy when needing to change the build structure, for example.

from fabric import api as fabpi

The Fabric library, (here used in its fabric3 fork) is an all-purpose ad-hoc library for operations.

Again, a full tutorial is beyond the scope of this article.

fabpi.local("uname -a", capture=True)

However, the advantages of running Fabric from Jupyter notebook are big. Because Fabric is specifically designed for ad-hoc operations, there is often no way to know exactly what someone did. Even with a fabfile, the logging is often lacking.

Running the functions from a notebook means an official log of what was done -- that can easily be attached to the relevant ticket, or to the post-mortem. (Either the post-mortem that the operations were meant to fix, or the ones they inevitably caused).

import docker
client = docker.DockerClient(base_url='unix://var/run/docker.sock')

The Docker client is also available as a Python library. Once again, the possibilities are endless.

[im for im in client.images.list() if (im.tags or [''])[0].startswith('debian')]

[<Image: 'debian:latest'>,
 <Image: 'debian:stable-slim', 'moshez/debian:2017-10-26T10-58-56-455022'>]

import boto3

The boto3 library is an API for the Amazon Web Services -- which includes everything from load balancers, through orchestrating containers, to sending e-mail.

The Jupyter console is a great adjunct to the AWS web console -- while results can often be inspected in the web console, any actions done from the notebook can be repeated, tweaked, and automated.

ec2 = boto3.client('ec2', region_name='us-west-2')

Hank Green recently shared the embarrassing secret to his productivity, which he summarises as:

Everything creative I do, I do my best to get it 80% of the way to as good as I can make it and go no further. I just don’t try to get it to 100%.

I recommend watching the video or reading the transcript, as he does a great job of explaining why and how this works.

I want to try to adopt this attitude in 2018. I have prided myself—perhaps without justification—on producing high-quality work but I expend way too much time fine-tuning and end up writing, speaking, and coding less than I would like.

That said, there might be a difference between creative projects that get released and then abandoned and software engineering projects that get deployed and then maintained forever.

With software projects like that, there’s an amazing power to continuously applying effort to the same code-base to make it incrementally better. Slow, steady, 80% efforts that acculumate over time to make something beautiful, maintainable and useful.

But you never know when the budget is going to be yanked from under you. In the worst case, you’ll be on the hook for running something in production without having the time or opportunity to fix the problems you find.

It’s the fear of this that sometimes drives me to reach for 100% as good as I can do. I guess the only thing I can do about this is ignore the fear.

Have you ever made conscious efforts to improve your productivity by lowering your ambitions around quality? How did it go?

Since the inception of wheels that install Python packages without executing arbitrary code, we need a static way to encode conditional dependencies for our packages. Thanks to PEP 508 we do have a blessed way but sadly the prevalence of old setuptools and pip versions make it a minefield to use.

Ten tricks to level up your Python.

(Thanks to Tom Prince and Nelson Elhage for suggestions for improvement.)

Nowadays, almost all text will be encoded in UTF-8 -- for good reasons, it is a well thought out encoding. Some of it will be in Latin 1, AKA ISO-8859-1, which is popular in the western world. Less of it will be in other members of the ISO-8859 family (-2 or higher). Some text from Japan will occasionally still be in Shift-JIS. These encodings are all reasonable -- too reasonable.

What about more interesting encodings?

>>> "hello".encode('cp1140')
b'\x88\x85\x93\x93\x96'

MASK = (1 << 8) - 1
with open('input', 'rb') as fin, open('output', 'wb') as fout:
    while True:
        c = fin.read(1)
        if not c:
            break
        c = c & MASK # <--- this right here
        fout.write(c)

>>> source = b'I\xe2\x9d\xa4U'
>>> declaration = source.decode('utf-8')
>>> declaration.encode('punycode')
b'IU-ony'

>>> source = b'I\xe2\x9d\xa4me'
>>> source.decode('utf-8').encode('punycode')
b'Ime-4r6a'

"It's ok to mess up your own room."

Sometime there is a problem where the design is obvious -- at least to you. Maybe it's simple. Maybe you've solved one like that many times. In those cases, just go ahead -- use Test-Driven-Development, lint your code as you're writing, and push a branch full of beautiful code, ready to be merged.

This is the exception, rather than the rule, though. In real life, half of the time, we have no idea what we are doing. Is a recursive or iterative solution better? How exactly does this SaaS work? What does Product Management actually want? What is, exactly, the answer to life, the universe and everything?

A lot of the time, solving a problem becomes with exploratory programming. Writing little snippets, testing them out, writing more snippets, throwing some away when they seem to be going down a bad path, saving some from earlier now that we understand the structure better. Poking and prodding at the problem, until the problem's boundaries become clearer.

This is, after all, why dyamic languages became popular -- Python became popular in web development and scientific computing precisely because in both places, "exploratory programming" is important.

In those cases, every single rule about "proper software development" goes straight out the window. Massive functions are fine, when you don't know how to break them. Code with one letter variables is fine, when you are likely to throw it away. Code with bad formatting is fine, when you are likely to refactor it. Code with no tests is fine, if it's doing the wrong thing anyway. Code with big commented out sections is fine, if those are likely to prove useful in an hour.

In short, every single rule of "proper software development" goes out the window when we are exploring a problem, testing its boundaries. All but one -- work on a branch, and keep your work backed up. Luckily, all modern version control systems have good branch isolation and easy branch pushing, so there is no problem. No problem, except the social one -- people are embarrassed at writing "bad code". Please don't be. Everyone does it. Just don't merge it into the production code -- that will legitimately annoy other people.

But as long as the mess is in your own room, don't worry about cleaning it up.