Planet Twisted

May 11, 2017

Hynek Schlawack

Please Fix Your Decorators

If your Python decorator unintentionally changes the signatures of my callables or doesn’t work with class methods, it’s broken and should be fixed. Sadly most decorators are broken because the web is full of bad advice.

by Hynek Schlawack ( at May 11, 2017 12:00 PM

May 01, 2017

Hynek Schlawack

Sharing Your Labor of Love: PyPI Quick and Dirty

A completely incomplete guide to packaging a Python module and sharing it with the world on PyPI.

by Hynek Schlawack ( at May 01, 2017 12:00 PM

April 29, 2017

Moshe Zadka

April 19, 2017

Glyph Lefkowitz

So You Want To Web A Twisted

As a rehearsal for our upcoming tutorial at PyCon, Creating And Consuming Modern Web Services with Twisted, Moshe Zadka, we are doing a LIVE STREAM WEBINAR. You know, like the kids do, with the video games and such.

As the webinar gods demand, there is an event site for it, and there will be a live stream.

This is a practice run, so expect “alpha” quality content. There will be an IRC channel for audience participation, and the price of admission is good feedback.

See you there!

by Glyph at April 19, 2017 03:29 AM

Moshe Zadka

Twisted Tutorial Webinar

Glyph and I are giving a tutorial about Twisted and web services at PyCon. In order to try it out, we are giving a webinar. Please come, learn, and let us know if you like it!

by moshez at April 19, 2017 03:14 AM

April 17, 2017

Itamar Turner-Trauring

Learning without a mentor: how to become an expert programmer on your own

If you're an intermediate or senior programmer you may hit the point where you feel you're no longer making progress, where you're no longer learning. You're good at what you do, but you don't know what to learn next, or how: there are too many options, it's hard to get feedback or even tell you're making progress.

A mentor can help, if they're good at teaching... but what do you do if you don't have a mentor? How do you become a better programmer on your own?

In order to learn without a mentor you need to be able to recognize when you're learning and when you're not, and then you need to choose a new topic and learn it.

How to tell if you're learning

If you're not getting any feedback from an expert it can be tricky to tell whether you're actually learning or not. And lacking that knowledge it's easy to get discouraged and give up.

Luckily, there's an easy way to tell whether you're learning or not: learning is uncomfortable. If you're in your comfort zone, if you're thinking to yourself "this isn't so hard, I can just do this" you're not learning. You're just going over things you already know; that's why it feels comfortable.

If you're irritated and a little confused, if you feel clumsy and everything seems harder than it should be: now you're learning. That feeling of clumsiness is your brain engaging with new material it doesn't quite understand. You're irritated because you can't rely on existing knowledge. It's hard because it's new. If you feel this way don't stop: push through until you're feeling comfortable again.

You don't want to take this too far, of course. Pick a topic that is too far out of your experience and it will be so difficult you will fail to learn anything, and the experience may be so traumatic you won't want to learn anything.

Choosing something to learn

When choosing something to learn you want something just outside your comfort zone: close enough to your existing knowledge that it won't overwhelm you, far enough that it's actually new. You also want to pick something you'll be able to practice: without practice you'll never get past the point of discomfort.

Your existing job is a great place to practice new skills because it provides plenty of time to do so, and you'll also get real-world practice. That suggests picking new skills that are relevant to your job. As an added bonus this may give you the opportunity to get your employer to pay for initial training or materials.

Let's consider some of the many techniques you can use to learn new skills on the job.


If you have colleagues you work with you will occasionally see them do something you think is obviously wrong, or miss something you think is the obviously right thing to do. For example, "obviously you should never do file I/O in a class constructor."

When this happens the tempting thing to do, especially if you're in charge, is to just tell them to change to the obviously better solution and move on. But it's worth resisting that urge, and instead taking the opportunity to turn this into a learning experience, for them and for you.

The interesting thing here is the obviousness: why is something obvious to you, and not to them? When you learn a subject you go through multiple phases:

  • Conscious ignorance: you don't know anything.
  • Conscious knowledge: you know how to do the task, but you have to think it through.
  • Unconscious knowledge: you just know what to do.

When you have unconscious knowledge you are an expert: you've internalized a model so well you apply it automatically. There's are two problems with being an expert, however:

  • It's hard for you to explain why you're making particular decisions. Since your internal model is unconscious you can't clearly articulate why or how you made the decision.
  • Your model is rarely going to be the best possible model, and since you're applying it unconsciously you may have stopped improving it.

Teaching means taking your unconscious model and turning it into an explicit conscious model someone else can understand. And because teaching makes your mental model conscious you also get the opportunity to examine your own assumptions and improve your own understanding, ending up with a better model.

You don't have to teach only colleagues, of course: you can also write a blog post, or give a talk at a meetup or at a conference. The important thing is to notice the places you have unconscious knowledge and try to make it conscious by explaining it to others.

Switching jobs

While learning is uncomfortable, I personally suffer from a countervailing form of discomfort: I get bored really easily. As soon as I become comfortable with a new task or skill I'm bored, and I hate that.

In 2004 I joined a team writing high performance C++. As soon as I'd gotten just good enough to feel comfortable... I was bored. So I came up with slightly different tasks to work on, tasks that involved slightly different skills. And then I was bored again, so I moved on to a different team in the company, where I learned even more skills.

Switching tasks within your own company, or switching jobs to a new company, is a great way to get out of your comfort zone and learning something new. It's daunting, of course, because you always end up feeling clumsy and incompetent, but remember: that discomfort means you're learning. And every time you go through the experience of switching teams or jobs you will become better at dealing with this discomfort.

Learning from experts: skills, not technologies

Another way to learn is to learn from experts that don't work with you. It's tempting to try to find experts who will teach you new technologies and tools, but skills are actually far more valuable.

Programming languages and libraries are tools. Once you're an experienced enough programmer you should be able to just pick them up as needed if they become relevant to your job.

Skills are trickier: testing, refactoring, API design, debugging... skills will help you wherever you work, regardless of technology. But they're also easier to ignore or miss. There are skills we'd all benefit from that we don't even know exist.

So read a book or two on programming, but pick a book that will teach you a skill, not a technology. Or try to find the results of experts breaking down their unconscious models into explicit models, for example:

Conclusion: learning on your own

You don't need a mentor to learn. You can become a better software engineer on your own by:

  • Recognizing when you're feeling comfortable and not learning.
  • Finding ways to learn, including teaching, switching jobs and learning skills from experts (and I have some more suggestions here).
  • Forcing yourself through the discomfort of learning: if you feel incompetent you're doing the right thing.

April 17, 2017 04:00 AM

April 13, 2017

Moshe Zadka

April 06, 2017

Itamar Turner-Trauring

You don't need a Computer Science degree

If you never studied Computer Science in school you might believe that's made you a worse programmer. Your colleagues who did study CS know more about algorithms and data structures than you do, after all. What did you miss? Are you really as good?

My answer: you don't need to worry about it, you'll do just fine. Some of the best programmers I've worked with never studied Computer Science at all.

A CS education has its useful parts, but real-world programming includes a broad array of skills that no one person can master. Programming is a team effort, and different developers can and should have different skills and strengths for the team to succeed. Where a CS degree does give you a leg up is when you're trying to get hired early in your career, but that is a hurdle that many people overcome.

What you learn in Comp Sci

I myself did study CS and Mathematics in college, dropped out of school, and then later went back to school and got a liberal arts degree. Personally I feel the latter was more useful for my career.

Much of CS is devoted to theory, and that theory doesn't come up in most programming. Yes, proving that all programming languages are equivalent is interesting, but that one sentence is all that I've taken away from a whole semester's class on the subject. And yes, I took multiple classes on data structures and algorithms... but I'm still no good at implementing new algorithms.

I ended up being bored in most of my CS classes. I dropped out to get a programming job where I could just write code, which I enjoyed much more. In some jobs that theory I took would be quite useful, but for me at least it has mostly been irrelevant.

Writing software in the real world

It's true that lacking a CS education you might not be as good as data structures. But chances are you have some other skill your colleagues lack, a unique strength that you contribute to your team.

Let me give a concrete example: at a previous job we gave all candidates a take home exercise, implementing a simple Twitter-like server. I reviewed many of the solutions, and each solution had different strengths. For example:

  • Doing a wonderful job packaging a solution and making it easy to deploy.
  • Choosing a data structure that made access to popular feeds faster, for better scalability.
  • Discussing how the API was badly designed and could result in lost messages, and suggesting improvements.
  • Discussing the ways in which the design was likely to cause business problems.
  • Adding monitoring, and explaining how to use the monitoring to decide when to scale up the service.

Packaging, data structures, network API design, big picture thinking, operational experience: these are just some of the skills that contribute to writing software. No one person can have them all. That means you can focus on your own strengths, because you're part of a team. Here's what that's meant in my case:

  • I'm pretty bad at creating new data structures or algorithms, and mostly it doesn't matter. Sometimes you're creating new algorithms, it's true. But most web developers, for example, just need to know that hashmaps give faster lookups than iterating over a list. But I've had coworkers who were good at it... and they worked on those problems when they came up.
  • In my liberal arts degree I learned how to craft better abstractions, and writing as a form of thinking. This has proven invaluable when designing new products and solving harder problems.
  • From the friends I made working on open source projects I learned about testing, and how to build robust software. Many of them had no CS education at all, and had learned on their own, from books and forums and their own practice.

Job interviews

The one place where a Computer Science degree is unquestionably useful is getting a job early in your career. When you have no experience the degree will help; once you have experience your degree really doesn't matter.

It's true that many companies have an interview process that focuses on algorithmic puzzles. Unfortunately interviewing skills are often different than job skills, and need to be strengthened separately. And my CS degree doesn't really help me, at least: I've forgotten most of what I've learned, and algorithms were never my strength. So whenever I'm interviewing for jobs I re-read an old algorithms textbook and do some practice puzzles.

In short: don't worry about lacking a CS degree. Remember that you'll never be able to know everything, or do everything: it's the combined skills of your whole team that matters. Focus on your strengths, improve the skills you have, and see what new skills you can learn from your teammates.

April 06, 2017 04:00 AM

March 26, 2017

Itamar Turner-Trauring

Why and how you should test your software

This is a major rewrite second draft of a talk I'll be giving at PyCon 2017. Thanks to all those who commented on the first draft. Once again I would appreciate feedback, comments and contrasting points of view.

Why should you test your software? How should you test your software? Some people have easy answers to these questions.

Yolo programmers don't bother testing at all, happy to live in the moment. More serious programmers will tell you that you test your software in order to produce a Quality Product. To produce a Quality Product you must always write unit tests and integration tests and do QA. Neglect any of these and your code will fall into a bug-infested abyss.

While I'm much more sympathetic to this second view, I don't think it's a sufficient answer. Given how different software projects can be from each other it seems unlikely that one set of answers will fit everyone:

  • A mobile game is not the same as a medical device is not the same as an online store.
  • Every organization is different: a bootstrapped startup has very different resources than a massive multinational, and neither are very much like NASA.
  • Every project goes through different stages and phases. A product with no users is in a very different position than a product used by thousands of businesses.

What you need is not a single answer, but a way to choose the answers that match your situation and your needs. We'll start by considering the means you have available to you for testing. Then we'll consider why you would to test your software. Finally, we'll combine means and goals to see how you can choose to test your software.

What means of testing can you use?

As a starting point, let's consider the different means of testing available to you. Is the following code a test?

def test_add():
    assert add(2, 2) == 5

I would say that, yes, that's clearly a test. Says so right in the function name, even. The test proves that add() does what it ought to do: add two numbers and give us the result.

You've noticed, of course, that this test is wrong. Luckily our development process has another step: code review. You, dear reader, can act as a code reviewer and tell me that my code is wrong, that 2 + 2 is 4, not 5.

Is code review a form of testing? If you're trying to verify your code matches a specification, well, this is a test. You have a specification for arithmetic in your head ("2 + 2 = 4") and you are checking that the code follows it.

Let's consider code review as a form of testing, alongside automated unit tests. Even if they're both tests, they're also quite different. What is the core difference between them?

One form of testing is automated, the other is done by a human.

An automated test is consistent and repeatable. You can write this:

def test_add_twice():
    for i in range(10000000):
        assert add(i, i) == 2 * i

And the computer will run the exact same code every time. The code will make sure add() consistently returns that particular result for those particular inputs. A human would face some difficulties in manually verifying ten million different computations: boredom, distraction, errors, slowness.

On the other hand, a human can read this code and tell you it's buggy:

def add(a, b):
    return a + b + 1

Where the computer does what it's told, for good or for bad, a human can provide meaning. Only a human can tell what the software is for.

Now we can categorize tests by the means used: humans test for meaning, whereas automated tests ensure consistency.

Why should you test your software?

Next, let's consider goals.

The first possible goal of testing your software is to make sure it meets a specification. This goal is what most programmers think of when testing is discussed: it covers things like unit testing and manual testing by QA. And as we saw it also covers code reviews. Your software has certain required functionality, the specification, and you want to make sure it actually does so now and in the future.

Some of the requirements are high-level: an online store wants customers to be able to order a product they've added to their shopping cart. Other requirements are low-level implementation details, mostly of interest only to the programmers. You might want the verify_creditcard() to accept a credit card number as a string and throw a InvalidCreditCard exception if the credit card is invalid.

The sum of all these requirements is the specification. It might be written down in great detail, or it might be a notion in your head (e.g. "2 + 2 is 4"). Regardless, you test your software to make sure it does what it's supposed to.

Sometimes, however, testing can have a different goal. In his book "Lean Startup" Eric Ries talks about building software only to discover that no one actually wanted to use it. Spending much time testing to ensure your software meets the specification is a waste of time if no one will ever use your software.

Ries argues that you first need to figure out if a product will succeed, by testing out what he calls a "Minimum Viable Product" with potential users and customers. This is a very different form of testing: it's not about verifying whether your software meets the specification, it's about learning something you hadn't known before.

The second possible goal of testing your software is in order to gain knowledge. Let's look at another form of testing with this goal. "A/B testing" is a form of testing where you try out two variations and see which produces a better result. Perhaps you are testing a redesign of your website: you show the current design to 90% of your visitors, the new design to 10% of your visitors, and see which results in more signups for your product.

Notice that you have two specifications, and you've already implemented them both. The point of the test is to figure out which works better, to learn something new, not to verify if the implementation meets the specification.

Now we have an answer for why you should test: either to verify you meet the specification or to gain new knowledge.

How should you test your software?

Combine the two goals we've come up with (gaining knowledge and matching the specification) and the two means of testing (human and software) and you get four different forms of testing, each providing a more specific testing goal:

Goal: Knowledge
Means: Humans Understanding Users Understanding Runtime Behavior Means: Software
Correct Functionality Stable Functionality
Goal: Implement the specification

You must choose the appropriate ones to use for your particular needs and situation. Let's go through these four types of testing one by one and see when you should each.

Understanding Users

  • Will people buy your product?
  • Will a design change result in more signups?
  • Will users understand how your software works?

These are all questions that cannot be answered by comparing your software to a specification. Instead you need empirical knowledge: you need to observe what actual human beings do when presented with your software.

Relevant testing techniques include:

  • Usability tests.
  • Minimum viable product testing ("Lean Startup").
  • A/B testing.

Understanding Runtime Behavior

  • How does your software behave under load?
  • Does your software have race conditions?
  • Does your software break with unexpected inputs?

These questions can't always be answered by comparing your software to a specification. Once your software is complex enough you can't fully understand or predict how it will behave. You need to observe it actually running to understand its behavior.

Relevant testing techniques include:

  • Stress tests and soak tests.
  • Gathering tracebacks and exceptions from your production logs.

Correct Functionality

  • Does your software actually match the specification?
  • Does it do what it's supposed to?

It's tempting to say that automated tests can prove this, but remember the unit test that checked that 2 + 2 is 5. On a more fundamental level, software can technically match a specification and completely fail to achieve the goal of the specification. Only a human can understand the meaning of the specification and decide if the software matches it.

Relevant testing techniques include:

  • Manual user interface tests (e.g. a QA person using your website).
  • Code review.

Stable Functionality

  • Does your public API return the same results for the same inputs?
  • Is your code providing the guarantees it ought to provide?

Humans are not a good way to test this. Humans are pretty good at ignoring small changes: if a button changes from "Send Now" to "Send now" you might not even notice at all. In contrast, software will break if your API changes from sendNow() to send_now(), or if the return type changes subtly.

This means a public API, an API that other software relies on, needs stability in order to be correct. Writing automated tests for private APIs, or code that is changing rapidly, will result in high maintenance costs as you continuously update your tests.

Relevant testing techniques include:

  • Unit tests, integration tests, and other similar automated API tests.
  • Automated user interface tests (e.g. Selenium tests for websites).
  • Compiler checks and linters.

Applying the model

So what is all this good for?

Choosing how to test

First, our initial purpose: the model can help you choose what form of testing to do based on your goals.

Consider a startup building a product that it's not sure anyone wants. Writing automated tests may prove a waste of time, since it focuses on implementing a specification before figuring out what users actually want.

One possible alternative the Lean Startup methodology, which focuses on experiments or tests whose goal is finding what product will meet customers' needs. That means focusing on the Understanding Users quadrant. Once a product is chosen you spending the time to have more than the most minimal of specifications, at which point much more resources are applied to Correct Functionality and Stable functionality.

Recognizing when you've chosen the wrong type of testing

Second, the model can help you change course if you're using the wrong type of testing. For example, consider a hypothetical startup that writes tax preparation software (the details are inspired by a real company.) They wrote Selenium tests for their web UI at the same time were rapidly making significant changes to their UI.

Even with the tests their application was still buggy, and every time they changed the UI the tests would break. The tests didn't seem to improve quality, and they wasted developer time maintaining them. What were they doing wrong?

Their problem was that their system really has two parts:

  1. The tax engine, which is fairly stable: the tax code changes only once per year. Errors in the tax engine are a significant problem for users, and incompatible API changes are a problem for developers. This suggests the need for Stable Functionality tests, e.g. unit tests talking directly to the tax calculation engine. Correct Functionality could be ensured by code review and feedback from a tax accountant.
  2. The web-based user interface. The UI was changing constantly, which suggests Stable Functionality wasn't a goal yet. Correct Functionality was still a goal, so the UI should have been tested manually by humans (e.g. the programmers as they wrote the code.)

A basis for discussing testing

Finally, the model provides a shared terminology that can help you discuss testing in its broadest sense and its many differing goals.

  • Instead of getting into fruitless arguments about whether manual testing is a good idea or unit testing is a good idea, you can start with a model that shows very clearly the differences between them.
  • You can also discuss testing with other parts of the company (e.g. marketing) that are starting from a very different point of view.


Why should you test your software? Either to gain knowledge or to implement a specification.

How can you test your software? With humans or with software.

How should you test your software? Depending on your particular situation, choose the relevant forms of testing for understanding users, understanding runtime behavior, stable functionality or consistent functionality.

Questions or suggestions? Add a comment to the Hacker News or Reddit discussions linked below.

March 26, 2017 04:00 AM

March 20, 2017

Itamar Turner-Trauring

Dear recruiter, "open floor space" is not a job benefit

I was recently astonished by a job posting for a company that touted their "open floor space." Whoever wrote the ad appeared to sincerely believe that an open floor office was a reason to work for the company, to be mentioned alongside a real benefit like work/life balance.

While work/life balance and an open floor plan can co-exist, an open floor plan is a management decision much more akin to requiring long hours: both choose control over productivity.

The fundamental problem facing managers is that productivity is hard to measure. Faced with the inability to measure productivity, managers may feel compelled to measure time spent working. Never mind that it's counter-productive: at least it gives management control, even if it's control over the wrong thing.

Here is a manager explaining the problem:

I [would] like to manage [the] team's output rather than managing their time, because if they are forced to spend time inside the office, it doesn't mean they are productive or even working. At the same time it's hard to manage output, because software engineering tasks are hard to estimate and things can go out of the track easily.

In this case at least the manager involved understands that what matters is output, not hours in the office. But not everyone realizes is as insightful.

Choosing control

In cases where management does fall into the trap of choosing control over productivity, the end result is a culture where the only thing that matters is hours in the office. Here's a story I heard from a friend about a startup they used to work at:

People would get in around 8 or 9, because that's when breakfast is served. They work until lunch, which is served in the office, then until dinner, which is served in the office. Then they do social activities in the office, video games or board games, and then keep working until 10PM or later. Their approach was that you can bring your significant other in for free dinner, and therefore why leave work? Almost like your life is at the office.

Most of the low level employees, especially engineers, didn't feel that this was the most productive path. But everyone knew this was the most direct way to progress, to a higher salary, to becoming a team lead. The number of hours in the office is a significant part of how your performance is rated.

I don't think people can work 12 hours consistently. And when you're not working and you're in an open plan office, you distract people. It's not about hours worked, it's about hours in office, there's ping pong tables... so there's always someone asking you to play ping pong or distracting you with a funny story. They're distracting you, their head wasn't in the zone, but they had to be in the office.

A team of 10 achieved what a team of 3 should achieve.

Control through visibility

Much like measuring hours in the office, an open floor office is designed for control rather than productivity. A manager can easily see what each developer is doing: are they coding? Are they browsing the web? Are they spending too much time chatting to each other?

In the company above the focus on working hours was apparently so strong that the open floor plan was less relevant. But I've no doubt there are many companies where you'll start getting funny looks from your manager if you spend too much time appearing to be "unproductive."

To be fair, sometimes open floor plans are just chosen for cheapness, or thoughtlessly copied from other companies because all the cool kids are doing it. Whatever the motivation, they're still bad for productivity. Programming requires focus, and concentrated thought, and understanding complex systems that cannot fit in your head all at once and so constantly need to be swapped in and out.

Open floor spaces create exactly the opposite of the environment you need for programming: they're full of noise and distraction, and headphones only help so much. I've heard of people wearing hoodies to block out not just noise but also visual distraction.

Dear recruiter

Work/life balance is a real job benefit, for both employers and employees: it increases productivity while allowing workers space to live outside their job. But "open office space" is not a benefit for anyone.

At worst it means your company is perversely sabotaging its employees in order to control them better. At best it means your company doesn't understand how to enable its employees to do their job.

March 20, 2017 04:00 AM

Moshe Zadka

March 12, 2017

Itamar Turner-Trauring

Unit testing, Lean Startup, and everything in-between

This was my first draft of a talk I'll be giving at PyCon 2017. I've since rewritten it extensively, so you should just read the much improved second draft.

March 12, 2017 05:00 AM

March 05, 2017

Itamar Turner-Trauring

Why you're (not) failing at your new job

It's your first month at your new job, and you're worried you're on the verge of getting fired. You don't know what you're doing, everyone is busy and you need to bother them with questions, and you're barely writing any code. Any day now your boss will notice just how bad a job you're doing... what will you do then?

Luckily, you're unlikely to be fired, and in reality you're likely doing just fine. What you're going through happens to almost everyone when they start a new job, and your panicked feeling will eventually pass.

New jobs make you incompetent

Just like you, every time I've started a new job I've had to deal with feeling incompetent.

  • In 2004 I started a new job as a C++ programmer, writing software for the airline industry. On my first day of work the VP of Engineering told me he didn't expect me to fully productive for 6 months. And for good reason: I barely knew any C++, I knew nothing about airlines, and I had to learn a completely new codebase involving both.
  • In 2012 I quit my job to become a consultant. Every time I got a new client I had to learn a new codebase and a new set of problems and constraints.
  • At my latest job I ended up writing code in 3 programming languages I didn't previously know. And once again I had to learn a new codebase and a new business logic domain.

Do you notice the theme?

Every time you start a new job you are leaving behind the people, processes and tools you understand, and starting from scratch. A new language, new frameworks, new tools, new codebases, new ways of doing things, people you don't know, business logic you don't understand, processes you're unfamiliar with... of course it's scary and uncomfortable.

Luckily, for the most part this is a temporary feeling.

This too will pass

Like a baby taking their first steps, or a child on their first bike ride, those first months on a new job will make you feel incompetent. Soon the baby will be a toddler, the child will be riding with confidence. You too will soon be productive and competent.

Given some time and effort you will eventually learn what you need to know:

...the codebase.

...the processes, how things are done and maybe even why.

...the programming language.

...who to ask and when to ask them.

...the business you are operating in.

Since that's a lot to learn, it will take some time, but unless you are working for an awful company that is expected and normal.

What you can do

While the incompetent phase is normal and unavoidable, there is still something you can do about it: learn how to learn better. Every time you start a new job you're going to be learning new technologies, new processes, new business logic. The most important skill you can learn is how to learn better and faster.

The faster you learn the faster you'll get past the feeling of incompetence when you start a new job. The faster you learn the faster you can become a productive employee or valued consultant.

Some skills are specific to programming. For example, when learning new programming languages, I like skimming a book or tutorial first before jumping in: it helps me understand the syntax and basic concepts. Plus having a mental map of the book helps me know where to go back to when I'm stuck. Other skills are more generic, e.g. there is considerable research on how learning works that can help you learn better.

Finally, another way I personally try to learn faster is by turning my mistakes into educational opportunities. Past and present, coding or career, every mistake I make is a chance to figure out what I did wrong and how I can do better. If you'd like to avoid my past mistakes, sign up to get a weekly email with one of my mistakes and what you can learn from it.

March 05, 2017 05:00 AM

February 19, 2017

Itamar Turner-Trauring

When AI replaces programmers

The year is 2030, and artificial intelligence has replaced all programmers. Let's see how this brave new world works out:

Hi! I'm John the Software Genie, here to help you with all your software needs.

Hi, I'd like some software to calculate the volume of my house.

Awesome! May I have access to your location?

Why do you need to access my location?

I will look up your house in your city's GIS database and use its dimensions to calculate its volume.

Sorry, I didn't quite state that correctly. I want some software that will calculate the dimensions of any house.

Awesome! What is the address of this house?

No, look, I don't want anything with addresses. You can have multiple apartments in a house, and anyway some structures don't have an address, or are just being designed... and the attic and basement doesn't always count... How about... I want software that calculates the volume of an abstract apartment.

Awesome! What's an abstract apartment?

Grrrr. I want software that calculates the sum of the volumes of some rooms.

Awesome! Which rooms?

You know what, never mind, I'll use a spreadsheet.

I'm sorry Dave, I can't let you do that.


Just a little joke! I'm sorry you decided to go with a spreadsheet. Your usage bill for $153.24 will be charged to your credit card. Have a nice day!

Back to the present: I've been writing software for 20 years, and I find the idea of being replaced by an AI laughable.

Processing large amounts of data? Software's great at that. Figuring out what a human wants, or what a usable UI is like, or what the real problem you need to solve is... those are hard.

Imagine what it would take for John the Software Genie to learn from that conversation. I've made my share of mistakes over the years, but I've learned enough that these days I can gather requirements decently. How do you teach an AI to gather requirements?

We might one day have AI that is as smart as a random human, an AI that can learn a variety of skills, an AI that can understand what those strange and pesky humans are talking about. Until that day comes, I'm not worried about being replaced by an AI, and you shouldn't worry either.

Garbage collection didn't make programmers obsolete just because it automated memory management. In the end automation is a tool to be controlled by human understanding. As a programmer you should focus on building the skills that can't be automated: figuring out the real problems and how to solve them.

February 19, 2017 05:00 AM

February 11, 2017

Twisted Matrix Laboratories

Twisted 17.1.0 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 17.1!

The highlights of this release are:

  • twisted.web.client.Agent now supports IPv6! It's also now the primary web client in Twisted, with twisted.web.client.getPage being deprecated in favour of it and Treq.
  • twisted.web.server has had many cleanups revolving around timing out inactive clients.
  • twisted.internet.ssl.CertificateOptions has had its method argument deprecated, in favour of the new raiseMinimumTo, lowerMaximumSecurityTo, and insecurelyLowerMinimumTo arguments, which take TLSVersion arguments. This allows you to better give a range of versions of TLS you wish to negotiate, rather than forcing yourself to any one version.
  • twisted.internet.ssl.CertificateOptions will use OpenSSL's MODE_RELEASE_BUFFERS, which will let it free unused memory that was held by idle TLS connections.
  • You can now call the new twist runner with python -m twisted.
  • twisted.conch.ssh now has some ECDH key exchange support and supports hmac-sha2-384.
  • Better Unicode support in twisted.internet.reactor.spawnProcess, especially on Windows on Python 3.6.
  • More Python 3 porting in Conch, and more under-the-hood changes to facilitate a Twisted-wide jump to new-style classes only on Python 2 in 2018/2019. This release has also been tested on Python 3.6 on Linux.
  • Lots of deprecated code removals, to make a sleeker, less confusing Twisted.
  • 60+ closed tickets.

For more information, check the NEWS file (link provided below).

You can find the downloads on PyPI (or alternatively our website). The NEWS file is also available on GitHub.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
Amber Brown (HawkOwl)

by Amber Brown ( at February 11, 2017 10:08 AM

February 10, 2017

Glyph Lefkowitz

Make Time For Hope

Pandora hastened to replace the lid! but, alas! the whole contents of the jar had escaped, one thing only excepted, which lay at the bottom, and that was HOPE. So we see at this day, whatever evils are abroad, hope never entirely leaves us; and while we have THAT, no amount of other ills can make us completely wretched.

It’s been a rough couple of weeks, and it seems likely to continue to be so for quite some time. There are many real and terrible consequences of the mistake that America made in November, and ignoring them will not make them go away. We’ll all need to find a way to do our part.

It’s not just you — it’s legit hard to focus on work right now. This is especially true if, as many people in my community are, you are trying to motivate yourself to work on extracurricular, after-work projects that you used to find exciting, and instead find it hard to get out of bed in the morning.

I have no particular position of authority to advise you what to do about this situation, but I need to give a little pep talk to myself to get out of bed in the morning these days, so I figure I’d share my strategy with you. This is as much in the hope that I’ll follow it more closely myself as it is that it will be of use to you.

With that, here are some ideas.

It’s not over.

The feeling that nothing else is important any more, that everything must now be a life-or-death political struggle, is exhausting. Again, I don’t want to minimize the very real problems that are coming or the need to do something about them, but, life will go on. Remind yourself of that. If you were doing something important before, it’s still important. The rest of the world isn’t going away.

Make as much time for self-care as you need.

You’re not going to be of much use to anyone if you’re just a sobbing wreck all the time. Do whatever you can do to take care of yourself and don’t feel guilty about it. We’ll all do what we can, when we can.1

You need to put on your own oxygen mask first.

Make time, every day, for hope.

“You can stand anything for 10 seconds. Then you just start on a new 10 seconds.”

Every day, set aside some time — maybe 10 minutes, maybe an hour, maybe half the day, however much you can manage — where you’re going to just pretend everything is going to be OK.2

Once you’ve managed to securely fasten this self-deception in place, take the time to do the things you think are important. Of course, for my audience, “work on your cool open source code” is a safe bet for something you might want to do, but don’t make the mistake of always grimly setting your jaw and nose to the extracurricular grindstone; that would just be trading one set of world-weariness for another.

After convincing yourself that everything’s fine, spend time with your friends and family, make art, or heck, just enjoy a good movie. Don’t let the flavor of life turn to ash on your tongue.

Good night and good luck.

Thanks for reading. It’s going to be a long four years3; I wish you the best of luck living your life in the meanwhile.

  1. I should note that self-care includes just doing your work to financially support yourself. If you have a job that you don’t feel is meaningful but you need the wages to survive, that’s meaningful. It’s OK. Let yourself do it. Do a good job. Don’t get fired. 

  2. I know that there are people who are in desperate situations who can’t do this; if you’re an immigrant in illegal ICE or CBP detention, I’m (hopefully obviously) not talking to you. But, luckily, this is not yet the majority of the population. Most of us can, at least some of the time, afford to ignore the ongoing disaster. 

  3. Realistically, probably more like 20 months, once the Rs in congress realize that he’s completely destroyed their party’s credibility and get around to impeaching him for one of his numerous crimes. 

by Glyph at February 10, 2017 07:58 AM

Itamar Turner-Trauring

Buggy Software, Loyal Users: Why Bug Reporting is Key To User Retention

Your software has bugs. Sorry, mine does too.

Doesn't matter how much you've tested it or how much QA has tested it, some bugs will get through. And unless you're NASA, you probably can't afford to test your software enough anyway.

That means your users will be finding bugs for you. They will discover that your site doesn't work on IE 8.2. They clicked a button and a blank screen came up. Where is that feature you promised? WHY IS THIS NOT WORKING?!

As you know from personal experience, users don't enjoy finding bugs. Now you have buggy software and unhappy users. What are you going to do about it?

Luckily, in 1970 the economist Albert O. Hirschman came up with an answer to that question.

Exit, Voice and Loyalty

In his classic treatise Exit, Voice and Loyalty, Hirschman points out that users who are unhappy with a product have exactly two options. Just two, no more:

  1. Exiting, i.e. giving up on your product.
  2. Voicing their concerns.

Someone who has given up on your software isn't likely to tell you about their issues. And someone who cares enough to file a bug is less likely to switch away from your software. Finally, loyal users will stick around and use their voice when otherwise they would choose exit.

Now, your software has no purpose without users. So chances are you want to keep them from leaving (though perhaps you're better off without some of them - see item #2).

And here's the thing: there's only two choices, voice or exit. If you can convince your users to use their voice to complain, and they feel heard, your users are going to stick around.

Now at this point you might be thinking, "Itamar, why are you telling me obvious things? Of course users will stick around if we fix their bugs." But that's not how you keep users around. You keep users around by allowing them to express their voice, by making sure they feel heard.

Sometimes you may not fix the bug, and still have satisfied users. And sometimes you may fix a bug and still fail at making them feel heard; better than not fixing the bug, but you can do better.

To make your users feel heard when they encounter bugs you need to make sure:

  1. They can report bugs with as little effort as possible.
  2. They hear back from you.
  3. If you choose to fix the bug, you can actually figure out the problem from their bug report.
  4. The bug fix actually gets delivered to them.

Let's go through these requirements one by one.

Bug reporting

Once your users notice a problem you want them to communicate it to you immediately. This will ensure they choose the path of voice and don't contemplate exiting to your shiny competitor at the domain next door.

Faster communication also makes it much more likely the bug report will be useful. If the bug occurred 10 seconds ago the user will probably remember what happened. If it's a few hours later you're going to hear something about how "there was a flying moose on the screen? or maybe a hyena?" Remember: users are human, just like you and me, and humans have a hard time remembering things (and sometimes we forget that.)

To ensure bugs get reported quickly (or at all) you want to make it as easy as possible for the user to report the problem. Each additional step, e.g. creating an account in an issue tracker, means more users dropping out of the reporting process.

In practice many applications are designed to make it as hard as possible to report bugs. You'll be visiting a website, when suddenly:

"An error has occurred, please refresh your browser."

And of course the page will give no indication of how or where you should report the problem if it reoccurs, and do the people who run this website even care about your problem?

Make sure that's not how you're treating your users when an error occurs.

Improving bug reporting

So let's say you include a link to the bug report page, and let's say the user doesn't have to jump through hoops and email verification to sign up and then fill out the 200 fields that JIRA or Bugzilla think are really important and would you like to tell us about your most common childhood nightmare from the following list? We'll assume that bug reporting is easy to find and easy to fill it out, as it should be.

But... you need some information from the user. Like, what version of the program they were using, the operating system and so on and so forth. And you do actually need that information.

But the user just wants to get this over with, and every additional piece of information you ask for is likely to make them give up and go away. What to do?

Here's one solution: include it for them.

I've been using the rather excellent Spacemacs editor, and as usual when I use new software I had to report a bug. So it goes.

Anyway, to report a bug in Spacemacs you just hit the bug reporting key combo. You get a bug reporting page. In your editor. And it's filled in the Spacemacs version and the Emacs version and all the configuration you made. And then you close the buffer and it pre-populates a new GitHub issue with all this information and you hit Submit and you're done.

This is pretty awesome. No need to find the right web page or copy down every single configuration and environmental parameter to report a bug: just run the error-reporting command.

Spacemacs screenshot

Also, notice that this is a lot better than automatic crash reporting. I got to participate and explain the bits that were important to me, it's not the "yeah we reported the crash info and let's be honest you don't really believe anyone looks at these do you?"-vibe that one gets from certain software.

You can do much the same thing with a command-line tool, or a web-based application. Every time an error occurs or the user is having issues (e.g. the user ran yourcommand --help):

  1. Ask for the user's feedback in the terminal, or within the context of the web page, and then file the bug report for them.
  2. Gather as much information as you can automatically and include it in the bug report, so the user only has to report the part they care about.

Responding to the bug report

The next thing you have to do is actually respond to the bug report. As I write this the Spacemacs issue I filed is sitting there all sad and lonely and unanswered, and it's a little annoying. I do understand, open source volunteer-run project and all. (This is where loyalty comes in.)

But for a commercial product I am paying for I want to know that my problem has been heard. And sometimes the answer might be "sorry, we're not going to fix that this quarter." And that's OK, at least I know where I stand. But silence is not OK.

Respond to your bug reports, and tell your users what you're doing about it. And no, an automated response is not good enough.

Diagnosing the bug

If you've decided to fix the bug you can now proceed to do so... if you can figure out what the actual problem is. If diagnosing problems is impossible, or even just too expensive, you're not going to do fix the problem. And that means your users will continue to be affected by the bug.

Let's look at a common bug report:

I clicked Save, and your program crashed and deleted a day's worth of my hopes and dreams.

This is pretty bad: an unhappy user, and as is often the case the user simply doesn't have the information you need to figure out what's going on.

Even if the bug report is useful it can often be hard to reproduce the problem. Testing happens in controlled environments, which makes investigation and reproduction easier. Real-world use happens out in the wild, so good luck reproducing that crash.

Remember how we talked about automatically including as much information as possible in the bug report? You did that, right? And included all the relevant logs?

If you didn't, now's the time to think about it: try to ensure that logs, core dumps, and so on and so forth will always be available for bug reports. And try to automate submitting them as much as possible, while still giving the user the ability to report their specific take on the problem.

(And don't forget to respect your user's privacy!)

Distributing the fix

So now you've diagnosed and fixed your bug: problem solved! Or rather, problem almost solved. If the user hasn't gotten the bug fix all this work has been a waste of time.

You need fast releases, and you need automated updates where possible. If it takes too long for fixes to reach your users then for for practical purposes your users are not being heard. (There are exceptions, as always: in some businesses your users will value stability over all else.)

A virtuous circle

If you've done this right:

  1. Your users will feel heard, and choose voice over exit.
  2. You will get the occasional useful bug report, allowing you to improve your product.
  3. All your users will quickly benefit from those improvements.

There is more to be done, of course: many bugs can and should be caught in advance. And many users will never tell you their problems unless you ask.

But it's a good start at making your users happy and loyal, even when they encounter the occasional bug.

February 10, 2017 05:00 AM

January 31, 2017

Jonathan Lange

Announcing haskell-cli-template

Last October, I announced servant-template, a cookiecutter template for creating production-ready Haskell web services.

Almost immediately after making it, I wished I had something for building command-line tools quickly. I know stack comes with a heap of them, but:

  • it’s hard to predict what they’ll do
  • adding a new template requires submitting a PR
  • cookiecutter has existed for ages and is pretty much better in every way

So I made haskell-cli-template. It’s very simple, it just makes a Haskell command-line project with some tests, command-line parsing, and a CircleCI build.

I wanted to integrate logging-effect, but after a few months away from it my tired little brain wasn’t able to figure it out. I like command-line tools with logging controls, so I suspect I’ll add it again in the future.

Let me know if you use haskell-cli-template to make anything cool, and please feel free to fork and extend.

by Jonathan Lange at January 31, 2017 12:00 AM

January 30, 2017

Jonathan Lange

Announcing graphql-api: Haskell library for GraphQL

Late last year, my friend Tom tried to convince me that writing REST APIs was boring and repetitive and that I should give this thing called GraphQL a try.

I was initially sceptical. servant, the REST library that I’m most familiar with, is lovely. Its clever use of Haskell’s type system means that all the boring boilerplate I’d have to write in other languages just goes away.

However, after watching Lee Byron’s Strange Loop talk on GraphQL I began to see his point. Being able to get many resources with the same request is very useful, and as someone who writes & runs servers, I very much want clients to ask only for the data that they need.

The only problem is that there isn’t really a way to write GraphQL servers in Haskell—until now.

Introducing graphql-api

Tom and I put together a proof-of-concept GraphQL server implementation called graphql-api, which we released to Hackage today.

It lets you take a GraphQL schema and translate it into a Haskell type that represents the schema. You can then write handlers that accept and return native Haskell types. graphql-api will take care of parsing and validating your user queries, and Haskell’s type system will make sure that your handlers handle the right thing.

Using graphql-api

Say you have a simple GraphQL schema, like this:

type Hello {
  greeting(who: String!): String!

which defines a single top-level type Hello that contains a single field, greeting, that takes a single, required argument who.

A user would query it with something like this:


And expect to see an answer like:

  "data": {
    "greeting": "Hello World!"

To do this in Haskell with GraphQL, first we’d define the type:

type Hello = Object "Hello" '[]
  '[ Argument "who" Text :> Field "greeting" Text ]

And then a handler for that type:

hello :: Handler IO Hello
hello = pure greeting
   greeting who = pure ("Hello " <> who <> "!")

We can then run a query like so:

queryHello :: IO Response
queryHello = interpretAnonymousQuery @Hello hello "{ greeting(who: \"World\") }"

And get the output we expect.

There’s a lot going on in this example, so I encourage you to check out our tutorial to get the full story.

graphql-api’s future

Tom and I put graphql-api together over a couple of months in our spare time because we wanted to actually use it. However, as we dug deeper into the implementation, we found we really enjoyed it and want to make a library that’s genuinely good and helps other people do cool stuff.

The only way to do that, however, is to release it and get feedback from our users, and that’s what we’ve done. So please use graphql-api and tell us what you think. If you build something cool with it, let us know.

For our part, we want to improve the error messages, make sure our handling for recursive data types is spot on, and smooth down a few rough edges.


Tom and I want to thank J. Daniel Navarro for his great GraphQL parser and encoder, which forms the basis for what we built here.

About the implementation

graphql-api is more-or-less a GraphQL compiler hooked up to type-based executing (aka “resolving”) engine that’s heavily inspired by Servant and uses various arcane type tricks from GHC 8.

We tried to stick to implementing the GraphQL specification. The spec is very well written, but implementing it has taught us that GraphQL is not at all as simple as it looks at first.

I can’t speak very well to the type chicanery, except to point you at the code and at the Servant paper.

The compiler mostly lives in the GraphQL.Internal.Validation module. The main idea is that it takes the AST and translates it into a value that cannot possibly be wrong.

All the syntax stuff is from the original graphql-haskell, with a few fixes, and a tweak to guarantee that Name values are guaranteed to be correct.

by Jonathan Lange at January 30, 2017 12:00 AM

January 29, 2017

Itamar Turner-Trauring

Does your job contradict your beliefs?

As an employee your work is chosen by the owners and managers of the company: you choose the means, they choose the ends. Becoming an employee doesn't mean abdicating your moral responsibility, however. Even if someone else has chosen the goals you are still responsible for your own actions.

As an employee you need to ask yourself if the goals you're working for are worthwhile, and if the people you are working for deserve the power to direct your actions.

Let me tell you about the first time I was forced to ask myself this question.

The past is a different country

I grew up Israel, where as a Jewish citizen I was subject to the draft. The State of Israel is both Zionist and democratic, inherently contradictory ideals: a nation united by ethnicity, language and land versus the natural rights of individual human beings. Most Israeli citizens are indeed Jewish, but a large minority are Arab.

The contradiction is even more acute outside the borders of Israel, in the occupied territories of the West Bank and Gaza, where Israel's Jewish military rules millions of Palestinian Arabs. Whatever their opinion of the Occupation, most Israeli Jews are Zionists. They believe Israel must remain a Jewish state with a Jewish army, a bulwark against Palestinian terrorism and Israel's hostile Arab neighbors.

At age seventeen I had my first contact with the military: I was summoned to an initial screening at the Bakum, a military base that processes new recruits. The base was covered with asphalt, concrete, barbed wire and dirt, which turned to mud when it rained. The buildings were ugly concrete blocks and flimsy metal shacks that had seen better decades. Everything, inside and out, was painted in shades of khaki, gray and rust.

At the Bakum I was examined by a doctor. He gave me a physical profile of 97, the highest possible value; urban legend has it that the extra three points are taken off for circumcision. A profile of 97 meant I would be assigned combat duty, known in Israel as kravi.

As a kravi soldier I would have been required, as other Israeli soldiers have, to demolish homes, evict families, delay ambulances at checkpoints, threaten to shoot civilians for violating curfews. I didn't believe these actions protected Israel from Palestinian terrorists.

So I took the easy way out: I signed up for the Academic Reserve. I would go to college as a civilian, studying computer science, then serve in the army as a programmer for six years.

In the summer of 1999, after my first year in college, I went through a month of unstrenuous basic training for non-combatants. I hated every minute of it, the half-hearted brainwashing and the petty sadism. The following year, bored and unmotivated, I dropped out of college and started a software company.

Sooner or later the Academic Reserve office would notice I'd stopped taking classes, and I would be drafted into a kravi unit. My mother suggested I write a letter to the army – via a distant relative who worked for the Department of Defense – explaining my qualifications as a programmer and requesting my skills be put to use. Perhaps I could reach an accommodation allowing me to work on my company in my spare time.

I was summoned to a number of interviews in the following months, at office buildings, anonymous houses in residential areas, even a war monument behind a military base. None of them went well. I thoughtlessly, in retrospect perhaps deliberately, wrecked my chances at getting a security clearance, telling an interviewer my political views: I did not believe in the leadership of the country or the military. It was unlikely I would be trusted with classified material.

My final interview was different: I actually liked the soldiers I met. They were intelligent and sympathetic and I was sure I'd enjoy working with them, but I left the interview feeling miserable. I reached not so much a decision as a self-diagnosis: I could never be a soldier, I could not give up my right and duty to make my own decisions.

In the present

I did manage to get out of military service, but that's a longer story. But before that I spent literally years refusing to admit to myself that this was not a job I was willing to take, whatever the social expectations.

Being an employee is not quite the same as being a soldier. But these days when I'm looking for a job I try to be more aware of what I am willing to do. I won't even bother applying to companies that do anything related to "defense" or the military, for example.

You will have your own criteria, of course, but I would urge you to consider whether your current employment matches your beliefs.

January 29, 2017 05:00 AM

January 26, 2017

Itamar Turner-Trauring

Coding skills you won't learn in school: Object Ownership

There are many skills you need to acquire as a programmer, and some of them are not part of the standard software engineering curriculum. Instead you're expected to learn them by osmosis, or by working with someone more experienced. David MacIver covers one such skill: tracking which type a value has.

Another skill you need is an understanding object ownership in your code: knowing which part of your code owns a particular object in memory, and what its expectations are for access. Lacking this understanding you might write code that causes your program to crash or to suffer from subtle bugs. Even worse, some programming languages won't even provide you with facilities to help you in this task.

Learning by osmosis

Here's how I learned this skill. When I was in university I was once had to implement a Red-Black Tree in C. I'd basically skipped all the classes for the intro to C class, and still gotten a perfect grade, so as far as the school was concerned I knew how to code in C.

In reality I had no clue what I was doing. I couldn't write a working tree implementation: my code kept segfaulting, I couldn't keep the structure straight. Eventually I turned in a half-broken solution and squeaked by with a grade of 60 out of 100.

I spent the next 5 years writing Python code, and then got a job writing C and C++. I did make some mistakes, e.g. my first project crashed every night (you can hear that story by signing up for my newsletter), but in general I was able to write working code even though I hadn't really written any C or C++ for years.

What changed?

I believe the one of the key skills I learned was object ownership, as a result of all the concurrent Python I was writing, plus the fact that C++ has a better model than C for object ownership. Let's see what I learned over those years.

Object ownership for memory deallocation

Consider the following C function:

char* do_something(char* input);

Someone is going to have deallocate input and someone is going to have to deallocate the returned result of do_something(). But who? If two different functions try to deallocate the same allocation your program's memory will be corrupted. If no one deallocates the memory your program will suffer from memory leaks.

This is where object ownership comes in: you ensure each allocation has only one owner, and only that owner should deallocate it. The GNOME project's developer documentation explains how their codebase makes this work:

Each allocation has exactly one owner; this owner may change as the program runs, by transferring ownership to another piece of code. Each variable is owned or unowned, according to whether the scope containing it is always its owner. Each function parameter and return type either transfers ownership of the values passed to it, or it doesn't. ... By statically calculating which variables are owned, memory management becomes a simple task of unconditionally freeing the owned variables before they leave their scope, and not freeing the unowned variables.

GNOME has a whole set of libraries, conventions and rules for making this happen, because the C programming language doesn't have many built-in facilities to deal with ownership.

C++, on the other hand, has built a broad range of utilities for just this purpose. For example, you can wrap an allocation in a shared_ptr object. Every time it is copied it will increment a counter, every time it is deallocated it will decrement the counter. When the counter hits zero the wrapped allocation will be deallocated. That means you don't need to track ownership for purposes of deallocation: the shared_ptr is the owner, and will deallocate at the right time.

This can be simplified even further by using languages like Java or Python that provide garbage collection: the language runtime will do all the work for you. You never have to track ownership for purposes of deallocating memory.

Object access rights

Even when memory allocation is handled by the language runtime, there are still reasons to think about object ownership. In particular there is the question of mutation: modifying an object's contents. Memory deallocation is the ultimate form of mutation, but normal mutation can also break your program.

Consider the following Python program:

words = ["hello", "there", "another"]
counts = wordcount(words)

What do you expect to be printed? Typically you'd expect to see ["hello", "there", "another"], but there is another option. You may also get [] printed if wordcount() was implemented as follows:

def wordcount(words):
    result = Counter()
    while words:
        word = words.pop()
        result[word] += 1
    return result

In this implementation wordcount() is mutating the list it is given. Reintroducing the concept of an object owner makes this clearer: each object is owned by a scope, and that scope might not want to grant write access to the object when it passes it to a function.

Unfortunately in Python, Java and friends there is no real way for a caller to tell whether a function will mutate an input, nor whether a parameter to a function can be mutated. So you need to learn a set of conventions and assumptions about when this will happen and when it's safe to do so: you build a mental model of object ownership and access rights. I suspect most Python programmers wouldn't expect wordcount() to mutate its inputs: it violates the mental model we have for object ownership and access.

The concept of private attributes (explicit in Java, by convention in Python) is one way access rights are controlled, but it doesn't solve the problem in all cases. When conventions don't help and you're uncertain you have to refer to API docs, or sometimes even the implementation, to know who can or might modify objects. This is similar to how C programmers deal with memory allocation.

Interestingly, C and C++ have a mechanism that can often solve this problem: const. You can define arguments as being const, i.e. unchangeable:

std::map<string,int> wordcount(const vector<string> &words);

If you try to mutate the argument the compiler will complain and prevent you from doing so.

Other approaches: Rust vs. functional languages

Outside of const the C++ features for object ownership management grew over time, as library code. The Rust programming language, in contrast, provides object ownership as a feature within the compiler itself. And this applies both to ownership for purposes of memory allocation but also for purposes of mutation. Rust attempts to provide this features while still providing the full power of C and C++, and in particular control over memory allocation.

Where Rust code requires the programmer to make explicit decisions about object ownership, functional languages take a different approach. In functional languages like Haskell or Clojure if you call a wordcount function you know the arguments you pass in won't be mutated, because objects are immutable and can't be changed. If objects can't be mutated it doesn't matter who owns them: they are the same everywhere.

The need to track object ownership in your head or in code for mutation control is obviated by making objects immutable. Couple this with garbage collection and you need spend much less time thinking about object ownership when writing purely functional code.

Summary: what you need to know

Depending on the programming language you're coding in you need to learn different models of thinking about object ownership:

  • When you're writing functional code you don't have to think about it much of the time.
  • When you're writing Java/Ruby/Python/JS/Go you need to think about object ownership as it applies to mutation: who is allowed to mutate an object? Will a function mutate an object when you don't expect it? If you're writing concurrent code this becomes much more important: conventions no longer suffice, and access needs to be explicitly controlled with locks.
  • When you're writing Rust the compiler understands a broad range explicit annotations for object ownership, and will enforce safe interactions.
  • When you're writing C++ you can rely on const for mutation control, up to a point, and on library code for automatic memory deallocation.
  • When you're writing C you can rely on const for mutation control, up to a point, and the rest is up to you.

Next time you're writing some code think about who owns each object, and what guarantees the owner expects when it passes an object to other code. Over time you'll build a better mental model of how ownership works.

And if you're writing in a non-functional language consider using immutable (sometimes known as "persistent") data structures. You can get much of the benefits of a functional language by simply reducing the scope of mutation, and therefore how much you need to track object ownership.

January 26, 2017 05:00 AM

January 19, 2017

Itamar Turner-Trauring

Specialist vs. Generalist: which is better for your career?

One of the decisions you'll need to make during the course of you career as a software developer is whether you should become:

  1. A specialist, an expert in a certain subject.
  2. A generalist, able to take on a wide variety of different work.

Miquel Beltran argues that specialization is the path to choose. At the end of his essay he suggests:

Stick to one platform, framework or language and your professional career will be better on the long run.

I think he's both right and wrong. Specialization is a great career move... but I don't think being a generalist is bad for your career either. In fact, you can be both and still have a good career, because there are two distinct areas in which this question plays out.

Getting hired is not the same as getting things done

Getting hired and getting things done are two different tasks, and you need different skills to do each.

When you're trying to get hired you are trying to show why you are the best candidate. That means dealing with the company's attitude towards employees, and the reason they are hiring, and the way they approach their work. It's not about how well you'll do or your job, or how good a programmer you are, or any of that: it's just about getting your foot in the door.

Once you're in, once you're an employee or a consultant, what matters is the results you deliver. It doesn't matter if you've only spent a few weeks previously writing iOS apps so long you do a good job writing an iOS app after you've been hired. And if you've spent years as an iOS developer and you fail to deliver, the fact you specialize in the iOS apps isn't going to help you.

Since getting hired and doing the work are separate tasks, that means you need to separate the decision to be a specialist or generalist into two questions: which will help you get hired, and which will make you better at actually doing your job?

Specialization is a marketing technique

If the question is how you should get hired then you are in the realm of marketing, not engineering. Specialization is a marketing technique: it's a way to demonstrate why you should be hired because you are an expert in your specialty.

Because specialization a marketing technique, specialization doesn't necessarily need to map to specialization on an engineering level. Let me give some examples from my career.

In 2001 I started contributing to an open source Python project, a networking framework called Twisted. I have used this experience in a variety of ways:

  • In 2004 got a job offer from a company that was writing Java, because I had recently added multicast support to Twisted and they wanted to use multicast for an internal project. I had a little experience writing Java, but mostly they wanted to hire me because I was specialist in multicast.
  • I turned that job down, but later that year I got a job at ITA Software, writing networking code in C++. I didn't know any C++... but I knew stuff about networking.
  • When I left ITA I spent a couple years doing Twisted consulting. I was a Twisted specialist.
  • At my latest job I got hired in part because I knew networking protocols... but also because I had experience participating in open source projects.

While all these specializations are related, they are not identical: each job I got involved being a specialist in a different area.

It's not what you can do, it's what you emphasize

Now, you could argue that the reasons I got hired enough are close enough that I am indeed a specialist: in networking or distributed systems. But consider that earlier in my career I did a number of years of web development. So back in 2004 I could have applied to web development jobs, highlighted that part of my resume, and relegated my open source networking work to a sentence at the end.

You likely have many engineering and "soft" skills available to you. Instead of focusing on one particular skillset ("I am an Android developer") you can focus on some other way you are special. E.g. If you're building a consulting pipeline then maybe it's a some business vertical you specialize in, to differentiate yourself from all the other Android developers.

But if you're marketing yourself on a one-off basis, which is certainly the case when you're applying for a job, you can choose a specialty that fits the occasion. Here's how my former colleague Adam Dangoor does it:

Pick one thing from what they talk about that you think is probably the least pitched-to aspect. E.g. if they’re a Python shop everyone will say that they know Python well. But you can spot that e.g. they need help with growing a team and you have experience with that. It could very well be that 10 other candidates do too, but you just say that and you’re the one candidate who can grow a team.

Specialist or Generalist?

So which should you chose, generalist or specialist?

When it comes to engineering skills, or just learning in general, my bias is towards being a generalist. When I went back to school to finish my degree I focused on the humanities and social science; I didn't take a single programming class. You may have different biases then I do.

But engineering skills are fundamentally different than how you market yourself. You can be a generalist in your engineering skills and market yourself as a specialist. In particular, when applying for jobs, you should try to be a specialist in what the company needs.

Sometimes a technical specialty is exactly what they want: you have some set of skills that are hard to find. But often there's a bit more to it than that. They might say they need an Android expert", but what they really need is someone to ship things fast.

They're looking for "an Android expert" because they don't want a learning curve. So if you emphasize the times you've delivered projects quickly and an on schedule you might get the job even, though another candidate had a couple more years of Android experience than you do..

In short, when it comes to engineering skills I tend towards being a generalist, but that may just be my personal bias. When marketing yourself, be a specialist... but there's nothing keeping you from being a different specialist every time you apply for a new job.

January 19, 2017 05:00 AM

January 18, 2017

Jack Moffitt

Servo Talk at LCA 2017

My talk from was just posted, and you can go watch it. In it I cover some of the features of Servo that make it unique and fast, including the constellation and WebRender.

Servo Architecture: Safety & Performance by Jack Moffitt, LCA 2017, Hobart, Australia.

by Jack Moffitt ( at January 18, 2017 12:00 AM

January 12, 2017

Jonathan Lange

Announcing grafanalib

Late last year, as part of my work at Weaveworks, I published grafanalib, a Python DSL for building Grafana dashboards.

We use it a lot, and it’s made our dashboards much nicer to maintain. I’ve written a blog post about it that you can find it on the Weaveworks blog.

by Jonathan Lange at January 12, 2017 12:00 AM

January 11, 2017

Itamar Turner-Trauring

Your Job is Not Your Life: staying competitive as a developer

Are you worried about keeping your programming skills up-to-date so you can stay employable? Some programmers believe that to succeed you must spend all of your time learning, practicing and improving your craft. How do you fit all that in and still have a life?

In fact, it's quite possible to limit yourself to programming during work hours and still be employable and successful. If you do it right then staying competitive, if it's even necessary, won't require giving up your life for you job.

What does it mean to be "competitive?"

Before moving on to solutions it's worth understanding the problem a little more. The idea of "competitiveness" presumes that every programmer must continually justify their employment, or they will be replaced by some other more qualified developer.

There are shrinking industries where this is the case, but at the moment at least demand for programmers is quite high. Add on the fact that hiring new employees is always risky and worrying about "competitiveness" seems unnecessary. Yes, you need to do well at your job, but I doubt most programmers are at risk of being replaced a moment's notice.

Instead of worrying about "competitiveness" you should focus on the ability to easily find a new job. For example, there are other ways you improve your chances at finding a new job that have nothing to do with your engineering skills:

  • Living below your means will allow you to save money for a rainy day. You'll have more time to find a job if you need to, and more flexibility in what jobs you can take.
  • Keep in touch with old classmates and former colleagues; people you know are the best way to find a new job. Start a Slack channel for ex-coworkers and hang out. This can also be useful for your engineering skills, as I'll discuss later on.

Moving on to engineering skills, the idea that you need to put in long hours outside of work is based both on the need to become an expert, and on the need to keep up with changing technology. Both can be done on the job.

Becoming an expert

You've probably heard the line about expertise requiring 10,000 hours of practice. The more hours you practice the better, then, right?

In fact many of the original studies were about number of years, not number of hours (10 years in particular). And the kind of practice matters. What you need is "deliberate practice":

... deliberate practice is a highly structured activity, the explicit goal of which is to improve performance. Specific tasks are invented to overcome weaknesses, and performance is carefully monitored to provide cues for ways to improve it further. We claim that deliberate practice requires effort and is not inherently enjoyable.

Putting aside knowledge of particular technologies, the kinds of things you want to become an expert at are problem solving, debugging, reading unknown code, etc.. And while you could practice them on your own time, the most realistic forms of practice will be at your job. What you need to do is utilize your work as a form of practice.

How should you practice? The key is to know your own weaknesses, and to get feedback on how you're doing so you can improve. Here are two ways to do that:

  1. Code reviews: a good code reviewer will point out holes in your design, in the ways you've tested your code, in the technology you're using. And doing code reviews will also improve your skills as you consider other people's approaches. A job at an organization with a good code review culture will be valuable to your skills and to your career.
  2. Self-critique: whenever you make a mistake, try to think about what you should have noticed, what mental model would have caught the problem, and how you could have chosen better. Notice the critique is not of the result. The key is to critique the process, so that you do better next time.

I write a weekly newsletter about my many mistakes, and while this is ostensibly for the benefit of the readers I've actually found it has helped me become a more insightful programmer. If you want to learn how to make self-critique useful than just an exercise in negativity I recommend the book The Power of Intuition by Gary Klein.

Learning technical skills

Beyond expertise you also need technical skills: programming languages, frameworks, and so on. You will never be able to keep up with all the changing technologies that are continuously being released. Instead, try the following:

  • Switching jobs: when you're looking for a new job put some weight on organizations that use newer or slightly different technologies than the ones you know. You'll gain a broader view of the tools available than what you'd get a single company.
  • Building breadth: instead of learning many technologies in depth, focus on breadth. Most tools you'll never use, but the more you know of the more you can reach for... and building breadth takes much less time.
  • Find a community: you'll never know everything. But knowing many programmers with different experiences than you means you have access to all of their knowledge. You can find online forums like Subreddits, IRC, mailing lists and so on. But if you don't feel comfortable with those you can also just hang out on Slack with former coworkers who've moved on to another job.

Your job is not your life

All of the suggestions above shouldn't require much if any time outside of your job. If you enjoy programming and want to do it for fun, by all means do so. But your job shouldn't be the only thing you spend you life on.

If you would like to learn how to to get a job that doesn't overwhelm your life, join my free 6-part email course.

Join the course: Getting to a Sane Workweek

Don't let your job take over your life. Join over 800 other programmers on the journey to a saner workweek by taking this free 6-part email course. You'll learn how you can work reasonable hours and still succeed in your career a programmer.

Unsubscribe at any time. Powered by ConvertKit

January 11, 2017 05:00 AM

January 06, 2017

Itamar Turner-Trauring

The fourfold path to software quality

How do you achieve software quality? How do you write software that actually works, software that isn't buggy, software that doesn't result in 4AM wake up calls when things break in production?

There are four different approaches you can take, four paths to the ultimate goal. Which path you choose to take will depend on your personality, skills and the circumstances of your work.

The path of the Yolo Programmer

The first path is that of the Yolo Programmer. As a follower of the famous slogan "You Only Live Once", the Yolo Programmer chooses not to think about software quality. Instead the Yolo Programmer enjoys the pure act of creation; writing code is a joy that would only be diminished by thoughts of maintenance or bugs.

It's easy to look down on the Yolo Programmer, to deride their approach a foolish attitude only suitable for children. As adults we suppress our playfulness because we worry about the future. But even though the future is important, the joy of creation is still a fundamental part of being human.

When you have the opportunity, when you're creating a prototype or some other code that doesn't need to be maintained, embrace the path of the Yolo Programmer. There's no shame in pure enjoyment.

The path of the Rational Optimizer

In contrast to the Yolo Programmer, the Rational Optimizer is well aware of the costs of bugs and mistakes. Software quality is best approached by counter-balancing two measurable costs: the cost of bugs to users and the business vs. the cost of finding and fixing the bugs.

Since bugs are more expensive the later you catch them, the Rational Optimizer invests in catching bugs as early as possible. And since human effort is expensive, the Rational Optimizer loves tools: software can be written once and used many times. Tools to find bugs are thus an eminently rational way to increase software quality.

David R. MacIver's post The Economics of Software Correctness is a great summary of this approach. And he's built some really wonderful tools: your company should hire him if you need to improve your software's quality.

The path of Mastery

The path of Mastery takes a different attitude, which you can see in the title Kate Thompson's book Zero Bugs and Program Faster (note that she sent me a free copy, so I may be biased).

Mastery is an attitude, a set of assumptions about how one should write code. It assumes that the code we create can be understood with enough effort. Or, if the code is not understandable, it can and should be simplified until we can understand it.

The path of Mastery is a fundamentally optimistic point of view: we can, if we choose, master our creations. If we can understand our code we can write quality code. We can do so by proving to ourselves that we've covered all the cases, and by learning to structure our code the right way. With the right knowledge, the right skills and the right attitude we can write code with very few bugs, perhaps even zero bugs.

To learn more about this path you should read Thompson's book; it's idiosyncratic, very personal, and full of useful advice. You'll become a better programmer by internalizing her lessons and attitude.

The path of the Software Clown

The final path is the path of the Software Clown. If Mastery is a 1980s movie training montage, the Software Clown is a tragicomedy: all software is broken, failure is inevitable, and nothing ever works right. There is always another banana peel to slip on, and that would be sad if it weren't so funny.

Since the Software Clown is always finding bugs, the Software Clown makes sure they get fixed, even when they're in someone else's software. Since software is always broken, the Software Clown plans for brokenness. For example, if bugs are inevitable then you should make sure users have an easy time reporting them.

Since banana peels are everywhere, the Software Clown learns how to avoid them. You can't avoid everything, and you won't avoid everything, but you can try to avoid as many as possible.

If you'd like to avoid the many mistakes I've made as a software engineer, sign up for my Software Clown newsletter. You'll get the story of one of my mistakes in your inbox every week and how you can avoid making it.

These are the four paths you can take, but remember: there is no one true answer, no one true path. Try to learn them all, and the skills and attitudes that go along with them; you'll become a better programmer and perhaps even a better person.

Avoid my programming mistakes!

Get a weekly email with one of my many software and career mistakes, and how you can avoid it. Here's what readers are saying:

"Are you reading @itamarst's "Software Clown" newsletter? If not, you should be. There's a gem in every issue." - @glyph

I won't share your email with anyone else. Unsubscribe at any time. Powered by ConvertKit

January 06, 2017 05:00 AM

January 02, 2017

Itamar Turner-Trauring

When software ecosystems die

How much can you rely on the frameworks, tools and libraries you build your software on? And what can you do to reduce the inherent risk of depending on someone else's software?

Years ago I watched a whole software ecosystem die.

Not the slow decline of a programming language that is losing its users, or a no longer maintained library that has a newer, incompatible replacement. This was perma-death: game over, no resurrection, no second chances.

Here's what happened, and what you can learn from it.

The story of mTropolis

Back in the 1990s the Next Big Thing was multimedia, and in particular multimedia CD-ROMs. The market leader was Macromedia Director, a rather problematic tool.

Macromedia Director started out as an animation tool, using a sequence of frames as its organizing metaphor, which meant using it for hypermedia involved a rather bizarre idiom. Your starting screen would be frame 1 on the timeline, with a redirect to itself on exit, an infinite busy loop. Remember this started as animation tool, so the default was to continue on to later frames automatically.

When you clicked on a button that took you to a new screen it worked by moving you to another frame, let's say frame 100. Frame 100 would have a "go to frame 100" on exit to made sure you didn't continue on to frame 101, and then 102, etc.

Then in 1995 mTropolis showed up, a newer, better competitor to Director. It was considered by many to be the superior alternative, even in its very first release. It had a much more suitable conceptual model, features that were good enough to be copied by Director, and a loyal fan base.

In 1997 mTropolis was bought by Quark, maker of the the QuarkXPress desktop publishing software. A year later in 1998 Quark decided to end development of mTropolis.

mTropolis' users were very upset, of course, so they tried to buy the rights off of Quark and continue development on their own.

The purchase failed. mTropolis died.

Market leader or scrappy competitor?

The story of mTropolis had a strong impression on me as a young programmer: I worked with Director, so I was not affected, but the developers who used mTropolis were dead in the water. All the code they'd built was useless as soon as a new OS release broke mTropolis in even the smallest of ways.

This isn't a unique story, either: spend some time reading Our Incredible Journey. Startups come and go, and software ecosystems die with them.

Professor Beekums has an excellent post about switching costs in software development. He argues that given the choice between equivalent market leader and smaller competitor you should choose the latter, so you don't suffer from monopoly pricing.

But what do you do when they're not equivalent, or it's hard to switch? You still need to pick. I would argue that if they're not equivalent, the market leader is much safer. Macromedia was eventually bought by Adobe, and so Director is now Adobe Director. Director was the market leader in 1998, and it's still being developed and still available for purchase, almost 20 years later.

mTropolis may have been better, but mTropolis wasn't the market leader. And mTropolis is dead, and has been for a very long time.

Making the choice

So which do you go for, when you have the choice?

If you're dealing with open source software, much of the problem goes away. Even if the company sponsoring the software shuts down, access to the source code gives you a way to switch off the software gradually.

With Software-as-a-Service you're back in the realm of choosing between monopoly pricing and chance of software disappearing. And at least with mTropolis the developers still could use their licensed copies; an online SaaS can shut down at any time.

Personally I'd err on the side of choosing the market leader, but it's hard to give a general answer. Just remember: the proprietary software you rely on today might be gone tomorrow. Be prepared.

January 02, 2017 05:00 AM

December 23, 2016

Ralph Meijer


For me, Christmas and Jabber/XMPP go together. I started being involved with the Jabber community around the end of 2000. One of the first things that I built was a bot that recorded the availability presence of my online friends, and show this on a Christmas tree. Every light in the tree represents one contact, and if the user is offline, the light is darkened.As we are nearing Christmas, I put the tree up on the frontpage again, as many years before.

Over the years, the tooltips gained insight in User Moods and Tunes, first over regular Publish-Subscribe, later enhanced with the Personal Eventing Protocol. A few years later, Jingle was born, and in 2009, stpeter wrote a great specification that solidifies the relationship between Christmas and Jabber/XMPP.

Many things have changed in those 16 years. I've changed jobs quite a few times, most recently switching from the Mailgun team at Rackspace, to an exciting new job at VimpelCom as Chat Expert last April, working on Veon (more on that later). The instant messaging landscape has changed quite a bit, too. While we, unfortunately, still have a lot of different incompatible systems, a lot of progress has been made as well.

XMPP's story is long from over, and as such I am happy and honored to serve as Chair of the XMPP Standards Foundation since last month. As every year, my current focus is making another success of the XMPP Summit and our presence with the Realtime Lounge and Devroom at FOSDEM in Brussels in February. This is always the highlight of the year, with many XMPP enthousiasts, as well as our friends of the wider Realtime Communications, showing and discussing everything they are working on, ranging from protocol discussions to WebRTC and IoT applications.

Like last year, one of the topics that really excite me is the specification known as Mediated Information eXchange (MIX). MIX takes the good parts of the Multi User Chat (MUC) protocol, that has been the basis of group chat in XMPP for quite a while, redesigned on top of XMPP Publish-Subscribe. Modern commercial messaging systems, for business use (e.g. Slack and HipChat), as well as for general use (e.g. WhatsApp, WeChat, Google's offerings), have tried various approaches on the ancient model of multi-part text exchange, adding multi-media and other information sources, e.g. using integrations, bots, and cards.

MIX is the community's attempt to provide a building block that goes beyond the tradional approach of a single stream of information (presence and messages) to a collection of orthogonal information streams in the same context. A room participant can select (manually or automatically by the user agent) which information streams are of interest at that time. E.g. for mobile use or with many participants, exchanging the presence information of all participants can be unneeded or even expensive (in terms of bandwidth or battery use). In MIX, presence is available as a separate stream of information that can be disabled.

Another example is Slack's integrations. You can add streams of information (Tweets, continuous integration build results, or pull requests) to any channel. However, all participants have no choice to receive the resulting messages, intermixed with discussion. The client's notification system doesn't make any distinction between the two, so you either suffer getting alerts for every build, or mute the channel and possibly miss interesting discussion. The way around it is to have separate channels for notifications and discussion, possibly muting the former.

Using MIX, however, a client can be smarter about this. It can offer the user different ways to consume these information streams. E.g. notifications on your builds could be in a side bar. Tweets can be disabled, or modeled as a ticker. And it can be different depending on which of the (concurrent) clients you are connected with. E.g. the desktop or browser-based client has more screen real-estate to show such orthogonal information streams at the same time, a mobile client might still show the discussion and notifications interleaved.

All-in-all MIX allows for much richer, multi-modal, and more scalable interactions. Some of the other improvements over MUC include persistent participation in channels (much like IRC bouncers, but integrated), better defined multi-device use (including individual addressing), reconnection, and message archiving. I expect the discussions at the XMPP Summit to tie the loose ends as a prelude to initial implementations.

I am sure that FOSDEM and the XMPP Summit will have many more exciting topics, so I hope to see you there. Until then, Jabber on!

by ralphm at December 23, 2016 01:28 PM

December 20, 2016

Itamar Turner-Trauring

The one technology you need to learn in 2017

If you want to become a better programmer or find a better-paying job, you might wonder if there's a particular technology you should learn in the coming year. Once you learn this technology you will become far more productive, and provide far more value to current or future employers.

There is something you can learn, but it's not a new web framework, or a programming language, or an IDE. In fact, it's not a technology at all.

It's better than that.

Much better.

Why you need to learn this

Imagine you're a contractor for an online pet-supplies store, selling supplies for cats and dogs.

One Monday the owner of the store asks you to add a new section to the website, selling supplies for aardvarks. Being a skilled designer as well as a programmer you can update the website design, then change the database, write some new code... By the end of the week you've launched the new Aardvark section.

Next Monday the owner is back. This week you're adding badgers, so you add another section of the design, update the code to support another pet, and again you're done after a week. The next week is crabs, and so on and so forth until by the middle of the year you've added zebras.

Unfortunately by this point your website has 30+ sections, and almost all of them are for pets no one wants or owns. No one can find the cat and dog supplies amid the noise, sales are plummeting, and the owner of the store can't afford to pay you any more.

Knowing a better web framework or a more suitable programming language would have let you code the weekly animal's section faster. But coding the wrong thing faster is just as much a waste of time.

Even if you knew every programming language and web framework under the sun you'd still be missing a key skill.

The skill you need to learn

How could this problem have been prevented? Let's get in our time machine and go back to the beginning. Cars drive backwards, raindrops rush up to the clouds, flowers refold, the sun flies from west to east alternating with darkness...

One Monday the owner of the store asks you to add a new section to the website, selling supplies for aardvarks. And you think for a moment, and ask: "why do you want to do that? most people don't own aardvarks."

With a bit more probing you find out the reason the store owners wants to add aardvarks. Perhaps the store wants to sell supplies for all animals, in which case you can code generic animal support once, instead of adding a new one per week. Perhaps this is a misguided search engine optimization strategy, in which case you can suggest adding a blog.

If you ask and figure out the reason for a task, figure out the goal... that is an immensely useful skill. You can only succeed at a project if you know its goal.

Working towards goals

If you can figure out what your boss really needs, what your client's real problem is, you can actually solve their problem. Otherwise they're the ones who have to solve the problem; you're just a hired hand implementing someone else's solution.

If you can figure out what the real goal is you can make sure you're working towards it. You can avoid the irrelevant work, the nice-to-have work, the attractive but useless work: you can work just on what's needed. You won't be writing coding any faster, but you'll ship sooner and code far more effectively.

Technologies are useful and you need to learn them, but in the end they're just tools. Programming languages come and go, web frameworks come and go: what matters is what you choose to build with them.

Don't learn a technology, learn more fundamental skills. Figuring out root causes and why something needs to be done, discovering the real problem, not just the stated one: these skills will serve you this year, and next year and every year after that.

If you have these core skills, you'll be a far more valuable employee. If you also improve your bargaining position and negotiating skills you will be able to find a job on your own terms: a job with work hours you want, a job you love that doesn't own your life.

Join the course: Getting to a Sane Workweek

Don't let your job take over your life. Join over 800 other programmers on the journey to a saner workweek by taking this free 6-part email course. You'll learn how you can work reasonable hours and still succeed in your career a programmer.

Unsubscribe at any time. Powered by ConvertKit

If you would like to learn the skills you need to get a job that doesn't overwhelm your life, join my free 6-part email course.

December 20, 2016 05:00 AM

December 19, 2016

Glyph Lefkowitz

Sourceforge Update

When I wrote my previous post about Sourceforge, things were looking pretty grim for the site; I (rightly, I think) slammed them for some pretty atrocious security practices.

I invited the SourceForge ops team to get in touch about it, and, to their credit, they did. Even better, they didn't ask for me to take down the article, or post some excuse; they said that they knew there were problems and they were working on a long-term plan to address them.

This week I received an update from said ops, saying:

We have converted many of our mirrors over to HTTPS and are actively working on the rest + gathering new ones. The converted ones happen to be our larger mirrors and are prioritized.

We have added support for HTTPS on the project web. New projects will automatically start using it. Old projects can switch over at their convenience as some of them may need to adjust it to properly work. More info here:

Coincidentally, right after I received this email, I installed a macOS update, which means I needed to go back to Sourceforge to grab an update to my boot manager. This time, I didn't have to do any weird tricks to authenticate my download: the HTTPS project page took me to an HTTPS download page, which redirected me to an HTTPS mirror. Success!

(It sounds like there might still be some non-HTTPS mirrors in rotation right now, but I haven't seen one yet in my testing; for now, keep an eye out for that, just in case.)

If you host a project on Sourceforge, please go push that big "Switch to HTTPS" button. And thanks very much to the ops team at Sourceforge for taking these problems seriously and doing the hard work of upgrading their users' security.

by Glyph at December 19, 2016 01:19 AM

December 15, 2016

Itamar Turner-Trauring

Experts, True Believers and Test-Driven Development: how expert advice becomes a religion

If you've encountered test-driven development (TDD), you may have encountered programmers who follow it with almost religious fervor. They will tell you that you must always write unit tests before you write code, no exceptions. If you don't, your code will be condemned to everlasting brokenness, tortured by evil edge cases for all eternity.

This is an example of a common problem in programming: good advice by experts that gets turned into a counter-productive religion. Test-driven development is useful and worth doing... some of the time, but not always. And the experts who came up with it in the first place will be the first to tell you that.

Let's see how expert advice turns into zealotry, and how you can prevent it from happening to you.

Expert advice becomes a religion

The problem with experts is that they suffer from Expert Blind Spot. Because experts understand the subject so well, they have trouble breaking it down into concepts and explaining it in ways that make sense to novices.

Thus the expert may say "always write unit tests before you write your code." What they actually meant is this:

  1. Write unit tests before you write your code.
  2. Unless it's code that isn't amenable to unit testing, e.g. because unit tests aren't informative for this particular kind of code.
  3. Or unless it's code that you're going to throw away immediately.
  4. And technically you can write the test after the code, and then break your code and check the test is failing. But this is annoying and a bit more error prone so as an expert I'm not going to mention that at all.

A True Believer in TDD might start arguing with these claims. But consider that even Extreme Programming, where TDD originates, discusses types of coding where unit tests are unnecessary.

In particular a "spike" in Extreme Programming is an architectural prototype, used to figure out the structure of the code you're going to write. Since it's going to be thrown away you don't need to write unit tests when you write a spike. You can see this visually in this overview of Extreme Programming; the Architectural Spike happens before Iterations, and unit tests are written as part of Iterations.

If you're certain all code is amenable to unit testing, consider these two examples: unit tests aren't very helpful in testing a cryptographically secure random number generator. And if the director of a movie has asked you write some code to 3D render a "really awesome-looking explosion" you won't benefit much from unit tests either, unless you can write a unit test to determine awesomeness.

So if experts know unit-testing-before code isn't always the right answer, why do they say "always write unit tests before you write your code"? Expert blind spot: they can't imagine anyone would write unit tests when they shouldn't.

To the expert, a prototype and code requiring tests are obviously very different things with different goals. But that isn't so obvious to the novice listener.

The novice listener takes the expert at their literal word, and comes to believe that they must always write unit tests before writing code. The novice is now a True Believer. They tell everyone they know "always wrote tests before you write code," and they try to do so under all circumstances.

Thus good advice turns into a counter-productive religion, with unit tests often being written when they needn't or shouldn't be.

Avoiding the trap

How can you keep this from happening to you?

If you're an expert, make sure you explain the qualifications to your statements. Don't say "always do X." Instead say "you should do X, because Y, under circumstance A but not circumstance B."

If you're not an expert things get a bit harder. Consider that both the expert and the True Believer are making the same statement: "always write tests before you write code." How can you tell whether the person telling you this is an expert or a True Believer?

You need to take every piece of advice and figure out when and where it does not apply. An expert's assumptions will often be implicit in the topic they're writing about, so that may give you some hints.

If you're having a two-way conversation try to get them to qualify their statement: ask them to come up with the boundary cases, where this advice is no longer applicable. Bring up the cases you've found where their advice seemingly won't work and see what they say.

An expert, given enough prodding, will be able to come up with cases where their advice isn't applicable, and explain the reason why. But a True Believer won't back down, won't agree to any compromise: they know the The Truth, and it is absolute and unchanging.

Programming is a broad field, and what makes good software depends on your circumstances, goals and tools. There is almost never one answer that applies everywhere. Learn from the experts, but never become a True Believer.

December 15, 2016 05:00 AM

Glyph Lefkowitz

Don’t Stop Tweeting On My Account

Shortly after my previous post, my good friend David Reid not-so-subtly subtweeted me for apparently yelling at everyone using a twitter thread to be quiet and stop expressing themselves. He pointed out:

This is the truth. There are, indeed, important, substantial essays being written on Twitter, in the form of threads. If I could direct your attention to one that’s probably a better use of your time than what I have to say here, this is a great example:

Moreover, although the twitter character limit can inhibit the expression of nuance, just having a blog is not a get-out-of-jail-free card for clumsy, hot takes:

I screwed this one up. I’m sorry.

The point I was trying to primarily focus on in that post is that a twitter thread demands a lot of attention, and that publishers exploiting that aspect of the medium in order to direct more attention to themselves1 are leveraging a limited resource2 and thereby externalizing their marketing costs3. Further, this idiom was invented by4, and has extensively been used by people who don’t really need any more attention than they already have.

If you’re an activist trying to draw attention to an important cause, or a writer trying to find your voice, and social media (or twitter threads specifically) has helped you do that, I am not trying to scold you for growing an audience on - and deriving creative energy from - your platform of choice. If you’re leveraging the focus-stealing power of twitter threads to draw attention to serious social issues, maybe you deserve that attention. Maybe in the face of such issues my convenience and comfort and focus are not paramount. And for people who really don’t want that distraction, the ‘unfollow’ button is, obviously, only a click away.

That’s not to say I think that relying on social media exclusively is a good idea for activists; far from it. I think recent political events have shown that a social media platform is often a knife that will turn in your hand. So I would encourage pretty much anyone trying to cultivate an audience to consider getting an independent web presence where you can host more durable and substantive collections of your thoughts, not because I don’t want you to annoy me, but because it gives you a measure of independence, and avoids a potentially destructive monoculture of social media. Given the mechanics of the technology, this is true even if you use a big hosted service for your long-form stuff, like Medium or Blogger; it’s not just about a big company having a hold on your stuff, but about how your work is presented based on the goals of the product presenting it.

However, the exact specifics of such a recommendation are an extremely complex set of topics, and not topics that I’m confident I’ve thought all the way through. There are dozens more problems with twitter threads for following long-form discussions and unintentionally misrepresenting complex points. Maybe they’re really serious, maybe not.

As far as where the long-form stuff should go, there are very good reasons to want to self-host things, and very good reasons why self-hosting is incredibly dangerous, especially for high-profile activists and intellectuals. There are really good reasons to engage with social media platforms and really good reasons to withdraw.

This is why I didn’t want to address this sort of usage of twitter threading; I didn’t want to dive into the sociopolitical implications of the social media ecosystem. At some point, you can expect a far longer post from me about the dynamics of social media, but it is going to take a serious effort to do it justice.

A final thought before I hopefully stop social-media-ing about social media for a while:

One of the criticisms that I received during this conversation, from David as well as others who contacted me privately, is that I’m criticizing Twitter from a level of remove; implying that since I’m not fully engaged with the medium I don’t really have the right (or perhaps the expertise) to be critical of it. I object to that.

In addition to my previously stated reasons for my reduced engagement - which mostly have to do with personal productivity and creative energy - I also have serious reservations about the political structure of social media. There’s a lot that’s good about it, but I think the incentive structures around it may mean that it is, ultimately, a fundamentally corrosive and corrupting force in society. At the very least, a social media platform is a tool which can be corrosive and corrupting and therefore needs to be used thoughtfully and intentionally to minimize the harm that it can do while retaining as many of its benefits as possible.

I don’t have time to fully explore the problems that I’m alluding to now5 but at this point if I wrote something like “social media platforms are slowly destroying liberal democracy”, I’m not even sure if I’d be exaggerating.

When I explain that I have these concerns, I’m often asked the obvious follow-up: if social media is so bad why don’t I just stop using it entirely?

The problem is, social media companies effectively control access to an enormous audience, which is now difficult to reach without their intermediation. I have friends, as we all probably do, that are hard for me to contact via other channels. An individual cannot effectively boycott a communication tool, and I am not even sure yet that “stop using it” is the right way to combat its problems.

So, I’m not going to stop communicating with my friends because I have concerns about the medium they prefer, and I’m also not going to stop thinking or writing about how to articulate and address those concerns. I think I have as much a right as anyone to do that.

  1. ... even if they’re not doing it on purpose ... 

  2. the reader’s attention 

  3. interrupting the reader repeatedly to get them to pay attention rather than posting stuff as a finished work, allowing the reader to make efficient use of their attention 

  4. I’m aware that many people outside of the white male tech nerd demographic - particularly women of color and the LGBTQ community - have made extensive use of the twitter thread for discussing substantive issues. But, as far as my limited research has shown (although the difficulty of doing such research is one of the problems with Twitter), Marc Andreessen was by far the earliest pioneer of the technique and by far its most prominent advocate. I’d be happy for a correction on this point, however. 

  5. The draft in progress, which I've been working on for a month, is already one of the longest posts I’ve ever written and it’s barely half done, if that. 

by Glyph at December 15, 2016 03:02 AM

December 14, 2016

Glyph Lefkowitz

A Blowhard At Large

Update: I've written a brief follow-up to this post to clarify my feelings about other uses of the tweetstorm, or twitter thread, publishing idiom. This post is intended to be specifically critical of its usage as a self-promotional or commercial tool.

I don’t like Tweetstorms™1, or, to turn to a neologism, “manthreading”. They actively annoy me. Stop it. People who do this are almost always blowhards.

Blogs are free. Put your ideas on your blog.

As Eevee rightfully points out, however, if you’re a massive blowhard in your Tweetstorms, you’re likely a massive blowhard on your blog, too. So why care about the usage of Twitter threads vs. Medium posts vs. anything else for expressions of mediocre ideas?

Here’s the difference, and here’s why my problem with them does have something to do with the medium: if you put your dull, obvious thoughts in a blog2, it’s a single entity. I can skim the introduction and then skip it if it’s tedious, plodding, derivative nonsense.3

Tweetstorms™, as with all social media innovations, however, increase engagement. Affordances to read little bits of the storm abound. Ding. Ding. Ding. Little bits of an essay dribble in, interrupting me at suspiciously precisely calibrated 90-second intervals, reminding me that an Important Thought Leader has Something New To Say.

The conceit of a Tweetstorm™ is that they’re in this format because they’re spontaneous. The hottest of hot takes. The supposed reason that it’s valid to interrupt me at 30-second intervals to keep me up to date on tweet 84 of 216 of some irrelevant commentator’s opinion on the recent trend in chamfer widths on aluminum bezels is that they’re thinking those thoughts in real time! It’s an opportunity to engage with the conversation!

But of course, this is a pretense; transparently so, unless you imagine someone could divine the number after the slash without typing it out first.

The “storm” is scripted in advance, edited, and rehearsed like any other media release. It’s interrupting people repeatedly merely to increase their chances of clicking on it, or reading it. And no Tweetstorm author is meaningfully going to “engage” with their readers; they just want to maximize their view metrics.

Even if I cared a tremendous amount about the geopolitics of aluminum chamfer calibration, this is a terrible format to consume those thoughts in. Twitter’s UI is just atrocious for meaningful consideration of ideas. It’s great for pointers to things (like a link to this post!) but actively interferes with trying to follow a thought to its conclusion.

There’s a whole separate blog in there about just how gross pretty much all social-media UI is, and how much it goes out of its way to show you “what you might have missed”, or things that are “relevant to you” or “people you should follow”, instead of just giving you the actual content you requested from their platform. It’s dark patterns all the way down, betraying the user’s intent for those of the advertisers.

My tone here probably implies that I think everyone doing this is being cynically manipulative. That’s possibly the worst part - I don’t think they are. I think everyone involved is just being slightly thoughtless, trying to do the best that they can in their perceived role. Blowhards are blowing, social media is making you be more social and consume more media. All optimizing for our little niche in society. So unfortunately it’s up to us, as readers, to refuse to consume this disrespectful trash, and pipe up about the destructive aspects of communicating that way.

Personally I’m not much affected by this, because I follow hardly anyone4, I don’t have push enabled, and I would definitely unfollow (or possibly block) someone who managed to get retweeted at such great length into my feed. But a lot of people who are a lot worse than I am about managing the demands on their attention get sucked into the vortex that Tweetstorms™ (and related social-media communication habits) generate.

Attention is a precious resource; in many ways it is the only resource that matters for producing creative work.

But of course, there’s a delicate balance - we must use up that same resource to consume those same works. I don’t think anyone should stop talking. But they should mindfully speak in places and ways that are not abusive of their audience.

This post itself might be a waste of your time. Not everything I write is worth reading. Because I respect my readers, I want to give them the opportunity to ignore it.

And that’s why I don’t use Tweetstorms™5.

  1. ™ 

  2. Hi Ned. 

  3. Like, for example, you can do with this blog! 

  4. I subscribe to more RSS feeds than Twitter people by about an order of magnitude, and I heartily suggest you do the same. 

  5. ™ 

by Glyph at December 14, 2016 02:55 AM

December 10, 2016

Moshe Zadka

On Raising Exceptions in Python

There is a lot of Python code in the wild which does something like:

raise SomeException("Could not fraz the buzz:"
                    "{} is less than {}".format(foo, quux)

This is, in general, a bad idea. Exceptions are not program panics. While they sometimes do terminate the program, or the execution thread with a traceback, they are different in that they can be caught. The code that catches the exception will sometimes have a way to recover: for example, maybe it’s not that important for the application to fraz the buzz if foo is 0. In that case, the code would look like:

except SomeException as e:
    if ???

Oh, right. We do not have direct access to foo. If we formatted better, using repr, at least we could tell the difference between 0 and “0”: but we still would have to start parsing the representation out of the string.

Because of this, in general, it is better to raise exceptions like this:

raise SomeException("Could not fraz the buzz: foo is too small", foo, quux)

This way exception handling has a lot of power: it can introspect foo, introspect quux and introspect the string. If by some reason the exception class is raised and we want to verify the reason, checking string equality, while not ideal, is still better than trying to match string parts or regular expression matching.

When the exception is presented to the user interface, in that case, it will not look as nice. Exceptions, in general, should reach the UI only in extreme circumstances: and in those cases, having something that has as much information is useful for root cause analysis.

by moshez at December 10, 2016 04:42 AM

December 08, 2016

Itamar Turner-Trauring

Don't Get Stuck: 6 ways to get unstuck and code faster

A lot of my time as a programmer, and maybe yours as well, is spent being stuck. My day often goes like this:

  1. Write some code.
  2. Run the tests.
  3. "It failed."
  4. "Why did it fail?"
  5. "I don't know."
  6. "That makes no sense."
  7. "Seriously, what?"
  8. "That's impossible."
  9. "Lets add a print statement here."
  10. "And maybe try poking around with the debugger."
  11. "Oh! That's it!"
  12. "No wait, it isn't."
  13. "Ohhhhhhhh there we go."
  14. Run the tests.
  15. Tests pass.
  16. "Time for snacks!"

Given how much time you can end up wasting in this mode, Kaitlin Duck Sherwood points out that not getting stuck is one of the keys to programmer productivity. Imagine how much time I could save if I skipped steps 5 to 13 in the transcript above.

Not getting stuck will make you far more productive. Here are six ways to keep you from getting stuck:

Recognize when you're stuck

If you don't know you're stuck then you can't do anything about it, so the first step is having some way of measuring progress. Whenever you start a coding task you should have a time estimate in mind.

The time estimates should be short, no more than a few hours or a day, so a bigger project should be broken up into smaller tasks. The estimates don't have to particularly accurate, they just have to be in the right range: a task you estimate at a few hours should not require days of work.

Given an estimate of 4 hours, say, you can then recognize whether you're actually stuck:

  • If it's 10 minutes in and you have no idea what to do, then that's fine, there's plenty more time.
  • If you're 2 hours in and you haven't produced anything, then it's pretty clear you're stuck and need to make some changes.

Comparing actual time spent to the initial estimate tells you if you're making progress, and working in small chunks ensures you recognize problems quickly.

Ask for help

Now that you've recognized you're stuck, the next thing to do is find a solution. The easiest thing to do is talk to a colleague.

This is helpful in multiple ways:

  • You're forced to restate the problem in a way someone else can understand. Sometimes this is sufficient to help you solve the problem, even before they get to answering you.

In fact, this is so useful that sometimes you don't need a person, and talking to a rubber duck will do. I like debugging out loud for this reason, so occasionally I use a #rubberduck Slack channel so I don't distract my coworkers.

  • Your colleague may have an idea you don't, especially if they're experienced. For example, recently I was utterly confused why Java thought that assertThat(0.0, isEqual(-0.0)) was a failing test; it claimed 0.0 wasn't the same as -0.0.

Eventually I shared my confusion, and someone noted expression relies on Double.equals(), and then went and found the Double.equals() API documentation. And indeed, the documentation notes that new Double(0.0).equals(new Double(-0.0)) is false even though in Java 0.0 == -0.0 is true, because reasons.

Use or copy an existing solution

If you or your colleague can't find a solution on your own, you can try using some one else's solution. This can range from the copy/paste-from-StackOverflow fallback (but be careful, sometimes StackOverflow code is wrong or insecure) to copying whole designs.

For example, I built a multicast data distribution protocol. This is not a trivial thing to do, so I found a research paper and copied their design and algorithm. Designing such an algorithm would have been far beyond my abilities, but I didn't have to: someone smarter and more knowledgeable had done the hard work.

Find a workaround

Sometimes you're faced with an important bug you can't fix. Working around it may suffice, however, as you can see in this story shared by reader James (Jason) Harrison:

Several years ago, I was working many late nights on a new Wii game that was going to have gesture recognition. The first part of the game activities went as smoothly as could be expected and then we came to a new level where the player was supposed to bring the Wiimote up and then down quickly. This must have tripped on a bug in the system because this gesture could not be reliably recognized.

Replaying recorded motions demonstrated that the problem wasn’t “just” in the data form the Wiimote or in how the player made the motion but in the system. Instead of being deterministic, the system would work then not work. Looked for variables that were not being initialized, data buffers not being cleared, and all state that could possibly leak from previous inputs.

Unfortunately, all of the searching didn’t find the problem in time. So it was decided to reset the recognition system between inputs. While wasteful, it was the only thing that did fix the system and let us ship the milestone to the publisher. Left a comment in to find the problem later. Never did find it. Game was shipped with this fix.

Drop the feature

If you're working on a feature and it's taking forever, maybe it's time to consider dropping it. Can it wait until the next release? Do you actually need it?

A feature usually lands on the requirements list for a reason, it's true. But a surprising number of features are in the nice-to-have category, especially when it's taking far too long to implement them. If other approaches have failed to get you unstuck, find the person in charge and give them the choice of shipping late or dropping the feature.

Redefine the problem

Finally, another approach you can take is redefining the problem. The way you're approaching the problem may be making it needlessly complicated to solve, so revisiting the problem statement can help get you unstuck.

You can redefine the problem by relaxing some of the constraints. For example, maybe you're having a hard time finding a date picker that matches the website design. If the problem statement is "add a usable, good looking, date picker that matches our website style" then you might spend a while looking and not finding any that are quite right.

Often you can redefine the problem, though, to "find a minimal date picker so we can demo the feature and see if anyone cares." With that more limited scope you can probably use one of the options you initially rejected.

You can also redefine the problem by figuring out the real problem lies elsewhere. Havoc Pennington has a wonderful story about the dangerous "UI team": they will feel their mandate is to build UIs. But software that doesn't have a UI and "just works" is an even better user experience, if you can manage it.

In short, to keep from getting stuck you should:

  1. Break all your work up into small chunks.
  2. Estimate each chunk in advance, and pay attention to your progress against the estimate.
  3. When you recognize you are stuck: ask for help, copy an existing solution, find a workaround, drop the feature or redefine the problem.

I've learned most of this the hard way, over the course of 20 years of being stuck while writing software. If you'd like to avoid the many mistakes I've made as a software engineer during that time, both coding and in my career, sign up for my Software Clown newsletter. You'll get the story of one of my mistakes in your inbox every week and how you can avoid making it.

Avoid my programming mistakes!

Get a weekly email with one of my many software and career mistakes, and how you can avoid it. Here's what readers are saying:

"Are you reading @itamarst's "Software Clown" newsletter? If not, you should be. There's a gem in every issue." - @glyph

"Definitely subscribe if you want to learn some things that Itamar learned the hard way." -- Victor Algaze

I won't share your email with anyone else. Unsubscribe at any time. Powered by ConvertKit

December 08, 2016 05:00 AM

Moshe Zadka

Moshe’z Messaging Preferences

The assumption here is that you have my phone number. If you don’t have my phone number, and you think that’s an oversight on my part, please send me an e-mail at and ask for it. If you don’t have my phone number because I don’t know you, I am usually pretty responsive on e-mail.

In order of preference:

by moshez at December 08, 2016 03:03 AM

December 05, 2016

Jp Calderone

Twisted Web in 60 Seconds: HTTP/2

Hello, hello. It's been a long time since the last entry in the "Twisted Web in 60 Seconds" series. If you're new to the series and you like this post, I recommend going back and reading the older posts as well.

In this entry, I'll show you how to enable HTTP/2 for your Twisted Web-based site. HTTP/2 is the latest entry in the HTTP family of protocols. It builds on work from Google and others to improve performance (and other) shortcomings of the older HTTP/1.x protocols in wide-spread use today.

Twisted implements HTTP/2 support by building on the general-purpose H2 Python library. In fact, all you have to do to have HTTP/2 for your Twisted Web-based site (starting in Twisted 16.3.0) is install the dependencies:

$ pip install twisted[http2]

Your TLS-based site is now available via HTTP/2! A future version of Twisted will likely extend this to non-TLS sites (which requires the Upgrade: h2c handshake) with no further effort on your part.

If you like this post or others in the Twisted Web in 60 Seconds series, let me know with a donation! I'll post another entry in the series when the counter hits zero. Topic suggestions welcome in the comment section.

by Jean-Paul Calderone ( at December 05, 2016 12:00 PM

December 02, 2016

Moshe Zadka

Don’t Mock the UNIX Filesystem

When writing unit tests, it is good to call functions with “mocks” or “fakes” — objects with equivalent interface but a simple, “fake” implementation. For example, instead of a real socket object, something that has recv() but returns “hello” the first time, and an empty string the second time. This is great! Instead of testing the vagaries of the other side of a socket connection, you can focus on testing your code — and force your code to handle corner cases, like recv() returning partial messages, that happen rarely on the same host (but not so rarely in more complex network environments).

There is one OS interface which it is wise not to mock — the venerable UNIX file system. Mocking the file system is the classic case of low-ROI effort:

  • It is easy to isolate: if functions get a parameter of “which directory to work inside”, tests can use a per-suite temporary directory. Directories are cheap to create and destroy.
  • It is reliable: the file system rarely fails — and if it does, your code is likely to get weird crashes anyway.
  • The surface area is enormous: open(), but also, os.mkdir, os.rename, os.mknod, os.rename, shutil.copytree and others, plus modules calling out to C functions which call out to C’s fopen().

The first two items decrease the Return, since mocking the file system does not make the tests easier to write or the test run more reproducible, while the last one increases the Investment.

Do not mock the file system, or it will mock you back.

by moshez at December 02, 2016 05:34 AM

November 30, 2016

Itamar Turner-Trauring

The Not-So-Passionate Programmer: finding a job when you're just a normal person

When reading programming job postings you'll find many companies that want to hire "passionate programmers". If you're just a normal programmer looking for a normal job this can be pretty discouraging.

What if you're not passionate? What if you don't work on side projects, or code in your spare time?

What if you have a sneaking suspicion that "passionate" is a euphemism for "we want you to work large amounts of unpaid overtime"? Can you really find a job where you don't have to be passionate, where you can just do your job and go home?

The truth is that many companies will hire you even if you don't have "passion". Not to mention that "passion" has very little impact on whether you do your job well.

But since companies do ask for "passion" in job postings and sometimes look for it during interviews, here's what you can do about your lack of "passion" when searching for a job.

Searching for a job

The first thing to do is not worry about it too much. Consider some real job postings for passionate programmers:

  • "[Our company] is looking for Java Engineer who is passionate about solving real world business problems to join our team."
  • "We're looking for a senior developer to play a major role in a team of smart, passionate and driven people."
  • "This role is ideal for somebody who is passionate about building great online apps."

They all say "passionate", yes. But these are all posts from very different kinds of companies, with different customers, different technology stacks, and very different cultures (and they're in two different countries). Whoever wrote the job posting at each company probably didn't think very hard about their choice of words, and if pressed each would probably explain "passionate" differently.

It might be a euphemism for working long hours, but it might also just mean they want to hire competent engineers. If the job looks good otherwise, don't think about it too hard: apply and see how it goes.

Interviewing for a job

Eventually you'll get a job interview at a company that wants "passionate" programmers. A job interview has two aspects: the company is interviewing you, and you are interviewing the company.

When the company is interviewing you they want to find out if you're going to do your job. You need to make a good impression... even if insurance premiums, or content management systems, or internal training or whatever the company does won't be putting beating cartoon hearts in your eyes.

  • First, that means you need to take an interest in the company. Before your interview do some research about the company, and then ask questions about the product during the interview.
  • Second, since you can't muster that crazy love for insurance premiums, focus on what you can provide: emphasize your professional pride in your work, your willingness to get things done and do them right.

At the same time that you're trying to sell yourself to the company you should also be trying to figure out if you want to work for them. Among other things, you want to figure out if the word "passionate" is just a codeword for unpaid overtime.

Ask what a typical workday is like, and what a typical workweek is like. Ask how they do project planning, and how they ensure code ships on time.

Finally, you will sometimes discover that the employees who work at the company are passionate about what they do. If this rubs you the wrong way, you might want to find a different company to work for.

If you're OK with it you'll want to make sure you'll be able to fit in. So try to figure out if they're open to other ways of thinking: how they handle conflicts, how they handle diversity of opinion.

On the job

Eventually you will have a job. Most you'll just have a normal job, with normal co-workers who are just doing their job too.

But sometimes you will end up somewhere where everyone else is passionate and you are not. So long as your coworkers and management value a diversity of opinion, your lack of passion can actually be a positive.

For example, startups are often full of passion for what they're building. Most startups fail, of course, and so every startup has a story about why they are different, why they won't fail. Given the odds that story will usually turn out to be wrong, but passionate employees will keep on believing, or at least won't be willing to contradict the story in public.

As someone who isn't passionate you can provide the necessary sanity checks: "Sure, it's a great product... but we're not getting customers fast enough. Maybe we should figure out what we can change?"

Similarly, passionate programmers often love playing with new technology. But technology can be a distraction, and writing code is often the wrong solution. As someone who isn't passionate you can ensure the company's goals are actually being met... even if that means using existing, boring software instead of writing something new and exciting.

There's nothing wrong with wanting to go home at the end of the day and stop thinking about work. There are many successful software developers who don't work crazy hours and who don't spend their spare time coding.

Join the course: Getting to a Sane Workweek

Don't let your job take over your life. Join over 800 other programmers on the journey to a saner workweek by taking this free 6-part email course. You'll learn how you can work reasonable hours and still succeed in your career a programmer.

Unsubscribe at any time. Powered by ConvertKit

If you would like a job that doesn't overwhelm your life, join my free 6-part email course to learn how you can get to a sane workweek.

November 30, 2016 05:00 AM

November 25, 2016

Twisted Matrix Laboratories

Twisted 16.6.0 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 16.6!

The highlights of this release are:
  • The ability to use "python -m twisted" to call the new twist runner,
  • More reliable tests from a more reliable implementation of some things, like IOCP,
  • Fixes for async/await & twisted.internet.defer.ensureDeferred, meaning it's getting closer to prime time!
  • ECDSA support in Conch & ckeygen (which has also been ported to Python 3),
  • Python 3 support for Words' IRC support and twisted.protocols.sip among some smaller modules,
  • Some HTTP/2 server optimisations,
  • and a few bugfixes to boot!
For more information, check the NEWS file (link provided below).

You can find the downloads on PyPI (or alternatively our website). The NEWS file is also available on GitHub.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
Amber Brown (HawkOwl)

by Amber Brown ( at November 25, 2016 08:06 PM