Planet Twisted

Jonathan Lange: Simple made easy

2012-02-01T13:50:38+00:00

Rich Hickey did a great talk at Strange Loop called "Simple Made Easy". You should watch it.

When I tried to explain the talk to someone, I stumbled a lot and it was obvious to me that I didn't really understand it. So I'm going through it again and turning it into a blog post, purely for my own gain.

This is roughly the first half of the talk. Not much of my own analysis or opinion is inserted, and I've pretty much stuck with Hickey's illustrations and phrasings. Thus this post is pretty derivative. Oops.

Simple vs Easy

"Simple" means one thing, "easy" another. Simple is the opposite of complex. A thing is simple if it has no interleaving, if it has one purpose, one concept, one dimension, one task. Being simple does not imply one instance or one operation: it's about interleaving, not cardinality. Importantly, this means that simplicity is objective.

Easy is the opposite of hard, or difficult. A thing is easy if it's near to hand, if it's easy to get at (location), if it's near to our understanding (familiarity) or skill set or if it's within our capabilities. This means that ease is relative.

Speaking English is dead easy for me, but that doesn't mean that speaking English is intrinsically simple. I find French quite difficult. Little French children speak French all the time, and there's always a part of me that thinks, "Boy, those kids are clever, being able to speak a foreign language at that age", but that's silly. It's easy for them, it lies near to them.

This distinction between simple & easy is good one, and is useful in all sorts of areas. But how does it relate to software?

Constructs vs Artefacts

As programmers, when we make software we are working on constructs: source code, libraries, language concepts and so forth. Rich contends that we focus on the ease of use of those constructs: How many lines of code? How much boilerplate? Will new developers be familiar with our technology?

But all of this is secondary. What actually matters is the artefact, the running programs that users actually use. Does it do what it's supposed to do? Does it do it well? Can we rely on it working well? Can we fix problems when they occur? Can we change it? You know, the interesting problems.

Thus we need to be assessing our constructs – our code, our technology choices – based on the attributes of the artefacts that we'll create, not based on the experience of typing code in.

Limits

We can't make something reliable if we don't understand it. And, actually, everyone's understanding is pretty limited. We can all only hold a small number of things in our head at once.

When things are complex, many parts are tied together by definition. You can't pull out just one piece and consider it because it's intertwined with other pieces. This creates an extra burden to understanding a system and thus makes it difficult to reason about the system.

You do need to reason about a system, both to know what to change and to be able to do so without introducing defects. Tests, refactoring, rapid deployment and all that are great, but to make a change to the system safely & without fear still requires you to be able to reason about it. Every bug in your product that was found in the field passed the type checker and passed all of the tests. Your type system doesn't tell you what change to make next in order to get the software you want any more than guard rails on a highway tell you how to get to Grandma's.

Speed

Focusing on ease and ignoring simplicity means that you'll go really fast in the beginning, but will become slower and slower as the complexity builds.

Focusing on simplicity will mean that you'll go slower in the beginning, because you'll have to do some work to simplify the problem space, but making sure that you only have intrinsic complexity means that your rate of development will remain at a high constant.

There are no actual numbers for this.

Complicating constructs

Many complicating constructs are available, familiar, succinctly described and easy to use. But none of that matters to end users. What matters is the complexity they yield. This complexity is incidental, it's not intrinsic to the problem.

If we build things simply, then the resulting system is easier to debug, easier to change and easier to understand.

Compare a knitted castle to a castle made of Lego. The knitted castle might have been great fun to make, and might have been really easy if knitted using a loom and cutting edge knitting tools, but there's no way that it's easier to change than a Lego castle. It's not about the ease of construction, it's about the simplicity of the artefact.

How can we make software easier?

Well, we can install it to make it easier by location. We can learn it and try it to make it easier by familiarity. We can't do much about our capabilities though. If we want software to be easier to comprehend, we are going to have to bring it down to our level. We have to make it simpler.

Take Lisp as an example. It's hard for many people because they don't have a Lisp installed, or their editor doesn't support paren matching, but they can make it easier by installing a Lisp and getting a plugin for their editor. It's also hard because it's unfamiliar. Who'd have thought that parens could go on that side of the function? But you can gain that familiarity quickly enough.

But parens in Lisp are used for functions and for grouping data. That's hard to get your head around, and that's because it's complex. It braids together two distinct notions.

Jack Moffitt: The Numbers Behind the Twitter Data Silo

2012-01-30T20:47:00+00:00

The dark future of search is being foreshadowed by this Twitter vs. Google fight. The latest Twitter volley at Google is this quote (seen on GigaOm) from Twitter CEO Dick Costolo:

"Google crawls us at a rate of 1300 hits per second... They've indexed 3 billion of our pages," Costolo said. "They have all the data they need."

There's no doubt that 1,300 hits per second is a large number, but let's put that in perspective:

In February 2010, Twitter was at 50 million tweets per day. This is just under 600 tweets per second.
In June 2011, Twitter was at 200 million tweets per day. This is over 2,300 per second.
In October 2011, Twitter hit 250 million tweets per day or just under 3,000 per second.
They have spikes of over 7,000 tweets per second, with the largest (so far) being just over 25,000 tweets per second.

For part of 2010, Google was perhaps able to keep up with the stream at 1,300 requests per second. Somewhere between February and June, the average volume of tweets outpaced them.

Let's assume that they kept pace until June 2011, and that on June 1, Twitter went from somewhere in the range of 1,300 tweets per second to their reported 2,300 tweets per second. Google is 1,000 tweets behind per second.

By the end of the year, Google missed 15.5 billion tweets. They are two months behind if they didn't skip any, and the tweet volume did not increase. But it did increase by 25% or so by October, and surely it has grown more since then.

If Google has only indexed 3 billion pages so far, they have approximately 12 days of tweets at current volume. It's pretty hard to rationalize the 3 billion pages number against the 1,300 per second number. Was Google indexing at a much slower rate before? Did they not start until a few months ago?

Of course Google may be getting multiple tweets per request, perhaps by crawling the timelines of important users. But this means that they probably get a lot of requests that don't give them any new tweets, or else the timeliness of the data is poor.

No matter how you slice it, it appears Google would be unable to keep up. Even if they were keeping up now, Twitter's growth probably sets a time limit for which keeping up remains possible.

Perhaps Google is super clever, and can index only the right tweets. I think that it's more probable they have "enough" data to surface results for the super popular topics, and miss nearly everything in the long tail of the distribution. I expect that this adversely affects search quality, which one suspects is a high priority for the world's best search engine.

Google is no saint. They are just as guilty of the same data hoarding. If you ran these numbers for YouTube indexing, I think you will find the situation is much worse. I imagine that most of these data silo companies purposefully set their crawl rates too low for anyone to achieve high quality search results.

In the case of Twitter, the end result for users is even worse because Twitter's own attempts at search are terrible and are getting worse over time. At least Google makes a decent YouTube search, even if no one else can.

Even if Google could get all the tweets, they still would have very little to no Facebook data. I still think the best strategy in this situation for them is to create their own social data and use that instead. It's a tough road, but they seem to have little choice.

In the end, it's not about Google or Twitter or Facebook, but the stifling of innovation and competition around data. We can only hope that some federated solution or some data-liberal company wins out in the end.

Moshe Zadka: Why programmers are concerned about copyright law [Addendum]

2012-01-26T17:48:35+00:00

As we saw in the last edition, programmers are concerned about copyright law because the only way to universally enforce copyright law would be to take away all universal Turing machines and make sure that we cannot control them. How bad would that be?

The previous episodes focused on laws of math (generality of computation) and physics (quantum mechanics makes it easy to build general purpose computers). Now we concentrate on laws of economics in the industrial age. In the beginning of the industrial age, factories lowered the cost of goods dramatically from before — a textile factory is much more efficient than a tailor making suits, and so it is much cheaper to clothe ourselves. However, as factories evolved, it became more and more important to drive the cost even lower — whereas before the competition was with other tailors, now the competition was with other factories. So, economies of scale and efficient production lines developed. Soon after that, supply chain management and moving factories to the most cost-efficient places were developed.

Let’s say that you want to make T-shirts with political slogans. You’ve got your slogan writers, huddled and coming up with good slogans. They’re a sunk cost — you’ll pay them the same no matter how many T-shirts you make, or how much they cost. Now you build a big factory to make T-shirts, and print slogans on them. The company next door specializes in T-shirts with band logos. They build a big factory to make T-shirts and print logos on them. Someone realizes there is money to be made in this scenario — build a factory to manufacture T-shirts, and send them in large crates to places that will print stuff on them. Because their factory is bigger, their cost for making T-shirts are lower, and they pass some of the savings onto you, the political slogan company. In turn, you pass some of the savings onto your customers, and everyone is happier.

Lesson: When economies of scale hit, you want to try and buy standard components for the product you manufacture.

Now suppose you build a car. The anti-lock brake system need to figure out when the car is sliding, and start “pumping” the brake. You can build a system that will contain gyroscopes that attach to some handle that pulls a wire that pumps the brake. Or, you can

Buy an accelerorometer
Buy a general purpose chip
Write software for this chip, based on input from the accelorometer
Connect this software to a motor that controls the brake

Note that in this scenario, we source some standard parts. Even though the parts are more complicated, it turns out that we save a lot of money by using standard chips and accelerometers. Note that putting the software on the chip is just copying it to some standard storage device — extremely cheap. ABS brakes become cheaper, and people die less. Overall, a good thing.

The same sort of logic causes a lot of things that used to be done with handles, levers and pulleys to be done with a general purpose chip and some software. Your car is full of them. Your microwave has some. Your television has some. Each of those devices is basically a computer connected to some strange peripherals (brakes, microwave-rays or screens). In a world with strong copyright regulations, those devices’ software are locked to us, even though we own the device. Where darkness goes, evil deeds covered by the darkness soon follow. The police in Evil Regime Country wants to make sure nobody can run from them. They mandate all cars sold in ERC must have, in their software, a special switch that when a certain bluetooth signal is sent (many cars nowadays have computers connected to bluetooth receptors), the software controlling the automatic gear system makes sure the car will not go over second gear. If the ERC police are smart about using this signal, nobody will ever know.

In a world with strong copyright protection, big corporations control your life — and they are corruptible, and if darkness holds, corrupt.

Thomas Vander Stichele: GStreamer 0.11 Application Porting Hackfest

2012-01-26T10:16:27+00:00

I’m in the quiet town of Malaga these three days to attend the GStreamer hackfest. The goal is to port applications over to the 0.11 API which will eventually be 1.0 There’s about 18 people here, which is a good number for a hackfest.

The goal for me is to figure out everything that needs to be done to have Flumotion working with GStreamer 0.11. It looks like there is more work than expected, since some of the things we rely on haven’t been ported successfully.

Luckily back in the day we spent quite a bit of time to layer parts as best as possible so they don’t depend too much on each other. Essentially, Flumotion adds a layer on top of GStreamer where GStreamer pipelines can be run in different processes and on different machines, and be connected to each other over the network. To that end, the essential communication between elements is abstracted and wrapped inside a data protocol, so that raw bytes can be transferred from one process to another, and the other end ends up receiving those same GStreamer buffers and events.

First up, there is the GStreamer Data protocol. Its job is to serialize buffers and events into a byte stream.

Second, there is the concept of streamheaders (which is related to the DELTA_UNIT flag in GStreamer). These are buffers that always need to be send at the beginning of a new stream to be able to interpret the buffers coming after it. In 0.10, that meant that at least a GDP version of the caps needed to be in the streamheader (because the other side cannot interpret a running stream without its caps), and in more recent versions a new-segment event. These streamheaders are analogous to the new sticky event concept in 0.11 – some events, like CAPS and TAG and SEGMENT are now sticky to the pad, which means that a new element connected to that pad will always see those events to make sense of the new data it’s getting.

Third, the actual network communication is done using the multifdsink element (and an fdsrc element on the other side). This element just receives incoming buffers, keeps them on a global buffer list, and sends all of them to the various clients added to it by file descriptor. It understands about streamheaders, and makes sure clients get the right ones for wherever they end up in the buffer list. It manages the buffers, the speed of clients, the bursting behaviour, … It doesn’t require GDP at all to work – Flumotion uses this element to stream Ogg, mp3, asf, flv, webm, … to the outside world. But to send GStreamer buffers, it’s as simple as adding a gdppay before multifdsink, and a gdpdepay after fdsrc. Also, at the same level, there are tcpserversink/tcpclientsrc and tcpclientsink/tcpserversrc elements that do the same thing over a simple TCP connection.

Fourth, there is an interface between multifdsink/fdsrc and Python. We let Twisted set up the connections, and then steal the file descriptor and hand those off to multifdsink and fdsrc. This makes it very easy to set up all sorts of connections (like, say, in SSL, or just pipes) and do things to them before streaming (like, for example, authentication). But by passing the actual file descriptor, we don’t lose any performance – the low-level streaming is still done completely in C. This is a general design principle of Flumotion: use Python and Twisted for setup, teardown, and changes to the system, and where we need a lot of functionality and can sacrifice performance; but use C and GStreamer for the lower-level processor-intensive stuff, the things that happen in steady state, processing the signal.

So, there is work to do in GStreamer 0.11:

The GStreamer data protocol has not really been ported. gdppay/depay are still there, but don’t entirely work.
streamheaders in those elements will need adapting to handle sticky events.
multifdsink was moved to -bad and left with broken unit tests. There is now multisocketsink. But sadly it looks like GSocket isn’t meant to handle pure file descriptors (which we use in our component that records streams to disk for example)
0.11 doesn’t have the traditional Python bindings. It uses gobject-introspection instead. That will need a lot of work on the Flumotion side, and ideally we would want to keep the codebase working against both 0.10 and 0.11 as we did for the 0.8->0.10 move. Apparently these days you cannot mix gi-style binding with old-style binding anymore, because they create separate class trees. I assume this also means we need to port the glib2/gtk2 reactors in Twisted to using gobject-introspection.

So, there is a lot of work to be done it looks like. Luckily Andoni arrived today too, so we can share some work.

After discussing with Wim, Tim, and Sebastien, my plan is:

create a common base class for multihandlesink, and refactor multisocketsink and multifdsink as subclasses of it
create g_value_transform functions to bytestreams for basic objects like Buffers and Events
use these transform functions as the basis for a new version of GDP, which we’ll make typefindable this time around
support sticky events
ignore metadata for now, as it is not mandatory; although in the future we could let gdppay decide which metadata it wants to serialize, so the application can request to do so
try multisocketsink as a transport for inside Flumotion and/or for the streaming components.
In the latter case, do some stress testing – on our platform, we have pipelines with multifdsink running for months on end without crashing or leaking, sometimes going up to 10000 connections open.
Make twisted reactors
prototype flumotion-launch with 0.11 code by using gir

That’s probably not going to be finished over this week, but it’s a good start. Last night I started by fixing the unit tests for multifdsink, and now I started refactoring multisocketsink and multifdsink with that. I’ll first try and make unit tests for multisocketsink though, to verify that I’m refactoring properly.

Jonathan Lange: Undistract me

2012-01-23T17:44:37+00:00

Here's a thing that happens a lot to me: I'm doing some work, and as part of that work I need to run a command in my terminal that takes a little while. I run the command, look at it for about a second and then switch to doing something else – checking email, perhaps. I get deeply involved in my email checking, and then about twenty minutes later I switch back to the terminal and see the command has finished. For all I know, it finished nineteen minutes ago, and I was just too engrossed to notice it.

This is a big productivity sink for me, especially if the command happened to fail and need retrying. I'm not disciplined enough to just sit and watch the command, and I'm not prescient enough to add something to each invocation telling me when a command is done. What I want is something that alerts me whenever long running commands finish.

Well, that thing now exists, thanks to glyph's script that provides precmd and postcmd support to bash and a lot of help from Chris Jones of Terminator.

To use it right now:
$ bzr co lp:~jml/+junk/shell-tools
$ . shell-tools/long-running.bash
$ notify_when_long_running_commands_finish_install

You'll see that if you run a command that takes over 30 seconds to complete, it will pop up a notification, which should hopefully take you away from whatever it was you are doing and back to the task at hand.

If you look at the code, you'll see that it installs two hooks: precmd and preexec. preexec runs just before the shell launches a command, and precmd runs just before it prompts for the next command. Our preexec stores when the command was launched and the precmd checks to see if it finished within a certain time frame. If not, it sends out a notification.

Currently, you'll get a notification when you finish reading a long document, since the command finishes a long time after the command starts. Obviously this isn't ideal. I think the fix is to only send notifications when the shell doesn't have focus. Unfortunately, that's a little tricky and I think is going to be highly terminal specific.

Anyway, I'm a total shell newbie, so I'd love to know if there's any way this could be done better. Also let me know if you find this useful, or you know of someone who has already done this.

Jp Calderone: Cleaning Up Branch Checkouts

2012-01-21T19:29:35+00:00

Since Twisted development typically involves at least one branch per ticket, a Twisted developer can end up with a lot of branches checked out. For example, this morning I had 177 Twisted branches checked out on my laptop. Many of these were branches that I contributed code to, and perhaps even merged into trunk myself when they were complete. I could probably have deleted them at that point, but I usually can't be bothered. Besides, I put everything I have into the branch itself, by the time I'm merging it I'm done. Other branches are ones I've done code reviews on for other developers. I don't keep track of when these get merged into trunk as closely, since typically someone else is going to do those merges.

The incremental cost of another Twisted branch is pretty minimal. A few more megs used on my hard drive is barely noticable. The aggregate cost can get pretty high though (Seven GB for the 177 branches I had this morning). At some point this can cause problems.

Not all of these branches have been merged to into trunk, either, or I could just wipe them all out with ease. And while I try never to leave uncommitted changes in a branch checkout, nobody's perfect... What I really want to do is just get rid of the branches that just aren't relevant anymore.

So I use cleanup-local.py to deal with the mess. It looks at my branch checkouts, talks to the Twisted issue tracker to learn the state of the associated ticket (due to the naming convention for Twisted branches, it is easy to determine which ticket is associated with a branch, given just the branch name). Then it deletes all the checkouts associated with closed tickets (due to the Twisted workflow, if a ticket is closed, it is a very safe bet that you won't need its branch anymore).

The net result is that in (far) less time than it took to write this post, my laptop went from having 177 Twisted branches to having just 34. To save even more time, I could probably set this up as a weekly cron job or something similar. It's easy enough to run now, though, that I just do so manually once every couple of months to keep things tidy.

Here's a brief snippet from today's run:

Found password-comparison-4536-2 for ticket(s): 4536
Status of 4536 is assigned
Found pb-chat-example-4459 for ticket(s): 4459
Status of 4459 is closed
Removing closed: pb-chat-example-4459
Found plugin-cache-2409 for ticket(s): 2409
Status of 2409 is closed
Removing closed: plugin-cache-2409
Found poll-default-2234-2 for ticket(s): 2234
Status of 2234 is closed
Removing closed: poll-default-2234-2

Moshe Zadka: Why programmers are concerned about copyright law [Part 1 of 2]

2012-01-20T17:37:25+00:00

Welcome to a new experiment. I am going to try and explain why programmers tend to be concerned (one way or another) about copyright law, to the level where we compare it to slavery or tyranny. This is not going to be easy, since I intend this to be readable by people who are not programmers, and who never programmed. I am going to start from the very beginning, and it’s going to take a while.

Before we even start, I would like to point out an excellent description for why we usually expect explanations to be simpler then they really are. Please do read this. I’ll wait, really. If you think you do not need to read it, that’s actually evidence that you do — you expect my explanation to be simpler than it is, and that you do not need that bit of background knowledge…

Welcome back! One important concept introduced in the article linked above is “Word of Power”. (If you didn’t read it, now is the time to fix that issue…) I will try to introduce the new concepts using Words of Power, saying the word, and then linking it to the power behind it.

The first Word of Power will be “Universal Turing Machine”. You may remember a previous post of mine about Alan Turing, one of the greatest giants on whose shoulders we have the privilege to be standing on. But I want to start by talking about another great giant, John McCarthy. McCarthy wrote a paper with the somewhat unassuming name, Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I. A more appropriate title would have been “paving the way forward for programming for the next 50 years”, you see. What McCarthy did in the paper was to lay down a simple explanation of a system for defining computation. Nowadays, a lot of computation is done, appropriately, on electronic computers. Before we had those, however, “computer” was a job title of a person who applied computation rules to symbols on paper to get results — much like the rules we learned in primary school for doing long multiplication.

In both sense of the word “computer”, the goal is the same: apply rules to manipulate symbols. McCarthy defined a specific set of rules to manipulate symbols that he called “Lisp”. Lisp used S-expressions (short for “symbolic expressions”) to define computations. Read this sentence over again, because it might be the most important sentence in here: even though Lisp was a set of rules to manipulate S-expressions, those S-expressions define other computations. If you only learned to manipulate S-expressions, someone could write an S-expression that did, say, long multiplication. Then, if you manipulated the S-expression for long multiplication followed by S-expressions representing numbers, you would do long multiplication. Now comes the fun part — you could even write the rules for manipulating S-expressions using S-expressions. You might think that for this to work, the rules for manipulating S-expressions would be really complicated. This is not true — you can see the manipulation rules here. Sure, it’s a bit long, but doesn’t look like more than you needed to learn to do arithmetic, right?

Well, what use would rules for manipulating S-expressions when already know how to manipulate S-expressions? Maybe very little, if not for a few other thing. I hope you have seen a demonstration of the game of life, once upon a time. If you haven’t, I recommend that you read the wikipedia article. With 4 simple rules, much much simpler than McCarthy’s S-expression rules, much is possible. How much? Well, the proof is long and difficult, but mathematicians have found a Game of Life configuration that will manipulate S-expressions. So if you only knew to manipulate squares on graph paper according to the rules of the Game of Life, seemingly easier than manipulating S-expressions, we could just give you a (pretty hefty) Game of Life that would cause you to manipulate S-expressions. Wait, but manipulating S-expressions lets you do arithmetic, right? So you could do arithmetic too!

Wait, what about arithmetic? Well it turns out, if you just know how to add 1 to numbers, and do a bunch of other trivial things (like substitute numbers for other numbers), we can give you a instructions that will let you manipulate the Game of Life. Or other instructions, that will manipulate S-expressions. It turns out that if you can do interesting enough computations, it doesn’t matter a whole bunch what you know to do — you can find an initial input (S-expression, Game of Life configuration, or arithmetic instructions) that will let you do anything. Anything? Well, not quite!

Another interesting result (called “Rice’s Theorem”) is that you cannot write an S-expression (or any of the other things) to take any S-expression (or any of the other things) and say anything interesting about what it does to any input. Notice that I didn’t say “respectively”, and didn’t mean to — you can’t make an S-expression that will take any Game of Life configuration, and will say anything interesting about what it does. You can write S-expressions that will take some S-expressions (or …) and say something interesting about them — but not any S-expression (or …).

What has all this to do with our friend, Alan Turing? Well, Turing invented the original thing that stands for “…”, and he called it a Turing machine. A Turing machine has an internal state, and reads from and writes to a tape. The interesting thing about a Turing machine? You can write a Turing machine that will manipulate S-expressions. Or play the Game of Life. Or perform arithmetic. Any of those Turing machines can do anything S-expressions do! All of those are called “Universal Turing Machine”, because any computation possible by any other Turing machine, or S-expression, or Game of Life, or arithmetic, can also be done by them. You only need One Turing machine, or one S-expression, or one Game of Life configuration, or one set of arithmetic rules and you can do anything anyone else can do, if maybe somewhat slowly. Fortunately, or unfortunately, all those computations also have the same basic flaw — you can never write one of those to say anything interesting about any computation that comes their way (this will be important later!)

Let us recap:

Computations that manipulate symbols can be done either electronically or by a human, equally well, if not equally fast.
You can define a “universal computation”, that if you learn to do, anyone can get you to do any other computation by writing the correct input.
It is impossible for you to know anything interesting about any possible computational input.

Those three concepts (“a lot of interesting manipulations are equivalent”, “universal manipulations are possible” and “nothing interesting to say”) are the core of what computers are. A computer chip does very specific manipulations ([1]). However, programmers, by writing carefully thought out inputs, can get it to perform any computation ([2]). Lastly, it is impossible to write a computer chip that will be able to say anything interesting about any specific program ([3]). Of course, crafting these programs is difficult — this is why programmers use “programming languages”, which are even more “universal computation instructions”. A language is called “Turing complete” when you can use it to build a Turing machine (which would also have allowed you to build an S-expression manipulator, or Game of Life player, etc.). In short, all we have said so far applies equally well to most programming languages. There are niche non-Turing-complete languages, used where being able to say something interesting about any possible program is important — but here’s the rub — it’s actually hard to invent something which is not Turing complete. As we saw above, even very simple things will be able to build Turing machines.

Computers, languages and programmers are all becoming better at making computations that were “possible” a decade ago be “fast” now. There is an enormous economic pressure on that — after all, the more you can do “fast”, the more you can do “more and bigger”, and people like “more and bigger”.

Now, we come to a very simple computation — copying. A “copier” can be defined in multiple ways, but let’s suppose, for the sake of argument, that we just want to replicate the input twice (“have two copies”). It is easy to write an S-expression to do it. Therefore, any Turing complete environment can do it. But remember — it is impossible to say anything “interesting” about any possible S-expression, and therefore, it is impossible to write a computation that outputs the correct answer for “this S-expression is not a copier”. Wait, what? Yes, that’s right. If an S-expression has access to some input, it is impossible to know for sure that it will not copy it.

The next Word of Power I wish to introduce is “bits”. A “bit” (short for binary digit) is a place-holder for something that can be either 0 or 1. Anything that can represent at least two states (say, the light switch in your kitchen) is a bit. Now, let’s say that you have a set of symbols — say musical notes. I hope you have seen “Sounds of Music”, and remember the best song ever — “Do, a deer” (etc.). Our symbols are “Do, Re, Mi, Fa, Sol, La, Ti”. As the song makes the case, we can write any piece of music with just those (yes, I know about sharps, flats and octaves…bear with me). Now, let’s assign “bit patterns” to each note:

Do — 000
Re — 001
Mi — 010
Fa — 011
Sol — 100
La — 101
Ti — 110

Now, if we want to write the notes for the first line of “Twinkle, Twinkle” (Do Do Sol Sol Ti Ti Sol) we can instead write 000000100100110110100 in bits. If we have those bits, we can just take them in threes, and convert them back to the musical notes. We can do the same with any set of symbols — say, the alphabet. This is actually how, more or less, computers store text — they can convert the alphabet into bits, and save those bits. Then they can manipulate those bits using carefully crafted programs to, say, replace the word “Foo” with “Bar”. Or, say, copy them. If we have the bits to a fan fiction of Mickey Mouse having sex with Pluto, we can copy those too. If instead, we have the bits for an S-expression that will generate the bits for a fan fiction of Mickey Mouse having sex with Pluto, we could copy those too. But wait, this is funny — there is no way to know for certain that certain bits are not an S-expression that will produce the bits of Mickey Mouse having sex with Pluto.

A fan fiction of Mickey Mouse having sex with Pluto is a dangerous thing. It is, as the law and case law currently stands, a violation of copyright law. In general, it is not protected under the Fair Use doctrine, and this means that writing this fan fiction is illegal. Copying this fan fiction is illegal. What’s more, Disney has an incentive (or at least, believes it has an incentive) to prevent copying this fan fiction around. When incentive (or perceived incentive) and legal powers combine, the result is expected — Disney would dearly love to have an automatic way to prevent computers from copying this fan fiction. Or, say, from copying the bits that represent the video of “Cars 2″.

Remember what I said above — it’s impossible to have a way to know for certain which bits are actually an S-expression that will create the bits for the “Cars 2″ video. Although Disney has the incentive (this is a matter of economics and psychology, ultimately the results of the forces of evolution) and the legal powers (this is a matter of social convention), the math, hard and unyielding, doesn’t care. Math doesn’t care about evolution. Math doesn’t care about society. Math is math, and the math says that you can’t build a computer that will only copy things if they’re not S-expressions that produce the Cars 2 video, no matter how much you want to.

Join me next episode, when I explain the basics of cryptography, and how they pertain to the issue of copyright law.

Moshe Zadka: Why programmers are concerned about copyright law [Part 2 of 2]

2012-01-20T17:36:03+00:00

[A lot of the ideas in this edition awe a lot to Cory Doctorow. The responsibility for any mistakes or omissions are still mine.]

Recapping the previous episode:

It’s easy to define a system that is as powerful as any other computational system (Turing complete)
It’s impossible to say what any computation does (Rice’s theorem)
You can use bits (0s and 1s) to encode any sort of data — music, text or anything else.

In this episode, I will have a lot more to say about bits. Before I delve into bits, though, I want to talk about computers. While abstract systems that do computations abound, most modern computers are fairly similar. The theoretical computational system they are built to resemble are so-called “Von Neuman Machines“. A modern computer has lots of peripherals, but strip away the peripherals and you are left with:

A “microchip” which implements instructions like “Add Register1 and Register2 and put the Results in Register3″
Memory — mapping of “addresses” (index numbers) to “values”
Instructions on the microchip of the sort “Treat register 1 as an address, and fetch the value there into register 2″

The exact instruction set that the microchip implements depends on the type, but ultimately, as we saw before, it does not matter too much — all computational systems are equivalent. What is important is “Moore’s Law”, of which many variants exist but ultimately says that:

Microchips can perform more calculations per second every year.
The amount of memory available for a given price keeps growing.

An important computer peripheral, which almost all computers have a variant of, is a storage device. That ranges from a magnetic hard drive to a micro-SD card. All that is important, for our purposes, about those is:

Accessing them is slower than memory
They are bigger (have more addresses) than memory for the same price [a lot more -- frequently 10x or 30x]
They grow bigger for the same price every year

Another important computer peripheral is the network card. Networks connect computers to each other.

Now, let’s remember something we said earlier — there is no way to reliably detect any sequence of 0s and 1s that encodes a computation that we do not like — say, a Turing machine outputting a fanfiction about Micky Mouse having sex with Pluto. Therefore, if a network card allows you to send two distinct messages [if it only allows you to send one message, it's kind of a sucky network card], you can send some “illegal” Turing machine encoding. Moreover, you will be able to store this illegal Turing machine faster and more cheaply every year.

So as an inescapable conclusion of (1) The math (b) Moore’s law we see that the law cannot be enforced, and it costs less and less every year to evade this law. The laws of the universe (such as how easy it is to use Quantum Mechanics to implement computation on the stuff that is abundant on every ocean beach, or Rice’s theorem) care nothing for Walt Disney or a starving artist — they are simple there, immutable and unforgiving, and humans must learn to deal with them.

One way to deal with them would be to “follow the money”: enforce the law only when someone is breaking it for corporate-level gain. This is not the way the legal system has gone through. Instead, they turned to God — “Crypto. Really good, standards-defined crypto”.

So to understand what transpired, it is important to understand the basics of cryptography. From the beginning of time (as humans count time, I guess), people have communicated with each other — “Look, Ug, I found an antelope and killed it. Help me eat it?” Not long afterwards, eavesdropping begun — “Hey, everyone, Ung has an antelope.” The next level was to use codes — “Look, Ug, I found a You-know-what and you-know-what-ted it. Help me you-know-what it?”, and so the battle begun.

Codes, like in the example above, have to balance two issues. The person to whom they are intended must be able to decipher them (Ung better hope Ug will get the right message). The person to whom they are not intended for must not be able to decipher them (or once again, the whole village will know about the antilope). Fast-forward to Greek times, Caeasar had the eponymous cipher, based on shifting every letter in the alphabet by a certain amount and the Nazis had Enigma. Word to the wise: use neither of those, as they both have been “broken”. “Broken” is a term cryptographers use to say “the eavesdropper can read the messages which are not intended for them”. Cryptographers spend a lot of time trying to figure out how to read those…

But back to the basics: In computer-based cryptography, we have a secret, S. S can be considered, like everything else a computer handles, as a certain sequence of bits. For cryptography to work, two parties have to pre-agree on S. (Side note: I will not be covering public-key cryptography here.) Then, we must be able to compute the “encryption”: a computation that takes M, a message, and calculates a function that depends on M and S. Then we must be able to compute “decryption”: a function that takes the encrypted message, E, and S, and returns to us M. Next, it should be impossible (or at least, very hard), to compute M from E without S.

How do we know that something is “very hard”? Well, if we have a guess for S, we can check that the decryption gives us a plausible message. How hard is it to guess S? Returning to the quotation at the beginning of the section, “standards-based crypto” is usually at least 128 bits of S (also known as the “key size”). That means that there are 2 to the power of 128 options. It is largely agreed that the smallest time-frame relevant for computation is the time it crosses a photon (light particle) to cross a hydrogen atom. A hydrogen atom is about 10**-11 meters long. The speed of light is about 10**8 meters per second, which means it takes more than 10**-20 seconds, which is more than 2**-70 seconds. Thus, if we have 2**128 options, we need more than 2**58 seconds. 2**30 seconds is more than a year, so it will be more than 2**28 years, which is about 32 million years. I used various approximations above, but the conclusion still stands — and more and more modern crypto uses 256-bit sized keys, which is not quite the heat death of the universe, but the point still stands: you cannot guess and hope to win. However, it turns out to be an open problem whether any question where you can easily verify guesses (the formal name for that is NP) is “hard” (the formal name for “easy” problems is P). By open, I mean that many computer scientists have tried tackling this problem over the last 50 years, with no success (and little progress).

This means, in particular, we cannot know that any cryptographic computation is really “good”. However, what we can do, and the second part of why “really good” is followed by “standards-based”, is to ask really smart people to try and solve a cryptographic problem. If they can’t, after trying really hard, we assume that the problem cannot be solved, and we used it as our encryption mechanism.

In modern times, the names “Ug” and “Ung”, perfectly good though they may be, have fallen into disuse. Modern cryptographers usually talk about “Alice” and “Bob” wanting to transmit messages, and “Eve” wanting to listen to them surreptitiously.

So here is an example of a cryptographic system: if you buy a DVD in the store, it is encrypted with a special key. When you buy a DVD player in the store, is contains the key. Your DVD player is a computer, with special software that decrypts the DVD, with a special key, and then plays it. So Alice, which is the Hollywood studio, encrypted the movie contents. Then Bob, the DVD, decrypts it. Bob, to play the DVD, must have the key. You bought the DVD, so there is nothing stopping you from opening up the DVD player, and taking the key, is there?

Well, there are two things. Once is simple: Hollywood, before giving the DVD maker the key, make sure that the DVD player is hard to “tamper” with. The DVD maker must put the key in a special chip, glued to the microchip, and that self-destructs if people tamper with it. As you can imagine, creative people have found ways to defeat that self-destruction. And so, Hollywood convinced the US government to pass a law called “Digital Millenium Copyright Act”. The DMCA says that

It is illegal to tamper with the DVD that you bought and get the key out.
It is illegal to tell someone the key that you dug out of the DVD.
It is illegal to tell someone how to tamper with the DVD to get the key out.
It is illegal to tell someone where to find instructions on how to tamper with the DVD, or how to get the key.

You might have noticed that 2-4 are restrictions on speech. There are numbers that are so illegal, that not only are you not allowed to write them on a piece of paper and give them to your friend, but if someone spray-painted them on a building, you are not allowed to tell anyone where that building is. Whether you’re allowed to tell them where they can find a map with the building starred in it is, I believe, still up for debate.

The incredulity continues as you find songs on You Tube and Flag images that encode illegal numbers. This means that certain songs and flags are now illegal. In fact, it very well might be that the Wikipedia page on “Illegal numbers” is already illegal, since it contains data that can allow recovery of these numbers. I wish I were kidding, but I am not.

I, personally, am not a copyright extremist. I am not committed to abolish copyright. However, I think that understanding the math and physics of computation are important, because otherwise we end up making songs illegal. When Hollywood claims that copyright infringement might cause jobs lost, and that this necessitates stronger copyright law, we should first ask “Will this make singing songs illegal?”

This is why programmers are concerned about copyright law — because we understand all of the above. I hope, after reading this, you are also concerned!

Jack Moffitt: The More Things Change: A Review of The Soul of a New Machine

2012-01-20T10:20:00+00:00

Already in my career I've experienced enormous passion, burnout, extraordinary dedication to my team and projects, and depression. I'm sure many others have as well. Has it always been this way with technology? I often wonder if this rollercoaster is necessary, healthy, or normal.

I recently saw a recommendation for Soul of a New Machine, which tells the story of a team of engineers at Data General who built a new 32-bit computer in the late 1970s. The book is fascinating. Thirty year later, many of its descriptions of the project and the way the team worked and was treated could apply to any modern project.

The plot summary will no doubt sound familiar to you: A team of mostly young, mostly male engineers works grueling hours to build something amazing in too short an amount of time. They succeed, albeit a bit over their original schedule. Despite the project's commercial success, the team is denied both recognition and financial rewards and many end up leaving the company. Almost all of them ultimately enjoyed it and would (and did) do it again.

There were many pieces of this story that resonated with me.

Work is a Drug

On overworking Tom West, the manager of the team in the book, says:

That's the bear trap, the greatest vice. Your job. You can justify just about any behavior with it. Maybe that's why you do it, so you don't have to deal with all those other problems.

Why deal with the unpredictable world, when the controllable world of creation is available? It's code as escapist drug, and I love to get high on it. Mundane things like cleaning my house, and more serious ones like taking care of my health, are all easy to avoid while fixing bugs or starting a new project.

It's both possible and important to find a balance.

The team's secretary, who was much more than her title suggests, suffered and succeeded with the rest of the team. Even she says:

I would do it again. I would be very grateful to do it again. I think I would take a cut in pay to do it again.

Even as I recover from projects that burned me out, I am constantly thinking about how to do new ones. In fact, while I'm doing any project, I'm already thinking about doing another. This sounds like drugs again. But they are good drugs.

Harassment and Treatment of Women

The book describes how some team members tormented the lone female engineer. This is something that still happens today, and it's terrible. And people then wonder why there are so few women in our industry.

In addition to that, at the end when they hand out the peer awards, their award to the woman was for putting up with them, not for any of her actual accomplishments.

Betty Shanahan was that lone woman, and it looks to me that she deserved more than just an award for thick skin. She's the CEO of the Society of Women Engineers, and she was "a member of the design team for the first parallel processing minicomputer and manager of hardware design for subsequent systems." She later moved to the business side of technology, and I wonder if that had anything to do with her having to put up with the Eagle team's harassment.

How Something is Done is Important Too

Often we judge things by their properties, but one can also rightly judge something by how it is made. Shoes made from child labor are less good than those made in other ways.

Kidder, the book's author, discusses this:

In The Nature of the Gothic John Ruskin decries the tendency of the industrial age to fragment work into tasks so trivial that they are fit to be performed ony by the equivalent of slave labor. Writing in the nineteeth century, Ruskin was one of the first, with Marx, to have raised this now-familiar complaint. In the Gothic cathedrals of Europe, Ruskin believed, you can see the glorious fruits of free labor given freely. What is usually meant by the term craftsmanship is the production of things of high quality; Ruskin makes the crucial point that a thing may also be judged according to the conditions under which it was built.

By this kind of measure, is the work many teams do good? Is the Eagle computer that Tom West's team built really a success since the team worked much overtime, suffered divorces and other problems, and in the end received little to no reward?

I think it's time for entrepreneurs and workers in our industry to demand better. Our outputs will be better if they are made sustainably, and not just by the measure above. In retrospect, maybe the reviewers of LA Noire should have taken into the account the trials of its developers; it certainly would not have fared well.

Freedom of Expression

I want to hire resourceful people. I want to describe a general outline of a design and not have to describe it in intricate detail in order for them to build it.

It turns out that this is critical for happiness. If we're told exactly how to do something, it takes much of the creativity and fun out of the work.

Engineers are supposed to stand among the privileged members of industrial enterprises, but several studies suggest that a fairly large percentage of engineers in America are not content with their jobs. Among the reasons cited are the nature of the jobs themselves and the restrictive way sin which they are managed. Among the terms used to describe their malaise are *declining technical challenge; misutilization; limited freedom of action; tight control of working conditions*.

You must trust those you work with to be resourceful. If you don't trust them, you will end up micromanaging them into unhappiness, and you will also remove their valuable creative input from your product.

There is a balance to be struck with feedback. The Eagle engineers thought that the managers didn't appreciate their efforts, but in reality, some of this was them trying to stay out of the way. Kidder asked the Tom West's boss:

Had the Eagle project always interested him or had it grown in importance gradually?

"From the start it was a very important project."

Was he pleased with the work of the Eclipse group?

"Absolutely!" His voice falls. "They did a hell of a job."

But some members of the team felt that they had been rather neglected by the company.

"That doesn't surprise me," he says. "That's frequently the case. There's often a conflict in people's minds. How much direction do they want?"

I've had this same issue with investors as well. You don't want them to meddle with your company or your product, but you also want their advice and guidance. It's possible to go too far in either direction, but mostly you hear about stories where investors meddle too much. I personally think it's probably better to err on the side of too little help than to end up with too much meddling.

The Venture Capitalists

Even thirty years ago, the VCs had a bad rap. Tom West was asked in a Wired article years after the book's publishing why he stayed at Data General until he retired:

"You could do new products and companies within the company, rather than shag some venture capitalist and kill yourself for five years." To be an entrepreneur, he says, "you have to be interested in networking, even with fools."

This is another reason why I would prefer to bootstrap companies if at all possible.

Tom West ended up working on many interesting projects at Data General, but ultimately, none of them got the support or recognition they deserved. The other members of the Eagle team spread out and started or worked for new companies, and in general seemed much happier.

Final Thoughts

In the end, it's both a fascinating tale of heroism and creativity and a saddening tale of undervalued and underpaid engineers. I am both emboldened to keep following my passions and more mindful of its dangers. My troubles are not unique - not even modern. Thirty years after this book was written, I feel like it could have been written yesterday.

Glyph Lefkowitz: The Concurrency Spectrum: from Callbacks to Coroutines to Craziness

2012-01-20T06:23:39+00:00

Concurrent programming idioms are on a spectrum of complexity.

Obviously, writing code that isn't concurrent in any way is the easiest. If you never introduce any concurrent tasks, you never have to debug any problems with things running in an unexpected order. But, in today's connected world, concurrency of some sort is usually a requirement. Each additional point where concurrency can happen introduces a bit of cognitive overhead, another place you need to think about what might happen, so as a codebase adds more of them it becomes more difficult to understand them all, and it becomes more challenging to understand subtle nuances of parallel execution.

So, at the simplest end of the spectrum, you have callback-based concurrency. Every time you have to proceed to the next step of a concurrent operation, you have to create a new function and new scope, and pass it to the operation so that the appropriate function will be called when the operation completes. This is very explicit and reasonably straightforward to debug and test, but it can be tedious and overly verbose, especially in Python where you have to think up a new function name and argument list for every step. The extra lines for the function definition and return statement can be an impediment to quickly understanding the code's intentions, so what facilitates understanding of the concurrency model can inhibit understanding of the code's actual logical purpose, depending on how much concurrent stuff it has to do. Twisted's Deferreds make this a bit easier than raw callback-passing without fundamentally changing the execution dynamic, so they're at this same level.

Then you have explicit concurrency, where every possible switch-point has to be labeled somehow. This is yield-based coroutines, or inlineCallbacks, in Twisted. This is more compact than using callbacks, but also more limiting. For example, you can only resume a generator once, whereas you can run a callback multiple times. However, for a logical flow of sequential concurrent steps, it reads very naturally, and is shorter, as it collapses out the 'def' and 'return' lines, and you have to think of at least two fewer names per step.

However, that very ease can be misleading. You might gloss over a 'result = yield ...' more easily than a 'def whatever(result): return result; something(whatever)'. Nevertheless, if you have 'yield's everywhere you might swap your stack, then when you have a concurrency bug, you can look at any given arbitrary chunk of code and know that you don't need any locks in it, as long as you can't see any yield statements. Where you do see yield statements, you know that you have some code that needs to be inspected.

To continue down that spectrum, a cooperatively multithreading program with implicit context switches makes every line with any function call on it (or any line which might be a function call, like any operator which can be overridden by a special method) a possible, but not likely culprit. Now when you have a concurrency bug you have to audit absolutely every line of code you've got, although you still have a few clues which will help you narrow it down and rule out certain areas of the code. For example, you can guess that it would be pathological for 'x = []; ...; x.append(y)' to context switch. (Although, given arbitrary introspection craziness, it is still possible, depending on what "..." is.) This is way more lines than you have to consider with yield, although with some discipline it can be kept manageable. However, experience has taught me that "with some discipline" is a code phrase for "almost never, on real-life programming projects".

All the way at the end of the spectrum of course you have preemptive multithreading, where every line of code is a mind-destroying death-trap hiding every possible concurrency peril you could imagine, and anything could happen at any time. When you encounter a concurrency bug you have to give up and just try to drink your sorrows away. Or just change random stuff in your 'settings.py' until it starts working, or something. I never really did get comfortable in that style. With some discipline, you can manage this problem by never manipulating shared state, and only transferring data via safe queueing mechanisms, but... there's that phrase again.

Some programming languages, like Erlang, support efficient preemptive processes with state isolation and built-in super-cheap super-fast queues to transfer immutable values. (Some other languages call these "threads" anyway, even though I would agree with Erlang's classification as "processes".) That's a different programming model entirely though, with its own advantages and challenges, which doesn't land neatly on this spectrum; if I'm talking about left and right here, Erlang and friends are somewhere above or below. I'm just describing Python and its ilk, where threads give you a big pile of shared, mutable state, and you are constantly tempted to splash said state all over your program.

Personally I like Twisted's style best; the thing that you yield is itself an object whose state can be inspected, and you can write callback-based or yield-based code as each specific context merits. My opinion on this has shifted over time, but currently I find that it's best to have a core which is written in the super-explicit callback-based approach with no coroutines at all, and then high-level application logic which wraps that core using yield-based coroutines (@inlineCallbacks, for Twisted fans).

I hope that in a future post, I may explain why, but that would take more words than I've got in me tonight.

Twisted Matrix Laboratories: December Sprint Report

2012-01-17T04:15:40+00:00

Twisted sprint? Twisted sprint! Here's the final Twisted sprint report of 2011, from our December 10th event at Smarterer in Boston.

David Sturgis:

was Sprint Host and Food Wrangler
discussed miscellaneous twisted.web feature

JP Calderone:

investigated 64 bit Windows 7 IPv6 problems
finished #5383: Provide a library for simple valued named constants
finished #5084: Accept IPv6 address literals (with embedded scope ids) in IReactorTCP.listenTCP
reviewed #3420: twisted.web.client persistent connections
reviewed #1902: compatibility work-around for commercial SSH 2.0.12 misbehaviours
reviewed #5400: Change UDP port to have an explicit state machine, and no FileDescriptor dependency
reviewed #3648: twisted.cred.credentials.UsernameHashedPassword doesn't hash password strings when checkPassword

This was the last sprint for JP as a Bostonian. We will miss you!

Itamar:

worked on #5427: Improve core documentation index page
reviewed #5383: Provide a library for simple valued named constants

Alex Levy:

made headway on several website and documentation improvements

Glyph:

discussed and then worked on #1956: Make a less sucky producer/consumer API

I (Jessica McKellar):

reviewed #5427: Improve core documentation index page
reviewed #5429: Documentation index
reviewed #5422: pbgtk2.py example is excessively complex

Thank you David for organizing this, and Smarterer for hosting.

Thank you to everyone who closed out 2011 with contributions to Twisted!

Jack Moffitt: The Potentially Dark Future of Search

2012-01-12T10:12:00+00:00

Twitter sees Google's latest Google+ feature, integration into Google search, as anti-competitive, and it probably is. However, it brings to the surface some real issues with the future of search and of data.

Twitter's argument:

We're concerned that as a result of Google's changes, finding this information will be much harder for everyone. We think that's bad for people, publishers, news organizations and Twitter users.

Google's response was:

We are a bit surprised by Twitter's comments about Search plus Your World, because they chose not to renew their agreement with us last summer (http://goo.gl/chKwi), and since then we have observed their rel=nofollow instructions.

People have been digging into the semantics of nofollow (see Danny Sullivan and Luigi Montanez), but there is a much bigger issue.

Google and other established and up-and-coming search engines have no real way to include lots of data in their index. It's easy to imagine that the lack of access to Twitter and Facebook data was a motivator for Google+ in the first place.

Lots of sites now generate enough data that it is unrealistic to crawl them. For example, Youtube has more new content every day than they allow anyone to crawl. Twitter is essentially the same. This means there is no way to index this data without special arrangements with the provider. Twitter has closely guarded their firehose of data, but at least they have some mechanism to obtain it. Youtube, as far as I am aware, has no such mechanism.

My team and I ran into this problem head on trying to build Collecta, a real-time search engine. Access to the data was a primary blocker for many features and product ideas, and over the too short life of that company, access became significantly more difficult, not easier.

Google can build an effective search, even a real-time one, for Youtube, but no one else can. Twitter can build search for their data, but few others can, and their data access policies can and do change on a whim.

If Google believes that microblogging data will improve their search product, then a reasonable strategy to obtain that data is to try and build their own microblogging service to generate it. I can't fault Google for trying. If I thought Collecta could have effectively competed against Twitter for their audience, I would certainly have attempted that as well.

Google, Twitter, Facebook and others are hoarding silos of otherwise public data. Not only is this artificially limiting the features of their products, but it squashes the potential for new and exciting search applications. The search services that have sprung up are limited to your own data, aggregate results from service-specific search APIs, exist at the mercy of data providers, or make do with a tiny subset of the data. I don't think Google could have built their own search engine if the Web were similarly hostile.

One could argue for requiring these bits of data to be openly available, but unlike the data of the past, this data is expensive to publish and consume. Most of these services may not even have a mechanism to publish the data, even internally. Simply receiving the Youtube or Twitter firehoses (and not counting video or image media) would require significant engineering effort, and the rate of data generation is only accelerating.

I think we must push for open access to data, even if it is costly. These data wars benefit very few. If things don't change, the future of search is dark.

Jp Calderone: Learn About Twisted at PyCon 2012

2012-01-10T18:21:50+00:00

At PyCon this year I'll be presenting a tutorial to introduce Python programmers to Twisted. This tutorial has two goals. First, to give attendees a firm grasp of Twisted's concurrency model, both in the abstract and the concrete. Second, to remove the mystery around the tools Twisted provides for developing robust, testable concurrent applications. If you attend, you'll come away with an understanding of how event loops work and how to write code that works best in Twisted's event loop.

I am a long time core Twisted developer with real world experience building maintainable, scalable systems with Twisted. I've also presented similar introductory Twisted tutorials several times in the past, letting me learn the common sticking points and teaching approaches to help overcome them.

Check out the tutorial's page on the PyCon 2012 website for details about what will be covered. Come learn how to leverage Twisted and Twisted-based libraries to their fullest extent!

Moshe Zadka: Retroactive New Year’s Resolutions

2012-01-05T10:17:04+00:00

As is my tradition, I will be posting my New Year’s resolutions from last year, retroactively decided:

Get into a serious relationship

Get engaged

Move to a team at work where I fit in better

Move to a nicer apartment

Get a loan (to build up my credit history)

Go to a Less Wrong meet-up

Get a talk accepted at PyCon

Get more recommendations on my Linked In profile

Write Rationalist Fan Fiction

Contact my Congressperson

Thomas Vander Stichele: How do you manage mailing lists?

2012-01-02T16:00:57+00:00

Every new year is a time of cleaning. After getting back to Inbox 0, my next target is my mailing list subscriptions.

It must be something psychological, but I cannot bring myself to unsubscribe from some of these mailing lists. I don’t check on them daily, but once in a while it’s darn useful to search through my local copy of mails on, say, selinux, and find solutions for a problem I’m having.

However, all this mailing list mail brings me a lot of headache. My email client is slow, and I would want it to be fast for the real mail I’m getting (from actual people, needing actual work). It’s hard to track the mails that matter – all my list mail gets put into folders automatically with some procmail magic, but it also means that some of the things I should be paying more attention to are just another bold folder in Evolution somewhere down the mail tree. And lastly, the server where I host my mail shared with friends gets too much traffic, and syncing 3 different evolutions over IMAP with it is a big part of the burden.

I vastly prefered the newsreader model of old, and I think the de facto standard of mailing lists really is a mistake. But I’m not sure what to replace it with.

What I want:

have selected mailing list archives be available on my machines, locally
have them synced/updated automatically
have them out of the way of my normal mail usage unless when I need them

I’ve been considering getting a separate email account just for email lists for this purpose, although I don’t look forward much to having to change all my subscriptions, and would first like to hear from other people how this approach works out for them.

There used to be a push towards web-based mailing list subscriptions, but I don’t know if anyone is really seriously using that, and I would like to have the option of reading these mailing list archives offline.

How do you separate your ‘real’ mail from your mailing list mail? How do you handle them?

Thomas Vander Stichele: using xargs on a list of paths with spaces in a file

2011-12-30T18:18:45+00:00

Every few weeks I have to spend an hour figuring out exactly the same non-googleable thing I’ve already needed to figure out. So this time it’s going on my blog.

The problem is simple: given an input file listing paths, one per line, which probably contain spaces – how do I run a shell command that converts each line to a single shell argument ?

Today, my particular case was a file /tmp/dirs on my NAS which lists all directories in one of my dirvish vaults that contains files bigger than a GB. For some reason not everything is properly hardlinked, but running hardlink on the vault blows up because there are so many files in there.

Let’s see if wordpress manages to not mangle the following shell line.

perl -p -e 's@\n@\000@g' /tmp/dirs | xargs -0 /root/hardlink.py -f -p -t -c --dry-run

Thomas Vander Stichele: N900 life support

2011-12-29T09:35:44+00:00

I don’t want to simply be complaining about Nokia’s sad Linux story ending.

It’s obvious that things aren’t going to get better though for Maemo device owners.

Here are two things that caused me trouble over the last few months, and the fix that did it for me, in case you were suffering from the same problems.

Turns out that I simply had to re-authenticate the Sharing Account in Settings. No idea why – maybe Facebook changed some authentication system in the last few months ? Now it works again.

(As a side note, this plugin’s behaviour is really annoying when it comes to uploading photos. It will always try to upload as soon as you connect to a network, although usually all you get is access to some web page on which you have to authenticate, usually by paying, to get on to the net. The sharing plugin already blasts photos at facebook, then fails, gives you a non-useful error message, and then sits there forever without any option to retry. All you can do is cancel the transfer, in which case you will have to re-upload the photos from your library. After some time I figured out that a reboot caused it to retry all pending uploads on the next network connect, and then after that I figured out a kill of a sharing manager process did the same thing. But really, Nokia engineers – a simple ‘retry’ button was too hard ?)

I stumbled across this page and simply changed the AGPS server to google’s. Worked like a charm on the next connect. Nokia, I don’t know what you did to your AGPS server – surely other phones you have are using it too, not just the Maemo ones ?

I swore I wasn’t going to buy an N9 because there’s no point in buying an EOL’d phone if I plan to develop for it. The reviews when it actually came out almost persuaded me to get it, and the five minutes I got to play with Luis de Bethencourt’s phone got me really close. I’ve even seen plans in Belgium offering this phone! But really, there isn’t much point if Nokia isn’t going to support this phone any more and services are just going to get worse, and important parts of the stack remain closed and thus unfixable down the road.

This month’s Android course at work at least got me familiar developing for the phone and I was actually impressed by Eclipse this time around, and while Java still seems like a bitch to program in, the whole emulator setup is easy to use… Who knows, my next phone may in fact be an Android.

In the meantime, it’s nice to see that some of my fixes go noticed. That motivates me to possibly fix that other annoyance in erminig-ng – all-day events triggering an alarm at midnight and waking me up :)

Jonathan Lange: What are my projects?

2011-12-26T04:54:40+00:00

Launchpad doesn't really have any good mechanism for letting you review a list of "your" projects.

That's partly because there are a lot of different ways that it could do it. You could be the maintainer of a project, or its driver, or its bug supervisor, or you might have commit access to its trunk branch. All of this could be direct, or through membership of a team. It's tough.

Since it's that time of year when I review my projects, responsibilities, goals and the like and start to figure out what I want to do next year, I want to figure out what are my projects on Launchpad.

Luckily, Launchpad has an API. I can't use it to figure out what projects I'm the maintainer of, but I can use it to figure out what trunk branches I have commit access to. The link has the Python code for the script.

Really, Launchpad should allow me to curate my own list of projects, with input to that list coming from all of the sources mentioned above, as well as arbitrarily selecting projects.

Jp Calderone: May - December Reading List

2011-12-22T21:03:03+00:00

The History of the Peloponnesian War. Thucydides. (Books 2 - 8)
Root Cellaring: Natural Cold Storage of Fruits & Vegetables. Mike and Nancy Bubel.
The Fatal Shore: The Epic of Australia's Founding. Robert Hughes.
Special Topics in Calamity Physics. Marisha Pessl.
The Dirty Life: A Memoir of Farming, Food, and Love. Kristin Kimball.
The Worst Hard Time: The Untold Story of Those Who Survived the Great American Dust Bowl. Timothy Egan.
The Children of the Sky (Zones of Thought). Vernor Vinge.
The Gathering Storm (Wheel of Time). Robert Jordan and Brandon Sanderson.
Oresteia: Agamemnon, The Libation Bearers, and The Eumenides. Eschylus.
Towers of Midnight (The Wheel of Time). Robert Jordan and Brandon Sanderson.
The Clouds. Aristophanes.
Saturn's Children. Charles Stross.
The Fuller Memorandum (A Laundry Files Novel). Charles Stross.
Scratch Monkey. Charles Stross.

Jp Calderone: August - April Reading List

2011-12-22T20:43:46+00:00

Been a while since I let you all know what I've been reading:

Containment. Christian Cantrell.
Human Legacy Project. Christian Cantrell.
The Long Emergency. James Howard Kunstler.
Sethra Lavode. Steven Brust.
Star Soldier. Vaughn Heppner.
BIO-WEAPON. Vaughn Heppner.
Dzur. Steven Brust.
Fragile and Distant Suns: A Poul Anderson Collection. Poul Anderson.
Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software. Scott Rosenberg.
Spell Games. T. A. Pratt.
Blood Engines. T. A. Pratt.
Poison Sleep. T. A. Pratt.
Dead Reign. T. A. Pratt.
Bone Shop. T. A. Pratt.
Makers. Cory Doctorow.
Microcosm: E. coli and the New Science of Life. Carl Zimmer.
The Theory of Almost Everything. Robert Oerter.
Broken Mirrors. T. A. Pratt.
The Warrior's Apprentice. Lois McMaster Bujold
The Mountains of Mourning. Lois McMaster Bujold
The history of Herodotus. Herodotus. (except books 2 and 3)
Surely You're Joking, Mister Feynman! Richard P. Feynman.
The Name of the Wind. Patrick Rothfuss.
The Wise Man's Fear. Patrick Rothfuss.
Reluctant Swordsman. Dave Duncan.
The Golden Age of Science Fiction. Various
History of the Peloponnesian War. Thucydides. (Book 1)

Thomas Vander Stichele: ssh friction

2011-12-22T11:57:21+00:00

I haven’t been too good this year at removing friction from my workflow. Today I wanted to change that. And the random friction thrown my way today has to do with ssh.

You see, somewhere along the line I read that it is a good idea to create separate keys for separate identities. So I have an identity for all work-related stuff (which I consider ‘ring 1′: it’s unlikely to change but everyone can get fired or change jobs), one for personal stuff on machines I actually control (‘ring 0′: they’d have to pry it out of my dead hands), another for my ‘public online default’ identity (‘ring 2′: I can always pull a whytheluckystiff and pull myself of the net and reinvent myself), and then per-project identities (‘ring 3′: I may lose interest in being a fedora or gstreamer contributor without massive changes in my personality).

I started splitting ring 3 per project when it made sense – for example, Fedora recently enforced a key change even if your account wasn’t compromised and even if you already have a strong passphrase on your key (like I had), and of course a massive flamefest ensued. I shrugged and decided to split off a new key and set that on all my machines.

But the problem is, this whole tower of ssh doesn’t really work well in practice. I chose a long passphrase for the new fedora keys, but obviously I do not want to type that every time I clone a package or commit changes. So I use ssh-agent. In theory, ssh-agent adds your keys and asks you for the passphrase once, and is then able to offer those identities to the other side.

The problem is a lot of ssh servers out there only give you a few tries. So your ssh agent will offer identity by identity until it gets refused. If my fedora identity was added as the fourth identity I lose – I can’t clone a package.

Specifying IdentityFile in the ssh config is useless. It is poorly documented, but IdentityFile files actually come after your ssh-agent identities. So your agent blasts all the wrong keys at the host first, and you get denied.

So you can specify IdentityOnly to make sure that only the identity file you want is being used. Sadly in that case it will not use the ssh-agent at all, so it will ask you for the password to your key file – the whole reason you want agents to be used in the first place.

Now obviously ssh has all the pieces it needs to Do The Right Thing. If my config says to use this identity and this identity only, ssh should be able to request ssh-agent to present that identity, and that identity only, and make the login happen without any password.

Surely I must be missing something obvious. Surely one of you uberhackers out there has set up the same thing as me. Why don’t you comment about it here and help the rest of us?

Stephen Thorne: Explaining For/Else In Python

2011-12-19T14:06:27+00:00

In response to a thread on G+ I've decided to write a this article on 'how for/else works'.

I should start by saying that this isn't intuitive or simple. It's quite complicated, but it has a very clear use case. Let me start by stating as simply as I can the use-case.

for element in seq:
    if somecondition(element):
         dosomething(element) 
         break
else: 
     dosomething(defaultinput)

I think that example is reasonably clear, but it's important to explain what's happening here.

If a 'for' loop does not break, it's else is run, otherwise the else is not run. So:

The else suite is run if:
- the sequence is empty
- break is never executed
The else suite is bypassed if:
- break is executed
- a return statement in the loop is executed
- an exception raises in the for loop

I hope that's clear. Any comments or addendums or errors I've made, please let me know in the comments.

Edit: I don't think using an 'else:' on a for loop is a bad idea. But I think it should only be used in exactly the idiom presented in the example, anything with more complicated flow control or more than 8 lines should be re-factored to use a function call with 'return' in the appropriate places.

Glyph Lefkowitz: I'm Sorry It's Come To This

2011-12-15T10:20:01+00:00

If you want to be a great leader,

you must learn to follow the Tao.

Stop trying to control.

Let go of fixed plans and concepts,

and the world will govern itself.

- Tao Te Ching ch. 59,

as translated by S. Mitchell

I usually try not to get too political in my public persona – on blogs, twitter, IRC, mailing lists et cetera – and that's a conscious choice.

I work on open source software. I have for the last ten years. I am lucky enough to have founded a project of my own, but in open source, leaders are more beholden to their followers than vice versa. I depend on people showing up to effectively work for me, for free, on a regular basis. So, I try to avoid politics not because I don't have strong convictions (anyone who knows me personally can tell you that I certainly do) but because I don't want someone to avoid showing up and helping do some good in the world in one area, just because we might disagree in another.

This is a benefit of living in a free and democratic society: we have ways to dispute issues that we have strong feelings about, so we can cooperate on some things without having to agree on everything. It's rarely perfect but we can usually get some good stuff done, with rough consensus and running code.

Today though, there's a political issue which I can't ignore. The purpose of Twisted (the open source project which I founded) is to facilitate the transfer of information across the Internet. A new law, SOPA, is threatening to radically alter the legal infrastructure of the Internet in the United States, granting sweeping new powers to copyright cartels and fundamentally restricting the legal right to transfer any information, and to build tools that transfer it. Twisted is designed to make it easy to implement new protocols, to easily experiment with improvements to systems like the Domain Name System. SOPA might well make those potential improvements, and with only a little paranoid fantasizing, Twisted itself, illegal.

It's my view that this law is a blatantly unconstitutional restriction on free speech. It will kill job creation, at a time when our nation can scarce afford another blow to its economy. It will create the infrastructure to suppress political dissent, similar to the infrastructure in China and Syria, at a time when our corrupt political system needs dissent more than ever. It is the wrong thing at the wrong time.

This bill is being discussed in the house today. If you're in the US, call your representative right now.

(As always, I don't speak for anyone but myself; no one else has reviewed or endorsed these remarks.)

Twisted Matrix Laboratories: November Sprint Report

2011-12-11T00:11:44+00:00

November saw another successful Sprint at the Smarterer.com offices, thanks to a smaller but extremely busy group of developers:

Reviewmeister(ess) Jessica McKellar secured second place in the Twisted High Score list with an impressive number of reviews, including:

#3037: "loseWriteConnection breaks loseConnection" (now resolved)
#4262: "Multicast documentation is misleading" (now resolved)
#538: "Website-template.tpl is out of date"

She also resolved #1247: ("kqreactor timeout error (traceback)"), and opened #5385, #5386, and #5387, while resolving #5060: ("Correct python requirement in twisted/topfiles/setup.py").

Not to mention all the tickets reviewed at recent Sprints which have since been resolved! (Though, to mention them anyway, #1247, #2115, #2447, #2498, #2507, and #5370)

Eternal High Score Overlord JP Calderone made a fair showing as well, finishing #5371 ("In the developer guide, forbid new global reactor unit tests"), and splitting #4671 ("sometimes I want to have some constant values that are part of a set") into more managable tickets:

#5382: "Provide a library for valueless named constants"
#5383: "Provide a library for simple valued named constants"
#5384: "Provide a library for bitvector-like valued named constants"

Of which #5382 and #5383 saw immediate progress.

Returning Champion Sprinter Paul Swartz reviewed and otherwise worked on several tickets, including:

#2997: "Conch and getPeer"
#5352: "Do not register overlapping old DH key exchange messages"
#1902: "Compatibility work-around for commercial SSH 2.0.12 misbehaviours"
#3242: "Use python 2.5 'spwd' module instead of z3p secret 'shadow' module when available"

One-word Twisted celebrity Glyph was also on hand to assist with the reviews of #3242, #5382, and #1918

With considerably more words in his name, Itamar Turner-Trauring started porting a Nevow branch with IRI support (internationalized URLs) to Twisted as per ticket #5388, as well as helping resolve #3037 with Jessica.

Finally, Glyph, Itamar, and Dave Sturgis conferred on #288, and the related cluster of web-resource-model issues it involves; It is hoped that this will eventually result in a new Web Server URL Traversal API, or WSURLTA, not to be confused with the nordic kraken-headed demigod of lore.

Moshe Zadka: Contacting my congressperson

2011-12-09T18:49:59+00:00

I have decided to check which congressperson is responsible for my district, and send her a personal message urging her to vote against SOPA. Here is the text I used:

Dear Congresswoman Speier,

I am writing to you regarding the so-called “SOPA”, or “Stop Piracy Online Act” which will be up for debate and a vote soon (possibly as soon as December 15th). I am concerned about the implications of the bill for my rights, as well as those of others in the United States.

Let me prefix this by saying that the bill will do nothing to “Stop Online Piracy”. Technologies like Freenet[1], Tor[2] or VPN[3] already exist for dedicated pirates to work around any technical or legal hurdles the US government might throw in their path.

However, what the bill *will* do is to create a US equivalent of the so-called “Great Firewall of China” — a centralized place for the government to stop any communications it does not want. It wouldn’t work against pirates, of course, which would find any small hole in the implementation and smuggle everything through that (just like they do in China — a police state!). But it will mean that new forms of art or technology can be stopped by a unilateral act of the government.

I, like many of your constituents, work in the hi-tech industry. I depend on free communication, and unfettered access to information, for my livelihood. If you put obstacles in front of Silicon Valley’s business, competitors outside the US will out-innovate, and eventually out-compete us.

This bill does not just threaten my rights to free expression, and my access to new forms of communication. This bill threatens my ability to make money, all in the name of protecting out-moded business models.

Last but not least, you will notice that the amounts of television, music and movies available has not decreased with the internet — even if we just limit our search to mass-media. If you factor in over 50 hours a minute of video uploaded to YouTube, you will see that this bill is not necessary in order to “To promote the Progress of Science and useful Arts”[Article I, Section 8, Clause 8, United States Constitution]

I thank you in advance for your consideration, and hope that you will choose to vote according to the interests of your constituents — against the SOPA bill.

Yours,
Moshe Z.

[1]http://freenetproject.org/
[2]https://www.torproject.org/
[3]https://www.ipredator.se/

Duncan McGreggor: OpenStack at DreamHost

2011-12-07T02:49:46+00:00

So I guess this is old news now, but DreamHost is really into OpenStack :-)

(In fact, during recess, DreamHost asked if I would pass a note to OpenStack. I didn't look inside the note, but we can all guess what it said...)

I was hired specifically to work on cloud stuff here at DreamHost, and we've got a new team that's super-excited about this -- they're starting to gear up for increased contributions and community engagement, gettin' themselves some cloud. We've now got our own Launchpad team, we're working on a handful of blueprints, chatting it up on mail lists and IRC meetings -- you get the picture :-) Exciting times.

For official blog posts and other news items that highlight DreamHost's interest and involvement in OpenStack, check these out:

I'll be writing more about our OpenStack work later, but wanted to get a quick cloud-shout-out done before too much time passed...

P.S. We're hiring Python rock-stars!

P.P.S. Did I mention that DH is an AWESOME place to work? We got an award for that, two years in a row :-)

Duncan McGreggor: Occupy's Declaration of Independence

2011-11-21T22:08:49+00:00

Illustration by Peter Whitley

There's one thing I would really, really love to see under the tree this year -- under everyone's tree: Occupy the World.

For the first time in history, it seems that there might be enough momentum, enough communication, enough strength of individual convictions, and enough mass support to be able to have a world-wide, non-violent, revolution.

Taking the US as an example in this beautiful hope: imagine 2 or 3 million people showing up on the lawns of the US law-making machinery in Washington, D.C., issuing their declaration of independence and simply stating a fact: "Things are now going to change, we will not leave until we have the government that we want."

Far from mob rule, the 99 is intelligent, lucid, and a collection of THE overwhelming majority... regardless of old political parties. An enormous amount of discussion, insightful inspection, exploration of alternatives has been researched, written about, and promoted over the past 5 to 10 years, co-culminating in what we see around us today as the Occupy movement. I have the utmost faith in these thinkers (by which I mean everyone from Gar Alperovitz to my second cousins working in Detroit, MI automotive plants) and their (our!) ability to produce a new constitution that provides for the 99 fairly. Such a new system would stand in stark contrast to that of today's system: one that caters to policies driven by enormous, "legal" bribes or banking systems that continuously steal from the customers and shareholders, running off with the loot.

The 99 is saying it, and has been saying it for a while: "It's time for a change."

They've taken things further by showing an undeniable presence at the scenes of crimes (financial and governmental institutions). I say let's take the last logical step, and fix the problem: let's put a new system of government in place. Let's have radical, peaceful change. Let's do it with poise, grace, and while keeping the benefit of the entire planet foremost in our minds. Let's do it world-wide, and not stop until the job is done. Let's have a revolution.

Let's have the revolution.

Jonathan Lange: pyflakes update

2011-11-21T14:29:05+00:00

Thanks to radix, exarkun & dash, my branch to pyflakes to warn about duplicate definitions of classes finally landed. I did the work a year ago as an outrage-powered, opportunistic fix after I saw a co-worker struggle with tests weirdly not failing (Turned out it was a huge test module and there was another class at the bottom with the same name). I'm very happy to see it landed.

For those who haven't been paying attention, official pyflakes development is now taking place on Launchpad as part of the divmod.org project. The trunk of that project now has the best version of pyflakes known to man.

pyflakes is the best static Python checker. It's fast, and has very few false positives.

Update: A couple of people have asked me about lp:pyflakes. It's dead. It died when the divmorg.org trac instance died ages ago. Don't use it. To get pyflakes or any other divmod project, use lp:divmod.org.

Twisted Matrix Laboratories: Twisted 11.1.0 has been released!

2011-11-17T18:41:36+00:00

Twisted Matrix Laboratories is proud to announce the release of Twisted 11.1, thanks to the efforts of our release manager Thomas Hervé.

Highlights of the 185 tickets closed include:

The poll() reactor as default where applicable, instead of select() everywhere.
A new SSL implementation only relying on OpenSSL for cryptography, (not I/O) making it more robust.
Several improvements to the fresh HTTP/1.1 client implementation, including proxy and cookie support.
Thomas' personal favorite: a new howto has been published on test-driven development with Twisted.
A new abortConnection() method for immediately closing TCP and SSL connections, the oldest ticket closed by this release.

For more information, see the NEWS file. Note that this is the last release supporting Python 2.4 (the support on Windows stopped with 11.0).

Thanks to the supporters who have donated to Twisted via the Software Freedom Conservancy, and to the many contributors for this release! Download it now:

Thomas Vander Stichele: Fedora 16 upgrade

2011-11-13T13:03:28+00:00

A new Fedora, a new decision on which machines to upgrade. Usually I try to stagger the three machines I use most – my work desktop, my home desktop, and my laptop. I had updated work machine and laptop to F-15 when it came out, and kept my home desktop at F-14.

I actually have two or three root partitions on each of those machines, and I typically do a fresh install on a separate root, so I can try things, poke around, and make sure everything I will need works. When I do the install, I don’t mount my /home partition, because I don’t want to have the new version upgrade things for me on my user config.

I have a pretty long checklist by now that I go through on each install/upgrade, installing the packages I use a lot, setting up specific configuration, copy over ssh keys, …

I actually liked F-15 a lot, and though GNOME 3 has its issues (which I still want to document in a separate post), I overall enjoyed the experience. At home, I noticed myself using the windows key or moving my mouse to the top left corner expecting something to happen.

That is how you know you really are ready for GNOME3.

So I thought, what the heck, let’s get to upgrading all of them. I started with my laptop, as usual. That mostly went fine, except for hurdle number one. My laptop actually has /home encrypted. And I did not add it to my custom layout in anaconda. So, the system dropped me in a rescue shell after booting. It took me quite a while to figure out that I had to copy over /etc/crypttab from the old system. After that, things worked again.

Arguably, hurdle #1 may not be Fedora’s fault. Maybe normal users don’t encrypt home drives, or use custom partitioning like I do (although on a few fedora upgrades this saved my bacon when it turned out certain things I needed didn’t work in the new Fedora, like VMWare)

And yes, GNOME 3.2 is a slight improvement. Enough to make a difference at least. All the usual applications seem to work, so I can now mount my old /home directory.

That’s when I ran into hurdle number 2: the default uid/gid numbering change. My thomas user now was 1000:1000 as opposed to 500:500 on all my machines before Fedora 16.

In this day and age, I still have to shell it up to fix things like that:
find / -uid 500 -exec chown 1000 {} \; find / -gid 500 -exec chown :1000 {} \;
If I had less shame I’d tell you how embarassing it is if you do this for a few users on your system, and start thinking “let’s put this in a for loop”, and because it’s already 1 AM you start doing things like
for a in 0 1 3; do find / -uid 50$a -exec chown 10$a {} \; ; find / -gid 50$a -exec chown 100$a {} \; done
Note how I got the number of 0′s wrong in the first find, and how I actually forgot the : in the second. You can imagine how amusing it is to fix the effect of those commands.

But I’m a shameful person so I won’t tell you about this bit. Instead, suffice it to say that this took a long time.

Ok, so now /home is mounted on the laptop, and for the most part things worked fine.

On comes the weekend, so I turn to the home machine. I tend to keep the work machine for last, because I don’t want to spend work time on fixing distro problems. And I usually take a whole weekend to upgrade at home. The home machine turned out to be more of a problem. I ran headlong into hurdle number three. You see, there is this new thing called GPT for your partition table, and it is now the default, and it means that fdisk will no longer work, and now you should use gdisk (which sadly is not installed on the rescue bit of the install DVD, boo!), and this is all so we can have grub2, which is supposed to be better or something.

I’m sure one day I will be thankful. But on my home machine, I didn’t know any of this, and just had anaconda tell me something about the boot image being too large and there was no space for it and my system may not boot. (I am not sure why I did not run into this problem on my laptop – presumably, looking at the disk layout now, because I kept the original install, which includes Windows, and just shrunk that and added linux – so it’s probably the windows thing doing the booting). And sure enough, the Fedora 16 install did not boot. It dropped me into my friend, the shell.

So here’s the thing. This new way of doing things needs more space than your average MBR, so you actually need to create a primary partition for this, and it needs to be in the first 2 TiB. So you know what time it is now. It’s resize-o-clock time – I get to learn the joys and mysteries of shrinking ext4-on-software-raid so I can make space for this new partition, which doesn’t need to be big, apparently 5 MB is more than enough. Aren’t I happy now that I stubbornly stuck to having a /boot partition as the primary one on my machines, so I can just shrink that a little?

So shrinking an ext partition I already had down pat. I learnt about shrinking software raid partitions, and again I got into the land of not understanding which of the many types of numbers (sectors ? blocks ? bytes ? cilinders ? Mebi vs Mega ?) are understood the same way by the tools, or not understanding how much of those numbers you need to count extra because of the layer of indirection being added (encryption on logical volume on LVM on software RAID anyone ?). So to be safe I end up shrinking 10% on each layer of the onion as I go deeper – then let the tools handle growing to the maximum space again, since that’s the one thing they’re usually decent at.

But you know, if I’ve done all this, I want to get it right. I don’t want a stinking BIOS boot partition sitting after my /boot partitions. That’s not how F16 sets it up by default. But I have never actually moved a partition. So, download gparted, look at it, figure out how it can let me do that, make sure I ask it to count by cylinders so it doesn’t leave gaps, be puzzled at why it doesn’t let me fractions for MiB sizes of partitions, and work around it in some other way. And so I finally have those two software raid /boot-wearing partitions where I want them – sitting right behind this new BIOS partition.

I create a new partition in fdisk (which is what I’m used to), but I can’t actually set the partition type to EF02, which has four characters where I expect two. But really that is what BIOS BOOT should be.

And now the internet tells me I need to set some flag on it using a tool called parted – some flag called bios_grub. Except when I type that magical command that sets the flag, it tells me it can’t exist:
[root@otto ~]# parted /dev/sda set 6 bios_grub parted: invalid token: bios_grub Flag to Invert?

Isn’t this tool nicely written for only the writer of the tool instead of for human beings? Of course I don’t know this when it barfs this at me, but at the end of this story I figured a bunch of things out that this tool could have told me.

You see, invalid token just means that it doesn’t accept the flag named bios_grub. I know this because I’m a programmer so I know the programmer used a token parser – a thing normal people shouldn’t have to know about. What’s that you’re asking? Flag to Invert? How about the Belgian flag, I would quite like to see the colors go in the opposite direction. No, that’s a prompt to choose a different flag to invert than bios_grub. Apparently bios_grub is a flag, not a setting, and I’m trying to invert it, instead of setting it. Can you tell me what flags you do know about, dear parted ?

(parted) help set set NUMBER FLAG STATE change the FLAG on partition NUMBER

NUMBER is the partition number used by Linux. On MS-DOS disk labels,
the primary partitions number from 1 to 4, logical partitions from 5
onwards.
FLAG is one of: boot, root, swap, hidden, raid, lvm, lba, hp-service,
palo, prep, msftres, bios_grub, atvrecv, diag, legacy_boot
STATE is one of: on, off

Wait, what ? You do know about bios_grub ? But you don’t let me set it ?

I seriously spent 30 minutes on trying to figure that one out.

In the end, it’s because a) I should run gdisk b) parted won’t let you set that flag on a normal MBR drive c) gdisk should convert to using GPT and d) the messages gdisk prints by default are SUPER scary and the docs say that this is intentional to keep away stupid Windows users (I am not making this shit up). Well, that’s why I use software RAID, isn’t it ? How about we take our chances, dive in deep, and let this gdisk thing do the conversion to GPT on the first disk. Gulp.

OK, I got lucky. That actually worked. I can now create this partition, with the proper flag set. While I’m at it, why don’t we try this ‘sort partitions’ option in gdisk so that this new partition, which is now at the start, but listed as number 4 out of 4, shows up as number 1. Sure, it will renumber all other partitions, but let’s just hope that most things use UUID’s and labels and what not by now, and if not I should be able to figure things out.

In what feels like Day 5 in a two-day weekend, the system now boots! I actually see a new grub (wait, why is that text-mode only again ? Fedora guys, you spent years to make everything look graphical, because that was some huge important feature that mostly got in my way when it took longer than it was supposed to and I had no way to see why except reboot and remove quiet and rhgb from the options) and now you suddenly let grub2 take that back from you? Show us some spine, please), and the system shows me plymouth again. Until it doesn’t anymore, and drops me into a terminal screen.

Hurdle number four. Can you guess what it is ? Go on, take a stab. If you’ve updated your system, I’m sure you know the answer. I’ll give you some whitespace to think about it…

SELinux. Riding in to relabel my file system to save it from the evil people out there. And sure, it warns me. This may take a long time. And then it proceeds to throw asterisks in my face. Lots of asterisks. It’s not the first time this happens. But every time it does, I cannot help but wonder one thing.

Who thought it was a great idea to throw asterisks at the user? How many asterisks am I supposed to expect? Never mind that you can’t actually count them unless you glue your eyeball at the screen, because there are so many they actually scroll off at the top. You know, if you squint hard enough, you can see the maniacally laughing face of the programmer who thought this was a nice way of showing progress. Never mind that tools like fsck can show a progress bar that actually means something (if you trick it into sending data to file descriptor 0) in a sensible way – one line on the console, and visible progress towards an end goal of 100%.

If only I could guess what a long time is going to end up being. Is it a ‘get a drink’ amount of time? Or ‘watch some dexter’. Or nookie time? Or, get the hell out of the house and do all the shopping for the next three hours because there’s no way you’ll be doing anything useful with this system for that long?

So I do all of those things, twice, and one even four times times (I won’t tell you which but I ended up having to pee a lot), and I come back, and the system has rebooted, and there’s actually a GUI asking me to log in.

You know, this Fedora 16 better be frigging spectacular after this six day weekend.

I log in, follow my standard upgrade checklist, try out some of my tools. Media keys don’t seem to work as before for my prototype music player (it flashes a nasty forbidden sign at me), and even though I set up to have nothing happen on inserting audio CD’s (because my LEGO robot is inserting CD’s into an external drive about fifty times a day), Rhythmbox craps on and FORCES me to select which of the many CD’s with exactly the same name that audio CD might be. So, par for the course so far.

Maybe a reboot will fix that, it may not know about those settings until I have everything installed and upgraded. And if I reboot, I’d better convert my second drive to GPT and fix my /boot and set that flag and all that. So I do. And for some reason I can’t figure out how to tell software raid that sda2 and sdb2 (which are both still perfectly mountable as ext file systems and were part of the previous RAID-1 /boot array before I resized them) really are a software raid. So there’s this point where I’ve wasted more time on trying that then it would have taken me to actually manually type every byte on that /boot partition, and I just give up and recreate a software raid on those two partitions and copy stuff over.

And then I reboot. And won’t you know it. Effing goddamn selinux relabel all over again. In fact, this way too long entry was typed completely in less than half the time selinux took to complete some work it had already done an hour ago.

I better have a working system after this last relabel finishes. Now excuse me while I go make some comfort food, potatoes and beans and runny eggs with butter sauce. I’m going to eat it while my good friend Dexter comes back from a long holiday. It’s the only thing that is going to get me out of this weekend funk. And you know who I will be thinking about every time my friend Dexter tells me of a problem he solved…

Jonathan Lange: Automatic packaging update

2011-11-10T12:40:33+00:00

I just blogged on developer.ubuntu.com about the work we're doing with automatic packaging. Read it!

Thomas Vander Stichele: Package repository updated for Fedora 16

2011-11-10T08:38:57+00:00

Following up on last night’s post, I updated my package repositories with all packages I currently work on before sending them off to Fedora.

The repository is updated with chromaprint, gstreamer-chromaprint, longomatch, and some of my own projects: moap, mach, morituri, and savon.

Itamar Shtull-Trauring: The Euro crisis, in musical form

2011-11-10T04:03:14+00:00

DOOM.

Thomas Vander Stichele: Mach 1.0.0 “Madera” released

2011-11-09T22:56:01+00:00

Another November, another Fedora. 16 came out, so it was time to update mach again.

And today I thought, is there any reason mach isn’t 1.0 yet ? Am I going to do anything more to this piece of code before I want to call it that ?

And the answer is, no. It’s the first Python application I’ve written, and I’m not particularly proud of the code, but I’m happy I’ve made good use of it for so long, and that it helped push packaging approaches forward and sparked ideas for the Fedora build system.

Since I didn’t like the original code for mach2 (there was a version 1 which was Makefile-based), I started a rewrite with unit tests, better code layout, decent classes for abstracting distro-specific stuff, and so on.

The experience of how mock was created based off mach2 was a slightly sour one however, so I wasn’t really motivated to finish the mach3 rewrite. Sometimes that’s the drawback of open source – sure, forking is specifically allowed, so don’t whine about it when it happens. But when it’s done gratuitously, with no serious attempt at collaborating, it doesn’t feel like it’s in the spirit of open source.

Anyway, that was a long time ago. mach2 as it is today, is done. It really only needs updating for newer versions. As long as it works for me, it’s unlikely I will continue mach3, but who knows?

Enjoy the release!

Glyph Lefkowitz: Blocking vs. Running

2011-11-05T01:16:26+00:00

I've heard tell of some confusion lately around what the term "non-blocking" means. This isn't the first time I've tried to explain it, and it certainly won't be the last, but blogging is easier than the job Sisyphus got, so I can't complain.

A thread is blocking when it is performing an input or output operation that may take an unknown amount of time. Crucially, a blocking thread is doing no useful work. It is stuck, consuming resources - in particular, its thread stack, and its process table entry. It is sucking up resources and getting nothing done. These are resources that one can most definitely run out of, and are in fact artificially limited on most operating systems, because if one has too many of them, the system bogs down and becomes unusable.

A thread may also be "stuck" doing some computationally intensive work; performing a complex computation, and sucking up CPU cycles. There is a very important distinction here, though. If that thread is burning up CPU, it is getting work done. It is computing. This is why we have computers: to compute things.

It is of course possible for a program to have a bug where a program goes into an infinite loop, or otherwise performs work on the CPU without actually getting anything useful to the user done, but if that's happening then the program is just buggy, or inefficient. But such a program is not blocking: it might be "thrashing" or "stuck" or "broken", but "blocking" means something more specific: that the program is sitting around, doing nothing, while it is waiting for some other thing to get work done, and not doing any of its own.

A program written in an event-driven style may be busy as long as it needs to be, but that does not mean it is blocking. Hence, event-driven and non-blocking are synonyms.

Furthermore, non-blocking doesn't necessarily mean single-process. Twisted is non-blocking, for example, but it has a sophisticated facility for starting, controlling and stopping other processes. Information about changes to those processes is represented as plain old events, making it reasonably easy to fold the results of computation in another process back into the main one.

If you need to perform a lengthy computation in an event-driven program, that does not mean you need to stop the world in order to do it. It doesn't mean that you need to give up on the relatively simple execution model of an event loop for a mess of threads, either. Just ask another process to do the work, and handle the result of that work as just another event.

Jonathan Lange: How to feel better (or, some tips on refactoring)

2011-11-04T18:29:19+00:00

A few months back I gave a lightning talk at the Launchpad Thunderdome about how I do refactoring. It's very opinionated, and mostly applies to big, old code bases, but worth writing up anyway.

The core idea here is that very few things make me feel as good as deleting code. I love cleaning up code and the clean code base that results, and I'm sure that many others feel like me. As such, this is a guide on how to feel better.

1. Know your enemy

“Functionality is an asset, code is a liability”. Truer words were never spoken. Every line of code is a potential source of bugs and a barrier to understanding, and thus carries a maintenance cost.

Maintain an awareness of things that need refactoring. Here's a quick and incomplete list:

unused code – this can be deleted
boilerplate – this should become a function or class
wrong documentation – these should be updated
two ways of doing something – perhaps there should be one
bad names – change them to something that makes you think less

2. Keep a "yak stack"

Yak shaving is "any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you're working on." (Jargon File)

A few of us have extended the concept beyond the "actually necessary" to include anything that's making the task at hand more difficult and less fun but is not worth fixing right now. Hence the yak stack. Here's how it works:

Whenever you come across something in your code base that is difficult to understand or that slows you down: make a note of it. When you've finished the task at hand, fix the problem. If you encounter other things that slow you down, write them down. Work through the list.

3. Every option is wrong

In a big, old code base, there are probably many, many areas that need refactoring. Don't worry about which is the "best" place to start – there is no best place.

4. Start from green, stay green

Never, ever refactor while your tests are failing. Refactoring is about changing the shape of code while preserving its behaviour. It's much harder to be sure you're keeping the behaviour if you are comparing one set of thirty tracebacks with another set of thirty tracebacks. Better to compare a passing ("green") test run with another passing test run.

Run tests frequently. More often then you think you should. Commit often – think of it like quick save in a tough level of a video game. It frees you up to experiment more and means you have less in your head at any one time.

5. Do not nest

Don't begin a refactoring while you are in the middle of another refactoring. If you find you must, use tools like 'bzr shelve' to store your current diff and then work from the clean head of your branch.

6. Keep moving, leave a trail

Don't get bogged down in details, otherwise you'll never finish. Literally. Someone will come along and distract you and before you know it, three months will pass and your refactoring branch will be full of conflicts. If you see something you are unsure of, mark it with a XXX or a FIXME or a TODO or whatever works for you and then continue with what you are doing.

Tools like 'bzr todo' can make it really easy to check to see if you've added any XXX comments.

7. Translate tests with care

As said above, refactoring is about changing the shape of code while preserving its behaviour. When you update tests, you risk changing your definition of the system's behaviour – so be careful.

8. Confront uncertainty with destruction

If you see some code and you are not sure if it's needed, delete it. Doesn't matter if it's a whole function or just an odd line. If you have a test suite, and it was important, that will catch the failure. If you have version control, and it was important, one of your collaborators will notice and revert the change.

If it was important and neither of these happened, then your whole project has learned something new about itself, and that's probably worth the hassle. (Oh, add tests & better docs after this happens.)

9. Good grep tools

Remember that symbols aren't only referenced from files that match *.py. In big code bases there are often other sorts of files that refer to symbols in the main code. In Launchpad, for example, we have ZCML and doctest files that refer to symbols. When you want to know how something is used or you want to rename something, make sure you use a grep command that actually finds everything.

Ideally, you should be able to run this command faster than you can think about wanting to do it.

Personally, I use 'bzr grep' a lot. Others recommend 'ack'.

10. There will be failures

Mentally prepare yourself for the fact that the first two or three full test runs after your refactoring will fail. This is especially important for code bases that have multi-hour test run times.

If you think this way, then you won't be as discouraged when it actually happens.

11. Finish the job

Busy people refactoring a big code base are often tempted to apply a refactoring to only a single part. For example, some useful function is extracted from repeated boilerplate in a few other functions. However, many, many other instances in the code base continue to use the repeated boilerplate.

This is almost worse than just leaving the repeated boilerplate. There are now two idiomatic ways of doing the activity. Further, other developers who work on other parts of the code base probably won't find out about it, and might end up repeating your refactoring. Ugh. How is anyone new expected to get to grips with this code?

Similarly, if a class, function or concept is renamed, rename it everywhere, especially in the documentation.

It's difficult and often tedious, but it really is worth taking refactorings to completion. Apply them to the whole code base, or not at all.

Note that I'm referring to completeness, not perfection. If you block on perfection, you never get anything useful done. If you aim for frequent incremental improvements, you soar.

12. Read these books

I highly recommend "Refactoring" by Martin Fowler and "TDD by Example" by Kent Beck. I stole many of these ideas from them.

Over to you

This was very much just a dump of how I do refactoring when hacking on Launchpad. I'm always keen to learn more, and would love to hear about what works for you.

Jp Calderone: Don't Use Buildbot EC2 Features

2011-10-31T17:15:32+00:00

I just noticed that Buildbot spun up one EC2 instance 31 days ago and another one 14 days ago and left them both running.

Stephen Thorne: Gmail Keyboard Shortcuts

2011-10-28T12:26:00+00:00

I use Gmail a lot now. I also have come to know and love a few keyboard shortcuts. I can't imagine dealing with a large volume of email without them.

You have to turn them on under the mail settings first.

u will refresh the current mail search or your inbox if you're looking at your inbox.

u will return you to your mail search, label or inbox if you're looking at an email.

j/k to navigate between emails in your inbox view.

x selects an email, and dumps you back at the message list if you were in an email.

s applies a star to selected emails, or stars the email you're reading.

+/- will flag/unflag an email as important (if you're using priority inbox)

e archives selected emails, or archive an email you're reading.

r replies to an email.

m will mute selected email threads so it won't reappear in your inbox when more posts are made.

gl lets you type the name of a label you want to view.

? brings up the help.

Most important button to me is u, as it refreshes my inbox when my android phone has chimed to let me know that I've got an important email, and the gmail UI hasn't shown it yet. The androids. They always know.

(Photo from Geek Sugar)

Stephen Thorne: Appengine Development

2011-10-28T12:25:39+00:00

I've been quite enjoying writing some code for Google Appengine this week. Just a small project for querying a variety of book retailer websites for information.

I'm really impressed with it so far. Having the ability to simply import and use things like memcache and use it is excellent. It works locally with some kind of cache, and then when I deploy my app it uses a memcache on the server side seamlessly.

I'm also liking that the urlfetch infrastructure built into appengine supports async downloading of webpages, so i can fire off 5+ API queries simultaneously and then collect the results, instead of doing them in sequence.

Here's a random bit of code I threw together that lets me cache the results of doing API queries in memcache so that a user doing reloads won't cause more API requests to occur than are strictly necessary:

from google.appengine.api import urlfetch
from google.appengine.api import memcache
CACHE_TIMEOUT = 3600 # 1 hour 
class APIError(Exception):
     pass
def cache_result(rpc, url):
    result = rpc.get_result()
    if result.status_code == 200:
        data = result.content
        memcache.add(url, data, CACHE_TIMEOUT)
def apiquery(url):
     data = memcache.get(url)
    if data is not None:
        decoded_data = json.loads(data)
        return lambda:decoded_data
    logging.debug("Cache miss for %r", url)
    rpc = urlfetch.create_rpc()
     rpc.callback = lambda: cache_result(rpc, url)
    urlfetch.make_fetch_call(rpc, url, method=urlfetch.GET)
    def answer():
        result = rpc.get_result()
        if result.status_code == 200:
            data = result.content
             return json.loads(data)
        raise APIError, "Broken!"
    return answer

With the above code I can now do something like:

 queries = [apiquery(url) for url in generate_api_queries()]
for query in queries:
    result = query()
    ... # do stuff with result

I don't really like the variable names I've used, but it's an interesting enough bunch of code linking together a few bits and pieces so I thought I'd throw it on my blog.

Twisted Matrix Laboratories: Sponsored Development, October 2011

2011-10-25T18:28:42+00:00

I'm happy to report that I've just completed another two weeks of sponsored Twisted development. As usual, the issue tracker directed most of my work. I spent time on new development aimed at resolving tickets and I reviewed changes proposed by other developers with that aim.

All told, 59 tickets got some attention that they would not have received without the sponsored development. The result was 36 closed tickets and 17 other tickets unblocked for other developers to resume work on. Over the coming days or weeks a few more tickets will probably be resolved as a result of this work, as developers (including myself) respond to review feedback.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!

Apart from working to resolve tickets in the issue tracker, I also did some work on infrastructure. I:

cleaned up our BuildBot master configuration and published it
implemented improved error reporting for our API documentation build automation
converted the subversion repository for the Twisted website to bzr and made it public
moved some Windows build infrastructure to a new host with better network connectivity to try to reduce spurious build failures

For more details about a ticket, see <http://tm.tl/NNNN>.

The tickets I reviewed, with tickets now closed in bold:

#78 - abortConnection() method for transports

#2674 - t.i.defer.inlineCallbacks documentation issues

#2675 - Test timeout policy should be decided by runner

#3077 - twistd --help plugin list isn't alphabetized

#3078 - Dynamic ZSH tab completion for any commands using t.p.usage

#3350 - make each peer register as a producer in portforward

#3453 - twisted.web._auth.digest mostly ignores the value of the uri field

#3896 - Passing a unicode object to request.write corrupts the entire response

#3926 - twisted.positioning -- a better positioning framework

#4159 - Add pre-exec hook to spawnProcess

#4519 - Delay parsing of request body until Request.args is accessed

#4751 - t.c.telnet.ITelnetProtocol.unhandledSubnegotiation doesn't reflect reality

#4849 - Add stream compression to twisted.words.jabber

#5024 - MSI buildbot patches version to avoid breaking distutils MSI builder, but does it in the wrong place

#5040 - twisted.web.template is lacking patterns

#5044 - Client documentation should use explicit buildProtocol()s rather than magic "protocol = MyProtocol" on factories

#5085 - Accept IPv6 address literals (with embedded scope ids) in IReactorTCP.connectTCP

#5139 - Replace usage of os.path.walk in favor of twisted.python.filepath.FilePath.walk

#5156 - no ambiguous antecedents (plugins documentation edition)

#5192 - 100-continue support for twisted.web.client.Agent

#5194 - log.msg can fail if someone removes an observer at the wrong time

#5250 - finger15.tac patch

#5252 - Manhole should support CTRL-A and CTRL-E for home/end

#5253 - Add support for input history persistence in twisted.conch.manhole

#5263 - add multipart/form-data support to twisted.web.client.Agent for easy form posts

#5267 - allow twisted.internet.ssl.[Private]Certificate (and friends) to be backed with TLSLite rather than pyOpenSSL

#5271 - trial test method timeout support is fragile

#5275 - t.w.template does not always properly escape comments

#5279 - pass mode to FilePath.createDirectory()

#5282 - ILogObserver, log.msg, log.err, and logging.html are insufficiently stern about log observer thread safety

#5283 - opt_user unused in manhole twistd plugin

#5285 - win32eventreactor misses disconnect events that happen too soon

#5286 - twisted.web.template documentation doesn't cover very basic use-case of rendering more than one of something

#5288 - twisted.web.template.Tag.fillSlots documentation doesn't mention it returns self

#5301 - test_tcp_internals.PlatformAssumptionsTestCase fails when we leak fds

#5312 - Create some release automation for building Sphinx documentation

And tickets I developed:

#581 - CopiedFailure.getTraceback should include the result of CopiedFailure.getErrorMessage even if unsafeTracebacks is not set

#745 - writeSequence not well-tested

#1946 - ErrorHolder is basically untested

#2838 - _dumbwin32proc.Process does not implement IProcessTransport completely

#4603 - names should handle unknown records

#4666 - t.i.task.Cooperator scheduler objects must return an IDelayedCall

#4671 - sometimes I want to have some constant values that are part of a set

#5040 - twisted.web.template is lacking patterns

#5062 - ProtocolWrapper masks the name of the wrapped protocol in the logs

#5075 - IntNStringReceiver copies too much data in dataReceived, causing AMP to be slow

#5140 - Deprecate twisted.scripts.tkunzip

#5158 - Installing on Pypy fails because of CPython-specific extension modules

#5233 - win32eventreactor doesn't notice TCP connection is lost in certain (hopefully rare) cases

#5237 - Add SFTP example

#5273 - Improve the API documentation for MultiService.addService to clarify its use in comparison to Service.setServiceParent

#5278 - When connecting with endpoints, the application factory used is neither started nor stopped, and irrelevant start/stop messages are logged instead

#5285 - win32eventreactor misses disconnect events that happen too soon

#5291 - twisted.trial TestCase should support assertDictEqual from Python 2.7 unittest

#5292 - Listening ports used with wrapping factories could log better start messages

#5293 - Remove mktap and corresponding twistd options (mktap is broken in trunk)

#5299 - udp.Port schedules connectionLost call in the global reactor

#5303 - Default transport implementation doesn't write out bytes immediately

#5304 - Win32Reactor.spawnProcess is redundant

#5308 - Make UDP ports identify themselves when stopping

#5316 - Incorrect API link in twisted.web.proxy.Proxy docstring