Planet Twisted

Itamar Shtull-Trauring: Really?

2010-09-02T14:05:13+00:00

Mockingjay, some sort of popular young adult book
Freedom, by Jonathan Franzen, great American novelist du jour
Publication Manual of the American Psychological Association, Sixth Edition

Itamar Shtull-Trauring: One last thought

2010-09-02T12:17:41+00:00

To steal a line from Churchill, freedom of religion is the worst system possible, except for all the others that have been tried. But freedom of religion is a legal concept, and there's still the question of public attitudes. I think we'd all be better off if we swapped the roles of religion and sex in our society. Religion would be something you did in private, and politicians would spend their time explaining how awesome they are in bed instead of how much they love Jesus.

Wouldn't you rather the President said "Hope you get laid tonight, America" instead of "God bless America" after promising to spend billion of dollars killing random civilians in Afghanistan? At least with the former you don't have to think about how what kind of God would approve of mass murder...

Duncan McGreggor: HCI at Canonical

2010-09-02T10:56:08+00:00

uTouch

Back in March, I blogged about future possibilities (in a blue-sky sense) of multi-touch, mentioning the project management I was doing for MT hardware kernel driver support in Lucid (and then proceeding to dive into the deep end of speculation). It's now an Ubuntu cycle later, and holy crap... I'm having a hard time finding the words. I think the blog title says it all. But I'll try to elaborate :-)

Unless you've been living under a rock, you've probably noticed the big announcements we made a few weeks ago:

uTouch mail list announcement
Mark Shuttleworth's blog post
The Canonical uTouch announcement
Chase's blogging extravaganza (uTouch, MT architecture, Magic Trackpad, Trackpad update)

For the next few days, we were all over Google news. This was quite a shock, given that we'd been heads-down into the project for so long and hadn't really come up for air nor fully anticipated the impact (to others or ourselves). Needless to say, after the intense amount of work that the team had engaged in over the previous couple months, this was quite gratifying, if somewhat unexpected.

There has been a lot of discussion in blog posts, mail lists, IRC (#ubuntu-touch on freenode.net), Launchpad bugs and merge proposals, etc., so much so that touchscreens now pursue me feverishly when I sleep at night. I'm really not interested in writing more of the same :-)

As such, I want to mix things up a bit...

HCI Remixed

I've been reading an amazing anthology of essays on human-computer interaction. I still haven't finished the book (yeah, I've got about 10 in-progress titles on my nightstand), but am relishing every word in this particular collection. The book is HCI Remixed: Reflections on Works That Have Influenced the HCI Community.

While doing some research at the beginning of the Maverick development cycle, I came across HCI Remixed at the local library -- the title intrigued me and I couldn't resist. Weeks later, after having maxed out the number of times I could renew the book, I just purchased it -- I simply couldn't get enough of the book. Every essay I'd read up to that point was fantastic; each one provided volumes of information, experiences, insights, ideas for follow-up, etc. Whenever I finished one essay, I spent days and sometimes weeks reading up on references, pondering the past and future of human-computer interaction.

Due to the unusual nature of the book, describing it is surprisingly difficult. That being said, the MIT Press page gives you a great taste:

Over almost three decades, the field of human-computer interaction (HCI) has produced a rich and varied literature. Although the focus of attention today is naturally on new work, older contributions that played a role in shaping the trajectory and character of the field have much to tell us. The contributors to HCI Remixed were asked to reflect on a single work at least ten years old that influenced their approach to HCI. The result is this collection of fifty-one short, engaging, and idiosyncratic essays, reflections on a range of works in a variety of forms that chart the emergence of a new field.

If you're into HCI, learning from others, and discovering new sources of inspiration for your own work, this is simply a must-have book :-)

A Small Piece of History

By the time I checked the book out of the Golden public library, it was May and we had begun building the MT team. By July -- once it became clear how astounding the team's work was -- I realized that in 10 or 20 years I could very well be writing an article about Henrik, Chase, Stephen, Ikbel, and Rafi. Much like those in the book, I could be sharing the conversations I'd had with Stéphane Chatty, Mark Shuttleworth, Neil Patel, David Siegel, and John Lea. And that's only the crew which which I was collaborating or discussing directly. There are a lot of folks who've been working very hard on multi-touch infrastructure solutions and exploring ways of integrating these for several years (e.g., Peter Hutterer and Carlos Garnacho).

Though many foundations have been laid, as of yet (to the best of my knowledge), no Linux distribution has released a multi-touch stack that integrated gestures in a unified manner across everything from applications to window managers and beyond. This was something that Mark wanted us to provide to the open source world. In this spirit, the multitouch team hasn't just hacked things together to get a product out in time. A lot of generative, creative thought and care has gone into uTouch. A lot of original problem solving has taken place. Physics PhDs, kernel hackers, X.org hackers, driver creators, application integrators, toolkit gurus -- all of this knowledge was concentrated, applied, and used to distill a first approximation of what a gesture stack in Linux could look like, using the latest available technology and methodologies.

To be honest, we weren't really sure we could pull it off. There was a very good chance we could have failed at our task, quietly chalking up the loss as a lesson learned. Now that we've managed to shape these ideas into actual software, taken the threads of dreams and woven something real, we are thrilled to be engaging with others to see where all of us can take multi-touch and gestures from here.

Thanks to expert input from the wider open source community, we're already looking at ways in which we can improve upon the first version, ways of bringing new ideas and experiences to developers and users of multi-touch hardware running Linux. Things are only just warming up, and the greatest contributions have yet to be made. Every single person in the community has before them a world of possibilities for getting involved and creating the future human-computer interfaces for the free and open source world in the coming weeks and months. These are indeed exciting times.

Moshe Zadka: Android: reviewing after a week

2010-09-02T00:26:25+00:00

I got my android on Thursday evening, and it’s now Wednesday, so I feel I now have some perspective on it.

Technical things first: I got the Motorola Droid 2 from Verizon. It has a pull-out keyboard, which sold me on it. It is a little bulky.

First, the bad: no universal cut’n'paste. You almost forget how much it is taken for granted on today’s computers. I view this as the next gaping flaw the os should address. The browser deals with resizing suboptimally – I would like a mode where I force the text to be bigger and reflow. In a related note, you would expect mainstream websites to have a more mobile friendly version, no?

The good:
Turn-by-turn navigation
No crappy pre-loads
Pretty good app selection
Several decent games
A solid 5/10 on voice recognition
Nice ui
Amazing multitasking

If this continues, apple has their work cut out for them to stay ahead of the game.

Thomas Vander Stichele: long life external batteries for laptops ?

2010-08-26T18:19:27+00:00

I’m probably going to be doing some long flights pretty soon and I’d like to maximize my laptop time hacking on the plane.

Who of you have experience with big external batteries, what’s a good one to get, how long can I expect them to last and how long do they take to charge ?

Moshe Zadka: Pictures, Copyright and Licenses

2010-08-25T20:08:57+00:00

Recently, I find myself in the position of creating a lot of Facebook events, for reasons unrelated to the post.

When creating events on Facebook, you are given the option of attaching a picture. This picture, or thumbnails thereof, are used by the website wherever the event appears. The default was changed to be slightly less horrible than it used to be, but it is still a little embarrassing to use the default. Therefore, I find myself often needing pictures with a certain theme.

However, I am very careful about mistreating copyright works. I feel copyright is an outdated way of protecting creativity, but I am not willing to expose myself to C&Ds in the name of that belief just yet.

There are two good sources of pictures which can be used for those purposes:

Flickr — just make sure to go into “Advanced search” and click “yes” on the “Creative Commons” check-box.
Wikimedia Commons

Note that most pictures you find with a Google image search are not CC, free or in any way OK to use. However, the two sources above are a wonderful resource for finding great pictures without violating copyright.

Itamar Shtull-Trauring: An expert on money

2010-08-24T03:28:52+00:00

I'm proud to be a graduate of the Harvard Extension School. I got a wonderful liberal arts education, and it wasn't even that expensive. Unfortunately, when it comes to economics the Extension School is just as bad as the rest of Harvard: worse than useless.

For example, this fall you can take ECON E-1452: "Money, Banking and Financial Institutions." According to the syllabus, "this course is an analysis of money and its role in financial markets and the economy. It considers the impact banks and other financial institutions have made in the United States and internationally, as well as the events leading up to the financial crisis of 2008." The course textbook was written by Frederic S. Mishkin.

This is the same Frederic Mishkin who was apparently paid $124,000 to write a paper which stated that "it [is] unlikely that there are serious problems with safety and soundness in the [Icelandic] banking system," and though a financial meltdown might be possible, such "self-fulfilling prophecies are unlikely to occur when fundamentals are strong, as they are in Iceland."

The Icelandic banking system subsequently collapsed. Mishkin never disclosed in the report that he was paid to write it by the Icelandic Chamber of Commerce.

Apparently one Charles Ferguson is making a movie about the financial crisis, and he interviewed Mishkin on the subject:

(The CV has since been changed to include the correct title of the paper mentioned in the video.)

Itamar Shtull-Trauring: The Wider Point of View

2010-08-19T23:46:54+00:00

All fiction is a reflection of its time; only rarely does it poke its head through the mirror and take an active look around. Good science fiction is more likely to do this because its writers, like anthropologists, have internalized that the present is contingent, temporary, always provincial. Travel a bit in time or space, and the world is a different place, seen through alien eyes. But since fiction is a reflection of its time, often the same critical theme crops up again and again, the product of a particular soil and climate.

The last time I noticed this, the theme was the corruption of power. In book after book published in the same couple of years, thrillers and mysteries (all in the SF or fantasy genres) came to the same revelatory climax: the government or those in power were the villains. I'll omit the names of the authors for fear of spoilers.

This time the pattern I'm seeing is antagonistic reflections on religion, although the list of books is shorter so far. Stross' "The Fuller Memorandum" isn't quite as good as his previous two Laundry novels, although it'll still appeal to Lovecraft fans, and Ken Macleod's "The Night Sessions" (still!) hasn't made it to the US yet so I've yet to read it. Ian McDonalds "Ares Express" is quite good, as is China Mieville's "The Kraken", which I just finished reading. Strangely, the novel feels like Mieville is channeling Terry Pratchett, in his guise as humanist rather than humorist. There is some of the latter though, and Mieville can wield a sharp pun, e.g. in the climax of this short story.

All of these authors are British, though I couldn't say why. Perhaps now that religious discrimination is becoming fashionable in the US maybe we'll see some American authors chiming in. I do sympathize with the claim that religions whose members have been involved in religiously-motivated bloodshed should be looked upon with suspicion. The Spanish Inquisition, all the Jews slaughtered during the Crusades, the Protestants killed by other Protestants during the Reformation for believing in the wrong method of salvation (theological disputes are easy to win if the municipal executioner works for you), the Thirty Years War... violent bunch, these Christians. As one of many contemporary examples, there are all those avid readers of "The Left Behind" series (millions of copies sold!), excited about the impending death of my siblings in the coming apocalyptic wars in Israel.

Lets ban some churches first, and then maybe we'll talk about mosques.

Here's what Schaffer the Dark Lord has to say (or rather, sing) on the subject:

Moshe Zadka: Stargate Science

2010-08-16T03:52:53+00:00

There was one thing that always bothered me about “the science of stargate”. No, it’s not about the stargate’s FTL movement means that causality is violated — that’s a problem common to any FTL travel, and can be dealt with in a number of ways.

The bigger problem is something that was obviously invented for plot device purposes: “when you open a wormhole, matter can only move in the direction it was opened, but energy can move both ways”. How can it make sense? After all, matter and energy are interchangeable. Here is a possible solution: it pertains to the difference between fermions and bosons. In quantum mechanics, fermions are particles with spin of the form “n 1/2″. Bosons are particles with spin of the form “n”. There are many deep differences between them: for example, the Pauli exclusion principle, responsible for the structure of atoms, applies only to fermions.

Normal matter is made of atoms. Atoms are made of electrons, protons and neutrons. Electrons are fermions. Protons and neutrons are made of quarks, which are also fermions. In other words, atoms are made of fermions. Radio waves, the most common communication in the Stargate series across stargates, are made of photons. Photons are bosons.

It makes sense, of course, that Capt. Carter would simplify (“dumb down”) the science. Instead of saying “fermions” and “bosons”, the O’Neill would respond better to matter and energy. So there we have it: bosons can travel both ways, fermions can only travel one way, across a stable wormhole constructed from one stargate to another.

Jonathan Lange: unittest, part 3

2010-08-14T14:34:41+00:00

So far, we've talked about TestSuites, TestCases and TestResults. We've seen how these objects interact with each other and how they can generally be thought about as having more than one interface. TestResult has an interface for the TestCase and an interface used for querying the results, TestCase has an interface for test runners and an interface for test authors, and so forth.

Now we need to give some time to the bits that glue everything together: the test runner and the test loader.

TestRunner

You will not find a class in unittest.py called TestRunner. A test runner is simply something that takes user input about a test run – what tests to run, what manner to run them in, how to display the results – and does it.

Essentially, it does something like this:

  test = TestLoader().loadTests(user_specified_test_string)
  result = makeTestResult(options_specified_by_user)
  result.startTestRun()
  try:
    test.run(result)
  finally:
    result.stopTestRun()

And that's it.

You see that the test runner is responsible for instantiating the test loader and the test result. It's perhaps excusable for a test runner to be tightly bound to particular implementations of test loader and test result. Certainly, before TestResult grew startTestRun and stopTestRun it was inevitable: since the test runner was responsible for summarizing the results of a test run, overall responsibility for displaying the results was split between the runner and the result.

Nowadays, the tight coupling can be limited. If your test runner has an option to display stack traces as it gets them, then that's pretty much going to force you to use a particular result. However, you can still write your code internally such that someone could pass in a different result that still works, even though it doesn't do exactly what the user asked for.

TestLoader

From the point of view of interfaces and compatibility, this is a pretty boring class, and that's a good thing. The test loader's job is to find tests based on some user input and construct a single ITest object for them.

When it does more than this, one runs the risk of having the behaviour of a test suite depend too much on the runner itself. The ideal is to have the test suite run in any runner: trial, nose, unittest2, py.test, whatever.

Some TestLoaders provide hooks so that users with complicated test suites can customize the way their tests are loaded. Whenever the Trial TestLoader sees a test_suite() function in a module, it lets that function take charge of the loading.

The standard library in 2.7 has a new hook, inspired by an innovation in bzrlib, but slightly different. load_tests(loader, standard_tests, pattern) is given the loader used by the test runner, the tests that the loader would have loaded, and if appropriate, a glob used for matching test module files. The advantage of this hook is that it reduces the danger of customizations made to the loader, since the test suite has access to the same loader. It also makes custom loading easier by giving the standard tests as a starting point. bzrlib uses this to run the same set of tests against many implementations.

I think that's all I have to say about these two, which means that's pretty much all I have to say about unittest's API for test frameworks. Still one more post to go though: interfaces for test authors.

Let me know if I've missed anything, if anything here surprises you or contradicts something I said in the past or if things are unclear. The comments on the previous two posts have really helped!

Christopher Armstrong: Holy sh*t! QUAKE LIVE is launched!

2010-08-08T06:29:06+00:00

Alright, so what's the explanation for my long absence from the blog this time? QUAKE LIVE just went out of beta yesterday. We've released subscription services which give you access to new maps, a new game type (freeze tag), and the ability to start your own matches with your friends. It's at quakelive.com.

Since I joined the project I've been working on scalability improvements, but also new features like the invitation system and subscription management. I've even dipped into our PHP code recently. It's been really fun to work on and learn about a system that needs a lot of scalability in managing a combination of extremely dynamic and static, long-term statistical data. Python and Twisted are continuing to be a great fit for our work in the infrastructure.

Having scalability problems is a great thing, of course, so go try out the game and make me work more :-)

http://www.youtube.com/watch?v=uKtrmmIUv1k

Twisted Matrix Laboratories: Fluidinfo sponsors Twisted!

2010-08-08T03:11:00+00:00

Fluidinfo is Twisted's latest sponsor! We asked if they could share some of the joys of sponsorship with us, and they had this to say about it:

At Fluidinfo, we're heavy users of Twisted. All of our infrastructure depends on it. FluidDB, our social database, is entirely built on Twisted, and we've released several core parts of it as open source: txAMQP, txRDQ and txThrift. Not only that, but we have contributed to Twisted both with code and (albeit small) personal donations.

It's not just that sponsoring was the fair the thing to do, it has also produced tremendous results in a framework crucial to our business. When you donate to the TSF it's simple math that more bugs get fixed, but also the quality of the entire Twisted project is enhanced. And when that happens, all of our products are enhanced automatically without us writing one line of code. So sponsoring is not just an act of generosity, it's an investment in Fluidinfo. Using and sponsoring Twisted has been an indispensable "force multiplier" for a growing start-up like us.

– Esteve Fernandez

CTO, Fluidinfo Inc.

Moshe Zadka: Funny Moments from the Kitzmiller vs. Dover Trial

2010-08-08T02:04:19+00:00

The Kitzmiller vs. Dover trial has some doozies in the cross-examination of Michael Behe, one of the proponents of intelligent design (from the transcripts:

Q Telling the readers of Pandas that you were a critical reviewer of that book is misleading, isn’t it?

A I disagree. As I said, that’s not the typical way that the term “critical reviewer” is used, but nonetheless, in my opinion I don’t think it is misleading.

And a little bit later

Q And Mr. Dembski, who is the author of Design of Life, described you as a co-author of the book, correct?

A That’s what he does, yes.

Q That’s false, isn’t it?

A Again, I am not an author of the book, but William Dembski, several years ago, asked if I would contribute. And I explained to him that I did not have the time to do so. And he says well, perhaps, you know, in the future he could solicit material from me and then I would be one of the authors of the book. So, that’s correct.

Q So that makes you a co-author right now, Professor Behe?

A I certainly would not have listed myself now as a co-author, however, I think that he was anticipating my future participation in the project.

Q So that’s a true statement, Professor Behe, that you’re a co-author?

A It is not now a true statement but it might be in the future.

Later:

Q The statement that you’re an author, and Mr. Dembski’s statement is false too, isn’t it?

A That’s not what it says on the screen, sir. It says, “Who are the authors of Design of Life as you understand it?” And the way I read that is that he’s seeing into the future and seeing when this actually will be published and anticipating that I will participate in the publication of the book at that point.

Q Seeing into the future is one of the powers of the intelligent design movement?

And another one:

Q But you are clear, under your definition, the definition that sweeps in intelligent design, astrology is also a scientific theory, correct?

A Yes, that’s correct.

From the next page:

Q Starts “Among Organisms,” and going to the second sentence it says, “Design proponents have a realistic and more cautious approach to the use of homologies. They regard organisms which show great structural differences, such as starfish and chimpanzees, as having no common ancestry.” Correct.

A Yes, that’s what it says, but again, I read that as the fact that while other theories such as Darwinisms might make a commitment to common ancestry, a theory of intelligent design can live with what the data shows on that respect, because a theory of intelligent design does not speak to that; it just speaks to the effects of intelligence. So I see this as an accommodating sentence rather than as something that is required.

Also, this gem:

Q. Now you selected some of your peer reviewers?

A. No, I did not. I gave my editor at the Free Press suggested names, and he contacted them. Some of them agreed to review. Some did not.

Then, later:

Q. Okay. And then you go on to say that you still think — well, I’ll leave that. Your argument is that, even if the type III secretory system is a pre-cursor to the bacterial flagellum, is a subset, the bacterial flagellum is still irreducibly complex because that subset does not function as a flagellum?

A. That’s correct, yes.

Q. And, therefore, the bacterial flagellum must have been intelligently designed?

A. Well, again, the argument is that, there is — that when you see a purposeful arrangement of parts, that bespeaks design, so, yes.

He calls himself a scientist?

Q. Okay. And you said again that the strength of the inference is quantitative, but again you haven’t quantified it.

A. I have not put numbers on it, but one can kind of do intuitive judgments about these things.

(This is mostly a note-to-self in case I ever need to refer to these things. But I hope my readership enjoys it!)

Thomas Vander Stichele: middle mouse frustration

2010-08-05T15:34:01+00:00

Sometimes I get annoyed about a simple problem and want to fix it the right way and have it fixed forever. Right now is such a time.

Middle click stopped working. Not sure when – maybe on my last update or reboot ?

I know what’s wrong – xev tells me that clicking that middle mouse button tells me it thinks it’s button 6 when it should probably think it’s button 3.

Now, in the Old Days you’d tweak xorg.conf and add a funky option.

In the New Days Xorg is trying to be all about not needing xorg.conf and I support that vision. But, what is now the proper way to fix these issues ?

a) is this a bug, and should some hal file provide enough info for this mouse type to do the remapping ?

b) can evdev be configured on the fly and/or permanently to fix this mapping ?

c) which part of the stack is getting it wrong ?

If it helps, this particular machine is on Fedora 12, and the mouse is a simple Dell mouse with two buttons, a scroll wheel (the ‘third button’) and two up/down buttons.

Moshe Zadka: Discovering Miller’s God

2010-08-03T23:10:23+00:00

Kenneth Miller, also known as the guy who kicked Behe‘s ass in the Dover trial, wrote a book called “Finding Darwin’s God“. Kenneth is a talented writer, and I loved his book “Only a Theory“. But somehow, my reactions to FDG were more mixed.

First the compliments: it is exactly what it says on the tin: it is finding Darwin‘s God. Darwin lost his faith towards the end of his life — but the reason was his daughter’s death, and had nothing to do with the theory of evolution. The Darwin who wrote “Origins” was a believer. Kenneth Miller’s text reads almost like a whodunnit detective novel, searching for that God.

This is the part that weirded me out, in the beginning. The search is not for a God, some notion of divinity that is compatible with evolution. The search is for Darwin’s God, the same Darwin who studied Anglican theology. The search is for the Christian God, and the book itself is a fine book of Christian apologetics. Thus, Miller takes care to present evolution carefully and accurately, but takes no such similar care with presenting the views of other religions. He casually dismisses Einstein’s religious feelings as “not real religion”. It makes sense in context — he is on the hunt to prove that the Christian God (and related beliefs, like the divinity of Jesus and the immaculate conceptions) are perfectly consistent with what science knows.

This book is not really intended for people who accept evolution as hard science and wish to understand its compatibility with religious beliefs. This book’s real audience is Christians for whom evolution causes a crisis of faith. Miller’s goal, noble in its own way, it to reconcile the fact that the evidence for evolution are overwhelming with the need of Christians to keep their faith. The search is truly for Darwin’s God, the Christian God that Darwin believed in even as he wrote “Origins”.

If, like me, you have never read a book of Christian apologetics, this book is a wonderful example of its genre. If, however, you are interested in how evolution and religion, in general, can be reconciled — well, that’s not what the book attempts to do, and it does not do it.

Jp Calderone: June - July Reading List

2010-08-02T22:50:58+00:00

Forty Signs of Rain. Kim Stanley Robinson

Fifty Degrees Below. Kim Stanley Robinson

Sixty Days and Counting. Kim Stanley Robinson.

The Tipping Point: How Little Things Can Make a Big Difference. Malcolm Gladwell.

Brainbox. Christian Cantrell.

Altered Carbon. Richard K. Morgan.

The Second Ship (The Rho Agenda). Richard Phillips.

Immune (The Rho Agenda). Richard Phillips.

Moshe Zadka: Being a Jew in the Bay Area

2010-08-02T21:23:00+00:00

In Israel, you don’t have to think about being a Jew. It’s the default state. Others have to think about distinguishing themselves as not being Jews. For the first time in my life, I have to think about being a minority, and keeping my traditions while living in an apathetic nation.

OK, seriously, that’s hardly true. I mean, Palo Alto schools shut down for the high holidays. Still, it is a useful first approximation.

Diet: I never kept kosher in Israel. At 23yo, I became a vegetarian. This means that most kosher rules are irrelevant to me (still relevant: wine kashrut rules and dairy rules. Wine kashrut rules are (a) exotic (b) weird and (c) unless you demand a certificate, the fact that most wine is handled in an automated fashion should really be enough. Dairy: nobody sells camel or pork milk around here, as far as I know. I eat at non-kosher restaurants, which probably handle my food with non-kosher dishes, which makes my food technically non-kosher. The alternative would be all-home-cooking all the time, which is obviously unfeasible for a single guy. So I keep kosher as much as reasonable by default…

Shabbat: This post really started because I read the Shabbat Manifesto. Even the logo is an admission of failure: it shows Saturday as the 6th day of the week, using a Monday calendar. I usually set my preferences for a Sunday calendar whenever I can. The post is also infected by severe ludditeness: “avoid technology”. What’s technology? Why is a book not technological, but a car is? What about when books will all be e-books? I prefer to interpret it as “be a day from which you do not gain material benefits.” So no working (as much as possible), no going to the gym but yes to easy hikes, drives to the library or enjoying a leisurely breakfast at IHOP. I also try (managed to do it once) to go to a synagogue on Saturday morning.

Holidays: I try to observe most holidays, but I’m not amazingly good at it. I managed to learn about the Talmud on Shavuot (not a very good lesson, but oh well), I observed Yom Kippur, I observed Passover and so on.

Community: This is where I managed to hit pay-dirt. I managed to join a number of Jewish young-adult organizations, among them JND (where I am now serving as communications officer).

Jonathan Lange: unittest API, part 2

2010-08-02T11:15:15+00:00

In part 1 of this humble attempt to document the interfaces and contracts that unittest actually cares about, we talked about TestSuite and TestCase, how they both implement a common interface that's used for running tests, ITest and how they each implement their own interfaces, ITestSuite and ITestCase.

Now we're moving on to a much more complicated object, TestResult, to see how we can pick apart the ways it interacts with the rest of the system.

TestResult

A TestResult object is all about dealing with the results of tests, as you might expect. However, it doesn't generally represent a single test result. You could say it represents the results of a number of tests, but I don't think that's terribly helpful.

Better to think of a TestResult object as an event handler. A TestResult object receives events from a test run and then does something with them.

Just as TestCase has a two-faced nature, presenting one interface to the testing framework and another to test authors, so to TestResult can be thought of has having many interfaces:

Its interface to a TestCase. This can be thought of as the test event handling interface
A result querying interface, normally used by a test runner
An interface for events that come from the test runner, the runner event handling interface.
An execution control interface.

Note that the result querying interface and the runner event handling interface together make up the interface between the TestResult and test runner.

Let's start with the test event handling interface. The methods below are the interface between TestCase.run() and TestResult. (I guess TestCase.debug too, but no one cares about it).

startTest(test): Called when test commences running. Although not enforced, it's impolite to provide any results for test before calling this.
stopTest(test): Called when test is completely finished. Although not enforced, it's impolite to provide any more results for test after calling this, unless you call startTest(test) again first.
addSuccess(test): Called when test has been shown to be successful. The default implementation does nothing.
addError(test, err): Called when test raises an unexpected error. err is a tuple such as you might get from sys.exc_info(). Calling this method for the first time must change the result of wasSuccessful().
addFailure(test, err): Called when test has failed one of its assertions. err is a tuple such as you might get from sys.exc_info().

The above interface is tightly coupled to the implementation of TestCase.run(). In particular, if you wish to add more kinds of results to your testing framework ("skip" results are a fairly common addition), then you must change both TestCase.run() and the TestResult interface.

If you do something like that, I recommend making sure that your modified TestCase can handle TestResult objects that do not provide the extensions to the interface that you need. One common way of doing this is to have the TestCase fall back to the primitive result types, e.g. "skip" might become "success" for a TestResult that doesn't know what skipping means.

Importantly, the interface between TestCase and TestResult has been fattened in Python 2.7.

addSkip(test, reason): Called when test is skipped. reason is a string explaining why the test was skipped.
addExpectedFailure(test, err): Called when test failed in a way that was expected. err is a tuple such as the one returned by sys.exc_info().
addUnexpectedSuccess(test): Called when test was expected to fail, but didn't.

The following interface is a way of learning about test results after they have happened, the result querying interface, and is part of the contract between the test runner and the TestResult.

wasSuccessful(): If there have been no errors and no failures, return True. Return False otherwise.
testsRun: An integer that is the number of tests that have been run.
errors: A list of tuples of (test, error_message) for all of the tests with unexpected errors, where test is an ITestCase and error_message is a string suitable for display to humans, generally containing a traceback.
failures: A list of tuples of (test, error_message) for all of the failing tests, where test is an ITestCase and error_message is a string suitable for display to humans, generally containing a traceback.

And of course, Python 2.7 fattens this interface again to have the following:

skipped: A list of tuples of (test, reason) for all of the skipped tests, where test is an ITestCase and reason is a string suitable for display to humans, generally containing a traceback.
expectedFailures: A list of tuples of (test, error_message) for all of the tests that were expected to fail and failed in the manner they were expected to, where test is an ITestCase and error_message is a string suitable for display to humans, generally containing a traceback.
unexpectedSuccesses: A list of all of the tests that unexpectedly succeeded. Members of the list are ITestCases.

In Python 2.7, TestResult also extended its interface to the test runner beyond simple result querying and into allowing the test runner itself to send two very important events to the TestResult, behold the runner event handling interface:

startTestRun(): Called before any tests have been run. It is impolite to provide any test results before calling this.
stopTestRun(): Called after all the tests have finished running. It is impolite to provide any test results after calling this. A TestResult object is generally not expected to handle any events at all after this method has been called.

Some test runners rely on TestResults to use those events to display the results to the user. These runners frequently do not use the result querying part of the interface.

There is one more interface that TestResult implements: the execution control interface:

stop(): Signal that the execution of further tests should stop now. Sets shouldStop to True.
shouldStop: If True, then test execution should stop. TestSuite.run() should monitor this value and stop execution if ever it is True.

This interface is mostly used as a way of handling KeyboardInterrupts cleanly.

Summary

If you want your TestResult object to work with standard Python TestCase objects, or any TestCase objects that try to stick close to the standard, then you must provide the test event handling interface described above. If you are writing your own test framework or test runner, you care about this, because you want to run everyone's unit tests.

If you want your TestResult object to work with the standard Python test runner before Python 2.7, then you must provide the result querying interface. If you are using the standard Python test runner, you care about this. For Trial or testtools, you must provide the runner event handling interface. For anything else, I'm afraid you are on your own.

Always provide the execution control interface.

Comments

In this documentation, I've been trying to describe the various interfaces without inserting too much of my own opinion about their design. However, I think some commentary might actually help to make things easier to understand.

By providing a querying interface for TestResult to be used by a test runner, the original designers of unittest practically insisted that responsibility for displaying the results of a test run be split between two different classes. The TestResult takes care of displaying incremental feedback from the running tests and the test runner takes care of displaying the summary. You can see evidence of this design in Python 2.6's unittest.py, where there's a hidden _TextTestResult subclass which has extra methods that are called only by a special TextTestRunner.

The addition of startTestRun() and stopTestRun() mean that now a TestResult object can be fully in charge of displaying its results. As such, providing a query interface and exposing details like the list of test failures somewhat vestigial.

I'm less happy with this post than the previous one. As such your critique is even more welcome.

Still to come: the interface for test authors and just what is a test runner anyway?

Update: Remove ambiguity in expectedFailures description (see comments). Thanks Aaron.

Jonathan Lange: unittest API, part 1

2010-07-29T16:56:11+00:00

It's a little known fact, but unittest actually has an API.

This isn't the API that you deal with when you write tests, but rather an API that unittest itself uses when running tests. You could think of it as two interfaces: one for test frameworks and one for test authors. Both APIs are real, but both are poorly documented and often misunderstood or abused.

TestCase

An instance of TestCase represents a single test. What you think of as a single test is up to you, but most of the time it's a unit test.

A TestCase object must provide the following methods.

This first list of methods can be thought of as a single interface, which these blog posts will call ITest given the lack of any better name.

countTestCases(): A method that returns the number of test cases this represents. It should always return 1.
run(result=None): Calling this method actually runs the test. result is a TestResult object. run must call result.startTest(self) when it commences running the test and result.stopTest(self) when it is finished. Between these calls it must call a method on result to signal the result of the test. run must never raise an exception, and its return value is ignored. If result is not provided, the TestCase is obliged to make one.
__call__(result): Identical to run(result), provided for backwards compatibility.
debug(): Calling this method runs the test without collecting its results. It may raise exceptions. This method is rarely called by test frameworks.

The following methods are specific to individual test case objects. We call this interface ITestCase.

id(): Should return a string that uniquely identifies the test. For Python tests, the fully-qualified Python name works well. The uniqueness of the id is not enforced.
shortDescription(): Should return a string that describes the test. Many test frameworks use this value to display test results.
__str__: Should return a string that describes the test. Frequently the same as either shortDescription() or id(). Many test frameworks use this value to display test results.

There is also a second interface, one that matters to code that subclasses TestCase. We'll deal with that in a later post.

TestSuite

A TestSuite represents nothing more or less than a bunch of tests.

A TestSuite must provide the ITest interface described above, with the differences that you would expect from something that represents many tests: countTestCases returns the number of tests in the suite; run runs many tests and thus calls result.startTest and kin many times over; debug is the same and can explode anywhere.

One difference is that TestSuite.run must stop running tests as soon as it detects that result.shouldStop is true.

In addition, TestSuite implements the following interface, which I'm giving the completely arbitrary non-existent name of ITestSuite.

addTest(test): Takes an ITest and adds it to the suite.
addTests(tests): Takes an iterable of ITests and adds them to the suite. Normally equivalent to [suite.addTest(test) for test in tests].
__iter__: All test suites must be iterable. Iterating over a test suite yields ITests. These may differ from the ITests provided to addTest and addTests.

In later posts, I hope to document TestResult, the subclassing interface of TestCase and tell you exactly what I think about test loaders, test runners and the like.

I'm blogging this partly because I don't know where else to write this up, but mostly because I need your help to make sure that I'm being clear and correct. Please comment with questions and corrections, and let me know if you find this at all helpful.

Moshe Zadka: Irony

2010-07-28T21:21:16+00:00

(Friends have shown me a stand-up routine about “Ironic” which prompted this.)

Picture this: you’re in your literature class. The teacher gives the definition of “irony”, and proceeds to rattle off a few examples of irony. I’m dating myself this, but back when I was in this situation (highschool lit) the song “Ironic” did not achieve cult-status yet, at least around my part. So my teacher did this, we copied this down to our notebooks, and eventually got tested on this.

The scene as detailed above has two interesting things about it: (a) there is absolutely nothing ironic about it and (b) it could never happen, as described, today.

I am imagining this scene happening today: the teacher says “today, we will discuss irony.” Without fail, one of the brighter lights in the class says, “oh, it’s like rain on your wedding day.” The teacher visibly sighs, rolls her eyes, and explains how there is absolutely nothing ironic about rain on your wedding day. I am pretty sure this is more or less how it goes in every classroom around the world: everyone thinks they know what irony is, because of the song, and they’re all horribly, dreadfully mistaken.

This is ironic. The ur-example, the immediate connection in our shared knowledge, of irony is a list of examples of things which are not ironic. I think that even if I tried, hard, I could not manage to associate something more ironic with the word ironic.

Thank you, Alannis, for creating the most delicious irony of all, and ensuring that irony can never be mentioned without causing irony. The world is perfect once again.

Thomas Vander Stichele: Live WebM stream from GUADEC

2010-07-28T14:46:56+00:00

Six years ago, we did the first large scale Ogg Theora stream from the 2004 GUADEC conference.

It was a dime on its side to get things ready for this year. I purposely removed myself from the organization, because for various reasons I’m not going to GUADEC this year, but I was hoping the rest of the company would do their part to get this working, and I just provided the necessary prodding along the way. I’ve been told one of the organisers in charge of this got ill at some point and communication went a bit south during that period, so I had some complaints from our support guys that they had to do last-minute rushing.

But the streams are live today, and a few developers here are giddily running around looking at the stream, the image, working on some typical bugs you get when you’re doing stuff like this for the first time (the artifacts on keyframes the encoder seems to have remind me a lot of the Theora bugs we had to squash back in the day, and obviously they are worse on still images, like, say, an empty conference room…)

Go check out the stream and make sure you have a WebM-enabled browser, like the Firefox 4.0 beta or latest Opera.

Congratulations to our intrepid hackers like Zaheer and Andoni for their hard work a few weeks ago on WebM, and I’ve been told Marc-André actually went to Holland just to deliver the encoders :)

Jonathan Lange: Python 3

2010-07-26T23:40:28+00:00

I would be much more sympathetic to the whole Python 3 endeavour if they had made a serious effort to keep the major 2.x releases mutually compatible.

Moshe Zadka: On Tautologies and Evolution

2010-07-26T21:02:02+00:00

Many people dismiss evolution as “survival of the fittest is a tautology — the fittest are the ones who survive.”

This is wrong for many reasons, but one of the problems with that is something I have yet to see among the writings of evolution. As is standard in defenses of evolution, I will begin my thesis with analogy to gravity. But before that, I would like to analyze what we mean by “tautology”. A “tautology” is something that is true by definition, or cannot be logically anything other than true. An example is “x=x”. X must always be equal to itself.

Now, open a math journal. Any math journal. Choose a paper. Any paper. I will do it, now, by choosing a paper from arxiv.org (it’s not strictly a math journal, but close enough for what I have in mind): I tried to be “random” about it by going to “new” and choosing the first one that I could half-way understand: All automorphisms of all Calkin algebras. As an aside, much like most readers of this, I’m not really qualified to judge it. But just reading the abstract, the author claims: “The Proper Forcing Axiom implies all automorphisms of every Calkin algebra associated with an infinite-dimensional complex Hilbert space and the ideal of compact operators are inner.” Assuming his proof is correct, this means that this thing that is fairly non-trivial to understand is a tautology. I am willing to bet that even if you specialize in the field, the above was not obvious to you. In fact, this is true of any math paper: it is completely filled with tautologies, and is completely and utterly non-obvious.

Let me give you another example of a tautology: given that the gravitational force is GMm/r**2, a “solar system” (one very heavy body and a bunch of significantly lighter bodies) will move in close approximations to ellipses (that is not an exact phrasing, but a proper estimate of the approximation is outside the real of this post). That is a tautology, and is also fairly non-trivial.

Now, here is another: assuming that genes are passed in a mostly-faithful (small number of mutations) to the next generation, and that genes correlate to reproductive success, we expect to see “evolution”. This is a mathematical theorem, in that the axiom can be modeled mathematically and “evolution” (meaning many of the phenomena under “evolutionary biology” such as exaptation, adaptation, complexity and variety) be proven. Note that the initial axiom is not a tautology: there is no a-priory logical reason to assume changes in an individual’s life-style won’t change their genetic code. However, the initial axiom, despite not being a tautology, meaning logically true, is factually true: many experiments have been done on this. By the way, it’s as true as GMm/r**2: it’s a useful approximation, not the complete truth: there are epigenetic pressures on heredity. Just like in gravitation theory, we use a simple model to derive predictions, and the correlation of the predictions with the model gives us evidence that the model is true.

Summary: a significant part of evolutionary biology is a tautology. But tautology is not a bad word — it means that a significant part of biology is mathematics. This is good. Mathematical modeling is the best modeling.

Jonathan Lange: pyflakes-doctest

2010-07-20T13:08:10+00:00

Someone at Canonical (I don't know who) wrote something cool once to get Pyflakes (the best Python linter) to run on doctests.

It has recently been deleted from the Launchpad tree, but since it's so useful I thought I'd make it available.

Download pyflakes-doctest whenever you'd like. If you can get it into Pyflakes trunk, then you'll become even more wonderfully, deliciously fabulous and creamy than you undoubtedly are already. Get rid of the Launchpad-specific stuff though.

Moshe Zadka: Making the Web a More Secure Place

2010-07-15T22:09:31+00:00

The current phishing attempts are ludicrously easy to mount, and they are this ludicrous because the current web’s security model is fundamentally broken.

If Joe Q. Random discovers a new site (maybe a forum site dedicated to discussing cats — Mr. Random is a big cat afficionado), he needs to create an account. He creates a username, and a password, and will be asked to supply an e-mail address to confirm the registration. Random does so.

Now, all someone needs to do to get Random’s credentials is to fake a Realistic Mail from SillyCatForum.com saying “someone has responded to your message. Click <here> to read the reply.” Random clicks, and is presented with the login screen. Random enters his credentials. What R. didn’t notice was that the link actually went to hackersrus.com. But two seconds later, he’s on the real SillyCatForum.com, mildly bemused by there not being a response to any of the messages. “Maybe someone deleted it, or something”, thinks Random, without a thought. Random’s cat-forum credentials are now in the hands of Hackers’R'Us. Security — failed. Faking the realistic e-mail is a child’s play — all someone from Hackers’R'Us needs to do is to create two accounts, reply from one to the other, and presto!, copy’n'paste the e-mail.

“Why is Random clicking on things sent to him in the e-mail?” cry all those holier-than-thou security experts. Because, they seemed to have forgot, the web has trained him to do it. E-mailed links is the normal notification mechanism on the web.

How can we solve this? Simple, do away with passwords.

All sent links should contain “credential information” which will auto-log the user. If the user needs to log in, he can click for “send me credentials”, which will send him a credentialed link. E-mail providers can ask for a cellphone number, to which they will send a one-use (ONE USE) code, which will log the user in. They can also, on creation, ask for an address they can snail-mail a one-use code, in case the user loses his cellphone. (Naturally, you might have to pay for that — but on the off-chance that this happen, to your primary e-mail address, paying 10$ to restore it should be within the means of most people.)

Of course, most sites should not even need that much. Just offer the chances of a Facebook Connect/Open ID log-in, and don’t even try asking for a “credentials”. Let someone else handle credentials for you.

Christopher Armstrong: How to send good unicode email with Python

2010-07-15T16:16:51+00:00


# coding: utf-8

# Python's email API is simple and easy to use!!!!!!!!!!!!!!

# Requirements:
# * UTF-8 headers
# * UTF-8 body
# * prefer quoted-printable to base64 transfer-encoding.
# * Don't escape "From" at the beginning of a line in the message - it's not
#   the 1800s any more

from cStringIO import StringIO
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.header import Header
from email import Charset
from email.generator import Generator


subject = u'Hello あ'
recipient = u'Bあb '
from_address = u'Bかb '

html = u'<html><body>Hey böb!\nFrom Jack, I got enhanced pills!</body></html>'
text = u'Hey böb!\nFrom Jack, I got enhanced pills!'

# Override python's weird assumption that utf-8 text should be encoded with
# base64, and instead use quoted-printable (for both subject and body).  I
# can't figure out a way to specify QP (quoted-printable) instead of base64 in
# a way that doesn't modify global state. :-(
Charset.add_charset('utf-8', Charset.QP, Charset.QP, 'utf-8')


# This example is of an email with text and html alternatives.
multipart = MIMEMultipart('alternative')

# We need to use Header objects here instead of just assigning the strings in
# order to get our headers properly encoded (with QP).
# You may want to avoid this if your headers are already ASCII, just so people
# can read the raw message without getting a headache.
multipart['Subject'] = Header(subject.encode('utf-8'), 'UTF-8').encode()
multipart['To'] = Header(recipient.encode('utf-8'), 'UTF-8').encode()
multipart['From'] = Header(from_address.encode('utf-8'), 'UTF-8').encode()

# Attach the parts with the given encodings.
htmlpart = MIMEText(html.encode('utf-8'), 'html', 'UTF-8')
multipart.attach(htmlpart)
textpart = MIMEText(text.encode('utf-8'), 'plain', 'UTF-8')
multipart.attach(textpart)

# And here we have to instantiate a Generator object to convert the multipart
# object to a string (can't use multipart.as_string, because that escapes
# "From" lines).

io = StringIO()
g = Generator(io, False) # second argument means "should I mangle From?"
g.flatten(multipart)

# Pass the result of this to your SMTP library of choice.
print io.getvalue()

edited: The last part in a multipart message is the preferred one, so I moved the HTML part to the bottom.

edited AGAIN: I found out that in order to avoid ridiculous "From" quoting, I needed to use a Generator object instead of multipart.as_string().

Twisted Matrix Laboratories: Sponsored Development, July 2010

2010-07-12T19:47:25+00:00

Hello readers,

It's that time once again. I've just spent a very productive two weeks dedicated to Twisted developments, thanks to the support of the Twisted sponsors and the Software Freedom Conservancy.

There were quite a few tickets to review when I got started this time. Here's the list (italicised tickets are now resolved):

#411 - Returning a Deferred from the callback of another Deferred too many times results a RecursionError

#1384 - Document the meaning of trial's terminal output stanza

#1702 - Kill Defer.setTimeout

#2485 - 2to3 cross-compilation tools

#2661 - Deprecate IFinishableConsumer so it can be deleted

#2680 - abdapi ConnectionPool doesn't allow for synchronous close

#3546 - Add custom time zone support to twisted.log.FileLogObserver

#3586 - I want to install twisted without a c compiler

#3802 - win32 SerialPort requires a reactor that provides "addEvent" method, but doesn't have a good error message if it gets a different one

#4038 - DeprecationWarning: the MimeWriter module is deprecated; use the email package instead

#4045 - Twisted's Failure fake '_Frame' object lacks f_locals

#4138 - A fresh Twisted checkout should support "setup.py sdist"

#4299 - Remove deprecated twisted.web.trp

#4328 - Deprecate twisted.python.text.docstringLStrip

#4378 - Error handling in NetstringReceiver broken

#4419 - IRCClient.noticed() default is prone to causing loops

#4477 - UNIX ports fail to log their path when stopped

#4511 - trial(1) man page says "I don't know why this is in trial." for --disablegc

#4519 - Delay parsing of request body until Request.args is accessed

#4520 - pb.CopiedFailure.throwExceptionIntoGenerator breaks in Python 2.6.

#4522 - Speed up `reactor.spawnProcess()` by only closing open file descriptors

#4527 - Change either documentation or code of t.i.test.reactormixins.ReactorBuilder.requiredInterface

#4531 - Static analysis of module attributes in twisted.python.modules

#4536 - Credentials materials are compared unsafely throughout Twisted

#4540 - t.n.hosts.searchFileFor does not close the file

#4555 - Replace loop-like uses of map() with a loop

#4566 - Improve lore2sphinx buildbot results for `projects/conch`

#4567 - Improve lore2sphinx buildbot results for `projects/core/development`

Additionally, I had lots of time to spend worked on other tickets:

#526 - [PATCH] make t.w.microdom respect case sensitivity

#989 - Default log observer chokes on unicode messages

#1784 - disttrial --hosts=kunai,takkun,muon twisted

#2179 - Parameterize the root directory of the FTP server for non-anonymous logins

#3169 - Add FilePath.descendant(segmentsToChild) to replace fp.child(a).child(b).child(c)

#3242 - use python 2.5 'spwd' module instead of z3p secret 'shadow' module when available

#3595 - the interaction of the fireOnOneErrback and consumeErrors parameters to DeferredList is not well documented

#3690 - SerialPort never calls connectionLost(reason)

#4300 - Write some unit tests for Deferreds

#4473 - strports.endpoint

#4487 - report multiple tests with the same results at the same time

#4492 - deprecatedModuleAttribute emits warning twice when deprecated module is imported

#4494 - FTPRealm home directory code hardcodes '/home'

#4500 - Use Sphinx for Twisted Documentation

#4501 - trial's error reporting (especially --tbformat=emacs) does not like unicode exceptions

#4503 - domish gets confused by spaces in xmlns names

#4504 - Document reactor.listenMulticast's listenMultiple keyword argument

#4513 - twisted.names server sends incorrect authority section

#4517 - documentation for blockingCallFromThread doesn't exactly say the given callable can return a Deferred

#4528 - t.news.database should use t.m.smtp.sendmail and the email package

#4534 - Update twisted.application.service.IProcess to use Attributes

#4535 - zippath.child('..') does not raise InsecurePath

#4536 - Credentials materials are compared unsafely throughout Twisted

#4539 - fileno() gets called on a ReadDescriptor after removing it from the reactor

#4542 - `LineReceiver` API documentation should indicate when delimiter is stripped etc

#4556 - KQueue reactor misspelled in reactorbuilder

You can find more details about these tickets in the Twisted issue tracker. To look up a ticket, visit http://twistedmatrix.com/trac/ticket/ticketnumber. For example, http://twistedmatrix.com/trac/ticket/4487

Many of these which are still open are now up for review and should be closed over the next few weeks as other developers have a chance to take a look and provide feedback.

A careful reading of this list reveals a few tickets related to our attempts to convert Twisted's documentation from Lore to Sphinx, an undertaking you already know about if you follow the Twisted mailing list. Kevin Horn has spearheaded this project and made great progress towards the goal. Even though the conversion isn't complete yet, there's already some really great looking results.

Jonathan Lange: Documentation again

2010-07-06T13:14:32+00:00

Documenting your code is not important. Well-documented code is not a thing to be desired nor a goal to be sought. A lack of documentation is not a problem to be solved. Everyone who tells you otherwise is lying.

What is important is that other programmers can easily understand your code and the intent behind it, and that you can understand the code and the intent six months later.

Documenting your code is merely a way of achieving that. It might even be necessary for it. But do not confuse interface with implementation and do not confuse what you want with how to get it.

Solve the problem of confusing, opaque code. Aim for code that can be easily understood. Document if necessary.

(it's pretty much always necessary)

Jonathan Lange: When do you start testing?

2010-07-05T14:17:37+00:00

Although I love TDD, I don't always use it for everything. I wrote a Python script the other day to clean up my music collection, and I didn't write any tests for it.

Someone asked me, what's the trade-off? At what point do you start writing tests?

For me, it's at one of a few points:

Somewhere between 100-200 lines of code I start feeling the need for tests
When I start feeling unclear in my head, I reach to tests to make them clear
If someone wants to work with me, tests instantly become mandatory
When I come back to some code after a while
When I notice my first bug

What about you? Do you always do TDD? Do you have any rules-of-thumb for when you do?

Twisted Matrix Laboratories: Twisted 10.1.0 released

2010-07-05T13:19:24+00:00

Twisted 10.1.0 is finally out. It's a month late, but it's finally out there and free from the shackles of British rule.

Highlights include:

Deferreds now support cancellation
A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
inotify support for Linux, which allows monitoring of file system events.
AMP supports transferring timestamps

Note also that this is the last release in which we will support Windows for Python 2.4.

For more information, see the NEWS file.

It's stable, backwards compatible, well tested and in every way an improvement. Download it the tarball, the Windows installer for Python 2.5 or the Windows installer for Python 2.6.

Update: We now have executable Windows installers for Python 2.5 and Python 2.6 to complement the MSIs.

Many thanks to Glyph Lefkowitz, who helped prepare the release, and the PyCon 2010 sprinters, who did so much of the work for this release.

Jonathan Lange: Releasing Twisted, and procedure in general

2010-07-04T13:45:43+00:00

I'm in the middle of releasing Twisted 10.1, following the procedure document that I wrote when I released Twisted 10.0. Having everything written down has been a wonderful aide so far, but doing the release twice has made me think about what it takes to breathe life into old procedure.

The very first step has already been taken, figure out what the process is and write it down. There are still some bits that are unknown and hazy, but I expect they'll be clear by the time we're done. Anyway, writing things down is only the beginning, after that, there are two things that I think we ought to do roughly concurrently.

The first is automate the existing procedure for which here are already many tickets filed, and the second is simplify the process itself. Are all of the steps in the process really needed? Why do we have so many tarballs? Why upload the tarballs to a server that is only periodically mirrored by the actual official download location? Why generate a PDF?

I don't want to start a discussion on the details here, but rather raise the need for Twisted to begin considering this simplification, and for myself to begin articulating some of my own thoughts about process in general.

The final, on-going step in revitalizing procedure is to delegate the task, either to another human being or better yet a machine. I wonder if it would be possible to have the Twisted release done by a monthly cronjob?

In summary, to revive an existing process:

Figure out what the process is and write it down
Simplify the process itself, reducing the number of steps
Automate as many of the steps as possible, thus combining them
Delegate the execution of the process

Moshe Zadka: Apples vs. Oranges

2010-07-02T18:38:55+00:00

Similarities:

Both are fruit
Both have about the same caloric value

Orange advantages:

Significantly more vitamin C

Apple advantages:

Does not require peeling
Bakes well

Warning: if I ever see you saying “it’s like comparing apples and oranges”, I will link you to this post.

Thomas Vander Stichele: Dell R815 power draw

2010-06-30T16:43:04+00:00

Dear intarweb,

I’ve been searching all over for data on what these fancy Dell R815 servers (which can house 4 12-core Opteron CPU’s for a total of 48 cores in 2U) draw in terms of power.

This handy capacity planner from Dell doesn’t have this machine listed yet.

Anyone out there know where I can find this info, or have one of these babies actually running ?

Stephen Thorne: The Days of Horror Story

2010-06-29T22:36:42+00:00

Quite a while ago, I wrote out some of my experiences at a previous employer, as a cathartic release on the tension they caused me. It helped a lot,

Todays The Daily WTF titled "Got Time?" reminded me of one of those horrible hacks I had to deal with when I was there.

We used to have one of those business rules. "X business days from now". Where weekends, and national public holidays, were not considered "business days". Because this business rule was observed in the Oracle Forms interface that was used by our operations staff, a way to take now() + N business days was needed.

# select * from days_of_week order by num;
 day_of_week num
 ----------- ---
...
 31/07/2005  401  
 01/08/2005  
 02/08/2005  
 03/08/2005  402
 04/08/2005  403

In this entirely ficticious example from my memory of this table, that's what the data looked like. All saturdays and sundays and public holidays were listed in the table with a blank 'num'. To find a date 10 business days from now, you would do: select day_of_week from days_of_week where num = (select num from days_of_week where day_of_week = today()) + 10.

Every so often we would "run out of days" and an email would be sent. Then a script would be written to add another batch of days to the table, then we'd be good until we run out again...

I've been at my current employer for over 5 years now, and am extremely happy I left tripe like what I've just dredged out of my memory behind.

Jonathan Lange: Better than a pie chart

2010-06-25T11:26:11+00:00

The bugs page for projects on Launchpad used to have a really ugly pie chart on them. It was a strange little graph, useful because it helped you feel like you were making progress as you pushed bugs from New to Triaged to In-progress to Fix Committed and then off the chart completely with Fix Released.

Pie charts suck though. They use two dimensions to represent one dimension of data, and they are always 100% full. In the case of bugs, there was no way to tell from the pie chart whether there were more or fewer bugs than last week.

Inspired by some sketches from mpt, I decided to implement what I think would be a nice graph to have on Launchpad, but to do so using the webservice API. I've put some code up on lp:everyday, and there's an example of the graph for the whole Launchpad project.

The code works by sucking down all of the bug tasks associated with a project over the API, then storing them in a desktopcouch database, then using that to generate some timeline data. The graph is made using flot, which is excellent.

For those of you who care about Launchpad API details, I've recently added a parameter to searchTasks called modified_since that lets you fetch only bug tasks for bugs that have changed since a particular date. It makes syncing so much faster.

Jp Calderone: filepath 0.1

2010-06-24T01:57:08+00:00

Hello all,

I'm happy to announce the initial release of filepath.

filepath is an abstract interface to the filesystem. It provides APIs for path name manipulation and for inspecting and modifying the filesystem (for example, renaming files, reading from them, etc). filepath's APIs are intended to be easier than those of the standard library os.path module to use correctly and safely.

filepath is a re-packaging of the twisted.python.filepath module independent from Twisted (except for the test suite which still depends on Twisted Trial).

The low number of this release reflects the newness of this packaging. The implementation is almost entirely mature and well tested in real-world situations from its time as part of Twisted.

You can find the package on PyPI or Launchpad:

http://pypi.python.org/pypi/filepath/0.1
https://launchpad.net/filepath

Jp Calderone: python-signalfd 0.1

2010-06-21T22:37:30+00:00

Hello all,

I'm happy to announce the initial release of python-signalfd. This simple package wraps the sigprocmask(2) and signalfd(2) calls, useful for interacting with POSIX signals in slightly more advanced ways than can be done with the built-in signal module.

You can find the package on PyPI or Launchpad:

http://pypi.python.org/pypi/python-signalfd/0.1
https://launchpad.net/python-signalfd

Thomas Vander Stichele: Low cost per core

2010-06-21T11:16:19+00:00

For work, I’m re-reviewing servers and systems with the simple-but-not-easy goal of lowering the basic monthly cost per core. The world of racks, servers, CPU’s and cores is a more complicated place than it was a few years ago, since in a few U’s you can put anything from a bunch of small cheap servers up to monster boards with four CPU sockets and 12 core CPU’s for a total of 48 CPU’s in a 2U space. And a look at a Blade system still makes me drool, although I’m still not sure in what case a Blade really makes sense.

In any case, I tried to do a little comparison (which is hard, because you end up comparing apples and oranges) using Dell’s online configurator.

On the one hand, filling racks with Poweredge R810, 4 x 8 core 1.86 GHz, Intel XeonL7555 machines, gets the price down 26 euro per core per month. Doing the same with Opterons, which surely aren’t as powerful as the Intel ones, I can get a Poweredge R815, 48 cores quad Opteron 2.2 GHz, 6174, for 48 cores total, at 9.53 euro per core per month.

And then I thought a Blade would be an even better deal, but it turns out that it isn’t really. The cost per core, with similar CPU’s, really did come out pretty much the same as the R810 based solution. Probably not that surprising in the end since if you fill a machine with cores, the CPU cost will start dominating. But somehow I thought that Blades would end up being cheaper for maximum core power.

Maybe I’m approaching this the wrong way ? If the main concern is cost per core in a datacenter, how would you go about selecting systems ?

Itamar Shtull-Trauring: A hero for our time

2010-06-21T01:51:47+00:00

I grew up in Kochav-Yair, an Israeli town named after Avraham "Yair" Stern. When an Israeli minister gives a speech extolling Stern's memory, I can't help read and wonder:

"Stern's words of eternal dedication and national Jewish pride have been contaminated in public discourse. They have been turned from words of national consensus to negative remarks seen as 'extreme' and 'uneducated.'"

In 1940 Stern founded a militant organization -- terrorist, even, to his enemies -- dedicated to fighting the British rulers of Palestine and founding a Jewish State. When Ayalon, Israel's Minister of Strategic Affairs, thinks of this eternal dedication and national Jewish pride, was he thinking of Stern's attempted negotiations with the Nazis (before the Final Solution, to be true) who were after all fighting the hated British? Or Stern's organization's participation in the massacre of Palestinian civilians in the town of Deir Yassin?

Twisted Matrix Laboratories: Twisted 10.1.0pre1 released – please test

2010-06-19T12:47:35+00:00

From the dank and grey depths of a London summer, a thin, reedy voice wanders across the waves of the Internet, saying that it's jolly well pleased to announce the first 10.1.0 pre-release.

We would very much appreciate it if you could download the tarballs, test them on your systems, and file tickets as appropriate.

Highlights include:

Deferreds now support cancellation
A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
inotify support for Linux, which allows monitoring of file system events.
AMP supports transferring timestamps

For more information, see the NEWS file.

If there is only positive feedback from the pre-release, we aim to do the final release some time later this month.