Planet Twisted

Paul Swartz: sciyoshi » Using Akismet/TypePad AntiSpam with Django's new comments framework

Thu, 02 Jul 2009 17:22:38 +0000

sciyoshi » Using Akismet/TypePad AntiSpam with Django's new comments framework: Quick drop-in to connect Askimet to Django’s comment system

Thomas Vander Stichele: wordpress I hate you

Thomas — Thu, 02 Jul 2009 12:52:15 +0000

After seeing that I possibly might have had some exploits run on my site again, I upgraded to wordpress 2.8

After reading up on hardening wordpress, the official site mentions AskApache, some plugin that helps hardening. I’m not too sure about it yet, because it wants to be writing .htaccess files in my directories and for that I have to open up more than I would want. But hey, let’s give it a go.

At some point it creates a username and password that you choose. I go on and configure stuff, not knowing very well which of its many modules I’m supposed to activate, or why.

I forget about it, and ten minutes later I check my mail. I have a mail from AskApache. With my login details. And the password in plaintext.

…

Is the WordPress security model just fundamentally broken ?

Thomas Vander Stichele: 5luendo birthday party

Thomas — Thu, 02 Jul 2009 09:37:28 +0000

Yesterday was cause for celebration. We got together to celebrate five years of the Fluendo Group!

The picture quality is bad, and not everyone is in it, but I just took it on a whim after marveling how many people were there. I didn’t even know all of them – yes it’s gotten to that point. 67 months ago I arrived in Barcelona without the company even being created…

We celebrated with mountains of cheese and rivers of wine which in the first year would have lasted us a few weeks and now only lasted an hour.

As magical accidents sometimes happen, today is also the day Fluendo received the certification confirmation from Dolby for our DVD player. It didn’t take long to land in the webshop, so finally our DVD player is up for sale! So you know what to get us for our birthday – a shop checkout with the dvd player in your cart.

Good timing – that means that at this year’s GUADEC/Desktop Summit I know what the answer will be to one of the most asked questions I get.

This is the first GUADEC I’m going to with Kristien in tow, I hope she can manage. I’ll be there from Monday through Friday, because the week is bookended by two weddings. Looking forward to a GStreamer summit on Thursday discussing 1.0…

Thomas Vander Stichele: Tuesday

Thomas — Wed, 01 Jul 2009 10:07:32 +0000

left work around 38 degrees C, got a haircut, went for some great tapas on my own reading Darkly Dreaming Dexter, went to a bar, met up with friends, an impromptu bbq plan was hatched, went to a lovely atico at Portal De L’Angel, barbecued in a soothing summer breeze, rode home on the back of a motorcycle hanging on for dear life. All in all a typical Barcelona summer Tuesday.

Paul Swartz: "Folks say that if you listen real close at the height of the full moon, when the wind is..."

Tue, 30 Jun 2009 17:58:22 +0000

“Folks say that if you listen real close at the height of the full moon, when the wind is blowin’ off Nantucket Sound from the nor’ east and the dogs are howlin’ for no earthly reason, you can hear the awful screams of the crew of the Ellie May, a sturdy whaler Captained by John McTavish; for it was on just such a night when the rum was flowin’ and, Davey Jones be damned, big John brought his men on deck for the first of several screaming contests.”

- this year’s winner of the Bulwer-Lytton Fiction Content, which challenges people to create the worst first line of a novel. Bulwer-Lytton, if you don’t know, was the one who first used “It was a dark and stormy night.”

Paul Swartz: YouTube tribute to MJ (via Slate V)

Tue, 30 Jun 2009 15:45:43 +0000

YouTube tribute to MJ (via Slate V)

Jonathan Lange: Twisted huh?

jml (noreply@blogger.com) — Tue, 30 Jun 2009 15:23:03 +0000

One of the hallway conversations I've had a few times so far at EuroPython goes like this:

A: Twisted's got some good stuff going for it, but I don't like the way it forces me to use Zope interfaces. It's too heavyweight?
B: Huh? What do you mean? It doesn't make you do that.
C: Yeah, there's lots of deep inheritance, which just feels really wrong.
A: Right, I have to write too much code to get something simple working.
B: Sorry, I have no idea what you mean. Can you give a concrete example?
A, C: I can't think of one right now.

To me, this is a very intriguing beginning to a real conversation. Do you know what A and C might mean here? I'd love to understand their points so that I can form an opinion.

Paul Swartz: quote from a mug at the Farmer’s Diner in Quechee, VT

Mon, 29 Jun 2009 16:32:03 +0000

quote from a mug at the Farmer’s Diner in Quechee, VT

Thomas Vander Stichele: Programming contest

Thomas — Mon, 29 Jun 2009 16:27:40 +0000

Jan and Arek entered this year’s ICFP programming contest. It’s a three day programming contest, so this morning they asked if they could swap their Friday project day for today to finish the contest. They seem to be in the top third at the moment.

Arek’s never been a fan of long meetings, but today’s standup meeting was particularly amusing with Arek urging everyone to keep focused and get out there quickly. They had less than two hours left on the clock.

Spot the seven differences

Amusingly, today they came to work with almost the same shirt on, by accident! I can only assume there is a big clothes factory in Poland where they have huge stock of the same fabric…

90 minutes left, knock them dead, guys!

Thomas Vander Stichele: mach 0.9.5 ‘MMM…’ released

Thomas — Sat, 27 Jun 2009 21:57:34 +0000

mach allows you to set up clean roots from scratch for any distribution or distribution variation supported.

This release of mach contains fixes for Python 2.6, and adds Fedora 10 and 11, while fixing the archived Fedora locations.

Get it from the mach project page.

Thomas Vander Stichele: moap vcs bisect

Thomas — Sat, 27 Jun 2009 19:32:07 +0000

Next step on this weekend’s yakshave: a first implementation of moap vcs bisect!

The interface is lifted from git, obviously, since that’s where most people will know the feature from.

I implemented it first with CVS, so I could fix this pychecker bug which was blocking Fedora from bumping the pychecker version from 0.8.17 (3 years old) to 0.8.18. And sure enough, it picked out the commit I broke.

While implementing and while dealing with CVS’s idea of how it stores CVS revisions and dates and so on, I googled and was amused to find this first hit on google for the words cvs and bisect. Clever Andy! And he cleverly sidestepped the problem I wrestled with by making the user specify two dates at the start instead of trying to figure it out from the checkout. And all in lisp too!

Then, to test that my VCS interface was sane, I implemented it for Subversion as well. That took about 15 minutes, since Subversion is much more sane than CVS. I tried the following command on a flumotion checkout:

moap vcs bisect reset; moap vcs bisect start; svn up; moap vcs bisect good; svn up -r 3000; moap vcs bisect bad; MOAP_DEBUG=4 moap vcs bisect run ./test.sh

With test.sh containing
test -e flumotion/component/consumers/gdp/gdp.py

(In other words, look for the commit that added this file.)

Sure enough, it picked out this commit:

[moap-trunk] [thomas@ana flumotion]$ moap vcs bisect diff Index: /home/thomas/tmp/flumotion/configure.ac =================================================================== --- /home/thomas/tmp/flumotion/configure.ac (revision 6909) +++ /home/thomas/tmp/flumotion/configure.ac (revision 6908) @@ -212,7 +212,6 @@ flumotion/component/combiners/switch/Makefile flumotion/component/consumers/Makefile flumotion/component/consumers/disker/Makefile -flumotion/component/consumers/gdp/Makefile flumotion/component/consumers/httpstreamer/Makefile flumotion/component/consumers/preview/Makefile flumotion/component/consumers/shout2/Makefile Index: /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am =================================================================== --- /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am (revision 6909) +++ /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am (revision 6908) @@ -11,7 +11,6 @@

SUBDIRS = \ disker \ - gdp \ httpstreamer \ preview \ shout2 Index: /home/thomas/tmp/flumotion/ChangeLog =================================================================== --- /home/thomas/tmp/flumotion/ChangeLog (revision 6909) +++ /home/thomas/tmp/flumotion/ChangeLog (revision 6908) @@ -1,16 +1,5 @@ 2008-06-20 Thomas Vander Stichele <thomas at apestaart dot org> </thomas>

- * configure.ac: - * flumotion/component/consumers/Makefile.am: - * flumotion/component/consumers/gdp (added): - * flumotion/component/consumers/gdp/gdp.py (added): - * flumotion/component/consumers/gdp/__init__.py (added): - * flumotion/component/consumers/gdp/Makefile.am (added): - * flumotion/component/consumers/gdp/gdp.xml (added): - Add a GDP consumer. - -2008-06-20 Thomas Vander Stichele <thomas at apestaart dot org> - * flumotion/component/producers/gdp/gdp.py: Add error for http://bugzilla.gnome.org/show_bug.cgi?id=532364 </thomas>

So, the feature is ready for testing. It could use some more documenting, and some additional goodies like accepting arguments to moap vcs bisect start for example.

Feedback appreciated!

Thomas Vander Stichele: jhbuild for python

Thomas — Sat, 27 Jun 2009 18:15:44 +0000

Still on the yak shave expedition.

I’ve written some simple scripts and files to set up and build python 2.3, 2.4, and 2.5 in separate prefixes to be able to test my software against these versions.

If you’re interested, in theory it should be really simple:

Install jhbuild
Check out my python jhbuild directory: svn co https://thomas.apestaart.org/thomas/svn/jhbuild/python
run ‘make’

As the README says, this should go on to build all versions of python, and install some scripts.

After that, you just run py-2.3 to go into a shell with Python 2.3 on your path.

Don’t say I never did anything for you.

Thomas Vander Stichele: Python disassembly

Thomas — Sat, 27 Jun 2009 17:19:02 +0000

As part of this weekend’s yakshave, I’m trying to implement a handler for STORE_MAP in pychecker. STORE_MAP is a new opcode in Python 2.6, which speeds up dict building.

So, for the first time I went under the hood of Python and figured out just enough to understand this problem. It was a lot less scary than I thought it was going to be!

It seems that using dis.dis(), one can easily dissassemble any python function into its opcodes. This shows clearly where the behaviour is different between python 2.5 and python 2.6.

Given the following function:
f = lambda: {'a': 1, 'b': 2}

Python 2.6 gives:
1 0 BUILD_MAP 2 3 LOAD_CONST 0 (1) 6 LOAD_CONST 1 ('a') 9 STORE_MAP 10 LOAD_CONST 2 (2) 13 LOAD_CONST 3 ('b') 16 STORE_MAP 17 RETURN_VALUE

I couldn’t find a good description of the output of dis.dis, but in my naiveness I am guessing the following:

The first 1 maps to the line number in the code object where the function is found.
The second column is the offset of the opcode and its arguments
The third column is the opcode name
The next column is the arguments for the opcode; in the case of CONST, it shows the index as well as the const object indexed

I am assuming each opcode takes one address location, and each argument takes two; that maps with the address pointers in front of the opcodes.

The opcodes are all documented.

So, in human terms:

we start with BUILD_MAP, saying that we’ll create a new dictionary on the stack, with 2 entries.
we load the constant with index 0 onto the stack (which happens to be the integer object ‘1′, the value of the ‘a’ key)
we load the constant with index 1 onto the stack (which happens to be the string object ‘a’, the key for the ‘1′ value)
STORE_MAP pops the key and the value off the stack, storing them in the dict. Note that the key was indeed loaded on the stack after the value. Now only the dictionary is left on the stack.
Repeat LOAD_CONST, LOAD_CONST and STORE_MAP for the next set
RETURN_VALUE returns the current value on the stack to the caller

Pretty simple, when you look at it twice.

For the same code, python 2.5 gives:
>>> dis.dis(f) 1 0 BUILD_MAP 0 3 DUP_TOP 4 LOAD_CONST 1 ('a') 7 LOAD_CONST 2 (1) 10 ROT_THREE 11 STORE_SUBSCR 12 DUP_TOP 13 LOAD_CONST 3 ('b') 16 LOAD_CONST 4 (2) 19 ROT_THREE 20 STORE_SUBSCR 21 RETURN_VALUE

This code is slightly longer and more complicated. Basically, LOAD_CONST, LOAD_CONST, STORE_MAP was implemented with DUP_TOP, LOAD_CONST, LOAD_CONST, ROT_THREE, STORE_SUBSCR

It looks like DUP_TOP was needed because STORE_SUBSCR consumes the dictionary off the stack, and ROT_THREE is needed because the arguments are pushed on the stack in the wrong order.

Seems like a nice and obvious improvement once you understand it. An exercise for the reader is to profile whether this change actually makes things faster in practice.

So, where does this leave me for pychecker ? It now looks deceptively simple. STORE_MAP simply pops off two items of the stack. There is nothing to check for, since we’re in a dictionary context. So all my implementation needs to do is to pop 2 items off the stack, and that’s it.

And thus it was commited to pychecker CVS. Popping one item off the yak stack!

Thomas Vander Stichele: This weekend’s yak shave

Thomas — Sat, 27 Jun 2009 12:17:54 +0000

The yak shave started yesterday evening. The yak stack is actually a forked one this time, both of the forks involving pychecker.

I might not remember everything in order, but in a nutshell the stack is something like this:

The original goal for this weekend at the beginning of the week was to release my cd ripper, morituri
Something in its pychecker run did not run with pychecker 0.8.17 (the version still in Fedora, from 2006), and worked with pychecker 0.8.18 (my own build). Fork point 1.
While releasing moap this week, I realized that Freshmeat changed their remote API, which I should fix before I do another release of anything. Fork point 2.

Fork point 1 continues here:

I mailed Fedora’s pychecker maintainer, offering my help, and sending a patch to update to 0.8.18, which I built and installed locally. He mailed back informing me about this pychecker bug with anaconda which was blocking the upgrade. Looking at that bug, it looked suspiciously like a bug triggered by code I added to pychecker last year.
However, I’d like to confirm so in the easiest way. git’s got this great feature, git bisect, and wouldn’it be nice if I could do that on pychecker now ? Hey, why not add bisection to moap ?
CVS is actually not a very manageable VCS if you want to do fancy stuff. It costs me a few hours to figure out how I should get a more-or-less usable date from a CVS checkout. The final solution is similar to the reply to my stackoverflow question
moap vcs bisect run is now implemented and finds the commit that broke anaconda’s pychecking. STACK POINTER IS CURRENTLY HERE.

Fork point 2 continues here:

moap’s make check didn’t work because pychecker complained about the following code:
ef func(): d = { 'a': 1, 'b': 2} print d.keys()

which triggers, in python 2.6, the following warning:
Object (d) has no attribute (keys)
I can’t let myself change code in moap without a working make check, so on to figuring out what’s wrong in pychecker
After lots of debugging and print statements, I figure out that pychecker dispatches Python opcodes, and it silently drops the ones it doesn’t know about. Python 2.6 added a new opcode, STORE_MAP, and so pychecker doesn’t properly handle the stack since it ignores the opcode. I should error out on those opcodes. STACK POINTER IS CURRENTLY HERE
Before I can fix that though, I decide I should make pychecker’s test suite error out on unknown opcodes.
Of course, this will error out differently for different python versions, so I need different python versions on this machine.
I could do that by hand, but I’d also like Twisted’s trial to run the testsuite, which also needs zope-interface in recent versions, which needs setuptools. So hey, why not set up jhbuild stuff to build all these python versions ?
Python 2.3 on my 64 bit machine doesn’t work with setuptools, so the newest Twisted without it is 1.3.0, and it takes a while to figure out why trial doesn’t run my testcases (it ends up being because 1.3.0 trial expects subclasses of twisted.trial.unittest.TestCase, not unittest.TestCase)

I’ll blog about the useful products of my yak shave separately, for those who don’t enjoy descriptions of yak shavings, only outcomes.

In general, I actually enjoy yak shaves. They’re massive treasure hunts, you learn a lot, and you end up fixing a nice bunch of things all over the stack if you persevere. But it’s probably more a mentality thing than anything else, and I really only indulge myself in these in my spare time.

Glyph Lefkowitz: My Threat Model

glyph (noreply@blogger.com) — Fri, 26 Jun 2009 19:47:03 +0000

As a "computer guy", I am sometimes called upon by friends and family to opine on what makes a computer or a network secure. Many of my colleagues are in the same situation. As a "networking guy", I get similar questions from even from experienced "computer guys".

Users have very peculiar ideas about security. Users — and I include myself in this grouping — will become confused even in areas of the computing experience where billions of dollars have been spent trying to make the experience as easy and comprehensible as possible. So it stands to reason that users will often be confused in the area of security, by its nature the least usable and comprehensible area of computing. Attacks are arcane, and, by definition, unexpected ways that software can be manipulated. Yet, these attacks are very relevant to users, who want to understand what, exactly, they are vulnerable to and how to defend against it.

It's basically impossible to try to understand computer security this way, let alone explain it.

The important thing to remember in any security situation is this: what do you have of value, and what is the threat to it? Computer security professionals call the answer to this question the "threat model". Stephen Colbert calls it the ThreatDown. No matter what you call it, it's important to enumerate the threats that you're defending against. Any security measure that you take which is not designed to protect you from a threat which you can, at the very least, imagine and describe, is just extra cost.

In my case, people ask me about three broad classes of user:

users who have networked computers in a home, and use them for checking email, browsing the web, online shopping, and games,
users who have networked desktop computers in a business, and use them for email, web, and business applications, and
users who have networked server computers that are running server applications.

These users all have roughly similar threat models, so I'm going to lump them together for the sake of simplicity, with a nod to a few specific situations.

I believe there are five major types of attacks which threaten average users on the internet today.

Automated attacks that attempt to connect to your computer and exploit a flaw in its operating system or in software that is running a server, and install malicious software on your computer.
E-mail attacks, which attempt to deliver a message which will exploit a flaw in your desktop e-mail client to install malicious software on your computer.
Browser attacks, which attempt to get your browser (either with or without your consent) to visit a site which will exploit a flaw in your browser software to install malicious software on your computer.
Phishing attacks, which attempt to convince you to disclose information about yourself, such as bank account numbers, passwords, or personal details that can be used to access those other things.
Snooping attacks, which attempt to read information in transit between you and another computer. Usually snooping attacks read passwords in an attempt to allow the attacker to impersonate you later.

Attacks 1-3 are all based on the same premise: software is flawed, and sometimes the flaws in it can be exploited to get it to do things that it should not do. There are multiple resources under threat here: your computer itself (i.e. its processing power), your network connection, and the data stored on your computer.

Attacks 4 and 5 are in a different class. They're attempting to get you to reveal information over the network, either with or without your knowledge. The resource under threat here is the information you are transmitting - in most cases, the information being sought is a token which allows you access to some resource; anything from a username and password to your facebook account (which allows for stealing your personal information or impersonating you) to a debit card number (which allows attackers access to the money in your bank account).

I have fairly simple ways to protect yourself against each of these types of attack. In a series of follow-up articles, I'll cover each of those strategies. They should cover a wide variety of attacks with a minimum of effort and cost. Of course, these defenses aren't perfect. It's possible that someone who knows much more about security than I do will correct me, but if so, that's so much the better.

More importantly, I will try to provide simple abstractions that allow you to reason about each type of attack without understanding the intricacies of the technology involved. A major reason I've decided to try to write about this is that security vendors play upon the intuitive (and wrong) understanding that most people have about computer security: equating it with physical security, making their security widget the digital "lock" for the digital "house" of your computer.

I am targeting this series at a fairly nontechnical audience. I realize that my audience here mostly rates pretty high on the nerd spectrum; my hope is that you will agree with what I say sufficiently that this will be a useful resource for you to refer your less technical friends and family. To maintain your interest, however, I'll also be embedding some details about the reasoning behind my own security practices. See you next time!

Update: I accidentally posted a draft of this rather than a final copy; some of the sentences and paragraphs were incomplete. I hope that I've now corrected this.

Jonathan Lange: Argh!

jml (noreply@blogger.com) — Thu, 25 Jun 2009 06:47:44 +0000

I upgraded karmic, and now Do & Banshee are broken. :(

Thomas Vander Stichele: moap 0.2.7 released

Thomas — Wed, 24 Jun 2009 20:41:20 +0000

moap is a swiss army knife for maintainers and developers.

This is MOAP 0.2.7, “MMM…”.

Coverage in 0.2.7: 1424 / 1899 (74 %), 109 python tests, 2 bash tests

Features added since 0.2.6:
- Added moap vcs backup, a command to backup a checkout to a tarball that
can be used later to reconstruct the checkout. Implemented for svn.
- Fixes for git-svn, git, svn and darcs.
- Fixes for Python 2.3 and Python 2.6

I’ve been fixing things left and right for python 2.6, and in the process I noticed that moap hasn’t had a release for over a year. This release contains mostly bug fixes collected over the year, and a new feature that isn’t implemented yet for all VCS’s. Basically it’s an automatic replacement for something I was doing manually every time I removed an old GNOME cvs/svn/git checkout: figure out what’s in that tree that’s not in the repository (diffs, unversioned files, …), so I can delete everything else and free some disk space.

The only problem with this release is that, after doing the release, I noticed that Freshmeat removed their XML-RPC interface. Apparently they have some new kind of interface they want people to use. Sigh. But that means 0.2.8 is right around the corner!

Thomas Vander Stichele: Recovering from a lost /var on Fedora/Red Hat/CentOS

Thomas — Wed, 24 Jun 2009 19:28:47 +0000

Last week, after upgrading my home desktop to F11, I had palimpsest tell me one of my disks was broken on the desktop machine. The desktop is running on two 250 GB drives in software raid. It was time to get new drives.

After a weekend of fiddling with new 1 TB disks for my home desktop, trying failure scenarios, making sure the system can boot from each of the two drives, and waiting for the 4 hour resync of the software RAID in between each step, I finally closed up the desktop machine and cleaned up under my desk again, thinking I was done with my halfyearly messing about with broken disks.

I guess I was tempting faith anyway. Doing a routine operation on my home server after all the configuration stuff I’d done to set up asterisk last week, suddenly an rsync aborted, a journal errored out, a partition changed to being mounted read-only, and the log was full of scary drive errors. Ouch.

Well, that’s why I keep around a big box of old drives - for when some drive fails and I want to tempt fate even more by reusing an old drive that’s probably going to fail real soon too. And anyway, I had just spent my hard drive piggybank on the new desktop drives.

Luckily, I seemed to have a 400 GB SATA drive lying around that used to belong to my media center. I don’t remember why I swapped it out, given that the media center has a 160GB drive for the OS (and two 1.5 TB raid drives for the data, of course), but this was a lucky break. I booted with a rescue cd, and tried copying the root filesystem of my CentOS 5.2 home server partition to this new drive. Which worked fine, except that /var was where I triggered an Input/Output error and some more drive errors in the kernel log.

So, powered off, took out the broken drive, and put it in a USB chassis. The advantage of a USB chassis is that you can easily just replug the drive to try again, instead of locking up your system terribly and having to reboot. Sadly, /var was broken beyond repair. I ran an e2fsck hoping to recover the contents, and that partly worked, but some of the important stuff is missing even from lost+found (apart from the annoying situation where you have to reconstruct file names, which I usually end up not bothering with).

But really, how important can /var be ? Turns out, rather important. As in, you need it to boot in the first place. And also, it holds your rpm database. Crap.

Some Googling gave me some posts on how to reconstruct your rpm database from log files (using –justdb –noscripts –notriggers). But to use those, you actually need those log files. Where are those ? On /var as well. Crap. And they’re not in lost+found either.

Ok, so time to get creative. Here’s what I ended up doing:

create /var/lib/rpm, and run rpm –rebuilddb to end up with an empty rpm database
Based on the contents of /usr/bin, figure out what packages ought to be installed:
rpm -qf /etc/* | grep 'not owned' | cut -f2 -d' ' > /tmp/unowned yum --enablerepo=c5-media --disablerepo=base --disablerepo=updates --disablerepo=addons --disablerepo=extras whatprovides `cat /tmp/unowned` | cut -f1 -d' ' | sort | uniq > /tmp/missing yum --enablerepo=c5-media --disablerepo=base --disablerepo=updates --disablerepo=addons --disablerepo=extras install `cat /tmp/missing`

This works by first listing all files that are not owned by rpm (on the first run, that’s all of them), figure out what packages can provide these files, then installing those packages.
Repeat the process for other important directories, like /bin, /sbin, /usr/sbin, /usr/lib, /usr/include, …
Clean up .rpmnew files that don’t actually contain differences:
find / -name *.rpmnew | sed s/.rpmnew//g > /tmp/rpmnew for c in `cat /tmp/rpmnew`; do echo $c; diff $c $c.rpmnew && mv -f $c.rpmnew $c; done
Same for *.rpmorig:
find / -name *.rpmorig | sed s/.rpmorig//g > /tmp/rpmorig for c in `cat /tmp/rpmorig`; do echo $c; diff $c $c.rpmorig && mv -f $c.rpmorig $c; done
Inspect the remaining ones, and merge changes.

While it’s not an experience I hope to repeat any time soon, it worked out surprisingly well!

Thomas Vander Stichele: Upgrading to F11

Thomas — Wed, 24 Jun 2009 14:44:03 +0000

I managed to completely skip updating to F10. All my machines (work desktop, home desktop, laptop, media center) where running F9 without any real problems I worried about.

But of course I was curious. And, especially with the move to python 2.6, things I care about where bound to break.

So, last weekend I took the plunge, and after little over a week here are my first impressions:

Overall F11 looks slick. Nice work on the artwork! I particularly liked the GDM background, looking like an ancient brushed metal object, reminding me of how I used to love playing Gods by the Bitmap Brothers.
Apparently anaconda now has bugzilla integration, allowing you to file a bug directly from inside anaconda. Luckily for Jeremy (who I assume still maintains it) it has some code in there to look for existing bug entries with the same backtrace. Very nice!
Of course, I wouldn’t have found out if I hadn’t run into exceptions in anaconda. I ran into while setting up two completely new hard disks with 2 software raid partition and LVM on the second one.
I first installed my work desktop, as usual putting the new installation on a separate partition, keeping my old one around in case the install goes wrong or F11 just isn’t stable enough for me. For me, that involves having a /mnt/alpha and /mt/omega partition between which I alternate. At some point I should figure out if other people do this too and if it makes sense for anaconda to support something like this and at least allow me to keep my GRUB configuration for the older installation. For now I do this manually, using a hugeupgrade text file I follow each time I upgrade, accumulating more and more steps each time I perform the procedure.
On my home machine, when I booted into F11, as usual my second monitor didn’t work (I have a Radeon GeCube Pro 2400). My own fault really - I should have tried to get a patch upstream into the default radeon driver the same way I sent a patch for the radeonhd driver that I still use. A rebuild later, I at least had the old radeonhd driver rebuilt to get my second screen working again.
Having the second screen now made me change my mind completely about the GDM wallpaper. That lion on the right hand side that I didn’t see before completely ruins the style for me. Sorry!
Upon logging in to the work desktop, I had no network. Completely puzzled as to why, until I figured out that I had to actually right-click on NetworkManager’s tray icon, choose to configure, and activate eth0 by default. After some browsing it seems that this was a deliberate choice to increase security. While I can possibly sympathize with the motivation for doing so, it really is terrible to change this by default and not provide *any* indication during or after installation. At the very least, the following things could have been done:
1. provide a clear notice during installation, and allow a user to choose to enable it anyway, assuming the security risk
2. the same, but during firstboot
3. after logging in, having the network manager tooltip say ‘the network is disabled by default in this new release, here’s how you enable it
I am not entirely sure what the security problems are with enabling the network after installation. The default firewall is pretty locked down, SELinux is enabled by default, and there’s no way I can install updates without the network anyway. But I’m sure that I could find huge bikeshedding threads on fedora-devel about this if I really cared why this was decided.
Upon logging in to the home desktop, I was greeted with a tooltip saying that one of my drives was going bad. That was a nice touch! Really good idea to have something like that be monitored by default. This prompted me to ponder to finally replace my desktop’s 250 GB PATA drives with real SATA drives - a story for another post.
Various deprecation warnings pop up running various Python programs, including my own. Flumotion needed a patch for running against 2.6 (I rebuilt and pushed to F11). So I have some cleanup ahead, and I should revisit pychecker soon.
The first piece of functionality I checked was Evolution’s Google Calendar integration. It still seems a bit shaky, given that I had to restart Evolution a few times as it froze doing stuff with the net, but it does seem to work. That means I will finally be able to accept work invitations done through Outlook and get them on my Google Calendar! Awesome. Now if only I didn’t have to manually configure each of the ten calendars I’m interested in…
At work, when I played a video using XVideo, my machine instantly froze. Seems to be a known bug. The intel drivers are being rewritten. I’ve never quite understood why rewriting is an excuse for breaking stuff that worked (I should check if Firewire video finally works reliably now when I have the chance, for example), but all in the name of progress I guess.
I don’t know why it’s happening, but once in a while my screens blank. Even in the middle of doing stuff. If I were a gamer I’d be hugely annoyed as my character would be shot through the head in that split instant. The closest bug I can find is this one, where I commented. Hugely annoying bug because I don event know how to begin debugging a bug like this that I can’t catch in the act.
PulseAudio integration in GDM seems a bit fragile. I have my pulseaudio configured to send audio to my media center pulseaudio server. Sometimes, after choosing a username in GDM, it doesn’t manage to play the audio sample related to that action, and then GDM is stuck there not showing me the password entry dialog. Pretty sure it’s due to blocking on pulseaudio, because when I kill it the password dialog appears. Pretty painful bug for new users though.

All in all, not a bad first week experience, and seems like a solid release. Now, off to rebuild bits and pieces, and clean up Python 2.6 deprecation warnings…

Glyph Lefkowitz: A Chicken in Every Pot and a Python on Every Port

glyph (noreply@blogger.com) — Tue, 23 Jun 2009 17:12:26 +0000

Twisted Matrix Labs is bent on world domination. We spend so much time working at the level of fine-grained minutæ that we sometimes forget the overarching plan. So here's a step back: what is Twisted for?

Most people know at least part of Twisted's origin story. I was working on a text-based game, and I wanted a networking layer, and discovered that there was really nothing available. I decided to write something general to base the game's networking core on, so I would be able to use production-quality protocols rather than toy "just for this game" stuff.

However, it wasn't just about the game. That's a good thing, too, because the game has been falling behind quite a bit. My game was just one example of code that you might want to write that could talk to a network, and my frustration was that despite large amounts of code being written to talk to networks, very little of it was directly usable by other code, and even less of it could be combined. A major culprit here is that most networking software is written in C, where there is a stark contrast between "application" and "library"; a conscious, deliberate effort has to be made to expose functionality as a library, both in the code and in the build process.

Then there's the security situation. A 2007 analysis of different types of vulnerability reports that buffer overflows were only recently overtaken by web application attacks, but are still the #2 for vulnerabilities overall, and #1 for OS vendor advisories. Again, why is everybody still using all this network software written in C? You can't even have a buffer overflow in most high-level languages. (The even more depressing thing here is that, as the web development community has moved to higher-level languages, the majority have moved to the worst possible high-level language. The vulnerability listings for web applications in that same report mostly have to do with flaws in PHP.)

So, the goal of Twisted is to provide a high-quality, high-level, secure implementation of every protocol spoken on the Internet. We've achieved a lot, but there's still a long way to go. Netcraft no longer seems to have any data on Twisted, because it is too low in the "other" category. There's no site I'm aware of that does server market-share for DNS servers, but I'm betting that Twisted remains low in this category as well.

I believe Twisted remains popular in a growing segment of the network applications market, that is to say, applications that don't fit neatly into a single protocol. If you want to control DNS, HTTP, SIP, and XMPP from a single program, it's far easier in Twisted than in anything else. However, I think we can do better. I want Twisted to take on BIND, Apache, Asterisk and jabberd directly as a server in its own right, not an integration mechanism or library.

One major area where Twisted is lacking is in focused, purpose-specific developers. Apache has lots of people who are only interested in HTTP, Asterisk has people who are only interested in SIP, and libpurple has lots of people who are only interested in chat. Twisted, by contrast, has excellent generalists, but few individuals to focus on the individual details of a single application. I'm not sure how to recruit people who have that kind of monomaniacal focus to maintain individual components. I think it's the details that such people would notice which is holding us back from being more competitive in the general server "market", such as it is.

This is a chicken-and-egg problem. People interested in chat clients will often find libpurple before they find Twisted Words; people interested in web servers will often find Apache before they find Twisted Web. Part of this is the lack of relevant conveniences and features, but probably an even bigger part is just our lack of a coherent web presence for those interest groups. While I think that a lot of people looking for these things would be delighted to find something as easy to script and re-shape as Twisted is, they don't start out by looking for an omni-server platform.

So go update the web site, and take over the world!

Duncan McGreggor: Mac OS X: Execute Shell Commands via Icon-Clicks

Duncan McGreggor (oubiwann@gmail.com) — Sun, 21 Jun 2009 04:09:57 +0000

My main development machine is a custom PowerBook running Ubuntu natively. I use it when I'm sitting on the couch, my office comfy chair, the futon, floor, etc. Every once in a while, though, I want to work at a desk from my 24" iMac. Just to mix it up a little. However, that box is my gaming and web-browsing machine: it runs Mac OS X and that's the way I want to keep it. So, if I'm going to do work on the iMac, I need to ssh into the machines that have the environments set up for development.

In the course of an average day of writing code, I'll connect to anywhere from 1 to 5 remote machines open up 5-10 ssh sessions in a terminal to each machine. If I'm at the iMac, this get's tedious. Today, it got tedious enough for me to do somthing about it. Here's what I want: to click on a Terminal icon and have an ssh connection automatically established to the box I need to work on. This it pretty easy on Linux and Windows, but I had no idea how to accomplish this on a Mac until tonight.

I thought I'd share my solution; others may like it... but I'm betting there are some pretty cool ways of doing this that didn't occur to me -- so feel free to share yours!

Profile Hack

From previous messing about with the open command, I knew I could open Terminal.app from the terminal:

open -n "/Applications/Utilities/Terminal.app"

This got me part way there... if only I could dynamically execute a command upon login... so, yeah, I did something nasty:

vi ~/.bash_profile

And then:

if [ ! -z "$REMOTE_CONNECTION" ]; then
    ssh $REMOTE_CONNECTION
    REMOTE_CONNECTION=""
fi

.command Files

I was stumped at that point, until some googling revealed a nifty trick I didn't know about:

Create a new file in your favorite editor, using the .command extension
Add the commands you want executed
Save it and chmod 755
Double-click it and enjoy

So here's what I added to rhosgobel.command:

REMOTE_CONNECTION=rhosgobel \
   open -n "/Applications/Utilities/Terminal.app"

The Obligatory Icon Tweak

I then used the standard "Get Info" trick of icon copying: "Get Info" for Terminal.app, copy icon, "Get Info" for all my .command files, paste icon.

Usage

Now, I just click my "Shells" menu, choose the destination, and start working on that machine. A new window or new tab opened with that instance of Terminal.app will give me a new session to that server, without having to manually ssh into it -- this is even more convenient than having an icon to double-click!

One bit of ugly I haven't figured out how to remove: when I open a shell to a remote server, there's another shell opened at the same time with a [Process completed] message.

Kevin Turner (LJ): brunchbox burger with pineapple

Fri, 19 Jun 2009 08:01:09 +0000

There's an open-source open-space conference Friday. It's possible I want to talk about something but have forgotten what it was. What do I host a session on?

Duncan McGreggor: A Sinfonia on Messaging with txAMQP, Part III

Duncan McGreggor (oubiwann@gmail.com) — Thu, 18 Jun 2009 21:28:53 +0000

A Sinfonia on Messaging:

Before we play our third voice in this three-part invention, we need to do some finger exercises. In particular, let's take a look at the concepts and tools we'll be using to implement and run our kilt store messaging scenario.

Messaging

The RabbitMQ FAQ has this to say about messaging:

Unlike databases which manage data at rest, messaging is used to manage data in motion. Use messaging to communicate between and scale applications, within your enterprise, across the web, or in the cloud.

Paraphasing Wikipedia's entry on AMQP:

The AMQ protocol is for managing the flow of messages across an enterprise's business systems. It is middleware to provide a point of rendezvous between backend systems, such as data stores and services, and front end systems such as end user applications.

AMQP Essentials

AMQP is a protocol for middleware servers -- servers that accept, route, and buffer messages. The AMQP specification defines messaging server LEGO blocks that can be combined in various ways and numbers, achieving any manner of messaging goals, whose final forms are as diverse as the combinations of the components.

For the visually inclined, below is a simple diagram of the AMQ protocol. I've put multiple virtual hosts in the diagram to indicate support for multiple server "segments" (domains in the most general sense). There could just as easily be multiple exchanges and queues in each virtual host, as well. Likewise for publishers and consumers.

I highly recommend reading the spec: it is exceedingly clear at both intuitive and practical levels. To better understand the diagram above, be sure to read the definition of terms at the beginning as well as the subsections in 2.1 about the messaging queue and the exhange. Don't miss the message life-cycle section either -- you'll be reminded of circuitry diagrams and electronics kits, which is what AMQP really boils down to :-)

The Advanced Messaging Queing Protocol specifies that the the protocol can be used to create exchanges, message queues, chain them together, and do all of this dynamically. Any piece of code that has access to an API for your AMQP server can connect to it and communicate with other code -- using or creating simple messaging patterns or deeply complex ones. And everything in between.

RabbitMQ Quickstart

RabbitMQ is a messaging system written in Erlang, but in particular, it is an implementation of AMQP. The RabbitMQ web site provides documentation on installing and administering the messaging server. I run mine on Ubuntu, but since I've got a custom Erlang install, I didn't install the package (I dumped the source in /usr/lib/erlang/lib). To participate in the code play for this blog series, you'll need to install RabbitMQ.

Once you've got it installed, you'll need to start it up. If you've used something like Ubuntu's apt-get to install RabbitMQ, starting it up is as simple as this:

sudo rabbitmq-server

If you've got a custom setup like mine, you might need to do something like this (changing the defaults as needed):

BASE=/usr/lib/erlang/lib/rabbitmq-server-1.5.5/
BIN=$BASE/scripts/rabbitmq-server

RABBITMQ_MNESIA_BASE=$BASE/mnesia \
  RABBITMQ_LOG_BASE=/var/log/rabbitmq \
  RABBITMQ_NODE_PORT=5672 \
  RABBITMQ_NODENAME=rabbit \
  $BIN &

A txAMQP Example

Now that we've got a messaging server running, but before we try to implement out kilt store scenarios, let's take a quick sneak peek at txAMQP with a simple example having the following components:

a RabbitMQ server
a txAMQP producer
a txAMQP consumer

From reading the spec, we have a general sense of what needs to happen in our producer. It needs to:

connect to the RabbitMQ server
open a channel
send a message down the channel

Similarly, our reading lets us anticipate the needs of the consumer:

connect to the RabbitMQ server
open a channel
create an exchange and message queue on the RabbitMQ server, binding the two
check for in-coming messages and consume them

I have refactored some examples that the author of txAMQP created and I've put them up here. Once you download the three Python files (and the spec file, one directory level up), you can run them in two separate terminals. In terminal 1, start up the consumer:

python2.5 consumer amqp0-8.xml

In terminal 2, fire off a message:

python2.5 producer amqp0-8.xml \
 "producer-to-consumer test message 1"

After running the producer with that message, you should see the same text rendered in the consumer terminal window. You can also fire the message off first, then start up the consumer. The message is sitting in a queue on your RabbitMQ instance and will be available to your consumer as soon as it connects.

Now that you see evidence of this working, you're going to be curious about the code :-) Go ahead and take a look. There are lots of comments in the code that give hints as to what's going on and the responsibilities that are being addressed.

If you are familiar with Twisted, you may have noted that the code looks a little strange. If you're not, you may have noticed that the code looks normal, with the exception of extensive yield usage and the inlineCallbacks decorator. Let me give a quick overview:

Ordinarily, Twisted-based libraries and applications use the asynchronous Twisted deferred idiom. However, there's a little-used bit of syntactic sugar in Twisted (for Python 2.5 and greater) that lets you write async code that looks like regular, synchronous code. This was briefly explored in a post on another blog last year. The Twisted API docstring for inlineCallbacks has a concise example.

Briefly, the difference is as follows. In standard Twisted code, we assign a deferred-producing function's or method's return value to a variable and then call that deferred's methods (e.g., addCallback):

def someFunc():
    d1 = someAsyncCall()
    d1.addCallback(_someCallback)
    d2 = anotherAsyncCall()
    d2.addCallback(_anotherCallback)

With inlineCallbacks, you decorate your function (or method) and yield for every deferred-producing call:

@inlineCallbacks
def someFunc():
    result1 = yield someAsyncCall()
    # work with result; no need for a callback
    result2 = yield anotherAsyncCall()
    # work with second result; no need for a callback

Visually, this flows as regular Python code. However, know that the yields prevent the function from blocking and execution resumes as soon as the deferred has a result (which is assigned to the left-hand side). Since this latter idiom is used in txAMQP, I use it in the examples as well.

Next, we finally reach our implementation!

References

Duncan McGreggor: A Sinfonia on Messaging with txAMQP, Part I

Duncan McGreggor (oubiwann@gmail.com) — Thu, 18 Jun 2009 17:59:07 +0000

A Sinfonia on Messaging:

Prelude

A complete messaging solution is a three-part invention of business need, architecture, and implementation. In its final form, these three voices blend in harmony, with each one taking a dominant role depending upon which part of the solution one examines.

Neither have I the ability nor skill to seamlessly weave three concepts together while clearly explaining their roles. Therefore, I will separate out the voices from each other and leave it as an exercise for the reader to construct an application and practice the principles involved, thus experiencing well-earned contrapuntal pleasures first-hand.

Introduction

As computing and data exchange systems increased in complexity over the past 30 years, so has the need for improvements -- and where possible, simplifications. Some of these efforts have been focused on decentralization of communications (shared, distributed load) and decoupling of messaging from applications (removing redundancy and increasing delivery speed/throughput). The first steps towards this were made in the 1990s with explorations in "middleware" application universe.

Messaging, as we now refer to it in the industry, arouse from those middleware adventures: out of the business drive to refactor old software as new services to wider, more sophisticated audiences. With many new services replacing a single, monolithic application, formal and well-architected solutions were needed for creating, editing, and deleting shared data.

From Wikipedia:

Message-oriented middleware (MOM) is infrastructure focused on message sending that increases the interoperability, portability, and flexibility of an application by allowing the application to be distributed over multiple heterogeneous platforms. It reduces the complexity of developing applications that span multiple operating systems and network protocols by insulating the application developer from the details of the various operating system and network interfaces.

AMQP (Advanced Message Queuing Protocol) is one of these protcols.

In a recent blog post about Ubuntu as a business server, Vaughan-Nichols provides evidence for Ubuntu's and Canonical's commitment to enterprise, saying "... the new [version of] Ubuntu also includes AMQP [...] support. AMQP is an important set of middleware and SOA [...] protocols."

AMQP has demonstrated itself as a compelling protocol for messaging solutions, even to the point of being included in two Linux distributions. The code included in this blog series is txAMQP, an asynchronous Python AMQP library built with Twisted.

Note that last year I had planned to write a blog series on messaging with Twisted and XMPP, but was unable to as a result of time constraints. These days, I'm working with AMQP instead of XMPP, but I still hold some hope that I'll be able to write an anolog for this series from the perspective of XMPP.

The Voice of Business

Dipping into the future, one of the goals for AMQP as developed by a special interest group is the following:

Decentralized, Locally Governed Federated Mesh of AMQP Brokers with standardized Global Addressing. The killer application for AMQP is transacted secure business messages between corporations - e.g. send a banking confirmation message to confirms@bank.com [...]

I find this rather exciting due to my interest in ultra large-scale systems; scenarios like the one described above are the seeds for tomorrow's ULS systems :-)

For now, though, let's look at a more immediate use for AMQP: a messaging protocol for shared services between departments in a small store. In this exercise, the voice of business is the primary melody; everything else (architecture and implementation) is done in support of this theme.

An Example

Fhionnlaidh MacGrudder creates hand-made kilts to order. He's got a sales guy who works with movie costume design shops and the like. He's got a web girl who wrote and maintains a custom store front app. He's got a friend who does shipping and billing for him (as well as some other local Glen Orchy artisans). Until now, these three business "groups" associated with the kilt shop have been maintaining their own records, sometimes copying and updating them manually from each others' various export files.

Fhionnlaidh's niece Fíona is programmer, business student, and is dating shipping guy's son. Horrified by the inefficiencies in her uncle's business processes (and tired of her boyfriend's father's complaints), she has proposed the following:

sales guy will maintain customer contact info offer this as a service to the store and billing
the web store will dynamically update displayed data when sales guy changes it
the CRM will dynamically update displayed data when a web store customer updates their info
MacGrudder will have a new web page he goes to where all pending orders are presented with their full details; changes can be updated by a customer in real-time until MacGrudder has started working on the order
billing/shipping guy will be notified instantly as soon as MacGrudder marks a kilt order as completed

This setup has the following benefits:

Contacts will be maintained in a single data store
There is zero latency between customer-driven updates and sales guy-driven updates
Customers have increased post-purchase flexibility with their orders
Shipping guy can plug into MacGudder's messaging and be notified when packages are ready for pickup
Everyone has more time for buttered scones and tea (especially shipping guy, who will no longer be making unneeded trips down the glen)

The following changes will be made to the current software:

Contacts will need to be merged into the CRM
A read/write data service for the contacts will need to be created
The CRM front-end will need to be upgraded to an AJAX-enabled version
The web store app will need to be updated to support AJAX
A new page will be created which displays the status of all orders and allows MacGrudder to change an order from "pending" to "in-progress" to "completed"
The current "new order" email notification code in the web app will need to be changed so that it uses the same messaging as MacGrudder's status page
A new service needs to be created for shipping guy so that he can choose to be notified about pending pickups by email or he can check a web page or even make a query directly to the service, thus preventing unnecessary trips to MacGrudder's isolated little shop
After all the work is done, somone's going to need to order more scones

This example is not meant to fully justify messaging for businesses, but rather to provide a simple use case for which we can write some simple (and less than robust) code. It is a toy, but a conceptually useful one with a solid, concrete foundation.

In the next installment, we'll review the business process (with diagrams!) and the explore the architecture of the system, before and after. Another post will take that architecture and combine it with MacGrudder's already extant infrastructure, reusing as much as possible. With that in place, we will have the opportunity to look at some RabbitMQ basics and some actual txAMQP code.

References

SOA in Practice
Message-oriented middleware on Wikipedia
AMQP on Wikipedia
The AMQP Spec
A synchronous Python library for AMQP
The async Python library for AMQP
A txAMQP example
An intro to messaging and py-amqplib

Duncan McGreggor: A Sinfonia on Messaging with txAMQP, Part II

Duncan McGreggor (oubiwann@gmail.com) — Thu, 18 Jun 2009 17:58:53 +0000

A Sinfonia on Messaging:

After writing the last blog post, I found a fantastic site that focuses on messaging in the enterprise. I have really enjoyed the big-picture overview I get from some of the Martin Fowler signature books in this series, so I ordered a copy of this one too.

On the web site, the authors give a nice example for messaging integration in Chapter 1. They provide a more detailed, supplier-version of the kilt store (we're doing "manufacturing" as opposed to distribution) with "Widget-Gadget Corp", but the basic principles are the same. I highly recommend reading that entire page. I used it as the basis for much of this post.

Business Process Overview

At a top-level, we have the following business process for MacGrudder's kilt store:

These are represented by a sales guy or a web store, a third-party billing service, MacGrudder, and a third-party shipping service.

Up until now, the sale process could be either a user deciding to buy something in the online store or the sales guy engaging with a customer. Both generated orders; neither shared resources. The web app interfaced with the payment gateway operating by billing/shipping guy. The sales guy had to call in his orders to the billing/shipping guy. Once orders were charged/approved, a printout was handed to MacGrudder, who then created the ordered kilt. Once completed, he'd set it aside for shipping guy to come box it up and slap a label on it.

The Voice of Architecture

We're now ready to weave in the second voice of our three-part invention. MacGrudder's original infrastructure consists of silos of applications, functionality, data, and process. We want to interconnect these separated areas in order to reduce long-term overhead incurred by redundant components and data. Practically, we want to see the following changes:

Unified orders: at the end of the sales process, there should be one abstraction of the "order", regardless if the source was the web store, a phone call, or the sales guy. The order abstraction will be a message (or series of messages, for orders with multiple items; we'll be addressing only the simple case of a single item).

Unified validation and billing: at the end of the order creation process, there should be a validated order or an invalid order (e.g., if there were insufficient funds). At the end of the billing process, there should be an approved order that has be paid for via a single means (e.g., a single payment gateway, without bothering billing guy for manual entry). Additionally, once an order has been validated, messages should be sent to other components in the system.

Unified status: at the end of the manufacturing process, both the shipping guy and customers should be aware that the product has been completed and is ready to be sent: the shipping guy can connect to our messaging system (probably via a service) and the customer can be notified by email or by checking the order status in the web kilt store.

In the next installment, we will finally start looking at the code. We'll look at the "unified orders" messaging solution after covering some basics with RabbitMQ and Twisted integration, and then see how far we get with implementation details and descriptions. Unified validation, billing, and status might have to be pushed to additional posts.

References

Enterprise Integration Patterns, Chapter 1
SOA in Practice

Moshe Zadka: Hacker News: Groupthink’R'Us

moshez — Thu, 18 Jun 2009 06:42:45 +0000

Do you want to write a post that will appear in Hacker News?

Here are a few blog headers that are sure to win you votes. Just write some filler and publish:

Any post containing the words “iterate” and “learn”. Bonus points if you can weave in words about how failing is good and failing fast is better. In your post, deride the need for any planning with an explanation that “you cannot assume anything”, and explain how the best way to succeed is to iterate quickly.
Explain why ad-supported products are a bad idea. Bonus points if you quote pointless statistics, explain about freemium or announce why nano-, pico-, or milli- payment systems will finally enable people to pay for stuff.
Explain why start-ups are better than big companies. Use one of the common themes of flexibility and ability to take risks.

Seriously, all of these subjects have been done to death. We get them, really we do. The pros and cons have been explained, and making a more extreme post with an even more black-and-white picture is not what we want. And honestly? Debunking those is just as bad (done to death, we know it). Feel free to write such a post if it adds to your ego to have something rise to the top of HN. I’ll be here, waiting for more interesting posts there…

Glyph Lefkowitz: Why Phones Lost

glyph (noreply@blogger.com) — Wed, 17 Jun 2009 23:39:35 +0000

This morning I was reading Antonio Rodriguez's "Path Dependence And Smartphones" post, where he muses about different perspectives on the "smartphone" market; in particular the European / American divide outlined by Tomi Ahonen in "A Tale Of Two Smartphones". Antonio tends to think a few years ahead of his time, so I'm always interested in his take on trends like this. It seems that he's very cautiously optimistic that the "user customized mobile computer" thing is an important trend, but he also notes that Mr. Ahonen believes that the "[smartphone] operating system and any applications had ZERO bearing on the decision [of which phone to buy]. Not for mass market consumers"; and maybe we're looking at this the wrong way.

As I started thinking about my own reactions to this, I realized: I've heard this tune before. Remember when pundits used to talk about "convergence" between television and computers? Since the advent of the computer, futurists have been predicting the dawn of a strange new device: part computer, part television, part telephone, part vacuum cleaner. What would it look like?

Well, a few months ago, I feel like Paul Graham answered that question pretty definitively. For years, we've wondered what you would get if you mixed computers and televisions. In Mr. Graham's words: "We now know the answer: computers."

As a child of the digital age — I've been using computers with keyboards, mice, color displays, and networking almost as long as I've been able to read — I always found this conclusion somewhat obvious. A few of the early computers that I had the opportunity to use, an Atari 800 and an Amiga 1000, both used televisions as monitors, so I have always thought of a television as an output device — you could plug it into a VCR, a computer, or a cable box, but fundamentally it was just a bag of pixels.

I remember the exact moment that it dawned on me that computers were going to take over from TV: I was 14 years old, playing Myst for the first time, and monkeying with the configuration of system extensions that were loaded on my computer in order to squeeze the last few ounces of performance so that the video clips in the game would play smoothly. I remember thinking, "This is just a problem with RAM and CPU. In a few years computers will have so much of both that you'll be able to play full screen video without even turning off any extensions."

I, uh, had a pretty limited idea of how optimization worked at the time (the video was still jerky even after I turned off all my extensions), but I am frequently reminded of this insight when I am watching YouTube movies on my LCD "television". That television, by the way, is just a monitor for a computer that runs Ubuntu so I can watch Hulu and YouTube. I think maybe I have cable bundled with my internet service, because it's cheaper that way but I've never plugged it in to anything.

I didn't realized how powerful articulating this particular idea is until recently though, because I didn't realize just how much money is spent protecting obsolete infrastructure from the relentless onslaught of microprocessor technology. Phone companies — which, increasingly, are combination cable/phone/internet companies — are stuck between a rock and a hard place. As Internet service providers, they are a facilitator of the transition, and make a huge amount of money selling network services to people to make their computers more useful. But, as cable companies, they want people to think that television is some special, extra expensive thing that needs to be delievered over a different cable. As phone companies, both wired and wireless, they want people to think that voice and SMS data are special, extra expensive things that need to be delivered via special, magical wireless signals that can't be reduced to the simple and banal "internet". At the same time, especially as wired phone companies, they want the cost savings that comes from doing all of their networking as plain old IP, with no actual pesky phone circuits to worry about. Except they still want to sell you the service as if the phone were a different thing from your "internet" connection. (Whenever I see an ad for Comcast Digital Voice, I can't help but think, "Do you think that's air you're breathing?".)

There's still a lot of speculation in each of these industries that some new, hybridized technology is going to create a special and unique relationship with the consumer. But that's one thing Mr. Ahonen got right: the consumer doesn't care about your "operating system". They don't care about your "applications". They just care what they can do with their technology, and they care how much it costs to do so. The thing is, computers do more, and cost less, than any other specialized, dedicated technology. If your industry is fighting computers in the hopes of holding on to some residual value, you are going to lose. Here's a simple formula:

Computer + X = Computer

Consider a few specific examples: the convergence of computers with television has resulted in three general categories of technology: YouTube (and other flash video sites, such as Hulu), Tivo (and other DVRs), and digital cable boxes with on-demand technology. YouTube is a program you run on a computer to watch videos. A Tivo is a computer (running linux) that is running a program to let you watch and record videos. And those cable boxes are computers (running some crappy cut-down embedded OS) that let you watch videos on the cable company's terms. Whether or not your customers care about choice, all these things are computers because it's fundamentally cheaper and easier for the vendors to produce these things out of commodity PC components rather than specialized "media" electronics.

But Mr. Graham neatly outlined that trend already, so let's move on to other industries. What happens when you add a computer to an accounting ledger? You get a computer program (like BusinessMind, or QuickBooks) which lets you do accounting on your computer. Computers and books? The Kindle, which is a hand-held computer that lets you read books. If you look a bit deeper, you'll find that the Kindle is actually a computer program that can run places other than its dedicated device. Only crafty marketing folks prevent it from being more widely accessible; say, on your desktop or "television".

Let's get to the point of this whole schpiel: phones. Phones are already computers, pure and simple. They are just small computers with microphones and speakers, and soon, cameras and screens. You can look at the exciting developments in the world of phones and see that this is so. What are the hottest phones of the last few years? The iPhone, which is a small Macintosh computer, and the G1, which is a small Linux PC. Microsoft would have you believe that their small Windows PCs are equally relevant, even if they are clearly an also-ran in this category. (Disclosure: I actually have a Windows Mobile phone, and I'm fairly happy with it, but I'll be glad when I can finally ditch it for Android.) None of these "phones" does anything interesting in the area of phone-ness. They don't have particularly awesome voice quality or particularly awesome reception or even particularly awesome voicemail, although the iPhone certainly raised the bar. They're just better computers than the previous generation of "phones"; computers that can run a wider variety of programs.

However, phones are still computers with weird restrictions, restrictions that are purely a function of the "path dependence" that Antonio mentions, which dragged them out of the muck and the mire of the telecom industry. SMS is my favorite example of this: 10¢ to send a 140 character message. How much does a tweet cost on twitter? How much does an instant message cost on AIM, or Google Talk, or any IRC network you please? If you were billed at SMS rates to read this post, it would have cost you $10; the cost of a decent paperback. I know I'm wordy, but I'm not that wordy. If you were charged at SMS rates for a day's worth of casual web browsing, images and all, you'd probably have to take out a mortgage just to pay for it. Phone companies have been able to sustain the myth that SMS data is somehow special and deserves to be treated as sacred and precious, fully 1000 times more expensive than the regular bytes you get off the internet, even at the obscene prices they charge for usage-based data plans.

SMS is particularly egregious, but voice isn't that much different. Phone companies charge such ridiculous rates for "voice" data that Skype built an entire profitable business around giving people the same service for free, and only making money by piggybacking on the phone companies' greed and charging you when sending voice messages over phone networks rather than the internet. I can't imagine casting the wasteful overhead of legacy phone networks in any sharper relief.

So, we're not there yet, but the market pressure is tremendous to treat data as data, regardless whether it's voice, or SMS, or IM, or "internet" (in other words: everything else, including voice and SMS and IM messages which are sent via different mechanisms). Until the advent of the recent crop of smartphones, it was difficult and expensive to get an unlimited data plan. Now, unlimited data plans are the norm, except for "tethering" - using your phone as a proxy for your laptop. The phone companies are still desperate to convince you that you should pay $60 per month for the privilege of having a USB dongle that you can plug into your laptop rather than just using the mobile IP endpoint — which, by the way, probably aleady has a USB port — that's already in your pocket.

The "mass market" user might not care about operating systems or APIs, but they do understand that a bill with seventeen different break-out metered sections is a bald-faced attempt to rip them off, and a flat-rate or easy to understand pay-as-you-go plan with one number on it is better.

To the extent that phones are not yet interchangeable, unrestricted mobile IP endpoints, it is due to the high barrier to entry to telecom providers, lack of regulation of misleading pricing schemes, and the symbiotic relationship between government and the telecom industry. However, if one wireless carrier moves to provide simpler billing with more features, the others are forced to follow suit - even more so than cable companies and land-line providers, who can hold their customers hostage via development deals with local governments. So, this progression is happening, albeit slowly. For example, when AT&T introduced its iPhone plans, many of the other metered PDA and Blackberry plans, both on AT&T and other providers, began receding from their marketing materials.

Fifteen years ago ... ugh, I feel old. Let's say ... ten years ago, my computer was barely powerful enough to dedicate all of its processing power to playing one low-resolution movie that took up maybe half the screen. I was still paying for internet over a phone line with a cap on the number of hours I could use it. Today, I have real-time two-way video connection to anywhere in the world, 24-7, for a single flat rate. I own a device that fits in the palm of my hand which contains days worth of continuous music, a library of dozens of books, and connects to the internet.

So, back to that "mass market consumer". Maybe they don't care about my Python console or IRC chat or SSH access applications, but most "mass market" people do listen to music and read books. And they're going to care about those features being on their phones, and remaining cheap enough that they can use those features without worrying that they'll go broke if they feel like changing out their playlist. Also - nobody is really a "mass market" consumer, anyway. You might not be technical, but maybe you're a golfer, or a swimmer, or a finance nerd. You want to be able to check the weather on your mobile, or update your latest personal best lap time, or get updates when stocks hit certain price threshholds. Nobody cares what APIs these apps use, or even whether you call them "apps", but everybody has one extra thing they'd like their mobile to do.

The increasingly ubiquitous, user-customizable, network connected, commodity pocket computer is exactly the technology that is going to deliver that. It's going to have to become commoditized, which means it's going to be standardized, and secured, which means it's not going to be locked up in carrier notions of what's a "text message" and what's a "voice call" and allow for precise price segregation of every different type of data.

In the future, almost every device will be a computer, albeit with specialized peripherals to assist with performing tasks. If we're lucky, they will be networked together in standard ways to allow us to control all of them in a consistent and convenient way.

This progression towards computers is good for all of us. Trust the computer. The computer is your friend.

Paul Swartz: America's Top Public High Schools | Newsweek Best High Schools | Newsweek.com

Mon, 15 Jun 2009 20:50:27 +0000

America's Top Public High Schools | Newsweek Best High Schools | Newsweek.com: B-CC (my HS alma mater) ranks 55th, up from 64th last year. Sure, it’s a bit of a bogus measurement (AP and IB tests taken / graduating seniors) but it’s nice to be listed.

Paul Swartz: CrossFit Faq: Explain the Works with Names (the Girls)

Mon, 15 Jun 2009 20:50:26 +0000

CrossFit Faq: Explain the Works with Names (the Girls): Since there’s a thunderstorm outside, I think I’ll be doing ‘Angie’ from my apartment.

Paul Swartz: Last year, I did “what’s in your bag?” on...

Mon, 15 Jun 2009 15:39:39 +0000

Last year, I did “what’s in your bag?” on Twitter. To mix it up a bit, here’s a photo of the applications I’m running. It’s pretty boring:

pidgin (IM client, at the top left and top right)
Firefox 3.0.11 (top middle, open to Google Reader)
GNU Emacs 23.0.91.1 (bottom left, with code in the left pane and magit in the right)
Miro trunk (bottom right; it’s got Joe Biden because I’m watching Sunday’s Meet the Press)

What’s running on your desktop?

Paul Swartz: Photo

Thu, 11 Jun 2009 02:01:56 +0000

Moshe Zadka: Managing Large Twisted Projects

moshez — Tue, 09 Jun 2009 09:14:07 +0000

Introduction

When working on a Twisted-centric project, it can be hard to recruit programmers. While Python, especially in recent years, has achieved main-stream status, Twisted is not quite there yet. It seems to be the case that recruiting Python programmers is, today, a solved problem. In the case where programmers unfamiliar with Python need to be recruited, resources (both free, cheap and expensive) abound for getting them up to par.

In the Twisted world, especially for large projects, programmers who have no previous experience with Twisted will need to be recruited. In all software projects, time is of the essence. An efficient, effective, way to get Python programmers into Twisted is desired. This is the case in VMware Israel (formerly B-hive), where the flagship product AppSpeed has multiple portions written in Twisted.

Teaching Resources

The broadest tutorial for Twisted is the “finger tutorial”. It leads the reader from writing the most basic finger server into a finger server supporting IRC and XML-RPC backends, going through a variety of Twisted programming techniques. Its broadness is also its chief downside, causing many newbies to learn many concepts which are irrelevant to their immediate needs, hiding the forest inside a density of trees.

The Twisted howto collection is another excellent resources. Two primary howtos are the “Writing Servers” which covers, in a focused, no-nonsense manner, the steps needing to write a server for a new protocol. Very little depth and “why” are covered, which is often as much as an upside as a downside. The other important howto is “Deferred Execution”, covering the basics of deferreds. Several other resources for covering deferreds in-depth are available.

Architecture

It is often the case that one, or more, of the people beginning a project have experience in Twisted. It is worthwhile to think how to attempt having a clean interface to those parts which need more such experience so that people writing other parts can hold their own with less such experience. Protocol details, for example, are often uninteresting, have a natural API and require non-trivial coding to deal with all the possibilities of packet portions arriving. The best wrapper for such details, if at all possible, is with an object whose methods return deferreds, and expose as few of the protocol details as possible.
An infrastructure containing such objects will help people learn Twisted in parts, while becoming productive programmers on the project early on.

Another useful part of the architecture is to compensate for beginners’ mistakes. The most popular such mistake is trying to do too much in a single callback, halting the reactor loop. Two solutions, often used in tandem, are useful for this problem. One is to limit the damage by trying to separate loosely-coupled tasks into separate processes, and using infrastructure such as AMP to communicate between those. In case one reactor is paused for too long, the others will still function correctly. The other solution is watchdogging: have a relatively-frequent “LoopingCall” start early in the infrastructure do some measurable action (sending a UDP packet, touching a file) and have a separate watchdog check for this action. If too long passes without the action being performed, the reactor is not functioning up to par — and the process should be rebooted. This, of course, necessitates correct recovery from rebooting: a rudimentary persistence framework, to say the least.

In places where long-running algorithms are required, or are merely easier to write, “deferToThread” is a useful abstraction. It allows the Twisted-beginner to shed all thoughts about events, and to write regular Python code. Those deferred-returning APIs will come in handy as they are used through reactor.blockingCallFromThread. It is useful to make sure blocking API meant to run in a different thread lives in separate modules — it makes code reviews easier when knowing in advance whether “time.sleep” or “reactor.callLater” is a bug.

Deferred-oriented infrastructure, a communication platform and a watchdog are three things most large Twisted projects will sport at one point or another. The sooner they are done, and the better, the better overall quality of the project will be — especially in a project which is introducing a large portion of its programmers to Twisted.

Summary

In most big Twisted-oriented projects, a large portions of the programmers, even if Python experts, will not know Twisted before starting on the project. It is useful to plan for this in advance, allowing an easy Twisted initiation for the beginner. This can be done in the three prongs of focusing on unavoidable Twisted issues first (deferreds and non-blocking rather than low-level APIs), compensating in the architecture for common mistakes and allowing a transfer of non-Twisted Python knowledge by allowing threaded abstractions.

Thomas Vander Stichele: matplotlib

Thomas — Fri, 05 Jun 2009 13:43:44 +0000

Is python’s matplotlib and pylab just a twisty little maze of global variables and mass imports, making it impossible to learn your way around by introspection ? Or is it me ?

I am getting lost in the difference between Axes, figure()’s and plot()’s…

I’m sure this would all make more sense to me if I could recall my vague Matlab knowledge from university.

Glyph Lefkowitz: Who Wants To Know?

glyph (noreply@blogger.com) — Thu, 04 Jun 2009 21:27:50 +0000

Alternate Titles:

I Have No Log File, Yet I Must Scream

So You Logged It, Now What?

This is why I don't want to have LOG_DEBUG in Twisted

Sometimes, when writing a program, you feel compelled to make the program emit some output which is peripheral to its operation. The question is - who wants to know about that information?

Maybe you're debugging the program, and you insert a simple 'print' statement to get some information about it. Maybe your program is a network server, and you are recording the fact that a message was received and processed. Maybe you're maintaining an old library routine, and you want it to emit a message that points to a newer, better version of that routine which is now preferred. Finally, regardless of what kind of program you're writing, maybe it has produced an error that a user or administrator will need to deal with, and you would like to show it to them.

This activity is referred to in several different contexts depending on how the messages are delivered, but it is most commonly known as "logging". It is critical to the operation of many, many different kinds of programs. Unfortunately, it is one of the most poorly-understood and poorly-implemented areas of software in general. Software is a veritable cornucopia of poorly-understood and poorly-implemented ideas, so that's really saying something. You can see some of the more hilarious and visible examples of developers getting this wrong in the "Pop-Up Potpourri" series on the Daily WTF.

It might seem odd that I lump together funny dialog boxes with "logging". A dialog box is a little square on your screen; a log message is some text in a file somewhere. But they are very much the same thing, and they fail in very much the same way. Log files just do it less visibly.

The point that I hope to communicate here is that for every producer of information, there is a consumer. When most programmers need to produce a "log message", however, they are thinking only of getting the information out of their program in some format, any format; not how that information is going to be used later.

When I say "most programmers", I most definitely include myself. I'm probably guiltier than most. one of the reasons I'm writing about this in the first place is to work out some better approaches to the problem.

Consider this output from the "tomboy" desktop sticky-notes program on Ubuntu Hardy. If I start it from the command-line, I see this:

[DEBUG]: NoteManager created with note path "/home/glyph/.tomboy".
[INFO]: Initializing Mono.Addins
[DEBUG]: AddinManager.OnAddinLoaded: Tomboy.Tomboy
[DEBUG]:            Name: Tomboy.Tomboy,0.10
[DEBUG]:     Description:
[DEBUG]:     Namespace: Tomboy
[DEBUG]:         Enabled: True
[DEBUG]:            File: /usr/lib/tomboy/Tomboy.exe
[DEBUG]: Updating note XML to newest format...

It goes on for several hundred more lines just at startup, and continues to produce messages as the program runs. These messages are diligently classified into categories: DEBUG and INFO. I'm sure they're useful to someone. But why am I seeing them? I just wanted to start a program to put some sticky notes on my desktop, and none of this information is useful to that task.

I have to imagine that pretty much all of these messages are useful only to Tomboy's developers. But, worse than the fact that I see them is the fact that if something really interesting happened — I discovered a critical bug, let's say — all of that log output which is being splatted onto my screen is going nowhere. It is a book written on water. (Well, a book written on video memory, which is pretty much the same thing.) Meanwhile, thanks to the bug-reporting facilities in Ubuntu, I'm sure that I could opt to give the Tomboy developers a huge ton of mostly useless information, like the contents of my registers at the time that it crashed.

Consider not just the placement of the messages (on my screen, where I certainly don't care about them) but their formatting. Who is that elaborate right-justification of labels in the "DEBUG" output for, anyway? It isn't for me, I don't want to see these messages in the first place. I doubt it helps the developers, either; rather than just grepping for '[DEBUG]: File', now they need to put in a regular expression to collapse whitespace, or count the number of spaces that the justification happens to put in. Presumably if this output is useful at all, it is useful in a search.

Text Formatting and the Inevitable Descent into Log-Level Hell

The right-aligned pretty-printing is a beautiful illustration of a very common anti-pattern in logging: trying to convey structured information by messing with a textual format. A developer wanting to write a message indicating that there is a problem with the program, left with the extremely narrow confines of a logging API which just takes a string, will often do something like this:

log("*****")
log("THIS SHOULD NEVER HAPPEN! HELP!!!")
log("*****")

Of course, this frantic wording doesn't help the output go anywhere but silently into a log file where it will be ignored. But, perhaps if this is some server software, an administrator will notice this message and set up an alert that makes their blackberry buzz when they notice those particular words show up in the log file so they can ssh in and look for problems.

Then the developer gets chastised by his manager for his un-informative error message, and updates it to be something clearer:

log("Serious Error: phase inducers have been depolarized. Contact engineering immediately.")

Of course this breaks the administrators' alerts, so after much discussion between programmers and admins, log levels are added so that admins only get alerts when something "really bad" happens, where "really bad" is an agreed upon flag:

log2(SERIOUS_ERROR, "phase inducers have been depolarized. Contact engineering immediately.")

Okay. Now we've got a log level so admins can tell when their pagers should go off. Except, different developers have different ideas about what "serious" means.

log2(SERIOUS_ERROR, "OMG I lost my cat Mittens. Where is my cat?")

Clearly this is an abuse of the new "severity" flag that was added, but the cat-engineering team thinks that loss of a cat is pretty serious, so we add a new thing, a log "system".

log3(SYSTEM_CATS, SERIOUS_ERROR, "OMG I lost my cat Mittens. Where is my cat?")

Most logging systems stop in this general vicinity, but we still haven't solved the problem, which is that the log message has no structure and you can't tell what's going on without groveling around in a bunch of text files with regular expressions or manually reading each message. Which cat was lost? Which phase inducer was depolarized? How do we get from a log message or alert to this information? The 'log levels' solution to this problem is clearly untenable:

logRidiculous(SYSTEM_CATS, ALERT_IF_YOU_LIKE_CATS, O_RLY, YA_RLY, SERIOUS_BUSINESS, BUT_NOT_TOO_SERIOUS, CAT_LEVEL("Mittens"), "OMG I lost my cat Mittens. Where is my cat?")

More importantly, if you're writing a library, you have a bunch of other problems. This diagnostic information needs to be logged somewhere, but what if this library is being used on a user's desktop machine? Some of these messages are relevant to them as well. How do you tell the user who is using a GUI that a cat has been lost? How do you show them the picture of Mittens so they will recognize her if they see her?

Everyone agrees that log messages need some "small amount" of information associated with them, but very few people can agree on what that information should be. Even at the simplest layer, the idea of a "level", there are lots of open questions. Is the "debug" level for a programmer trying to debug something on their test rig, or is it for administrators trying to debug something in production? Should there be a difference between those two things? How serious does a problem have to be before warranting a "critical" classification?

Once you're using logging code written by more than one programmer, or worse yet, more than one team, you're going to be facing this problem.

The Particular Problem of Libraries

This is, of course, my main interest, since this is where the rubber meets the road for Twisted. Libraries need to communicate to several different audiences:

We need to tell developers using the library about the correct way to use the library at runtime.
We need to tell administrators of systems using the library about the status of the library and tasks they may need to perform to keep it functioning well. (Clear your caches, restart the server, install a security update...) We also need to provide administrators with information they can mine for statistics about how the library is performing; how many requests handled, where its resources are going, etc.
We need to notify users of applications using the library about things that the library is doing which may be relevant to them. (A new message has arrived, a new printer is available... obviously this depends heavily on what the library does.)

Libraries, especially event-driven engines such as Twisted, libevent, and glib, have a particularly difficult time because they have to deal with all of these audiences simultaneously. However, I think that any application or server which needs to do some kind of logging or user notification needs a subset of these features, so if any logging system could solve this problem, it could solve pretty much all logging problems.

Type of Information by Type of Audience

Developers, Developers, Developers

Many languages don't have a solution to developer communication at all. Python has one — the warnings module — but it is in many ways inadequate.

The warnings module doesn't easily let you selectively see which libraries you want to see warnings for. If I'm developing an application A using libraries M, N, and O, which themselves have dependencies on X, Y, and Z respectively, I don't want to see warnings that M caused in X or that O caused in Z; those are problems for the maintainers of M and O.

I am maintaining only A, so I want to see warnings caused by my application in M, N, and O. I can try to filter specifically by module, but unfortunately the
only way of determining which library caused which issue is by directly
examining stack depth, which is unreliable at best and misleading at
worst. Even if I could filter very accurately, it's hard to get a stand-alone report of warnings and deal with them as they're supposed to be. Warnings show up to end-users as well, and to administrators looking at applications in production. It's worth putting up with that to have at least some solution for communication with developers, but it would certainly be better if it didn't happen.

Finally, it's easy to generate a huge amount of warning noise (and, especially as of Python 2.6, many libraries do). With that much noise and no reporting functionality it's hard to the warnings you care about.

A better solution for communicating for developers would be one which:

allowed developers to declare somewhere what code they are working on and what code they are just using
recorded relevant warnings to a log file which was optimized, perhaps with an associated tool, for locating and removing the sources of the warnings
allowed end-users to easily communicate their warning data to developers without inundating them with irrelevant noise while using the application

Administrators

To communicate with administrators there is a huge variety of options, but many of them depend on a lot of ad-hoc hackery by the admins themselves, which means they are inconsistent and therefore there is little reusable technology or standardized APIs available.

Right now the gold standard for talking to admins seems to be just writing strings into a text file and hoping they have some facility to read it.

A better solution for communicating with administrators would be one which:

preserved structured data in an analysis-friendly format, rather than formatting it in human readable messages. (For most UNIX admins, I imagine some kind of structured text would be best, so "grep" would still work but more advanced tools could also be brought to bear. I'm not sure what the tools in the Windows world look like. The "Event Viewer" looks like maybe it's a step in the right direction, but its UI is incredibly primitive.)
provided easily-accessible hooks for dispatching different types of events to ad-hoc code to wire up to existing notification systems - without significantly altering the behavior of the system doing the logging, if the logging hooks were broken, as admin-written code tends to be a bit flaky
included an enumerated list of events which administrators could inspect before they happened to run across them in log files

Although it's a crappy format in many ways, the Common Log Format for HTTP might serve as a good example. Unfortunately it's too purpose-specific to extend to do more than what it already does, but lots of tools have been written to produce lots of interesting data from even that very simple standard.

Desktop Users

There are two popluar cases for communicating with end-users. One case is that you're actually running a program on their desktop and you want to tell them something. Another is that you've got some code running in a web application which wants to tell them something "out of bounds".

On the desktop, there are fairly standard "notification" APIs for popping up little bubbles. On the web, there are emerging conventions for these notifications, like a bar that descends from the top of the page to mimic the firefox 'do you want to remember this password?' UI element. A good example of this is Stack Overflow's notification banner.

Unfortunately both of these have a problem with scale and with timing. If your application suddenly encounters a large number of errors, it will flood the user's screen. If the user isn't present when a notification occurs (or navigates away), the bubbles or banners may disappear.

A better way to talk to end-users would be:

for desktop applications, a mini-email interface, which records notifications in a scrolling list so that users can inspect notifications that occur while they're away.
for web applications, a standard API so that multiple applications on a single site (or even, potentially, on different sites) can drop notifications into a queue which can be displayed appropriately. (Since websites tend to have strong preferences to control their own design, an actual standard widget might not be possible, but it would be nice.)
for both of these, a standard protocol which would enable notifications to be easily streamed to different computers or mobile devices without needing to reconfigure
a specific classification of messages at the API level, saying, "I want to tell the end-user about this". Messages about crashes, etc, should be displayed as an option to send information to the developer. In the context of a web application this can be done automatically and silently; in a desktop app there would need to be some channel set up for sending that information.

Let's not forget that administrators are users too. Everything that happens in a server's notification APIs should be able to be trivially filtered and redirected to administrator's desktop machine (or their phone) so they can immediately notice when something has gone wrong.

Appendix A: Optimization and Dynamic Instrumentation

This doesn't fit into the "figure out what you need to say and who you need to say it to" theme of the rest of the requirements here, but it is nevertheless important. If you make heavy use of a logging system, especially one where you have lots of messages that are logged "just in case" and rarely displayed, you will quickly discover that it's consuming a lot of resources.

The work to calculate what goes into a log message shouldn't be done unless the message is actually going to be emitted. Similarly, it should be easy to dynamically add and remove log events from particular methods or particular objects without modifying their code. The best way to do this is to keep the logging entirely separate and add it at the level of methods and functions, rather than ad-hoc in the middle of application code. Sometimes, of course, you want to emit log messages at a very particular point, but in general it should be easy to add instrumentation to a method without modifying its code, to avoid cluttering up application logic with lots of "just in case" debug messages.

What Can Do This?

I'm not aware of any logging system that can already do these things. We'd have to write a new one. This essay was largely composed due to my desire to understand what I thought a "good" logging system would do. It might be too ambitious.

There are a few fundamental units which are missing from most logging systems. While lots of logging systems have various ways to indicate subtle and nuanced levels of urgency, few have a way to indicate who a message might be relevant to. Logging systems also generally don't have a good way to associate structured information along with a log message.

Twisted's logging mechanism does have a free-form dictionary associated with each message, but that's not much help unless you impose your own structure on it, which means you have to build all this infrastructure anyway. It is, at least, possible to treat the Twisted system as a kernel which this could be based upon.

In order to produce an enumeration of events before the events are actually logged, this API will require pre-declaration of log messages. This might be too much of a burden, since sometimes you want to just stick a log message into the middle of some code so that you can see if it happens. So, in practice, it's more likely that pre-declaration will need to be optional and you'll need to be able to associate ad-hoc data and have it still be persisted along with your message.

There will need to be some way for communicating both the structured data of an event and the human-readable text associated with that event — preferably in a way which can be internationalized.

There's also a bunch of UI, web, and protocol standardization work that would need to be done. Luckly, that's independent of the actual log machinery; if it already existed it would be a trivial matter to hook it up. In the meanwhile, something that did all of this but just used existing facilities, like the desktop notification spec, status icons and email, would still be immensely useful.

Jp Calderone: April - May Reading List

Thu, 04 Jun 2009 13:52:42 +0000

Green Mars. Kim Stanley Robinson.
Blue Mars. Kim Stanley Robinson.

Paul Swartz: the peanuts I could be eating with Geico

Thu, 04 Jun 2009 01:15:01 +0000

the peanuts I could be eating with Geico

Jonathan Lange: pyflakes now warns about unused local variables

jml (noreply@blogger.com) — Wed, 03 Jun 2009 22:54:44 +0000

I finally got around to finishing my fix for Divmod bug #2718 -- Warn about unused variables in methods in pyflakes. Last night, the magnificent Jean-Paul Calderone reviewed and landed my patch. This means that if you are using pyflakes trunk (either from Subversion trunk or from the Launchpad Bazaar import), pyflakes will spot code like:

def foo(bar):
baz = bar + 2
return 12

and generate a warning like:

example.py:2: local variable 'baz' is assigned to but never used

I use pyflakes hooked up to flymake, so it's always running all the time on every Python file I'm working on. Relying on it has become as second-nature as relying on syntax highlighting. There's a whole class of mistakes I don't make any more, simply because it's on.

However, the main way it helps me is when I'm refactoring code. When extracting a function or changing a variable name, pyflakes acts like a sort of todo list for me. Now that it shows unused local variables, it's getting dangerously close to perfect.

To get pyflakes quickly, bzr branch lp:pyflakes.

(Edit: Grammar fix)

Thomas Vander Stichele: git merge conflict

Thomas — Tue, 02 Jun 2009 10:42:39 +0000

For every positive experience with git there’s still more than enough negative ones to balance out.

Today, I got into the situation where I was updating my gstreamer modules and one of them was apparently still in conflict. Unhelpfully, git just says:
You are in the middle of a conflicted merge.
without telling you what to do about it.

Googling revealed lots of people in the same situation, and git reset –hard would work. It looks like that would throw away my changes though. Of course that’s one way out of the conflict.

I want to know what the other way is - the one where you get a change to merge conflicts.

Apparently the conflict was in a generated config file, so naively I deleted it because I wanted to check out that file again from the repository (probably an svn-ism that stuck with me). I know that in this case I don’t actually need to be able to merge my changes, but I would like to know what would have been the correct way to get out of this situation.

Here’s what I tried before I gave up and did a reset:

[gst-git] [thomas@level gst-plugins-ugly]$ git pull --rebase You are in the middle of a conflicted merge. [gst-git] [thomas@level gst-plugins-ugly]$ git diff diff --cc win32/common/config.h index 18dbcc4,abf941a..0000000 deleted file mode 100644,100644 -- a/win32/common/config.h +++ /dev/null [gst-git] [thomas@level gst-plugins-ugly]$ git checkout win32/common/config.h error: path 'win32/common/config.h' is unmerged [gst-git] [thomas@level gst-plugins-ugly]$ git reset -- win32/common/config.h win32/common/config.h: locally modified

Thomas Vander Stichele: git bisect

Thomas — Mon, 01 Jun 2009 15:46:07 +0000

Today I finally had a reason to be happy about GStreamer having switched to GIT.

I added actual encoding support to my CD ripper this weekend. I’m only supporting lossless encoding for now. Sadly, in practice it seems the FFMpeg ALAC encoder crashes, and the wavpack one hangs for me, so I’m left with .wav (meaning no compression) or .flac

And then today I realized that some of my encoded .flac files had a strange bug in them. I wasn’t sure where it was coming from, but some of my encoded files didn’t play with mplayer, gave an error when using flac -d, but still worked completely fine with GStreamer and totem.

I first tried to find a different file from the GStreamer media testsuite to reproduce the symptom. south.mp3 and benow.mp3 owrked fine, but sugar.ogg reproduced the problem.

I also tried it with my installed version (0.10.8, while git master is at 0.10.15.1) and that worked fine.

So that gave me two points to bisect inbetween.

Then I read up on git bisect, and started playing with it. It isn’t particularly nice to do by hand; most checkouts change enough of the autotools files that it has to rerun them most of the times. Then configure changes win32/common/config.h which generates a local change. The common submodule also gets in the way. I got away with just compiling the flac plugin with make -C ext/flac before each test, so that sped up things. But there’s definitely potential for human error.

Basically, you start by doing git bisect start; git bisect bad; git checkout (known good commit); git bisect good

This will leave you somewhere in the middle between the bad and the good commit. Rebuild, do your test, then either type git bisect bad or git bisect good based on the test result. Repeat until you’re at the last commit.

That helped me find where the bug happened (see the bug report).

Of course I wondered immediately if I could automize this, since I wanted to make sure that the same commit broke my original files and I do not want to do that manual bisection again…

It turns out you can; that’s what git bisect run is for.

So, given the following two shell scripts:

test.sh

#!/bin/bash

git submodule update

make -C ext/flac

# rebuilds can touch these files leaving you with local changes
git checkout win32/common/config.h

gst-launch -v filesrc location=/home/thomas/gst/media/medium/sugar.ogg ! oggdemux ! vorbisdec ! audioconvert ! audio/x-raw-int,width=16,depth=16 ! flacenc ! filesink location=test.flac
flac -d test.flac -f
# flac -d exits with 1 when it fails
exit $?

and bisect.sh:

#!/bin/bash

git bisect start
git bisect bad
# 0.10.8 release
git checkout c186d67f40827be349f97d810a45243c874b73f7
git bisect good
git bisect run ./test.sh

I can now just run ./bisect.sh and it will do the whole process automatically.

Try it if you’re curious on a checkout of gst-plugins-good. Change test.sh to point to your sugar.ogg file (which you can get from GStreamer’s test repository).

At the end, the output should show something like:
df707c666433a78d3878af6f055698d5756226c4 is first bad commit
commit df707c666433a78d3878af6f055698d5756226c4
Author: (HIDDEN TO PROTECT THE GUILTY)

The ‘run’ command really is what makes the bisection useful for me. Now, back to bug fixing….

Ralph Meijer: PubSubHubbub

ralphm — Sun, 31 May 2009 13:09:49 +0000

PubSubHubbub is a protocol and reference implementation for doing publish-subscribe using web hooks, polling in feeds triggered by a ping from the publisher, and POSTing Atom entries to notify subscribers. The notification part is similar to what I've been working on for the publish-subscribe stuff at Mediamatic Lab, where we spiced up Idavoll with an HTTP interface to bridge the gap between XMPP Publish-Subscribe and HTTP speaking entities.

Although I spend a lot of time working on XMPP based publish-subscribe, I understand the reasons for going for a full HTTP-based approach. XMPP can be intimidating for developers of web applications. While the differences between XMPP and HTTP are important (stateful connections, asynchronous processing, etc), the fact that it is different is reason often enough. Hosting facilities don't always offer ways to do XMPP, and there is not nearly enough running code out there to make it easier for people to play with these technologies to spice up their web application with non-IM XMPP functionality. Having platforms like Google App Engine provide sending and handling raw XMPP stanzas as part of the API would surely help.

That said, PubSubHubbub has two separate sides to it, the publishing part and the notification part. There's nothing that prevents a hub to do the publishing part using regular XMPP publish-subscribe. Instead of fetching the Atom Feed over HTTP every time, it could use autodiscovery to find out the publish-subscribe node and upgrade by subscribing to it instead. Similarly, the notification part could send out XMPP notifications. Combined with existing HTTP aggregator, that combination is very similar to how the aggregator for Mimír works.

I'm still not convinced that PubSubHubbub is the answer to the efficient exchange of updates on social objects, but I do think it is a good way to make smaller entities be part of a federation of social networking sites. Likely, we'll see a hybrid approach, to begin with.