Tim Bell preparing to get his OpenStack on

"When we're running a complex fabric of apps on over 5,000 servers across three data centers, we need a lean and nimble approach to software development and operational implementation. Without a DevOps approach, we wouldn't be able to push code into production as fast or as efficiently as we do, and our customers would not be happy! Today's developers demand up-to-the-hour security and performance updates to Internet infrastructure, so we aim to deliver just that with DevOps."

"The CERN Agile Infrastructure project aims to develop CERN's computing resources and processes to support the expanding needs of LHC physicists and the CERN organisation."

One artist's conception of what success will look like for OpenStack at CERN

Despite the minor hiccoughs along the way, CERN is aiming for success. (Given the lack of Combine and forced relocation programs, they're already doing better than Black Mesa's Anomalous Materials team.) Plans are in place for an initial pre-production service, OpenStack deployment this year. Following that, they will be moving towards 300,000 virtual machines on 15,000 hosts spread across two data centres by 2015.

The OpenStack community is supporting them in their efforts with fantastic new features, high-quality discussions on the mail lists, and real-time interaction on the IRC channels. In an act of reciprocity and community spirit, operators at CERN have volunteered to contribute back to the OpenStack community with regard to operations best practices, reference architecture documentation, and support on the operators' mail list.

To see how other institutions were taking this news, I spent several days waiting on hold. In particular, Aperture Science could not be reached for comment. However, Ops team member Belmiro Rodrigues Moreira did say that there's an audio file being circulated at CERN of Cave Johnson threatening to "burn down OpenStack" ... with lemons. Given Aperture Science's failure record with time machine development, it's generally assumed to be a prank audio reconstruction. CloudStack developers are considered to be the prime suspects, seeing how much time they have on their hands while waiting for ant to finish compiling the latest Java contributions.

When asked what advice he could give to shops deploying OpenStack, Tim said simply: "Remember, the cake is a lie. Don't get distracted and don't stop. Just keep hacking."

Alyx, explaining to her dad why she loves DreamHost

Couldn't have said it better myself.

In closing, and interestingly enough, one of DreamHost's employees has an uncle who works at the Black Mesa Research Facility. Though his teleportation research team was too busy for an extended interview, his daughter did mention that she is a DreamHost customer and can't wait to use OpenStack while interning at CERN next summer. After all, that's what she uses to auto-scale her WordPress blog (she's in our private beta program).

It's a small world.

And, thanks to Tim and the rest at CERN, a safer one, too.

1. an interactive Python prompt was rendered, e.g., ":>>"
2. the banner was getting written to the terminal, and
3. the terminal screen refreshed with the prompt at the top

• twisted.conch.recvline.RecvLine.connectionMade calls t.c.recvline.RecvLine.initializeScreen
• t.c.recvline.RecvLine.initializeScreen does a terminal.reset, writes the prompt, and then switches to insert mode. But this is a red herring. Since something after initializeScreen is causing the problem, we really need to be asking "who's calling connectionMade?"
• t.c.manhole_ssh.TerminalSession.openShell is what kicks it off when it calls the transportFactory (which is really TerminalSessionTransport)
• openShell takes one parameter, proto -- this is very important :-)
• openShell instantiates TerminalSessionTransport
• TerminalSessionTransport does one more thing after calling the makeConnection method on an insults.ServerProtocol instance (the one I had tried overriding without success), and as such, this is the prime suspect for what was preventing the banner from being properly displayed: it calls  chainedProtocol.terminalProtocol.terminalSize
• chainedProtocol is an insults.ServerProtocol instance, and its terminalProtocol attribute is set when ServerProtocol.connectionMade is called.
• A quick check reveals that terminalProtocol is none other than the proto parameter passed to openShell.

Last week I posted a survey to gauge interest in commercial Twisted support contracts to the Twisted mailing list:

http://twistedmatrix.com/pipermail/twisted-python/2012-May/025537.html

If you think this might be applicable to your interests and you didn't see the initial posting or haven't had a chance to respond yet, please take a minute or two to fill it out (it's very short, no essay questions at all). Thanks!

Given the theory of relativity, C (the speed of light) has absolute meaning. Since speed is distance/time, this means that we have a standard way of converting time into distance. Instead of measuring time and distance in different units, we should be using the same units.

Hence, a second (in meters), would be 299,792,458. Of course, that’s quite a distance — so instead, let’s do the standard thing and use prefixes to scale it down to useful numbers.

1 nanosecond =~ .3m =~ 0.984 feet (if you need less than 3% accuracy, a nanosecond is a foot)
1 microsecond =~ 0.3km =~ 0.1875 miles

I’m roughly .495 nanoseconds tall. My wife is around .525 nanoseconds. I have to drive about 160 microseconds to get to work — for most of it, the legal speed limit is around 340 micros. (Note — speed has no more units, but it does have scale — the legal speed limit is 340*10e-6). Anyone else up to the task of starting to measure distance in nano- and micro- seconds in real life?

• trystack.org needs volunteer admins -- a great opportunity for improving the dev <-> ops interface
• CERN is deeply invested in DevOps, and wants to share data and possibly help with defining reference architectures
• The same goes for the University of Melbourne!
• There was a good corporate presences in the session too, rallying around a new and improved DevOps community: HP, Yahoo, AT&T, Canonical, Rackspace, DreamHost, and more (sorry if I forgot you!) 

I pretty much never drink and hack, and last Friday’s evening is a good reason why. I was having a rare beer and managed to spill part of it on my keyboard and desk. So I turned the keyboard around, started cleaning it as fast as I could, forgetting to actually unplug it. I called it a night because nothing good was going to come from that night anymore.

And on Saturday morning I noticed that my INBOX was gone. Hm, is it really gone? Yep, gone from my laptop too. Crap, must have deleted it on the server by accident while cleaning my keyboard…

And because my NAS is a little full lately, I haven’t been as diligent with backups as I normally have been. Hm, and the modest cache on my N900 isn’t very useful either…

Luckily, evolution on my work machine was shut down for some reason, so yay, it has a reasonably fresh cache of my INBOX!

Except that it’s not all that straightforward to actually get this cache back into Evolution. Just copying its contents to an existing or new folder doesn’t do anything. The files themselves are split up versions of the actual email, assumingly because the evo guys thought it would be faster to search header and body by splitting them off from the attachments and saving them separately, inventing their own caching format. Which is fine, but makes it impossible to actually restore a backup with…

After lots of Googling, I stumbled upon this tool that did the trick for me. A lot of hours wasted over a bunch of emails… But what would happen if I really lost my IMAP server mail ? Run this script by hand on all the folders ? Shudder…

This was originally concieved as a response to A Conversation with Guido about Callbacks over at Duncan McGreggor’s blog.

First, a jQuery-ish example of using some callbacks:

def handleResponse(resp):
    def handleParsed(xmlObj):
          firstStyle = xmlObj.xpath('//link[rel="stylesheet"]')[0]

        def handleAnotherResponse(anotherResp):
              print anotherResp

        getPage(firstStyle['href'], handleAnotherResponse)

    asyncParser(resp.body, onComplete)

getPage("http://example.com", handleResponse)

Now this is an absurd example of callback soup. Or is it? I’ve read a lot of JavaScript, and it isn’t really uncommon to see multiple levels of calls which take a callback. The JavaScript actually usually looks cleaner and is easier to read than our contrived Python because JavaScript has multi-statement anonymous functions that can be used to inline the callback definition into the function call. (That feature has other readability side-effects though, such as the pattern of defining a single argument anonymous function to invoke a single, single argument function.)

Now, when you write callbacks like this, it’s no wonder people tend to dislike them. Especially when contrasted with the more straightforward non-callback example.

resp = getPage('http://example.com')
xmlObj = parser(resp.body)
firstStyle = xmlObj.xpath('…')[0]
print getPage(firstStyle['href'])

4 lines of code vs 8 lines of code (11 including almost mandatory blank lines.) Is it any wonder that Guido prefers this approach? No, absolutely not. That first example is an absolutely terrible but perfectly normal piece of callback using code. Let’s enumerate some of the ways in which it is bad.

1. Everything is out of order.
2. Excessive indentation. Flat is better than nested, but close is better than far, so nested wins.
3. The desire for locality causes function definitions to be scattered among statements.
4. You’re defining a lot of non-reusable, non-testable functions.

I’m sure you’ve come up with plenty more of complaints (including non-PEP8 camelCase names) but these are the 4 big points I’d like to discuss further.

Order matters.

Most of us read left to right and top to bottom. Even if your primary language doesn’t read left to right and top to bottom your primary programming language almost certainly does. When we make todo lists we give them an order, items at the top are more important. Shopping lists are sometimes ordered starting with vegetables and moving towards more time fragile things like ice cream, because that’s the way the stores are laid out. Computer programs are written line by line in the order we want instructions executed, because that is how computers work.

Order matters. On first encounters with code order matters a lot.

Related statements should be close.

Locality is the topic of both issues 2 & 3. And it can best be summed up as, related things should be close. It’s the most basic element of organization for everything from kitchens to code.

Anytime you’re reading anything you’ll find yourself glancing back to reread a few lines, or sentences, or even words as soon as you’ve encountered something that you don’t understand or doesn’t quite make sense. The further you have to look the longer it’s going to take to get your answer.

Reduce, Reuse, Retest.

We write functions to enhance the readability, and testability of our code. We reduce the size of large functions by breaking them up into higher level discrete steps and creating functions to encapsulate those actions. We reuse common code by creating more functions. We retest, ok well retest doesn’t make much sense, but we increase testability by concentrating functionality into smaller more well defined units.

You would never inline all the socket calls necessary to make an HTTP request. Maintainable software is built out of well defined reusable and testable abstractions, and there is no reason that callbacks should not also be reusable and testable as much as possible.

There is a better way.

I’ll admit, I have a love/hate relationship with callbacks. I view them as a necessary evil to do event-driven programming. There is of course a better way to think about callbacks and you may have already guessed it. Deferreds and Dataflow programming. A lot of people use deferreds for the first time much the same way they use plain callbacks.

d = getPage(…)
def handleResponse(resp):
    def handleParsed(xmlObj):
        firstStyle = xmlObj.xpath(…)
        def handleOtherResponse(otherResponse):
              print otherResponse

        d3 = getPage(firstStyle['href'])
        d3.addCallback(handleOtherResponse)

    d2 = asyncParser(resp.body)
    d2.addCallback(handleParsed)

d.addCallback(handleResponse)

Now, this example, though still contrived is not unrealistic, it is in fact perfectly functional and probably exists in more than a few Twisted using code bases. Of course it’s terrible for all the same reasons our plain callback example was terrible.

So clearly it is not the better way. So what is?

Deferreds as a dataflow abstraction.

Here you have a acyclic directed graph of data through the callback chain. The individual operations being performed is less important than the flow of information through the structure. An event causes a result to be available, it is put into the deferred and moves along the directed graph through the callback chain until an error is encountered. Then we move to the errback chain will proceed until the error has been handled at which point we can go back to the callback chain, or until we run out of errbacks.

The result from one operation flows directly into the next.

getPage(getFirstStyle(parse(getPage("http://example.com")))

The above is a dataflow program. Our imperative example from earlier is not, because the grouping of operations is a side effect of style, any number of operations could be interspersed which may or may not have side effects that influence future operations.

d = getPage("http://example.com")
d.addCallback(parse)
d.addCallback(getFirstStyle)
d.addCallback(getPage)
d.addCallback(printResult)

Now, everything is order, related operations are close, and the structure of higher level operation is broken up into discrete and independently testable units.

By this point I hope I have made it obvious about why “thinking in Deferreds” is better than “thinking in callbacks” but if I have not imagine this example from a fictional streaming API and maybe you’ll understand why “thinking in dataflows” is a good thing.

urls.flowTo(getPage)
    .flowTo(parse)
    .flowTo(getFirstStyle)
    .flowTo(getPage)
    .flowTo(stdout)

urls.put("http://example.com")
urls.put("http://google.com")

Now we have encapsulated not only how we handle one result, but rather a potentially infinite number of events. Now we have the potential for building some realtime distributed systems.

That is all.

Go read about Storm and Orc

I’ve been having fun recently on a new project where I put myself through all sorts of pain by nesting git submodules into team submodules into platform submodules and so on. The goal here is to be able to tag a root repository and thus identify exact commit hashes of all the submodules to any level. This was an idea Andoni had when he was working on livetranscoding in response to a request of mine where I want to be able to use a single ‘tag’ to identify a complete deployment.

That’s been working better than I expected, and I even hacked git-submodule-tools so that I can do git rlog and get a recursive git log between two root version tags, and get a list of every commit between the master and all submodules. That’s pretty neat for writing out release notes.

However, the way I embedded submodules causes a bit of pain when going back and forth. One of my hackers once gave me a PS1 bash prompt that includes info of which git branch you’re on in your shell prompt. So today I decided to extend that a little, and I now have this:

(b:release-0.2.x d:deploy-pro-2012-03-29) [thomas@otto platform]$ ls
Makefile platform puppet RELEASE-0.2.1
(b:release-0.2.x d:deploy-pro-2012-03-29) [thomas@otto platform]$ cd puppet/pro/
(s:puppet/pro b:release-0.2.x d:v0.2.1) [thomas@otto pro]$

This is showing me submodule name, branch, and description of the current commit.

If you want this for your prompting fun too, here’s the github repo

In the near future, simple portknocking for fun and profit with bash!

What is Klein?

klein is a micro-framework built out of widely used components that aims to be production ready out of the box.

Inspired by frameworks such as flask and bottle it features:

1. A minimalist API.
2. A fast, event-driven, production-ready web server.
3. Complete support for a wide range of asynchronous client APIs.

It’s routing is provided by werkzeug and it’s webserver & event-loop by Twisted.

An Example

from klein import run, route

@route('/')
def hello(request):
    return 'Hello, world!'

run('localhost', 8080)

This is the most simple example possible. It’s actually simpler than the first bottle example. This is the example you want to use in the “how fast can I serve a hard coded static string” benchmark you post to Hacker News

This post will not contain one of those benchmarks. Lets break it down.

from klein import run, route

run and route are the most basic APIs and it should be pretty clear what they are for. In this example we’ve imported the top-level functions which actually operate on a global instance of a the class klein.Klein.

This class may be used directly to facilitate a more flask-like usage. For example:

from klein import Klein

app = Klein()

@app.route('/')
def hello(request):
    return 'Hello, world!'

app.run('localhost', 8080)

Next we have the actual handler definition:

@route('/')
def hello(request):
    return 'Hello, world!'

The route decorator is built on top of Werkzeug and passes all unknown arguments directly to werkzeug.routing.Rule.

You may have noticed that unlike both flask and bottle handler functions in klein take an explicit request argument, which is an IRequest provider. This is primarily because “Explicit is better than Implicit”, but also because anything else would greatly complicate the implementation.

At this point we have our feet firmly planted in the world of Twisted. Though our example does not do anything asynchronous, or Twisted specific the full power of Twisted is available to us. You may return a str, unicode, twisted.web.template.Element, or twisted.web.resource.IResource, and klein will handle all of them for you. You may of course also return a twisted.internet.defer.Deferred which fires with any of the above as it’s result and we will handle that as well. The README of course has examples covering the full range of return values.

Finally we are ready to run the server.

run('localhost', 8080)

This starts up a twisted web server listening on port 8080 and only accessible to localhost. You can listen any specific IP address, or ‘0.0.0.0’ for all IP addresses. As a bonus logging to stdout is set up for you.

Work in Progress

klein is of course not complete, as no software is ever complete. It does however have a comprehensive test suite and builds on Twisted’s own heroicly extensive tests. We have lots of ideas about how to improve it and don’t plan to stop developing it. It is not currently used in production anywhere but it’s only a matter of time.

If you try it give us feedback in #twisted.web on freenode. And of course fork it on github.

The jury is still out on puppet as far as I’m concerned.

On the one hand, of course I relish that feeling of ultimate power you are promised over all those machines… I appreciate the incremental improvements it lets you make, and have it give you the feeling that anything will be possible.

But sometimes, it is just so painful to deal with. Agent runs are incredibly slow. It really shouldn’t take over a minute for a simple configuration with four machines. Also, does it really need to be eating 400 MB of RAM while it does so ? And when running with the default included web server (is that webrick ?), I have to restart my puppetmaster for every single run because there is this one multiple definition that I can’t figure out that simply goes away when you restart, but comes back after an agent run:
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Class[Firewall::Drop] is already defined; cannot redefine at /etc/puppet/environments/testing/modules/manifests/firewall/drop.pp:19 on node esp

And sometimes it’s just painfully silly. I just spent two hours trying to figure out why my production machine couldn’t complete its puppet run.

All it was telling me was
Could not evaluate: 'test' is not executable

After a lot of googling, I stumbled on this ticket. And indeed, I had a file called ‘test’ in my /root directory.

I couldn’t agree with the reporter more:

I find it incredibly un-pragmatic to have policies fail to run whenever someone creates a file in root which matches the name of an executable I am running.

• Books
• Books in the Sky
• Yeah, I Know: Go Social
• Human Data History
• Reality Merges
• Conclusion

• the secure location of our huts/houses/castles
• what we presented about ourselves in adornment/fashion (public data)
• what we could carry with us in bags/crates/vehicles

• DreamHost Pledges Support to OpenStack Project
• DreamHost and OpenStack sitting in a tree...
• DreamHost uses Dell’s Crowbar as a lever to raise automation (more on Crowbar here) 
• Web Host DreamHost Contributing Code for Ceph File System to OpenStack
• Ceph and OpenStack at OSCON 2011

• check the cache for a value
• get the value if present
• add a value if not present
• delete a value

On the bad side of life, I was planning to go to an awesome Calcotada in Lleida today, but I spent last night awake until 4:30 with an upset stomach, so I had to cancel and stay home feeling like shit.

On the good side of life, I really had no excuse left to not do a little long overdue hacking on Digital Audio Database.

I still use it regularly to listen to music, but the GNonLin-based player is just really not very stable. I should really just rewrite it using simply adder just like roughly ten years ago, but my brain won’t be able to do that. So instead I decided to clean up the web-based WebSockets using player I prototyped at OVC last year.

I started with some refactoring, clearly defining model/view/controller base classes and adapting the player and playerview classes to them.

WebSocket code seems to need an update every few months – I pulled the latest revision of txWebsocket on my fork, so my recent browsers actually play music again.

Since the last time I hacked on this, I actually added my 1500+ freshly ripped cd’s, in FLAC format – which browsers don’t actually support.

So, first off, I added an option for the scheduler – responsible for picking tracks, and picking audio files to represent them – to filter by extension. It’s not ideal, but it will do for now, and I punched that filter through the levels of abstraction in DAD. I now start it filtering on .mp3 and .oga, and so Chrome can play back all the tracks the scheduler throws at it.

The web-based player just loads tracks and timing info from the scheduler relative to page load time. I’ve been wanting to make that absolute for a while, so I did just that – the player server schedules tracks for epoch seconds now through websockets.

I had an entertaining half hour listening to the awesome echo effects obtained by having three chrome pages simultaneously playing the jukebox schedule – each page being slightly out of sync with the others.

As I’ll be wanting to use a smallish computer for music playback using a browser, I adapted the code to not use localhost any longer, but do everything with relative URL’s. Voila – the laptop now plays music too, a little bit more out of sync, and of course through its own speakers, adding to the eerie effect.

As an encore, I wanted to stumble my way through some jquery code, to which I’m a certified newbie. I want a nice background slideshow related to the current artist, and I pulled together echonest and bgstretcher-2 as an experiment.

That seems to work relatively well, except that the slideshow plugin doesn’t let you reload a new set of images to cycle through. And some of the other ones I tried instead after that seemed to have the same problem.

Oh well, it’s a start. If anyone knows of a good jquery background slideshow plugin that lets me update the list of url’s for images at any time, let me know!

• an interactive Python prompt for entering code directly using the iPhone keyboard
• a secondary, linear "keyboard" that one can use in conjunction with the main keyboard, extending one's ability to type faster
• multiple options for working with/preserving one's code (email, saving to a file, viewing command history)

First, my public-service announcement: if you went to PyCon, please make sure to answer the PyCon 2012 survey.

Second, my self-service: my crashing talk and lightning brain-hack talk.

Now that this is out of the way — this was a unique PyCon for me in many ways. First PyCon I could commute to, instead of staying in a hotel. I’m still torn about what I want to do for next year — commuting has advantages, but so does staying in a hotel. Driving 30 minutes back and forth was decidedly not fun. Being a local with a car was decidedly fun — being a local with a car who has been to this convention center 3-4 times before was even more fun. I knew the area, I could help others out and so on.

The other way it was unique is that I had tons of colleagues in this PyCon. There was the constant temptation to sit with them for lunch, and the constant fun of introducing them to everyone. I also tried my hand at recruiting at this PyCon, with limited success.

The last way it was special, if not quite unique, is that I have given a talk after 3 conferences where I did not. I forgot how much work it is to prepare a talk, and how nervous it makes me. Actually giving a talk is fun, but preparing for it was a bit nerve-wrecking. I was tweaking the presentation until the last day, making sure that it was exactly what I wanted. I definitely have tons of ideas for next year’s presentation. I also intend to submit a poster session — those were a lot of fun. I’ll probably try to go in with a partner for the poster session though.

On Friday, I thought it would be my “duty day”. I chaired a few sessions, and then had to give my talks. The keynotes were awesome, and the talks I chaired ranged between OK and Good (nothing that blew me out of the water, but nothing that had me going “oh, dear lord, make it stop”). What was apparently a mistake in the schedule had every speaker get an extra 10 minutes — that they used well, so I did not mind. For my talk, I was really happy with the way the green room worked this time. The runner took my laptop, connected it and all I had to worry was about getting wired up and then talking. I have to say, watching my video was kinda disappointing — I definitely need to do better on the “um”, “like” and “uuuuh” fronts. A lot of people liked my talk, though. At the Loggly party, I ate, and talked to a lot of fun people. After 90 minutes of it, I was full, I was starting to get tired and drove back home.

On Saturday, I gave a lightning talk in the morning. That was kind of an awesome experience, giving a talk to a room this big. Of course, only the non-commuters who knew how awesome lightning talks are were there, but still. I then hung out in the green room which was awesome/mistake, as I got roped into chairing the session where one talk was opposite of the “shooting squirrels with water guns” talk (guess why nobody volunteered to chair it). It was an OK session, all in all, though I thought the talk that was basically reading out loud a PyPI howto was mildly meh. For dinner, I went with a friend to the Mongolian Grill place, and then came back for my birthday. That was pretty awesome, partially because I got to introduce my wonderful wife to everyone.

Sunday had way more people in the lightning talks, and was kind of a short day. I drove to the Subway place for take-outs with a couple of people from the Twisted sprints, and then helped the Twisted newbies get started. I finally crashed and decided to go home and get a good night’s rest.

This work is licensed under a Creative Commons Attribution 3.0 Unported License.

def update_counter():
    fp = file("counter.txt")
    s = fp.read()
    counter = int(s.strip())
    counter += 1
    # If there is a crash before this point,
    # no changes have been done.
    fp = file("counter.txt.tmp", 'w')
    print >>fp, counter
    fp.close()
    # If there is a crash before this point,
    # only a temp file has been modified
    # The following is an atomic operation
    os.rename("counter.txt.tmp", "counter.txt")

def update_counter():
    fp = file("counter.txt")
    s = fp.read()
    counter = int(s.strip())
    counter += 1
    # If there is a crash before this point,
    # no changes have been done.
    fp = file("counter.txt.tmp", 'w')
    print >>fp, counter
    fp.close()
    # If there is a crash before this point,
    # only a temp file has been modified
    os.remove("counter.txt")
    # At this point, the state is inconsistent*
    # The following is an atomic operation
    os.rename("counter.txt.tmp", "counter.txt")

def recover():
    if not os.path.exists("counter.txt"):
        # The permanent file has been removed
        # Therefore, the temp file is valid
        os.rename("counter.txt.tmp", "counter.txt")

def update_counter():
    files = [int(name.split('.')[-1])
                  for name in os.listdir('.')
                      if name.startswith('counter.')]
    last = max(files)
    counter = int(file('counter.%s' % last).read().strip())
    counter += 1
    # If there is a crash before this point,
    # no changes have been done.
    fp = file("tmp.counter", 'w')
    print >>fp, counter
    fp.close()
    # If there is a crash before this point,
    # only a temp file has been modified
    os.rename('tmp.counter', 'counter.%s' % (last+1))
    os.remove('counter.%s' % last)

# This is not a recovery routine, but a cleanup routine
# Even in its absence, the state is consistent
def cleanup():
    files = [int(name.split('.')[-1])
                  for name in os.listdir('.')
                      if name.startswith('counter.')]
    files.sort()
    files.pop()
    for n in files:
        os.remove('counter.%d' % n)
    if os.path.exists('tmp.counter'):
        os.remove('tmp.counter')

0.log:
  ['add', 'key-0', 'value-0']
  ['add', 'key-1', 'value-1']
  ['add', 'key-0', 'value-2']
  ['remove', 'key-1']
  .
  .
  .

1.log:
  .
  .
  .

2.log:
  .
  .
  .

## First, some utility functions:

## Get the level of a file
def getLevel(s)
    return int(s.split('.')[0])

## Get all files of a given type
def getType(tp):
    return [(getLevel(s), s)
                 for s in files if s.endswith(tp)]

## Get all relevant files
def relevant(d):
    files = os.listdir(d):
    mlevel, master = max(getType('.master'))
    logs = getType('.log')
    logs.sort()
    return master+[log for llevel, log in logs if llevel>mlevel]

## Read in a single file
def update(result, fp):
    for line in fp:
        value = json.loads(line)
        if value[0] == 'add':
            result[value[1]] = value[2]
        else:
            del result[value[1]]

## Read in several files
def read(files):
    result = dict()
    for fname in files:
        try:
            update(result, file(fname))
        except ValueError:
            pass
    return result

## The actual data store abstraction.
class Store(object):
    def __init__(self):
        files = relevant(d)
        self.result = read(files)
        self.fp = None
        self.level = getLevel(files[-1])
        self._next()
    def _next(self):
        self.level += 1
        if self.fp:
            self.fp.close()
        self.fp = file('%3d.log' % self.currentLevel, 'w')
        self.rows = 0
    def get(self, key):
        return self.result[key]
    def add(self, key, value):
        self._write(['add', key, value])
    def _write(self, value):
        print >>fp, json.dumps(value)
        fp.flush()
        self.rows += 1
        if self.rows>200:
            self._next()
    def remove(self, key):
        print >>fp, json.dumps(['remove', key])
        fp.flush()

## This should be run periodically from a different thread
def compress(d):
    files = relevant(d)[:-1]
    if len(files)<2:
        return
    result = read(files)
    master = getLevel(files[-1])+1
    fp = file('%3d.master.tmp' % master, 'w')
    for key, value in result.iteritems():
        print >>fp, json.dumps(['add', key, value])
    fp.close()

def activate_due():
    scheduled = rs.smembers('scheduled')
    now = time.time()
    for el in scheduled:
        due = int(rs.get(el+':due'))
        if now<due:
            continue
        rs.sadd('activated', el)
        rs.delete(el+':due')
        rs.sremove('scheduled', el)

def recover():
    inconsistent = rs.sinter('activated', 'scheduled')
    for el in inconsistent:
        rs.delete(el+':due') #*
        rs.sremove('scheduled', el)

def forking_server():
    s = socket.socket()
    s.bind(('', 8080))
    s.listen(5)
    while True:
        client = s.accept()
        newpid = os.fork()
        if newpid:
            f = client.makefile()
            f.write("Sunday, May 22, 1983 18:45:59-PST")
            f.close()
            os._exit()

## Process one
class SchedulerResource(resource.Resource):
    isLeaf = True
    def __init__(self, filepath):
        resource.Resource.__init__(self)
        self.filepath = filepath
    def render_PUT(self, request):
        uuid, = request.postpath
        content = request.content.read()
        self.filepath.child(uuid).setContent(content)
s = server.Site(SchedulerResource(filepath.FilePath("things")))
reactor.listenTCP(8080, s)

## Process two
rs = redis.Redis(host='localhost', port=6379, db=9)
while True:
    for fname in os.listdir("things"):
        when = int(file(fname).read().strip())
        rs.set(uuid+':due', when)
        rs.sadd('scheduled', uuid)
        os.remove(fname)
    time.sleep(1)

## Process three
rs = redis.Redis(host='localhost', port=6379, db=9)
recover()
while True:
    activate_due()
    time.sleep(1)

## Process four
rs = redis.Redis(host='localhost', port=6379, db=9)
channel = pika.BlockingConnection(pika.ConnectionParameters(
              'localhost')).channel()
channel.queue_declare(queue='active')
while True:
    activated = rs.smembers('activated')
    finished = set(rs.smembers('finished'))
    for el in activated:
        if el in finished:
            continue
        channel.basic_publish(exchange='',
                              routing_key='active',
                              body=el)
        rs.add('finished', el)

## Process five
# It is possible to get "dups" of bodies.
# Application logic should deal with that
channel = pika.BlockingConnection(pika.ConnectionParameters(
              'localhost')).channel()
channel.queue_declare(queue='active')
def callback(ch, method, properties, el):
    syslog.syslog('Activated %s' % el)
channel.basic_consume(callback, queue='hello', no_ack=True)
channel.start_consuming()

## In a Twisted process
def beat():
    file('beats/my-name', 'a').close()
task.LoopingCall(beat).start(30)

## Watchdog
while True:
    now = time.time()
    timeout = dict()
    for heart in glob.glob('hearts/*'):
        beat = int(file(heart).read().strip())
        timeout[heart] = now-beat
    for heart in glob.glob('beats/*'):
        if not os.path.isfile(heart):
            mtime = 0
        else:
            mtime = os.path.getmtime(heart)
        problem = 'problems/'+heart
        if (mtime<timeout[heart] and
           not os.path.isfile(problem)):
            fp = file('problems/'+heart, 'w')
            fp.write('watchdog')
            fp.close()
    problemTimeout = now-30
    for problem in glob.glob('problems/*'):
        if os.path.getmtime(problem)<problemTimeout:
            subprocess.call(['restart-system'])
    time.sleep(1)

<a rel=”license” href=”http://creativecommons.org/licenses/by/3.0/”><img alt=”Creative Commons License” style=”border-width:0″ src=”http://i.creativecommons.org/l/by/3.0/80×15.png” /></a><br />This work is licensed under a <a rel=”license” href=”http://creativecommons.org/licenses/by/3.0/”>Creative Commons Attribution 3.0 Unported License</a>.

This is a follow-up for my earlier post.

Here I intend to give some practical advice! But first, allow me to digress — why are you up there speaking, and why are these people listening to you?

If they really were consumed by a desire to know, they would read your blog posts about the topic. They are not there because they are missing knowledge. They are there because they think you will be entertaining. Optimize for being entertaining, and you might be able to teach them something. Optimize for teaching, and you will probably be boring, and they will learn nothing.

Remember: public speaking is a performance art. Treat your technical talk as a performance, and you will do much better. Now, performance doesn’t mean you make it into a circus: you want to give the audience what they expect, more or less. In a technical talk, they expect to be fascinated by stories of the interaction between computers and people.

Hollywood writing has a concept called “Save the Kitten”. Near the beginning of the movie, the hero does something good (“saving the kitten”) which makes the audience care about him. Now, when he does something stupid and has to go on a quest to fix it, you care enough about him to follow along. Likewise, you need to set yourself up as a hero in the beginning of the talk — subtly.

The talk I went to about metaclasses set the speaker as a hero because he outlined a Prometheus narrative of bringing fire from the Gods to the mortals. The talk about stopping to write classes set the hero as the person who helps clean up other people’s code. In my talk, I told a story about visiting a customer and seeing how they didn’t notice many errors because of our crash recovery system. I could outline many other examples, but this ultimately has to be about you.

Apologies are rarely entertaining. If you are already a good speaker, you can usually find an entertaining way to apologize but as a novice just refrain from that at all. A self-deprecating joke would do much better if something screwed up.

• Wrong: “Sorry, I completely forgot to talk about something 3 slides ago…”
• Right: “Oh, hey, three slides ago I should have told you something else. Let’s go jump in the time machine and fix it…”

Put pictures on your slides. People like looking at pictures. Have them somewhat related to your talk, although it’s ok if the connection is obscure. Puzzled people are a bit distracted, but they are not bored. They might not listen to you — but at least they will not walk away thinking that they are glad to have caught up on their sleep.

Put less words on your slides. If you must put words, have them be a “teaser” that you explain. There is a reason shows on TV starts with a teaser — people will keep their attention to find a solution. If you put code, try to make it less than 10 lines, including comments. It is ok to use code in a style that would not pass muster in production to illustrate a point.

Practice your talk. Do it in front of a video camera. I am still working on reducing my “ums” and “likes”. Accents, however, are not bad. Sure, they make the audience work harder to understand you, which works against the “teaching” goal. But they work for the “entertaining” goal because, let’s face it, accents are funny.

Lastly, remember that unless you’re the human cannon, people are not watching you to see you fail. They want to see you succeed. Draw confidence from people, not fear.

Perhaps the reason we have not seen single-page html applications that connect directly to peers without an intermediate server is that browsers cannot easily listen on a local port. They can open outgoing connections all day long, but WebRTC may be the first web standard that allows the browser to listen on a port. If there are others, please let me know.

Of course, the WebRTC spec looks overly complicated for the incredibly simple thing I want to do. I just want the browser to be able to listen on a port like any other process on the machine can. Sure, there are security implications, but these exist for everything the browser exposes to web applications, and there's an entire class of Peer-to-Peer web apps that simply cannot easily be written using current web technologies.

There are many examples of a Peer-to-Peer experience being delivered to users using a client-server architecture. ChatRoulette, Omegle, and even more recently, products like Google Hangouts. These applications must be implemented by using servers in the middle to connect the peers, making scaling them much harder than it would be if browsers could just listen.

There is an opportunity to explore a generic Client-Agent-Peer architecture, where Clients (Browsers) talk to an Agent server using HTTP to configure the state of the Agent, which would then be contacted by the Peer on behalf of the Client when the Browser is not online. When the Browser is online, the Agent can refer the Peer directly to the Client. When the Browser is offline, the Agent can handle the request itself using a cached copy of the material the Browser was sharing, or it can just decline to fulfill the request.

Reverse HTTP was my attempt to push out the simplest thing that could possibly work to get browsers to talk to each other. It didn't really go anywhere in terms of being implemented in actual browsers. Coincidentally, someone else had the same idea around the same time, and implemented Reverse HTTP in terms of actual HTTP Requests encoded in Responses, and vice versa. This makes it possible to write a pure javascript client rather than needing the browser to support the Upgrade protocol itself.

Really, though, it would be very nice if all of the hacks and tricks and workarounds weren't necessary, and I could just listen on a port with javascript. I'll keep dreaming.

After last week’s Linode incident I was getting a bit more worried about security than usual. That coincided with the fact that I found I couldn’t run puppet on one of my linodes, and some digging turned up that it was because /tmp was owned by uid:gid 1000:1000. Since I didn’t know the details of the breakin (and I hadn’t slept more than 4 hours for two nights, one of which involving a Flumotion DVB problem), I had no choice but to be paranoid about it. And it took me a good half hour to realize that I had inflicted this problem on myself – a botched rsync command (rsync arv . root@somehost:/tmp).

So I wasn’t hacked, but I still felt I needed to tighten security a bit. So I thought I’d go with something simple to deploy using puppet – port knocking.

Now, that would be pretty easy to do if I just deployed firewall rules in a single set. But I started deploying firewall rules using the puppetlabs firewall module, which allows me to group rules per service. So that’s the direction I wanted to head off into.

On saturday, I worked on remembering enough iptables to actually understand how port knocking works in a firewall. Among other things, I realized that our current port knocking is not ideal – it uses only two ports. They’re in descending order, so usually they would not be triggered by a normal port scan, but they would be triggered by one in reverse order. That is probably why most sources recommend using three ports, where the third port is between the first two, so they’re out of order.

So I wanted to start by getting the rules right, and understanding them. I started with this post, and found a few problems in it that I managed to work out. The fixed version is this:
UPLINK="p21p1"
#
# Comma seperated list of ports to protect with no spaces.
SERVICES="22,3306"
#
# Location of iptables command
IPTABLES='/sbin/iptables'

# in stage1, connects on 3456 get added to knock2 list
${IPTABLES} -N stage1
${IPTABLES} -A stage1 -m recent --remove --name knock
${IPTABLES} -A stage1 -p tcp --dport 3456 -m recent --set --name knock2

# in stage2, connects on 2345 get added to heaven list
${IPTABLES} -N stage2
${IPTABLES} -A stage2 -m recent --remove --name knock2
${IPTABLES} -A stage2 -p tcp --dport 2345 -m recent --set --name heaven

# at the door:
# - jump to stage2 with a shot at heaven if you're on list knock2
# - jump to stage1 with a shot at knock2 if you're on list knock
# - get on knock list if connecting t0 1234
${IPTABLES} -N door
${IPTABLES} -A door -m recent --rcheck --seconds 5 --name knock2 -j stage2
${IPTABLES} -A door -m recent --rcheck --seconds 5 --name knock -j stage1
${IPTABLES} -A door -p tcp --dport 1234 -m recent --set --name knock

${IPTABLES} -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
${IPTABLES} -A INPUT -p tcp --match multiport --dport ${SERVICES} -i ${UPLINK} -m recent --rcheck --seconds 5 --name heaven -j ACCEPT
${IPTABLES} -A INPUT -p tcp --syn -j door

# close everything else
${IPTABLES} -A INPUT -j REJECT --reject-with icmp-port-unreachable

And it gives me this iptables state:

So the next step was to reproduce these rules using puppet firewall rules.

Immediately I ran into the first problem – we need to add new chains, and there doesn’t seem to be a way to do that in the firewall resource. At the same time, it uses the recent iptables module, and none of that is implemented either. I spent a bunch of hours trying to add this, but since I don’t really know Ruby and I’ve only started using Puppet for real in the last two weeks, that wasn’t working out well. So then I thought, why not look in the bug tracker and see if anyone else tried to do this ? I ask my chains question on IRC, while I find a ticket about recent support. A minute later danblack replies on IRC with a link to a branch that supports creating chains – the same person that made the recent branch.

This must be a sign – the same person helping me with my problem in two different ways, with two branches? Today will be a git-merging to-the-death hacking session, fueled by the leftovers of yesterday’s mexicaganza leftovers.

I start with the branch that lets you create chains, which works well enough, bar some documentation issues. I create a new branch and merge this one on, ending up in a clean rebase.

Next is the recent branch. I merge that one on. I choose to merge in this case, because I hope it will be easier to make the fixes needed in both branches, but still pull everything together on my portknock branch, and merge in updates every time.

This branch has more issues – rake test doesn’t even pass. So I start digging through the failing testcases, adding print debugs and learning just enough ruby to be dangerous.

I slowly get better at fixing bugs. I create minimal .pp files in my /etc/puppet/manifests so I can test just one rule with e.g. puppet apply manifests/recent.pp

The firewall module hinges around being able to convert a rule to a hash as expressed in puppet, and back again, so that puppet can know that a rule is already present and does not need to be executed. I add a conversion unit test for each of the features that tests these basic operations, but I end up actually fixing the bugs by sprinkling print’s and testing with a single apply.

I learn to do service iptables restart; service iptables stop to reset my firewall and start cleanly. It takes me a while to realize when I botched the firewall so that I can’t even google (in my case, forgetting to have -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
) – not helped by the fact that for the last two weeks the network on my home desktop is really flaky, and simply stops working after some activity, forcing me to restart NetworkManager and reload network modules.

I start getting an intuition for how puppet’s basic resource model works. For example, if a second puppet run produces output, something’s wrong. I end up fixing lots of parsing bugs because of that – once I notice that a run tells me something like
notice: /Firewall[999 drop all other requests]/chain: chain changed '-p' to 'INPUT'
notice: Firewall[999 drop all other requests](provider=iptables): Properties changed - updating rule
I know that, even though the result seems to work, I have some parsing bug, and I can attack that bug by adding another unit test and adding more prints for a simple rule.

I learn that, even though the run may seem clean, if the module didn’t figure out that it already had a rule (again, because of bogus parsing), it just adds the same rule again – another thing we don’t want. That gets fixed on a few branches too.

And then I get to the point where my puppet apply brings all the rules together – except it still does not work. And I notice one little missing rule: ${IPTABLES} -A INPUT -p tcp –syn -j door

And I learn about –syn, and –tcp-flags, and to my dismay, there is no support for tcp-flags anywhere. There is a ticket for TCP flags matching support, but nobody worked on it.

So I think, how hard can it be, with everything I’ve learned today? And I get onto it. It turns out it’s harder than expected. Before today, all firewall resource properties swallowed exactly one argument – for example, -p (proto). In the recent module, some properties are flags, and don’t have an argument, so I had to support that with some hacks.

The rule_to_hash function works by taking an iptables rule line, and stripping off the parameters from the back in reverse order one by one, but leaving the arguments there. At the end, it has a list of keys it saw, and hopefully, a string of arguments that match the keys, but in reverse order. (I would have done this by stripping the line of both parameter and argument(s) and putting those on a list, but that’s just me)

But the –tcp-flags parameter takes two arguments – a mask of flags, and a list of flags that needs to be set. So I hack it in by adding double quotes around it, so it looks the same way a –comment does (except –comment is always quoted in iptables –list-rules output), and handle it specially. But after some fidgeting, that works too!

And my final screenshot for the day:

So, today’s result:

• fixes for the recent module
• an implementation of –tcp-flags
• some small fixes for firewallchain
• A feature integration branch pulling all of these together

Now, I have a working node that implements port knocking:
node 'ana' {

$port1 = '1234'
$port2 = '3456'
$port3 = '2345'

$dports = [22, 3306]

$seconds = 5

firewall { "000 accept all icmp requests":
proto => "icmp",
action => "accept",
}

firewall { "001 accept all established connections":
proto => "all",
state => ["RELATED", "ESTABLISHED"],
action => "accept",
}

firewall { "999 drop all other requests":
chain => "INPUT",
proto => "tcp",
action => "reject",
}

firewallchain { [':stage1:', ':stage2:', ':door:']:
}

# door
firewall { "098 knock2 goes to stage2":
chain => "door",
recent_command => "rcheck",
recent_name => "knock2",
recent_seconds => $seconds,
jump => "stage2",
require => [
Firewallchain[':door:'],
Firewallchain[':stage2:'],
]
}

firewall { "099 knock goes to stage1":
chain => "door",
recent_command => "rcheck",
recent_name => "knock",
recent_seconds => $seconds,
jump => "stage1",
require => [
Firewallchain[':door:'],
Firewallchain[':stage1:'],
]
}

firewall { "100 knock on port $port1 sets knock":
chain => "door",
proto => 'tcp',
recent_name => 'knock',
recent_command => 'set',
dport => $port1,
require => [
Firewallchain[':door:'],
]
}

# stage 1
firewall { "101 stage1 remove knock":
chain => "stage1",
recent_name => "knock",
recent_command => "remove",
require => Firewallchain[':stage1:'],
}

firewall { "102 stage1 set knock2 on $port2":
chain => "stage1",
recent_name => "knock2",
recent_command => "set",
proto => "tcp",
dport => $port2,
require => Firewallchain[':stage1:'],
}

# stage 2
firewall { "103 stage2 remove knock":
chain => "stage2",
recent_name => "knock",
recent_command => "remove",
require => Firewallchain[':stage2:'],
}

firewall { "104 stage2 set heaven on $port3":
chain => "stage2",
recent_name => "heaven",
recent_command => "set",
proto => "tcp",
dport => $port3,
require => Firewallchain[':stage2:'],
}

# let people in heaven
firewall { "105 heaven let connections through":
chain => "INPUT",
proto => "tcp",
recent_command => "rcheck",
recent_name => "heaven",
recent_seconds => $seconds,
dport => $dports,
action => accept,
require => Firewallchain[':stage2:'],
}

firewall { "106 connection initiation to door":
# FIXME: specifying chain explicitly breaks insert_order !
chain => "INPUT",
proto => "tcp",
tcp_flags => "FIN,SYN,RST,ACK SYN",
jump => "door",
require => [
Firewallchain[':door:'],
]
}
}
and I can log in with
nc -w 1 ana 1234; nc -w 1 ana 3456; nc -w 1 ana 2345; ssh -A ana

Lessons learned today:

• watch iptables -nvL is an absolutely excellent way of learning more about your firewall – you see your rules and the traffic on them in real time. It made it really easy to see for example the first nc command triggering the knock.
• Puppet is reasonably hackable – I was learning quickly as I progressed through test and bug after test and bug.
• I still don’t like ruby, and we may never be friends, but at least it’s something I’m capable of learning. Puppet might just end up being the trigger.

Tomorrow, I need to clean up the firewall rules into something reusable, and deploy it on the platform.

• I did all of this in precise, and it seemed to work just fine. I substituted 'precise' for 'oneiric' wherever I saw it.
• I had to manually add my user to the libvirtd group. The instructions implied that this was not necessary.
• juju bootstrap finished very quickly for me and without any actual evidence that it had debootstrapped the whole OS.
• When I deployed services, I didn't get any feedback that anything was going on. The recommended debug-log command showed very little output for a while. My guess is that this was while packages were downloading.

• testing out development deployments of OpenStack using Vagrant (some successes, some blockers)
• testing out dev deployments of OpenStack using VirtualBox directly
• filed some bugs for issues in horizon regarding error feedback to users and how the documentation is generated
• dug into issues with logging and inconsistencies in datestamps
• uncovered some weirdness with the usage of gnu screen and hanging services/partial devstack installs due to sudo assumptions (devstack assumes a passwordless sudo, and will label an install as failed if it gets hung up on the apache log tail, waiting for a password, even if the install was successful and all the services started correctly)
• Doug Hellmann made his first commit upstream to OpenStack

• Thanks for Zenoss for the fun swag today (the Zebras are still staring at me... I think they're going to be making an appearance in ToyStory 5)
• Even more thanks to Zenoss for the offer to become an OpenStack Atlanta Sponsor (food, drinks, and swag)!
• Thanks to DreamHost for the AMAZING coffee and danishes from The Village Corner Restaurant/Basket Bakery. Seriously. That was the best coffee I have ever had. In. My. Life.
• Also, thanks DreamHost for the pizza and the sweet potato pies!

Elsewhere, I wrote about the beginning of growing season and some software I've written to help us out this year. The software I was talking about is very spartan right now. It tries to serve exactly our needs, with just enough user interface so that we can get at the information we need. If you notice where exactly on Launchpad I'm currently hosting it, you'll get some idea about how much effort I've put into making this a real, distributable, useful-to-anyone-else project so far.

What the software does at this point is this:

• Load data from a (semi-)structured file (csv, because it's easy to create and export data in this format using Open Office). The data it can load describes certain crops and certain varieties of those crops, including information about start and end of season, required growing days, anticipated yields, etc.
• Plan out a seed order, based on that yield data and additional product data (also in the input file). Doing this without wasting a ton of money ends up being something like a solution to the covering problem, due to discounts for buying greater quantities (sometimes unbelievable discounts, with marginal costs for additional seed ranging as low as 5% of the base cost). This is also a very tedious part of the program, as common suppliers offer seed in well over a dozen different package sizes (with "packages" with the same name containing different amounts of seed for different kinds of vegetables, and of course different vegetables requiring different amounts of seed to produce a particular yield).
• Predict various kinds of resource usage at each point in the season. Resources include things like bed feet (eg, we have 22 beds, each 100 feet long, so we have 2200 bed feet; our crop plan cannot exceed this, or we'll have plants that have nowhere to be planted), plug flag usage (where seeds are started and grow until they're hardy enough to be transplanted outside), and man hours (there are two of us, we don't want to plant so much that we would need to hire help to deal with it).
• Generate a schedule of when to seed each variety, when to expect to transplant them outdoors, and when to harvest them. The schedule can be displayed as a list or it can be generated as an iCalendar file and loaded into something like Google Calendar or Apple's iCal.

These are all pretty basic pieces of information that someone growing vegetables would want to know. On a small scale, they're the kinds of things you can plan out in your head, or keep track of on paper. As you want to do more, though, it can be overwhelming. For example, our schedule for this season has 376 events on it. I wouldn't have wanted to generate that manually.

There is also some rudamentary graphing functionality. This is for visualizing some of the pieces of information I mentioned above (eg plug flat usage). So far this part has been mostly for fun, as it's hard to make any additional specific decisions based on the graphs, as opposed to the textual, numerical output also generated. One thing it has been useful for, though, is sanity checking the output. It's easier to see a crazy spike or a mysterious plateau on a graph than in numerical data.

As far as the implementation goes, there's nothing really fancy going on here. I've added a lot of features that I hadn't originally planned on (or realized would be useful). As I mentioned, this is a new domain for me to be working in. There is some unit test coverage now, but I didn't start out doing test-driven development. This has bitten me a few times already, as some of the scheduling logic is subtle enough that I can't change it without introducing bugs. Fortunately that part of the code is somewhat well tested now. Well, not completely untested, at least. Development has been test-driven for a month or two now, so I expect things to get easier going forward.

Everything is written in Python, of course. I used vobject to generate the iCalendar output, with pytz to help with the timezone math (oh, timezones, how I loathe you). A pleasantly small amount of code suffices for that.

I used matplotlib and dateutil to generate the graphs. I have a tolerate/hate relationship with matplotlib. It clear does a lot of stuff, and I've seen people use it to good effect. Most of its functionality escapes me, though, and I can hardly learn about a new API without observing that it is completely terrible. Still, I used it because it can do the job, and better than the other options, in my experience.

For the highly tedious structure definition, I used a class from Epsilon. epsilon.structlike.record is a lot like the Python standard library collections.namedtuple. Any time I used the latter, though, I remember how it is implemented and I feel bad. So I stick to the former.

I also used Twisted and html5lib to write a simple web scraper to turn variety names into Johnny's product identifiers. Even if ordering seeds this way ends up being a one-off task, writing the scraper to get this information was definitely easier than chasing down product identifiers in a Johnny's catalog or from the Johnny's website, which each have their own... unique approach to organization. I asked Johnny's if they could make this information available in any sort of structured format and they told me they couldn't. Maybe I should sell it back to them?

Many features are still missing from the planning software. Some of them are simple, like reporting how many flats to seed in the iCalendar event it generates, instead of just reporting how many bed feet will be used after the seeds germinate and are transplanted out into the field. Others are a bit bigger, like having a more coherent model for the underlying data. I might want to put this off until the end of the season, when I might have a better idea if I've fully understood the underlying data myself.

I don't expect this to be useful to a lot of people. In case this sort of tool does appeal to you, though, I'd love feedback (particularly from people more experienced with planning and executing these kinds of agricultural tasks) - but no feature requests, please :)

Well, this sure has been a long time in the making.

Fluendo and Collabora have a checkered past which I won’t get into, but on paper it has always made sense for these two companies to collaborate and making GStreamer work commercially. One company specializes in products, the other in consulting (I’m sure you can figure out which is which), and complement each other perfectly to make GSstreamer more successful commercially.

I personally have always believed that we need to get GStreamer to other platforms and make them as easy to use as possible. Windows was an obvious target in the past, and now Android is another. There is a big difference between a successful open source project, and a commercially successful one. Flumotion’s Andoni Morales who came with me to the GStreamer 0.11 hackfest in Malaga is going to be working on this one SDK to rule them all.

Christian beat me to it in the blogosphere, but the word is now officially out! Feel free to read Fluendo’s press release.

• A fix to the GTK2 reactor preventing unnecessary wake-ups 
• Preliminary support of IPV6 on the server side
• Several fixes to the new protocol-based TLS implementation
• Improved core documentation's main page 

Today I needed an xml diff tool. There seem to be an xmldiff and a diffxml, neither of them packaged by Fedora at the moment. I found an old src.rpm for Fedora 6 for xmldiff 0.6.8

The src.rpm doesn’t rebuild as is for various reasons: its %check stage imports unittest2, so I disabled check. Then it didn’t find debuglist.files, so I disabled building a debug package. And then it found an installed egg file which it didn’t like, so I disabled checking for installed files.

Since I’m going to forget how I did this in the future when I will need this again for some obscure reason, and because if you ever build rpms you may find this useful, here is the command that did it for me for Fedora 15:
rpmbuild --rebuild -D '%check exit 0' -D 'debug_package %{nil}' -D '_unpackaged_files_terminate_build exit 0' xmldiff-0.6.8-1.fc6.src.rpm

Now, back to comparing xml files.