Planet Twisted

Twisted Matrix Laboratories: Migration Report

Tom Prince (noreply@blogger.com) — 2013-05-19T06:45:55+00:00

I have completed the migration scripts for deploying the services currently running on cube. They have been run against our new machine, dornkirk which is currently running with a snapshot of data.

It can currently be accessed by putting

66.35.39.66     twistedmatrix.com speed.twistedmatrix.com

in /etc/hosts. Please tests it, and verify that things appear to be working, but be aware that any changes will be lost, when the transition occurs.

At some point Monday or Tuesday, there will be some downtime for mail and the mailing lists, as mail-service is migrated to the new machine. For those that have accounts on cube, your data will be copied to the new machine at this point.

On Wednesday, at about 10 MDT (16 UTC), there will be downtime of all twisted services, as live data is transfered over. This may last up-to a couple of hours.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!

Thomas Vander Stichele: Organizing photo libraries

Thomas — 2013-05-18T11:51:02+00:00

The weather’s picking up so it’s time for spring cleaning around the house. When I moved back to Barcelona three years ago I took with me my old analogue photos and negatives, with the idea of sorting through them at some point and getting them digitized. And while I’m at it, maybe it’s time to pull all my various folders of photos together too and organize them.

Well, I finally started. I grouped the negatives, labeled them by year, put them in individual envelopes, and handed them off to a professional lab to scan them after doing a quick test run on one set (which turned out great, but it’s *really* annoying me that they scan to JPEG by default, charge 40% extra for TIFF, and use a non-multiple-of-8 resolution to scan at which means I can’t losslessly rotate the negatives. Yes, I’m anal.)

So now I pulled together all my various folders of photos, and before I start doing tagging and stuff like that, I want to organize them in a decent folder layout. Googling for ideas pretty much suggests that the way to go is

YYYY/MM/DD

with possibly some description together with the DD

I’m not really happy about that, however, because there are certain things I’d like to be able to do:

easily see where photos come from – did I make them ? did I get them from someone ? Did I download them from Facebook ?
Are these original files from a camera without editing ?
Are these the original scans ? From negatives ? From actual photos ? Or are they retouched, rotated, denoised, …
Are these photos SFW ? Can I point my media center slideshow to this directory and have it safely show any photos under it ? (What do you mean, you’ve never snowboarded at night in only your underwear, and mooning the photographer ?) Or maybe not even SFW, but simply watchable and reasonable quality or subject material?

I realize some of these issues can not be resolved simply with a directory layout. But I’m sure some of you must have had similar issues or come up with a slightly better layout ?

Point me in the right direction please.

Thomas Vander Stichele: morituri and Hidden Track One Audio

Thomas — 2013-05-10T20:08:31+00:00

I have tomorrow (saturday) blocked out for a whole day of morituri hacking as I will be home alone.

One of the things a lot of morituri users are puzzled by is its relentless drive to extract every single sample of audio from the CD. Currently, even if it’s a really short pre-gap, and most likely just an inaccurate master or burn, with no useful audio in it.

For me, that was a design goal of morituri – I want to be able to exactly reproduce a CD as is. That is to say, ripping a CD should extract *all* audio from the CD, and it should be possible to make a copy of that CD and then rip that copy, and end up with exactly the same result as from the original CD. (I’m sure there’s a fancy scientific term for that that I can’t remember right now)

To a lot of other people, it seems to be annoying and they don’t like having those small almost empty files lying around.

So I thought I’d do something about that, and that it might be useful as well to analyze my current collection of tracks and figure out what’s in there. Maybe I can find some hidden gems that I hadn’t noticed before?

So I added a quick task to morituri that calculates the maximum sample value (I didn’t want to use my own level element in GStreamer for this as I wanted to make sure it was actual digital zero; this should be done in an element instead though, but I preferred the five minute hack for this one).

And then I ran:

rip debug maxsample /mnt/nas/media/audio/rip/morituri/own/album/*/00*flac

Sadly, that turned up 0 as the biggest sample for all these tracks!

Wait, what? I spent all that time on getting those secret tracks ripped just to get none? That’s not possible! I know some of those tracks!

Maybe the algorithm is wrong. Nope, it works fine on all the regular tracks.

Oh, crap. Maybe morituri has been ripping silence all this time because my CD drive can’t get that data off. Yikes, that would be a bit of egg on my face.

No, it works if I check that Bloc Party track I know about.

Ten minutes of staring at the screen to realize that, while I was outputting names from a variable from the for loop over my arguments, the track I was actually passing to the task was always the first one. Duh. Problem solved.

As for what I found in my collection:

a cute radio jingle that brought back memories from a live bootleg I had made myself of Bloem. That’s from over ten years ago, but that must have been around the time I learned about the existence of HTOA and wanted to get one in
found unknown HTOA tracks on Art Brut’s Bang Bang Rock & Roll, Mew’s Half the world is watching me; not their best stuff
soundscapey or stagesetting tracks on QOTSA’s Songs for the Deaf, Motorpsycho’s Angels and Daemons at play And Blissard; not that worth it (the Blissard track was ok, but really quiet)
Pulp hid a single piano chord in a 2 second pre-gap on This is Hardcore; very curious. It’s not an intro to the first track, because it doesn’t fit with the sound at all.
Damien Rice hid a demo version of 9 Crimes (the first track) in the pregap; instead of piano and female vocals, he plays guitar and sings all the parts.
Got reacquainted with my favourite HTOA tracks: the orchestral quasi-wordless medley on the Luke Haines/Das Capital disc; the first Bloc Party album with a beautiful instrumental (up there with the hidden track at the end of Placebo’s first album; both bands delivering an atypical but stunning moodscape; the beautiful cover of Ben Kenobi’s Theme by Arab Strap on the Cherubs EP (no idea why that landed in my album dir, that needs to be fixed); the silly Soulwax skit for their second album.

Of course, Wikipedia has the last word on everything

I note that they think Pulp recorded a cymbal, not a piano. And now that I see the title of the QOTSA hidden track, I get the joke I think.

In total, on my album collection of 1564 full CD’s, I have 171 HTOA’s ripped, 138 tracks of pure digital silence, and only about 11 are actually useful tracks.

I expected to find more gems in my collection. I’ll go through ep’s, singles and compilations next just to be sure.

But with this code in hand, maybe it’s time to add something to morituri to save the silent HTOA tracks as pure .cue information.

Thomas Vander Stichele: Votes for talks at open source conferences

Thomas — 2013-05-07T11:53:26+00:00

I’ve never been a fan of voting for talks, because it tends to be poorly implemented under the guise of democracy. Of course it’s easy for me to talk, I’ve never organized anything at that scale.

I’ll give two examples on why I feel this way, one of which triggering today’s blog post.

First off, my colleague Marek submitted a talk to Djangocon. The talk was about how to use feat (a toolkit we wrote for livetranscoding) to serve Django pages, but in such a way that they can use Deferreds to remove the concurrency bottleneck of “1 request at a time” per process running Django.

Personally, to me, this is one of the most irritating design choices of Django – from the ground up it was built synchronously (which could have been fine in most places). But the fact that, when you get a request, you have to always synchronously respond to it (and block every other request for that process in the meantime) is a design choice that could have easily been avoided.

In our particular use case, it was really painful. If our website has to do an API request to some other service we don’t control that can easily take 30 seconds, our process throughput suddenly becomes 2 pages per minute. All the while, the server is sitting there waiting.

Yes, you can throw RAM at the problem and start 30 times more processes; or thread out API requests; or farm it out to Celery, and do some back-and-forthing to see when the call’s done. Or do any other number of workarounds for a fundamental design choice.

Since we like Twisted, we preferred to throw Twisted at the problem, and ended up with something that worked.

Anyway, that’s a lot of setup to explain what the talk was about. Marek submitted the talk to DjangoCon, and honestly I didn’t expect it to get much traction because, when you’re inside Django, you think like Django, and you don’t really realize that this is a real problem. Most people who do realize it switch away to something else.

But to my surprise, Marek’s talk was the most-voted talk! I wish I could link to the results, but of course that vote site is no longer online.

I guess I expected that would mean he’d be presenting at DjangoCon this year. So I asked him today when his talk was, and he said “Oh that’s right. I did not get accepted.”

Well, that was a surprise. Of course, the organising committee reserves the right to decide on their own – maybe they just didn’t like the talk. But if you ask your potential visitors to vote, you’d expect the most-voted talk to make it on the schedule no ?

The feedback Marek got from them was surprising too, though. Their first response was that this talk was too similar to another talk, titled “How to combine JavaScript & Django in a smart way”. Now, I’m not a JavaScript expert, but from the title alone I can already tell that it’s very unlikely that these two talks have many similarities beyond the word ‘Django’.

After refuting that point, their second reason was that they wanted more experienced speakers (but they didn’t ask Marek for his experience), and their third reason was that the talk was in previous editions of DjangoCon US/EU (it’s unclear whether they meant his talk or the JavaScript one, but Marek’s definitely wasn’t, and we couldn’t find any mention of the other talk in previous conferences. I’m also not sure why that even matters one way or the other. This email thread was in Polish, so I have to rely on Marek’s interpretation of it)

Personally, my reaction would have been to complain to the organizers or Django maintainers. Marek’s flegmatic attitude was much better though – after such an exchange, he simply doesn’t want to have anything to do with the conference.

He’s probably right – it’s hard to argue with someone who doesn’t want to invite you and is lying about the reasons.

The second example is BCNDevCon, a great conference here in Barcelona, organized by a guy who used to work for Flumotion who I have enormous respect for. I’ve never seen anyone create such a big conference over so little time.

He believes strongly in the democratic aspect, and as far as I can tell constructs the schedule solely based on the votes.

Sadly I didn’t go to the last one, and the reason is simply because I felt that the talks that made it were too obviously corporate. A lot of talks were about Microsoft products, and you could tell that they won votes because people’s coworkers voted on talks. I’m not saying that’s necessarily wrong – given that he worked at our company and has friends here, I’m sure people working here presenting at his conference have also done vote tending. It’s natural to do so. But there should be a way to balance that out.

I think the idea of voting is good, but implementation matters too. Ideally, you would only want people that actually are going to show up to vote. I have no idea how you can ensure that, though. Do you ask people to pre-pay ? Do you ask them to commit to pay if at least 50% of their votes make it in the final schedule, kickstarter-style ?

These two examples are on opposite extremes of voting. One conference simply disregards completely what people vote on. If I had voted or bought a ticket, I would feel lied to. Why waste the time of so many people? The other conference puts so much stock in the vote, that I feel the final result was strongly affected. I seriously doubt all those Windows 8 voters actually showed up.

Does anyone have good experiences with conference voting that did work? Feel free to share!

Twisted Matrix Laboratories: April end-month report

Tom Prince (noreply@blogger.com) — 2013-05-04T02:10:26+00:00

During the second half of the month, I have concentrated on the continuing development tools to automate the deployment of twisted infrastructure.

I've also spent some time developing some command-line tools for interacting with trac. It currently support view tickets, listing review tickets, and grabbing attachments (including applying them in git with an appropriate commit message). I plan to add support for doing reviews, as well.

Thomas Vander Stichele: If I was 16 years younger…

Thomas — 2013-05-03T21:30:25+00:00

I’d totally try and be the intern for pinboard.

The money is great for a summer job, but that’s not the important part. pinboard seems interesting, it’s a real service, and it’s (I assume) small enough to understand from top to bottom. Contrary to, say, a Google Summer of Code project, you get to touch a real existing service, and from what I can tell from the blog you get to do it with a smart and funny guy.

You’ve got five weeks left; even if you’re in the middle of exams right now, apply!

(And if you do, why not add the features to merge and rename tags while you’re at it?)

Jonathan Lange: Deliberately being naughty

Jonathan Lange (noreply@blogger.com) — 2013-04-29T13:23:33+00:00

I don't get a whole lot of time to hack on things outside of work. Maybe one or two slots of about two hours in a good week. What I want is for that time to be fun, creative, and hopefully useful for others.

When I do get such discretionary time, what I often end up doing is code review. I believe that finishing things is more important than starting them, and know how much it sucks to be waiting for someone to review your code.

So, I'll do a couple of code reviews (some of which might be strictly blocking on me). And then I'll run out of time.

This means I rarely work on anything I actually find fun. Code reviews are rarely fun.

There's a similar, related thing working on a project that's got a backlog of code reviews, like, say Twisted. Twisted has forty or fifty patches right now that are waiting for someone to review them. Do I want to fix bugs in Twisted? That would be contributing to the patch problem, so probably not. Do I want to do code reviews? Well, I guess I should, but I'll probably find something else to do instead.

I don't think Twisted should change its policy. I think the solution is for people like me to volunteer more time doing something they don't necessarily enjoy for the sake of achieving some broader, nobler outcome.

However, for me to sustain myself and keep enjoying open source and my chosen career, I've decided to make sure that I regularly schedule "naughty time". That's going to be time where I don't work on the thing that I should be working on, but instead play with something I find interesting at the time, even if it's not the best thing to do.

Future Foundries: Unittesting With Localized Patching

Itamar Turner-Trauring (noreply@blogger.com) — 2013-04-25T19:05:28+00:00

In my previous two posts, I gave examples of alternatives to patching in unittests: class-based and function-based parameterization. Nick Coghlan pointed out that my example of patching was a little bit of a strawman argument - the most global (and therefore the most side-effecty) way of doing patching. Here's what he wrote:

While your point about the risks of patching destructive calls is valid, if you're going to decry the practice of using mocks in tests, at least decry a version which uses them properly. In your first example you are patching the wrong module - you shouldn't patch os._exit (with potentially non-local effects), you should patch the module under test so that *in that module only*, the reference "os._exit" resolves to your patched function.

Most functions under test *aren't* destructive (so you'll get the expected test result failure), and by jumping straight to dependency injection in cases where you don't need it, you can end up adding a huge amount of complexity to your production code without adequate reason *and* give yourself additional code paths to test in the process. Dependency injection should be used only if there is a *production* related reason for adding it (and "this function is destructive, so we should use dependency injection rather than mocking to test it" is a valid reason).

For those non-destructive cases, you can avoid most of the non-local effects without adding complexity to the production code by localising your mock operation to as narrow a target as possible.

The version of patching he suggests is definitely a lot better than my initial example, so let's take a look. First, the module we're going to test:

import os

def exit_with_result(function):
    result = function()
    if result:
        os._exit(0)
    else:
        os._exit(1)

And now the patch-based tests, based on example code from Nick Coghlan (any mistakes were added by me):


import unittest
# Note that we don't import os, because we're not touching it!

import exitersketch

class FakeOS:
    EXIT_NOT_CALLED = object()
    CALLED_WITH_DEFAULT = object()

    def __init__(self, module):
        self.module = module
        self.exit_code = self.EXIT_NOT_CALLED

    def _exit(self, code=CALLED_WITH_DEFAULT):
        self.exit_code = code

    def __getattr__(self, attr):
        return getattr(self.original_os, attr)

    def __enter__(self):
        self.original_os = self.module.os
        self.module.os = self
        return self

    def __exit__(self, *args):
        self.module.os = self.original_os


class ExiterTests(unittest.TestCase):

    def test_exiter_success(self):
        with FakeOS(exitersketch) as fake:
            exitersketch.exit_with_result(lambda: True)
        self.assertEqual(fake.exit_code, 0)

    def test_exiter_failure(self):
        with FakeOS(exitersketch) as fake:
            exitersketch.exit_with_result(lambda: False)
        self.assertEqual(fake.exit_code, 1)


if __name__ == '__main__':
    unittest.main()

This version is definitely a superior form of patching: only one module's view is impacted. Notice also the use of __getattr__ to ensure overriding exitersketch.os only overrides os._exit and not other parts of the os module. Nonetheless, it still suffers from the inherent problem of patching: it's overriding more state than necessary. Thus it's still possible to call destructive functions by mistake if you rearrange your imports. For non-destructive functions it's still possible to have a test unexpectedly call a patched function, albeit only from code in the same module rather than globally. If you are going to use patching, though, making the patching as local as possible is definitely the way to go.

Future Foundries: Unittesting Without Patching: A Followup

Itamar Turner-Trauring (noreply@blogger.com) — 2013-04-25T18:40:41+00:00

I got a couple questions about my previous post asking why I didn't show the simpler, function-based style of parameterization. This style does make unittesting possible, and with less complexity than creating a new class:

import os

def exit_with_result(function, _exit=os._exit):
    result = function()
    if result:
        _exit(0)
    else:
        _exit(1)

The problem is that when you add arguments to a function, the parameterization leaks into your public API. This means that:

You need to document the fact that these extra arguments (e.g. _exit in the example above) should not be used.
*args and **kwargs can't be used at all.
Changing the function signature later on can be more difficult.
If you have large numbers of things you need to parameterize, the function definition gets pretty long and ugly.

In the class style in contrast the public API is not affected by the need to unittest.

What's more, you will often have a group of related functions using the same modules, functions or classes. By grouping them in a class, you can implement the parameterization hook once, rather than for every function. You can see an example of this in Crochet (specifically the Eventloop class), where the parameterized reactor is used by multiple functions. If the code you need to parameterize is already a method, setting the parameters in __init__ or as a class attribute is even more attractive, requiring only minimal additional complexity.

Update: If you go with this style of parameterization, you still need to assert that in the default case it actual calls the correct function (e.g. os._exit for exit_with_result). Probably the nicest way to do so is to use inspect.getcallargs.

Duncan McGreggor: OpenStack Developer Summit: Heat Followup

Duncan McGreggor (noreply@blogger.com) — 2013-04-23T23:55:39+00:00

Folks are finally starting to recover from the OpenStack Developer Summit that was held in Portland, Oregon recently. All reports indicate that it was a truly phenomenal experience, record-breaking in many ways, and something that has inspired incredible enthusiasm within the community. And that's great news, since there's an enormous amount of work to be done this release ;-)

Of particular importance to many in the community is the work around maturing the autoscaling feature in OpenStack Heat. There was a fantastic session at the summit, facilitated by the bow-tied and most dapper Ken Wronkiewicz (his notes from the Summit were published on the Rackspace blog).

In preparation for the session, the following resources were created:

That one in the middle is important, as it is also where notes were taken during the actual session itself (see the section entitled "ODS Session Notes"). Devs at Rackspace have started going through the notes from the session and started planning work around this -- all of which will be carried on in the open, on the OpenStack mail list (tagged with "[Heat]"), on Freenode, and on github/gerrit.

The discussion at the Summit indicated strong interest in building a REST API for the existing autoscaling feature. Needless to say, there is a lot involved in this, touching upon significant OpenStack components like Quantum, LBaaS, and Ceilometer. Once the appropriate code is in place, a REST API will need to be created, features will need to be expanded/added, etc., and we'll be off and running :-)

Lots to do, and lots of great energy and excitement around this to keep us all chugging through this cycle.

On that note, we'd like to send out a special "thanks" to all the countless folks who worked so hard to make ODS happen. This event anchors us in a most excellent way, providing the insight and fuel that supports future development work so well!

Duncan McGreggor: Tea Time at Rackspace SF

Duncan McGreggor (noreply@blogger.com) — 2013-04-23T22:48:13+00:00

One of the Rackspace teams here in the SF office doesn't to standups; it does "tea time" instead. A delightful change, to be sure. One of my coworkers published a Rackspace blog post about this today.

More than a gratutious reblog, I wanted to highlight this bit of lovely coincidence I came across while reading Turing's Cathedral last night:

"Afternoon tea— a ritual introduced at Fine Hall by Oswald Veblen, who, according to Herman Goldstine, “tried awfully hard to be an Englishman”— was served on real china daily at exactly three o’clock. According to Oppenheimer, “tea is where we explain to each other what we do not understand.”

Dyson, George (2012-03-06). Turing's Cathedral: The Origins of the Digital Universe (p. 90). Knopf Doubleday Publishing Group. Kindle Edition.

The part of the book where this quote occurred was discussing Fine Hall which was built for Von Neumann's computer team at the Princeton Institute for Advanced Studies.

And, yes, tea time is a great time for our team mates to explain to each other what they don't understand.

In case you missed it above, here's the link again:

http://www.rackspace.com/blog/tea-time-because-we-wont-stand-for-a-standup/

Twisted Matrix Laboratories: April mid-month Fellowship report

Tom Prince (noreply@blogger.com) — 2013-04-18T17:15:34+00:00

During the first half of the month, 43 tickets got some attention that they would not have received without the sponsored development. The result was 17 closed tickets and 24 other tickets unblocked for other developers to resume work on.

I set up an official git mirror that contains all the branches, including release branches and tags on github and documented my workflow on the wiki.

I've documened how our current infrastructure (cube), and with the help of Jonathan Stoppani, I've been working on scripts to automate the deployment of its successor (dornkirk).

The tickets I reviewed, that have now been closed:

Other tickets I have reviewed:

Other tickets I have worked on:

Future Foundries: Unittesting Without Patching

Itamar Turner-Trauring (noreply@blogger.com) — 2013-04-18T17:14:20+00:00

Python has the power to override any attribute on any module or class, but just because you can doesn't mean you should. This is true in regular code, but just as true of unittests. Many testing libraries (mock, Twisted's trial, py.test) provide facilities for overriding some piece of global state; you can also do so manually. Occasionally these facilities prove invaluable, but often they are used unnecessarily. Better alternatives are available.

Before I explain why patching is problematic, let's look at an example. Consider the following module:

import os

def exit_with_result(function):
    result = function()
    if result:
        os._exit(0)
    else:
        os._exit(1)

On the face of it patching is necessary to test this example. The tests would look something like this:

import unittest
import os

from exitersketch import exit_with_result


class ExiterTests(unittest.TestCase):
    def setUp(self):
        self.exited = None
        self.originalExit = os._exit
        os._exit = self.fakeExit

    def fakeExit(self, code=0):
        self.exited = code

    def tearDown(self):
        os._exit = self.originalExit

    def test_exiter_success(self):
        exit_with_result(lambda: True)
        self.assertEqual(self.exited, 0)

    def test_exiter_failure(self):
        exit_with_result(lambda: False)
        self.assertEqual(self.exited, 1)


if __name__ == '__main__':
    unittest.main()

Having seen patching, and seen that it works as a testing technique, why should we avoid it?

Patching is fragile. If the example above changed import os to from os import _exit, the patching would need to be modified. However, if you forgot to modify the patching, unexpected code will run. In this case, your test run will mysterious exit half way through. If the function you are attempting to patch is more destructive, worse things may happen: credit cards may get charged, data may get deleted, etc..
Patching leads to unexpected behaviour. Because patching is a global change, the patched code may be called not just by the function being tested, but by code it is calling which happens to use the same patched code.
Patching indicates bad design. Code code should be designed to be easily testable. Having to modify global state suggests that the code is not as modular as one might hope.

How to avoid patching? Parameterization, aka dependency injection. We refactor the code to accept the _exit function as a parameter. Notice the the public API has not changed:

import os

class _API(object):
    def __init__(self, exit):
        self.exit = exit

    def exit_with_result(self, function):
        result = function()
        if result:
            self.exit()
        else:
            self.exit(1)


_api = _API(os._exit)
exit_with_result = _api.exit_with_result

Our tests can now test both that _API.exit_with_result class has the correct behavior in general, and that the public exit_with_result is going to call os._exit in particular.

import unittest
import os

from exiter import _api, _API, exit_with_result


class ExiterTests(unittest.TestCase):
    def setUp(self):
        self.exited = None

    def fakeExit(self, code=0):
        self.exited = code

    def test_api(self):
        self.assertIsInstance(_api, _API)
        self.assertEqual(_api.exit, os._exit)
        self.assertEqual(exit_with_result, _api.exit_with_result)

    def test_exiter_success(self):
        _API(self.fakeExit).exit_with_result(lambda: True)
        self.assertEqual(self.exited, 0)

    def test_exiter_failure(self):
        _API(self.fakeExit).exit_with_result(lambda: False)
        self.assertEqual(self.exited, 1)


if __name__ == '__main__':
    unittest.main()

The same technique is useful when you are tempted to store some state in a module. Instead, store an instance of a class:

class _Counter(object):
    value = 0

    def increment(self):
        self.value += 1

    def value(self):
        return self.value


_counter = _Counter()
increment = _counter.increment
value = _counter.value

As I've demonstrated, patching can often be avoided by restructuring code to be more testable. The same Python features that make patching so easy also make avoiding patching just as easy. Given the choice, you should avoid changing global state when testing individual components.

Duncan McGreggor: Cruising HTTP with LFE

Duncan McGreggor (noreply@blogger.com) — 2013-04-18T15:32:00+00:00

In the last post, you learned how to get LFE running on Ubuntu. This one will give you some insight into how LFE can be used in something approaching real-world problems. In the next post, we're going to jump back into the lambda calculus, and we'll see some more LFE shortly after that.

Because Lisp Flavored Erlang is 100% compatible with Erlang Core, it has access to all the Erlang libraries, OTP, and many third-party modules, etc. Naturally, this includes the Erlang HTTP client, httpc. Today we're going to be taking a look at how to use httpc from LFE. Do note, however that this post is only going to provide a taste, just enough to give you a sense of the flavor, as it were.

If you would like more details, be sure to not only give the official docs a thorough reading, but to take a look at the HTTP Client section of the inets Reference Manual.

Note that for the returned values below, I elide large data structures. If you run them in the LFE REPL yourself, you can view them in all of the line-consuming glory.

Synchronous GET

Let's get started with a simple example. The first thing we need to do is start the inets application. With that done, we'll then be able to make client requests:
Now we can perform an HTTP GET: This just makes a straight-forward HTTP request (defaults to GET) and returns a bunch of associated data:

HTTP version
status code
reason phrase
headers
body

All of that data is dumped into our result variable. Here's the same GET but with pattern matching set up so that we can easily access all that data:
For those not familiar with Erlang patterns, we've just told LFE the following:

the return value of the function we're going to call is going to be a tuple composed of an atom ('ok) and another tuple
the nested tuple is going to be composed of a tuple, some headers, and a body
the next nested tuple is going to be composed of the HTTP version, status code, and status code phrase

If you'd like to learn more about using patterns in LFE, be sure to view the patterns page of the LFE User Guide.

Once the request returns, we can check out the variables we set in the pattern:

That's great if everything goes as expected and we get a response from the server. What happens if we don't?

Well, errors don't have the same nested data structure that the non-error results have, so we're going to have to make some changes to our pattern if we want to extract parts of the error reason. Pattern matching for just the 'error atom and the error reason, we can get a sense of what that data structure looks like:

Looking at just the data stored in the reason variable, we see:
If you check out the docs for httpc request and look under "Types", you will see that the error returned can be one of three things:

a tuple of connect_failed and additional data
a tuple of send_failed and additional data
or just unspecified additional data

In our example our additional data is a tuple of the address we were trying to connect to and the specific error type that for our failed connection.

Async GET

Now that we've taken a quick look at the synchronous example, let's make a foray into async. We'll still be using httpc's request function, but we'll need to use one of the longer forms were extra options need to be passed, since that's how you tell the request function to perform the request asynchronously and not synchronously.

For clarity of introducing the additional options, we're going to define some variables first: You can read more about the options in the httpc docs.

With the variables defined, let's make our async call: The sender receives the results, and since we sent from the LFE REPL, that's the process that will receive the data. Let's keep our pattern simple at first -- just the request id and the result data:
Needless to say, parsing the returned data is a waste of Erlang's pattern matching, so let's go back and do that again, this time with a nice pattern to capture the results. We'll need to do another request, though, so that something gets sent to the shell:
Now we can set up a pattern that will allow us to extract and print just the bits that we're looking for. The thing to keep in mind here is that the scope for the variables is within the receive call, so we'll need to display the values within that scope:
This should demonstrate the slight differences in usage and result patterns between the sync and async modes.

Well, that about sums it up for an intro to the HTTP client in LFE! But one last thing, for the sake of completeness. Once we're done, we can shut down inets:

Moshe Zadka: Public letter to Congresswoman Jackie Speier, CA14

moshez — 2013-04-16T17:33:31+00:00

Dear Congresswoman Speier,

I have already called your office, so I am sorry if this is redundant, but they did not have an answer for me. I am sending you this message because of my concern about HR 624, the “Cyber Intelligence Sharing and Protection Act”, also known as CISPA. It appears that it would be up for a vote in Congress soon. I am worried for my privacy, as well as for the privacy of other Americans, were this bill to be passed. This law violates due process by allowing private companies to hand data to the “government” even when they are under contract to not do so, without any court order. I am sure most people working for Federal agencies are good and honest people, but this allows one bad apple to grab whatever data they need for whatever private means they have with no judicial oversight. It also allows private companies to hack into computers as long as they believe “it is necessary” — in essence privatizing law enforcement, again without proper judicial oversight.

In the wake of the tragedy in Boston, the US must, of course, be ever-vigilant against threats. However, laws like CISPA, making our country more surveiled and less free, are essentially give terrorists what they want — to terrorize us into hurting out essential freedoms.

Thanks,
Moshe Zadka

Duncan McGreggor: Twisted SSH: Rendering a Log-in Banner/MOTD in Conch

Duncan McGreggor (noreply@blogger.com) — 2013-04-15T04:25:41+00:00

A few weeks ago, I pinged my peeps on #twisted asking why the banner for a custom SSH server wasn't rendering properly. After some digging around and some inconsistent results (well, consistently bad results for me), we weren't able to resolve anything, and I had to set the problem aside.

The Symptom
The first thing I had tried was subclassing Manhole from twisted.conch.manhole, overriding (and up-calling) connectionMade, writing the banner to the terminal upon successful connection. This didn't work, so I then tried overriding initializeScreen by subclassing twisted.conch.recvline.RecvLine. Also a no-go. And by "didn't work" here's what I mean:

In both Linux (Ubuntu 12.04 LTS, gnome-terminal) and Mac (OS X 10.6.8, Terminal.app), after a successful login to the Twisted SSH server, the following sequence would occur:

an interactive Python prompt was rendered, e.g., ":>>"
the banner was getting written to the terminal, and
the terminal screen refreshed with the prompt at the top

This all happened so quickly, that I usually never even saw #1 and #2. Just the second ":>>" prompt from #3. Only by scrolling up the terminal buffer would I see that the banner had actually been rendered. Even though I was doing my terminal.write after connectionMade and initializeScreen, it didn't seem to matter.

Discovery!
Some time last week, I put together example Twisted plugins showing what the problem was, and the circumstances under which a banner simply didn't get rendered. The idea was that I would provide some bare-bones test cases that demonstrated where the problem was occurring, post them to IRC or the Twisted mail list, and we could finally get it resolved. 'Cause, ya know, I really want my banners ...

While tweaking the second Twisted plugin example, I finally poked my head into the right method and discovered the issue. Here's what's happening:

twisted.conch.recvline.RecvLine.connectionMade calls t.c.recvline.RecvLine.initializeScreen
t.c.recvline.RecvLine.initializeScreen does a terminal.reset, writes the prompt, and then switches to insert mode. But this is a red herring. Since something after initializeScreen is causing the problem, we really need to be asking "who's calling connectionMade?"
t.c.manhole_ssh.TerminalSession.openShell is what kicks it off when it calls the transportFactory (which is really TerminalSessionTransport)
openShell takes one parameter, proto -- this is very important :-)
openShell instantiates TerminalSessionTransport
TerminalSessionTransport does one more thing after calling the makeConnection method on an insults.ServerProtocol instance (the one I had tried overriding without success), and as such, this is the prime suspect for what was preventing the banner from being properly displayed: it calls chainedProtocol.terminalProtocol.terminalSize
chainedProtocol is an insults.ServerProtocol instance, and its terminalProtocol attribute is set when ServerProtocol.connectionMade is called.
A quick check reveals that terminalProtocol is none other than the proto parameter passed to openShell.

But what is proto? Some debugging (and the fact that of the three terminalSize methods in all of twisted, only one is an actual implementation) reveals that proto is a RecvLine instance. Reading that method uncovers the culprit in our whodunnit: the first thing the method does is call terminal.eraseDisplay.

Bingo! (And this is what I was referring to above when I said "poked my head" ...)

Since this was called after all of my attempts to display a banner using both connectionMade and initializeScreen, there's no way my efforts would have succeeded.

Here's What You Do
How do you get around this? Easy! Subclass :-)

The class TerminalSessionTransport in t.c.manhole_ssh is the bad boy that calls terminalSize (which calls eraseDisplay). It's the last thing that TerminalSessionTransport does in its __init__, so if we subclass it, and render our banner at the end of our __init__, we should be golden. And we are :-)

You can see an example of this here.

Not sure if this sort of thing is better off in projects that make use of Twisted, or if it would be worth while to add this feature to Twisted itself. Time (and blog comments) will tell.

Epilogue
As is evident from the screenshot above (and the link), this feature is part of the DreamSSH project. There are a handful of other nifty features/shortcuts that I have implemented in DreamSSH (plus some cool ones that are coming) and I'm using them in projects that need a custom SSH server. I released the first version of DreamSSH last night, and there's a pretty clear README on the github project page.

One of the niftier things I did last night in preparation for the release was to dig into Twisted plugins and override some behaviour there. In order to make sure that the conveniences I had provided for devs with the Makefile were available for anyone who had DreamSSH installed, I added subcommands... but if the service was already running, these would fail. How to work around that (and other Twisted plugin tidbits) are probably best saved for another post, though :-)

Duncan McGreggor: Getting Started with LFE on Ubuntu

Duncan McGreggor (noreply@blogger.com) — 2013-04-15T02:06:50+00:00

For those that don't know, there is a fully Erlang Core-compatible Lisp-2 that runs on the Erlang VM and produces .beam files that can be used in any Erlang application. This manna from heaven is LFE, or Lisp Flavored Erlang. It was started about 5 years ago by Robert Virding, one of the co-creators of the Erlang programming language, and has inspired other similar efforts: Elixir (a Ruby-alike) and Joxa (a Lisp-1). (Incidentally, Robert has also created Prolog and Lua implementations that run on top of the Erlang VM!)

The new LFE docs site (a continuous work in progress) has some good introductory materials for the curious reader:

A Quick Start guide
An as yet incomplete User Guide (most of the introductory chapters are finished)
A tutorial on Erlang's Lightweight Processes in LFE.

This blog post aims to bring some of those hidden materials into the consciousness of Ubuntu users. If you are averse to Erlang syntax, LFE opens up a whole new world to you :-)

The examples below assume Ubuntu 12.10.

Getting Erlang

Erlang R15B01 comes with Ubuntu 12.10. If that's all you need, then this will suite you just fine:

$ sudo apt-get install erlang

If you are wanting to test against multiple versions of Erlang, you should check out the kerl project, which lets you install a wide variety of Erlang versions (including the latest releases) side-by-side.

You'll also need git, if you don't yet have it installed:

$ sudo apt-get install git

Currently, rebar is required to build all the LFE files. If you're going to be building LFE projects, you'll want this anyway ;-) Rebar will be in Ubuntu 13.04, but it's not in 12.10, so you'll need to get it:

$ wget https://github.com/rebar/rebar/wiki/rebar
$ chmod 755 rebar
$ sudo mv rebar /usr/local/bin

Getting and Building LFE

Here's what you need to do to build LFE:

$ mkdir -p ~/lab/erlang && cd ~/lab/erlang
$ git clone https://github.com/rvirding/lfe.git
$ cd lfe
$ make compile

If you looked at your ./ebin directory when you cloned the repo, you would have seen that there were no .beam files in it. After compiling, it is full of .beams ;-)

Sidebar: A common pattern in Erlang applications is the use of a deps directory under one's project dir where dependencies can be installed without conflicting with any system-wide installs, providing versioning independence, etc. Managing these with rebar has been very effective for projects, where simply calling rebar compile puts everything your app needs in ./deps. Projects that depend upon LFE are doing this, but we'll cover that in a future blog post.

Using LFE

With everything compiled, we can jump right in! Let's fire up the REPL, and do some arithmetic as a sanity check:

How about a message to stdout?

Any form starting with : is interpreted as a call to a module. The full form is (: <module name> <function name> <arguments>). As such, you can see that we're calling the format function in the (built-in) io module.

Also, it's good to know that there are certain things that you can't do in the REPL, e.g., defining modules, macros, functions, and records. Erlang expects that these sorts of activities take place in modules. However, we can explore a little more before we write our first module. Let's use the REPL's set form and lambda to define a function anyway (albeit, in a somewhat awkward fashion):

That wasn't too bad ;-) We're seeing the external module call, again -- this time to the math library. Now let's use a module of our own devising...

Creating Modules

In another terminal (but same working directory), let's create a module in a file called my-module.lfe, with the following content:

Note that the module name in the code needs to match the file name (minus the extension) that you used for the module.

Back in the REPL terminal window, let's compile this module and run the defined function:

Let's add another function to the module that demonstrates the benefits of Erlang's multiple-arity support:

Re-compiling and running the functions, we are greeted with success:

Lastly, let's convert the power function we defined in the previous section using our REPL-workaround to a "real" function, defined in our new module:

And then let's try it out:

Perfect.

(Of course, it's rather absurd to redefine pow to exp, when there is basically nothing to gain by it ;-) It's just a quick demo...)

Conclusion

There's lots more to learn; this has been just a small sip from a hot mug o' LFE.

However, it's definitely enough to get you started and, should you be interested in following along in future LFE blog posts, you'll have everything you need to get the most out of those.

Moshe Zadka: Babies — a guide

moshez — 2013-04-13T23:10:55+00:00

Some people have asked me to summarize the collection of links I posted about baby development. Here is my unauthorized summary. There are links to the original resources throughout the post.

A lot of the following information is based on, or inspired by, the Educaring philosophy as put down by Magda Gerber and taught by RIE. Educaring philosphy is based on trust in the child to initiate development, giving children safe areas to explore in an uninterrupted way, involving the child in all child-care activities, observing the child and responding to her needs and being consistent when limits are concerned. The central theme is respect for the child as a whole person in his own right.

When babies are born, they have one mode of communication — crying. Therefore, a lot of what parents do in the first few months is dealing with crying babies. The blogosphere, unsurprisingly, reacted with a wealth of information to the question “what do I do with a crying baby?”. The basic newborn care advice — check diaper, burp, and feed — only goes so far. If your baby is not giving you hunger signs — clenched fingers and fists over chest and tummy, flexed arms and legs, rooting, fast breathing or sucking noises or motions — feeding might stop the crying even if it is not the best idea.

After those, comes the “unexplained crying”. “Unexplained”, of course, is not a fact about the crying — it is a fact about the parent. Babies have feelings too and need to express them.

The diagnosis of “colic” as long stretches of unexplained crying is giving way to the concept of “PURPLE” crying. PURPLE stands for “Peak” (of crying) “Unexpected” “Resists soothing” “Pain-like face” “Long lasting” (several hours a day) “Evening” (also known as “the witching hour”, around twilight). Apparently all mammals go through their own equivalent of PURPLE crying, the human animal between 2 weeks and 3-5months.

Listening to the crying, being compassionate without trying to stop it, is important. Crying is a form of communication. When we do want to soothe the baby, CALMS ¹ is a popular approach. [C]heck in with yourself, and connect with your feelings — you want to calm yourself before calming the baby down. [A]llow yourself to take a breath and relax. [L]isten to what your baby is trying to tell you. [M]irror — empathize with the baby’s feeling, voicing his emotions as best you can. [S]oothe your baby. It is OK if the soothing initially makes the crying harder — the baby might need to voice the hurt feelings first. Sometimes, the end result of this approach is to “hold the baby through the tears”, being calm and letting the baby cry as long as she needs to. When soothing a baby, pick a tactic (singing, bouncing, etc.) and stick to it — switching tactics is a bad idea. Repetition, repetition, repetition. There are reasons to be wary of the “5S”.

Parents need to understand that sometimes the baby will cry, and this does not mean they are failing as parents. Unfortunately, that is exactly what attachment parenting is teaching.

Second only to crying in the amount of energy parents spend obsessing about is the baby’s sleep. Nursing kids to sleep does not make them sleep longer — putting them to sleep earlier does. Althought not too much earlier, and not too soon. Wait for them to display sleep signs, and put them to bed then. Babies’ “long-stretches” of sleep start from 3 hours at newborn, through 4 hours at two months, 4-5 at 4 months, 6 hours at 6 months and by 1 year old, 80% of babies sleep through the night. Routines, especially at 4 months and beyond, are useful for helping babies sleep. Babies are woken up by hunger, dreams or discomfort. Some babies fight sleep, and need to be helped there. SIDS is mostly a risk while sleeping, follow AAP recommendations to reduce risk: put babies on their backs to sleep, use a firm sleep surface, room-sharing but no bed sharing and keep soft objects and loose bedding out of the crib.

A distant third is how children play. Even babies can do independent play as long as they are placed, carefully, on their back until they have learned to roll to their tummies. It is OK if the baby does not “do” anything — just kicking their legs and staring is fine. The area, of course, must be safe — a no-”no” area, where everything possible is permissible. It is OK, and even good, to stay near the child and observe — but even if they seem frustrated, not to help them (this is like giving someone the solution to a riddle). Babies are perfectly capable of entertaining themselves this way. Babies can start having play-dates early on — if we believe them capable, if we are attentive to their moods, make it into a routine, and are patient with them. As always, observing them in quiet is important. Tummy time is important developmentally. Start slow, get down to your baby’s level and provide entertainment while the baby is on his tummy. Avoid having tummy time right after feeding, or when the baby is upset. Keep it calm and quiet, so the baby is not scared by what she cannot see. Be patient — the baby will eventually want tummy time.

Jack Moffitt: Servo Update: Upgrading Rust, GPU Rendering, and Automation

Jack Moffitt (jack@metajack.im) — 2013-04-12T16:40:00+00:00

I've been working on Servo for three weeks now. There's an enormous amount of work to do, and I want to capture what's going on and how it's progressing. This should be the first of many such updates on the project.

Day One

When I arrived, Servo no longer built at all, at least not on OS X. Servo often requires bleeding edge versions of Rust, and backwards incompatible changes to Rust are still happening on a regular basis. Since all of the contributors to Rust work on different platforms, when porting to a new Rust compiler, some platforms have gotten left behind. This was particularly acute this time because Rust 0.6 contained a lot of syntax changes, mostly things that got removed from the language, and many pieces of Servo were using syntax that was deprecated in Rust 0.5, and was finally deleted entirely in Rust 0.6.

Upgrading to Rust 0.6

Rust 0.6 removed a lot of keywords and syntax from the language. Porting Servo required modifying all the constants, many function declarations, many import statements, etc. These changes were largely mechanical. There were a few changes that weren't so easy.

Mutable fields are being removed from the language, and mutability will be controlled by the mutability of the struct itself. Not all of these had to be removed in Servo, but many of them did, and removing them often required slightly changing the data structures and their type signatures. In some cases this was trivial, but in a few cases these changes needed more care. In particular, lots of these changes bumped up against the Rust borrow checker, which ensures it's safe to hand out pointers to memory. There are still some bugs in the borrow check, and workarounds are not always straightforward.

It took me about a week and a half to work my way through all the dependent libraries and Servo itself at which point I had a build. By the end of that second week I had landed the language upgrade to servo as well as some Rust library changes that were needed. The end result is that Servo is now using Rust 0.6 syntax, but it requires a post-0.6 version of Rust due to the Rust changes not landing quite in time for the 0.6 release.

GPU Rendering

Servo uses many forms of parallelism, but one bit of low hanging fruit is to move to a fully GPU rendering path. Currently compositing is done on the GPU, but rendering to the various layers is done on the CPU. This is how most current browsers operate as well.

We're moving to rendering on the GPU as well which should speed up some things a bit. Instead of rendering in parallel to several layers, Servo will render directly into textures on the GPU which the compositor can use without doing CPU to GPU memory transfers.

This required upgrading the rendering stack to a newer version of Azure (Mozilla's drawing library) and a new version of Skia (the specific backend that Azure uses on OS X, Linux, and Android). Now that this part is done, we'll be adding texture layers to the renderer and switching drawing to those.

Automation

We're setting up build and testing automation for Servo now, which should help ensure Servo remains buildable on all platforms. Rust has an amazing set of tools for this already, which we are hoping to reuse fully. Buildbot machines run builds and tests, and a GitHub bot called Bors handles dispatching builds for patches that have been reviewed and merging pull requests that have passed tests.

For now this work will be on Linux, but we hope to expand it to cover OS X and Android as well in the near future. Once Servo is a little farther along, we plan to put up nightly snapshots so more people can follow along with our progress.

Other Work

There's tons of other work in progress on both Servo and Rust. The DOM bindings are getting improved, a new Rust scheduler that will make performance and I/O better is in progress, a more optimized C FFI in Rust should also land soon, and the rustpkg package manager is shaping up which we'll be switching to for more and more of Servo as it matures.

We need more help in lots of areas. Please join us in IRC in #servo or on the mailing list. We'll be trying to mark bugs and projects that are well suited for new contributors. If you want to work on Servo and write Rust code all the time, we're hiring.

Future Foundries: SSH Into Your Python Server

Itamar Turner-Trauring (noreply@blogger.com) — 2013-04-12T16:11:04+00:00

Have you ever wanted to see what's going on inside your Python server? With Crochet and Twisted, you can add a Python prompt to you process that is accessible via SSH, allowing you to poke around in the internals of your running program. Here's an example session to a Flask server:

$ ssh admin@localhost -p 5022
admin@localhost's password: ******
>>> app
<flask.app.Flask object at 0x28a96d0>
>>> app.url_map
Map([<Rule '/' (HEAD, OPTIONS, GET) -> index>,
 <Rule '/static/<filename>' (HEAD, OPTIONS, GET) -> static>])
>>> from twisted.internet import reactor
>>> reactor._selectables
{9: <SSHServerTransport #0 on 5022>, 3: <<class 'twisted.internet.tcp.Port'> of twisted.conch.manhole_ssh.ConchFactory on 5022>, 6: <twisted.internet.posixbase._UnixWaker object at 0x28a2510>}

The code to start the SSH server has quite a lot of boilerplate, so I filed a ticket to provide a utility function. If you're using the system Twisted, you may need to install Twisted's Conch package, e.g. apt-get install python-twisted-conch on Ubuntu.

import logging

from flask import Flask
from crochet import setup, in_reactor
setup()

# Web server:
app = Flask(__name__)

@app.route('/')
def index():
    return "Welcome to my boring web server!"


@in_reactor
def start_ssh_server(reactor, port, username, password,
                     namespace):
    """
    Start an SSH server on the given port, exposing a Python
    prompt with the given namespace.
    """
    from twisted.conch.insults import insults
    from twisted.conch import manhole, manhole_ssh
    from twisted.cred.checkers import (
        InMemoryUsernamePasswordDatabaseDontUse as MemoryDB)
    from twisted.cred.portal import Portal

    sshRealm = manhole_ssh.TerminalRealm()
    def chainedProtocolFactory():
        return insults.ServerProtocol(manhole.Manhole,
                                      namespace)
    sshRealm.chainedProtocolFactory = chainedProtocolFactory

    portal = Portal(
        sshRealm, [MemoryDB(**{username: password})])
    reactor.listenTCP(port, manhole_ssh.ConchFactory(portal),
                      interface="127.0.0.1")


if __name__ == '__main__':
    import sys
    logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
    start_ssh_server(
        5022, "admin", "secret", {"app": app}).wait()
    app.run()

Duncan McGreggor: The Lambda Calculus: A Quick Primer

Duncan McGreggor (noreply@blogger.com) — 2013-04-11T15:55:46+00:00

The λ-Calculus Series

A Brief History
A Quick Primer for λ-Calculus
Reduction Explained
Church Numerals
Arithmetic
Logic
Pairs and Lists
Combinators

To the untrained eye, the notation used in λ-calculus can be a bit confusing. And by "untrained", I mean your average programmer. This is a travesty: reading the notation of λ-calculus should be as easy to do as recognizing that the following phrase demonstrates variable assignation:

x = 123

So how do we arrive at a state of familiarity and clarity from a starting state of confusion? Let's dive in with some examples, and take it step at a time :-) Once we've got our heads wrapped around Alonzo Church's notation, we'll be able to easily read it -- and thus convert it into code! (We will have lots of practice in the coming posts to do just that.)

A Quick Primer for λ-Calculus

Here's one of the simplest definitions in λ-calculus that you're going to see: the identity function:

λx.x

This reads as "Here is a function that takes x as an argument and returns x." Let's do some more:

λxy.x

"Here is a function that takes x and y as arguments and returns only x."

λx.λy.xy

"An outer function takes x as an argument and an inner function takes y as an argument, returning the x and the y." Note that this is exactly equivalent to the following (by convention):

λxy.xy

Let's up the ante with a function application:

λf.λx.f x

"Here is a function that takes a function f as its argument; the inner function takes x as its argument; return the result of the function f when given the argument x." For example, if we pass a function f which returns its input multiplied by 2, and we supplied a value for x as 6, then we would see an output of 12.

Let's take that a little further:

λf.λx.f (f (f x))

"Here is a function that takes a function f as its argument; the inner function takes x as its argument. Apply the function f to the argument x; take that result and apply f to it. Then do it a third time, returning that result." If we had the same function as the example above and passed the same value, our result this time would be 48 (i.e., 6 * 2 * 2 * 2).

That's most of what you need to read λ-calculus expressions. Next we'll take a peek into the murky waters of λ-calculus reduction and find that it's quite drinkable, that we were just being fooled by the shadows.

Duncan McGreggor: The Lambda Calculus: A Brief History

Duncan McGreggor (noreply@blogger.com) — 2013-04-11T15:53:17+00:00

Over this past weekend I took a lovely journey into the heart of the lambda calculus, and it was quite amazing. My explorations were made within the context of LFE. Needless to say, this was a romp of pure delight. In fact, it was so much fun and helped to clarify for me so many nooks and crannies of something that I had simply not explored very thoroughly in the past, that I had to share :-)

The work done over the past few days is on its way to becoming part of the documentation for LFE. However, this is also an excellent opportunity to share some clarity with a wider audience. As such, I will be writing a series of blog posts on λ-calculus from a very hands-on (almost practical!) perspective. There will be some overlap with the LFE documentation, but the medium is different and as such, the delivery will vary (sometimes considerably).

This series of posts will cover the following topics:

A Brief History
A Quick Primer for λ-Calculus
Reduction Explained
Church Numerals
Arithmetic
Logic
Pairs and Lists
Combinators

The point of these posts is not to expound upon that which has already been written about endlessly. Rather, the hope is to give a very clear demonstration of what the lambda calculus really is, and to do so with clear examples and concise prose. When the gentle reader is able see the lambda calculus in action, with lines of code that clearly show what is occurring, the mystery will disappear and an intuition for the subject matter will quite naturally begin to arise. This post is the first in the series; I hope you enjoy them as much as I did rediscovering λ-calculus :-)

Let us start at the beginning...

A Brief History

The roots of functional programming languages such as Lisp, ML, Erlang, Haskell and others, can be traced to the concept of recursion in general and λ-calculus in particular. In previous posts, I touched upon how we ended up with the lambda as a symbol for the anonymous function as well as how recursion came to be a going concern in modern mathematics and then computer science.

In both of those posts we saw Alonzo Church play a major role, but we didn't really spend time on what is quite probably considered his greatest contribution to computer science, if not mathematics itself: λ-calculus. Keep in mind that the Peano axioms made use of recursion, that Giuseppe Peano played a key role in Bertrand Russell’s development of the Principia, that Alonzo Church sought to make improvements on the Principia, and λ-calculus eventually arose from these efforts.

Invented in 1928, Alonzo didn't publish λ-calculus until 1932. When an inconsistency was discovered, he revised it in 1933 and republished. Furthermore, in this second paper, Church introduced a means of representing positive integers using lambda notation, now known as Church numerals. With Church and Turing both publishing papers on computability in 1936 (based respectively upon λ-calculus and the concept of Turing machines), they proposed solutions to the Entscheidungsproblem. Though Gödel preferred Turing's approach, Rosser suggested that they were equivalent definitions in 1939. A few years later, Kleene proposed the Church Thesis (1943) and then later formally demonstrated the equivalence between his teacher's and Turing's approaches giving the combination the name of the Church-Turing Thesis (1952, in his Introduction to Metamathematics). Within eight years, John McCarthy published his now-famous paper describing the work that he had started in 1958: "Recursive Functions of Symbolic Expressions and Their Computation by Machine". In this paper, McCarthy outlined his new programming language Lisp, citing Church's 77-page book (1941, Calculi of Lambda Conversion), sending the world off in a whole new direction.

Since that time, there has been on-going research into λ-calculus. Indisputably, λ-calculus has had a tremendous impact on research into computability as well as the practical applications of programming languages. As programmers and software engineers, we feel its impact -- directly and indirectly -- on a regular, almost daily basis.

Future Foundries: Crochet: Background Operations for Threaded Applications

Itamar Turner-Trauring (noreply@blogger.com) — 2013-04-10T17:08:05+00:00

In my previous post I showed Crochet doing a blocking call against a Twisted API. In this example, you can see how Twisted and Crochet allow you to run an operation in the background. An HTTP request for a new user starts a download in the background, and a reference is stored in the user's session. Every time the user reloads the page, a check is made to see if the download is finished, and if it is done it is display. You can also see the stash()/retrieve_result() API in use, which allows temporarily storing results under a key suitable for serialization in a session object.

import logging
from flask import Flask, session, escape
from crochet import setup, in_reactor, retrieve_result, TimeoutError
setup()

app = Flask(__name__)


@in_reactor
def download_page(reactor, url):
    """
    Download a page.
    """
    from twisted.web.client import getPage
    return getPage(url)


@app.route('/')
def index():
    if 'download' not in session:
        # @in_reactor functions return a DeferredResult:
        result = download_page('http://google.com')
        session['download'] = result.stash()
        return "Starting download, refresh to track progress."

    # retrieval is a one-time operation:
    result = retrieve_result(session.pop('download'))
    try:
        download = result.wait(timeout=0.1)
        return "Downloaded: " + escape(download)
    except TimeoutError:
        session['download'] = result.stash()
        return "Download in progress..."


if __name__ == '__main__':
    import os, sys
    logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
    app.secret_key = os.urandom(24)
    app.run()

Duncan McGreggor: Interview with Erlang Co-Creators

Duncan McGreggor (noreply@blogger.com) — 2013-04-09T21:25:25+00:00

A few weeks back -- the week of the PyCon sprints, in fact -- was the San Francisco Erlang conference. This was a small conference (I haven't been to one so small since PyCon was at GW in the early 2000s), and absolutely charming as a result. There were some really nifty talks and a lot of fantastic hallway and ballroom conversations... not to mention Robert Virding's very sweet Raspberry Pi Erlang-powered wall-sensing Lego robot.

My first Erlang Factory, the event lasted for two fun-filled days and culminated with a stroll in the evening sun of San Francisco down to the Rackspace office where we held a Meetup mini-conference (beer, food, and three more talks). Conversations lasted until well after 10pm with the remaining die-hards making a trek through the nighttime streets SOMA and the Financial District back to their respective abodes.

Before the close of the conference, however, we managed to sneak a ride (4 of us in a Mustang) to Scoble's studio and conduct an interview with Joe Armstrong and Robert Virding. We covered some of the basics in order to provide a gentle overview for folks who may not have been exposed to Erlang yet and are curious about what it has to offer our growing multi-core world. This wend up on the Rackspace blog as well as the Building 43 site. We've got a couple of teams using Erlang in Rackspace; if you're interested, be sure to email Steve Pestorich and ask him what's available!

Future Foundries: Presenting Crochet: Use Twisted as a library

Itamar Turner-Trauring (noreply@blogger.com) — 2013-04-09T18:40:52+00:00

Twisted is an event-driven framework; by default it expects to run the reactor event loop in your main thread to drive your application. If however you're writing a Django or Flask application you may want to use Twisted as just another library. Unless you choose to use Twisted as a WSGI container, this requires you to run the reactor in a thread. Today I am happy to announce Crochet, which makes using Twisted even easier in this situation.

Here's an example program using Crochet, allowing it to easily use Twisted from a normal, blocking command-line tool:

from __future__ import print_function

from crochet import setup, in_reactor
setup()


@in_reactor
def mx(reactor, domain):
    """
    Return list of MX domains for a given domain.
    """
    from twisted.names.client import lookupMailExchange
    def got_records(result):
        hosts, authorities, additional = result
        return [str(record.name) for record in additional]
    d = lookupMailExchange(domain)
    d.addCallback(got_records)
    return d


def main(domain):
    print("Mail servers for %s:" % (domain,))
    for mailserver in mx(domain).wait():
        print(mailserver)


if __name__ == '__main__':
    import sys
    main(sys.argv[1])

When we run it on the command line, output looks this:

$ python mxquery.py gmail.com
Mail servers for gmail.com:
alt2.gmail-smtp-in.l.google.com
alt2.gmail-smtp-in.l.google.com
alt3.gmail-smtp-in.l.google.com
alt3.gmail-smtp-in.l.google.com
alt4.gmail-smtp-in.l.google.com
alt4.gmail-smtp-in.l.google.com
alt1.gmail-smtp-in.l.google.com
gmail-smtp-in.l.google.com
gmail-smtp-in.l.google.com

The library provides much more functionality, but that's the gist of it: it runs and stops the Twisted reactor for you, and wraps asynchronous results in a blocking API. If you'd like to try out Crochet, or learn more about its other features, visit Crochet's PyPI page.

Jp Calderone: There, I Fixed It

Jean-Paul Calderone (noreply@blogger.com) — 2013-04-08T22:49:59+00:00

Got a new hard drive for my Inspiron 1545 - an SSD, at last! Turns out it was probably just in time too, as I found an unrecoverable bad block on the WD it is replacing.

Just one problem. The SSD is smaller than the drive it is replacing. Not narrower (they're both 2.5" drives - which means they're both 2.75" wide). But thinner. I suppose that dimension isn't standardized by anybody. But surely that can't actually be a problem? After all, thinner still fits in the drive bay...

Guess it is a problem after all. Near as I can tell, the SATA port in the Inspiron requires the drive to completely fill the bay in order to force the all the contacts to ... contact.

Good thing the problem is that it's too thin, not too thick. I can fix this.

Problem solved.

Thanks for the well-engineered hardware, guys.

Twisted Matrix Laboratories: Join Twisted for Google Summer of Code and the Outreach Program for Women

Itamar Turner-Trauring (noreply@blogger.com) — 2013-04-05T01:40:57+00:00

Twisted is once again participating in Google's Summer of Code this year, as well as the GNOME Outreach Program for Women. If you're a student interested in working on Twisted as part of a paid internship, please visit our Google SoC page. We use best practices like in-depth code reviews and full coverage unit tests, so this is a great way to improve your technical skills whether you're a beginner or an expert programmer.

In addition, Twisted is once again participating the GNOME Outreach Program for Women, with the generous support of Mozilla, providing a paid internship for one woman to spend the summer participating in and contributing to Twisted. Unlike Google's program, the outreach program is not restricted to students; if you qualify, we do encourage you to apply to both. This internship is appropriate for any level of open source experience.

If you have worked on an open source project before, great! If not, we'll help you learn the development and communication tools we use as part of the internship. Some Python experience is a prerequisite, and a small initial contribution to Twisted is a part of applying (if this sounds intimidating, don't worry, we'll help you pick a task to complete and you'll have lots of support as you work through submitting your first patch.) Please check out the full OPW and GSoC descriptions and apply today!

Duncan McGreggor: Autoscale and Orchestration: the Heat of OpenStack

Duncan McGreggor (noreply@blogger.com) — 2013-04-04T02:19:53+00:00

Several months before I joined Rackspace last year, there were efforts under way to provide an Autoscaling solution for Rackspace customers. Features that we needed in OpenStack and Heat hadn't been released yet, and there were no OpenStack experts on the Autoscaling team. As such, the engineers began developing a product that met Rackspace customer needs, integrated with the
existing monitoring and load-balancing infrastructure, and made calls to OpenStack Nova APIs as part of the scaling up and down process.

At PyCon this year, Monty Taylor, Robert Collins, Clint Byrum, Devananda van der Veen, and I caught up and chatted about what their views were of the current status of autoscaling support in OpenStack Heat. It seems that the two pieces we need the most -- LBaas and support for external monitoring systems (perhaps via webhooks) -- are nascent and not ready for prime-time yet. Regardless, Monty and his team encouraged us to dive into Heat, contribute patches, and in general, release our work for consumption by other Stackers.

Deeply encouraged by these interactions, we took this information to Rackspace management and, to quote Monty Python, there was much rejoicing. Obviously OpenStack is huge for Rackspace. Even more, there is a lot of excitement about Heat, the existing autoscaling features in OpenStack, and getting our engineers involved and contributing to these efforts.

In the course of these conversations, we discovered that Heat was getting lots of attention internally. It turns out that another internal Rackspace project had been doing something pretty cool: they were experimenting with the development of a portable syntax for application description and deployment orchestration. Their work had started to converge on some of the functionality provided by Heat, and they had a similar experience as the Autoscaling team. The timing was right to contribute what they have learned and align all of their continued efforts with adding value to Heat.

Along these lines, we are building two new teams that will focus on Heat development: one

contributing to features related to autoscaling (not necessarily limited to Heat) and the other contributing to the ongoing conversations regarding the separation of concerns between orchestration and configuration management. Everyone -- from engineers to management -- is very excited about this new direction in which our teams are moving. Not only will it bring new developers to OpenStack, but it is aligning our teams with Rackspace's OpenStack roots and the company's vision for supporting the growing cloud community.

Simply put: we're pretty damned pumped and looking forward to more good times with OpenStack :-)

Twisted Matrix Laboratories: Twisted 13.0.0 released

Thomas Hervé (noreply@blogger.com) — 2013-04-03T11:20:55+00:00

On behalf of Twisted Matrix Laboratories, I am pleased to announce the release of Twisted 13.0.

Among the 70 tickets closed, we can see:

A new Introduction to Deferreds document.
A fix in twisted.web.template where attributes were not quoted properly, risking HTML injection.
Support for unicode domain names in twisted.names SRVConnector and Name classes, after a 12.3 regression
A workaround for platform limitations when trying to schedule events far in the future.

For more information, see the NEWS file here: NEWS.txt

Download it now from Twisted-13.0.0.tar.bz2 or Twisted-13.0.0.win32-py2.7.msi

Thanks to the supporters of Twisted via the Software Freedom Conservancy and to the many contributors for this release.

Duncan McGreggor: Maths and Programming: Whence Recursion?

Duncan McGreggor (noreply@blogger.com) — 2013-04-03T00:33:31+00:00

As a manager in the software engineering industry, one of the things that I see on a regular basis is a general lack of knowledge from less experienced developers (not always "younger"!) with regard to the foundations of computing and the several related fields of mathematics. There is often a great deal of focus on what the hottest new thing is, or how the industry can be changed, or how we can innovate on the decades of profound research that has been done. All noble goals.

Notably, another trend I've recognized is that in a large group of devs, there are often a committed few who really know their field and its history. That is always so amazing to me and I have a great deal of admiration for the commitment and passion they have for their art. Let's have more of that :-)

As for myself, these days I have many fewer hours a week which I can dedicate to programming compared to what I had 10 years ago. This is not surprising, given my career path. However, what it has meant is that I have to be much more focused when I do get those precious few hours a night (and sometimes just a few per week!). I've managed this in an ad hoc manner by taking quick notes about fields of study that pique my curiosity. Over time, these get filtered and a few pop to the top that I really want to give more time.

One of the driving forces of this filtering process is my never-ending curiosity: "Why is it that way?" "How did this come to be?" "What is the history behind that convention?" I tend to keep these musings to myself, exploring them at my leisure, finding some answers, and then moving on to the next question (usually this takes several weeks!).

However, given the observations of the recent years, I thought it might be constructive to ponder aloud, as it were. To explore in a more public forum, to set an example that the vulnerability of curiosity and "not knowing" is quite okay, that even those of us with lots of time in the industry are constantly learning, constantly asking.

My latest curiosity has been around recursion: who first came up with it? How did it make it's way from abstract maths to programming languages? How did it enter the consciousness of so many software engineers (especially those who are at ease in functional programming)? It turns out that an answer to this is actually quite closely related to a previous post I wrote on the secret history of lambda. A short version goes something like this:

Giuseppe Peano wanted to establish a firm foundation for logic and maths in general. As part of this, he ended up creating consistent axioms around the hard-to-define natural numbers, counting, and arithmetic operations (which utilized recursion). While visiting a conference in Europe, Bertrand Russell was deeply impressed by the dialectic talent of Peano and his unfailing clarity; he queried Peano as to his secret for success (Peano told him) and them asked for all of his published works. Russell proceeded to studied these quite deeply and eventually co-wrote the Principia Mathematica, which Alonzo Church (along with his grad students) sought to improve upon later. Alonzo Church ended up developing the lambda calculus based on his work around the Principia and his student, John McCarthy, later created the first functional programming language, Lisp, utilizing concepts from the lambda calculus (recursion and function composition).

In the course of reading between 40-50 mathematics papers (including various histories) over the last week, I have learned far more than I had originally intended. So much so, in fact, that I'm currently working on a very fun recursion tutorial that not only covers the usual practical stuff, but steps the reader through programming implementations of the Peano axioms, arithmetic definitions, the Ackermann function, and parts of the lambda calculus.

I've got a few more blog post ideas cooking that dive into functions, their history and evolution. We'll see how those pan out. Even more exciting, though, was having found interesting papers discussing the evolution of functions and the birth of category theory from algebraic topology. This, needless to say, spawned a whole new trail of research, papers, and books... and I've got some great ideas for future blog posts/tutorials around this topic as well. (I've encountered category theory before, but watching it appear unsearched and unbidden in the midst of the other reading was quite delightful).

In closing, I enjoy reading not only the original papers (and correspondence between great thinkers of a previous era), but also the meanderings and rediscoveries of my peers. I've run across blog posts like this in the past, and they were quite enchanting. I hope that we continue to foster that in our industry, and that we see more examples of it in the future.

Keep on questing ;-)

Twisted Matrix Laboratories: Report the Third

Tom Prince (noreply@blogger.com) — 2013-03-28T08:59:33+00:00

During the past three weeks, 23 tickets got some attention that they would not have received without the sponsored development. The result was 9 closed tickets and 8 other tickets unblocked for other developers to resume work on.

I also spent some time working on getting newly donated slaves setup, as well as documenting the setup of the main twistedmatrix.com server, in preparation for automating the redeployment of services to new hardware.

The tickets I reviewed, that have now been closed:

Other tickets I have reviewed:

Tickets I have closed:

#6316: Minor cleanup of pam tests in `twisted.test.test_newcred`.

Other tickets I have worked on:

Ashwini Oruganti: PyCon 2013: Down the Rabbit Hole

2013-03-23T22:53:00+00:00

So there was PyCon. And there was me.

Now there are two ways in which I can tell you how it was. I can use some really big words and give you a dry report of events. But that’s boring, and there’s enough boring stuff out there.

“Speak English!” said the Eaglet. “I don’t know the meaning of half those long words, and I don’t believe you do either!”

So here’s a recollection of whatever I can remember, without using many big words.

It started on March 12, at least the interesting part did.

I was welcomed into SJC with a PyLadies banner and a smiling Lynn Root. I also met Hynek, and we drove to the hotel and I guess that’s about it for the first day.

The next day, i.e. March 13, I went to the Convention Centre, met Yarko, who printed my badge. I met Steve Holden - no wait, didn’t really meet, I jumped into his way and talked to him - I wonder how weird it looked. I did some volunteering - moved boxes, helped out at the registration desk - was fun. I then trotted around the venue with Stuart, this time pretending to do some volunteering. Met Doug - he’s nice, and gave me more work. I saw Guido at the Language Summit, and he was getting coffee alone, but looked somewhat tired, so I decided to leave him alone. I wasn’t equally kind to Glyph though, and chatted a lot, everything from the history of Twisted to, well… DuckDuckGo.

I also wasn’t that kind to Asheesh, and badgered him loads. He called me weird because Twisted didn’t scare me. Jessica, who’s super-humanly awesome, by the way, if you didn’t already know, gathered lots of folks from Boston and OpenHatch, and we all hung out and ate food. Good food, actually, in case you were wondering. I also got to know the secret to how paulproteus is omnipresent on IRC (he’s everywhere you look!).

Later that night, I met nedbat and a few CPython core devs who told me stories about old Twisted days and some insider-info on Glyph, his name and identity and stuff. Not going to write about that though - if you want to know, get your own tickets to PyCon. We also discussed some cool MIT Media labs stuff, and I met Steve again, and he seemed to remember my name (I guess that means I must have weirded him out significantly earlier).

Moving on to the 14th of March. The day started with me meeting Itamar, who had a tutorial later that day. Then had lunch with some Twisted people, and then there was that tutorial. I met Jean-Paul for the first time, and learnt that I had caused him so much anguish during the summer that he decided to get a new job this summer. I apologised profusely, but I think the damage is done. Oh, and I threw my laptop on the floor, because you know, no one wants to not look like a fool in a room full of people.

Then I went back to my room, after revealing a nervous-wrecked self to Glyph and Ying. I woke up next day and went to attend the Keynote. Now, I was quite sleepy and tired and lost and fretting a bit over my talk, so I wasn’t expecting to find a 30 minute lecture too interesting. To my immense surprise, I actually enjoyed it. I met Michelle and a few other PyLadies, and we all got a Raspberry Pi each, and then I vanished until my talk.

I saw that there were actual people present in my talk, which was quite surprising. Oh wait, there were some really helpful people in the Green Room, a special mention must go to David, who was great at calming nerves - he asked me to take two bottles of water to the talk - and I did. Anyway, it seemed to have gone by okay - I did throw the mic on the floor in the middle of my talk (I guess I throw things around a lot) and then there were some issues with the computer, but anyway, I kept talking, it’s hard to shut me up. People were nice, they laughed at my jokes and Glyph let me call him a liar. After the talk, I saw that the Twisted core team was holding some signs up, which spelt “Go Ashfall/Ashwini”, and that’s nice enough to make you cry. I didn’t cry though, because everything was being recorded.

Later that day, I ate some more good food with the Twisted people, and heard more stories.

The next day, the 16th of March, I got to the Keynote. And then there was Glyph’s talk. He mentioned my name and referred to some stuff I’d said in my talk (and did not, unlike me, call me a liar - I guess he’s nice), and everyone is saying that this makes me a celebrity. I don’t think it works that way, but if you believe so and want an autograph or something, feel free to drop me a mail.

Then there was the PyLadies lunch, where I met more awesome people, and had some more good food. I attended Lynn’s talk next, and then went to dinner with Guido. Yes, the Guido who wrote Python. My friends insist that I must describe this part well, so here it is: we drove in his car to a steakhouse, and I sat to his left. Ate food. Drank lemonade. Heard some interesting stories. Yes, PyCon exposes you to loads of stories, it is cool that way. Then Guido dropped me back to the Convention Center where everyone got together and talked about goats, which was very informative.

Oh, also, Disney gave away Wreck-it Ralph action figures to all the speakers (and hence, me), and Paul Hildebrandt is one of the nicest and kindest people I have talked to. Just saying. Also, Armin Rigo is awesome - he’s perhaps the only incredibly intelligent person involved with programming I’ve met who seemed genuinely happy about life (and not grumpy at all).

The next day, which was also my last day at PyCon, I attended Guido’s Keynote, and then everyone agreed that I wasn’t lying when I said that I had become cynical about software. We had the annual Twisted dinner and I had an interesting conversation with Zooko, who accepted (and even adopted) my mispronunciation of IRC nicks without hesitation or revolt. I sat with Bradley and learnt a lot, with interesting insights from Duncan - about Usenet, about embarrassing email conversations, and about proprietary software pains, I should probably write more about that sometime.

And then there was the Twisted sprint, which I couldn’t attend for long, but everyone signed the Twisted book for me (also, Jessica mentioned me in the credits of that book - again, people tell me it makes me a celebrity of sorts).

Well, that’s that. I met most of the Twisted team, and they were all incredibly nice. But before I finish, I would like to thank the Python Software Foundation, PyLadies, and everyone involved with PyCon, for making it possible for me to attend, and I hope I see you all again next year!

So then, how was PyCon 2013? Welcoming? Engaging? Awesome? I guess you should decide for yourself.

Jack Moffitt: Joining Mozilla

Jack Moffitt (jack@metajack.im) — 2013-03-22T10:52:00+00:00

On Monday, I join Mozilla to work on Servo, a new and experimental web browser engine built on Rust, a new systems programming language. I am perhaps the first professional Rust programmer.

This will also be the first time in over a decade that I'm not working in a small company or a startup (usually both). I've been thinking for a while that it would be nice to work for a company that has real resources to solve problems, as opposed to being at the mercy of venture capitalists or the whims of users. Mozilla's mission statement is one that is easy for me to get behind, and they are doing very interesting things at Mozilla Research.

I enjoy working on difficult and important projects, and it's hard for me to imagine much that is more difficult or important than web browsers. It's an added bonus to be working in and (hopefully) contributing to a new programming language. I also love working with smart people, and Mozilla seems to have those in abundance.

This is going to be awesome.

David Reid: The Talks I Saw At Pycon

2013-03-22T07:00:00+00:00

PyCon 2013 was amazing. Here are video links to the talks I saw whose videos are currently available. I liked them all.

In no particular order:

Of course, these are just a few of the talks at the conference, you can find more at pyvideo.org.

Donovan Preston: Guido doesn't want non-portable assembly in Python and it's understandable

Donovan Preston (noreply@blogger.com) — 2013-03-22T04:10:13+00:00

(I wrote a long comment in the Hacker News discussion of Guido's slides about his plans for async io and asymmetric coroutines in Python 3.4, but I thought it was good enough to deserve a blog post)

From a certain perspective [Guido's desire to keep non-portable stack slicing assembly out of Python] is a rational decision. Because the CPython API relies so heavily on the C stack, either some platform-specific assembly is required to slice up the C stack to implement green threads, or the entire CPython API would have to be redesigned to not keep the Python stack state on the C stack.

Way back in the day the proposal for merging Stackless into mainline Python involved removing Python's stack state from the C stack. However there are complications with calling from C extensions back into Python that ultimately killed this approach.

After this Stackless evolved to be a much less modified fork of the Python codebase with a bit of platform specific assembly that performed "stack slicing". Basically when a coro starts, the contents of the stack pointer register are recorded, and when a coro wishes to switch, the slice of the stack from the recorded stack pointer value to the current stack pointer value is copied off onto the heap. The stack pointer is then adjusted back down to the saved value and another task can run in that same stack space, or a stack slice that was stored on the heap previously can be copied back onto the stack and the stack pointer adjusted so that the task resumes where it left off.

Then around 2005 the Stackless stack slicing assembly was ported into a CPython extension as part of py.lib. (By Armin Rigo. A million thanks from me for this.) This was known as greenlet. Unfortunately all the original codespeak.net py.lib pages are 404 now, but here's a blog post from around that time that talks about it.

Finally the relevant parts of greenlet were extracted from py.lib into a standalone greenlet module, and eventlet, gevent, et cetera grew up around this packaging of the Stackless stack slicing code.

So you see, using the Stackless strategy in mainline python would have either required breaking a bunch of existing C extensions and placing limitations on how C extensions could call back into Python, or custom low level stack slicing assembly that has to be maintained for each processor architecture. CPython does not contain any assembly, only portable C, so using greenlet in core would mean that CPython itself would become less portable.

Generators, on the other hand, get around the issue of CPython's dependence on the C stack by unwinding both the C and Python stack on yield. The C and Python stack state is lost, but a program counter state is kept so that the next time the generator is called, execution resumes in the middle of the function instead of the beginning.

There are problems with this approach; the previous stack state is lost, so stack traces have less information in them; the entire call stack must be unwound back up to the main loop instead of a deeply nested call being able to switch without the callers being aware that the switch is happening; and special syntax (yield or yield from) must be explicitly used to call out a switch.

But at least generators don't require breaking changes to the CPython API or non-portable stack slicing assembly. So maybe now you can see why Guido prefers it.

Myself, I decided that the advantages of transparent stack switching and interoperability outweighed the disadvantages of relying on non-portable stack slicing assembly. However Guido just sees things in a different light, and I understand his perspective.

Duncan McGreggor: Rendering ReST with Klein and Twisted Templates

Duncan McGreggor (noreply@blogger.com) — 2013-03-14T03:44:37+00:00

In a previous life, I spent about 25 hours a day worrying about content management systems written in Python. As a result of the battle scars built up during those days, I have developed a pretty strong aversion for a heavy CMS when a simple approach will do. Especially if the users are technologically proficient.

At MindPool, we're building out our infrastructure right now using Twisted so that we can take advantage of the super amazing numbers of protocols that Twisted supports to provide some pretty unique combined services for our customers (among the many other types of services we are providing). For our website, we're using the Bottle/Flask-inspired Klein as our micro web framework, and this uses the most excellent Twisted templating. (We are, of course, also using Twitter Bootstrap.)

Here's the rub, though: we want to manage our content in the git repo for our site with ReStructured Text files, and there's no way to tell the template rendering machinery (the flattener code) to allow raw HTML into the mix. As such, my first attempt at ReST support was rendering HTML tags all over the user-facing content.

This ended up being a blessing in disguise, though, as I was fairly unhappy with the third-party dependencies that had popped up as a result of getting this to work. After a couple false starts, I was hot on the trail of a good solution: convert the docutils-generated HTML (from the ReST source files) to Twisted Stan tags, and push those into the renderers.

This ended up working like a champ. Here's what I did:

Created a couple of utility functions for easily getting HTML from ReST and Stan from ReST.
Wrote a custom IRenderable for ReST content (not strictly necessary, but organizationally useful, given what else will be added in the future).
Updated the base class for "content" page templates to dispatch, depending upon content type.

Afterwards I was rewarded with some nicely rendered content on the staging MindPool site :-) (once the content text has been completed, we'll be pushing it live).

Kudos to David Reid for Klein and (as usual) to the Twisted community for one hell of a framework that is the engine of my internet.

Duncan McGreggor: GIMP 2.8 and the Taming of Two Decades' Graphics Habits

Duncan McGreggor (noreply@blogger.com) — 2013-03-14T03:44:13+00:00

At long last, I find myself in a 100% comfort zone with the GIMP. As a user, it's been a long road to get here ... I can't even imagine what the development teams have experienced over the past 16 years. Regardless, I am infinitely grateful for their efforts over that time, their hard work to bring the world a completely free, open source application for incredibly sophisticated image manipulation, drawing, and even painting.

I'll provide some more details on my experiences with GIMP 2.8, but first... a romp through the past!

I've been using graphics programs since my first exposure to the Macintosh Plus and SE machines in the late 1980s, where the head of the science department at Bangor High School often found himself without access to his machine (yeah, thanks to me...). As much as a science geek as I was, it was actually the graphics programs to which I was addicted. In particular, I was obsessed with using MacPaint to draw -- pixel by pixel -- a dithered bitmap of a photograph leaning next to the monitor.

About 6 or 7 years later, I was at it again doing graphics work on a non-profit's Mac (running System 7) with Photoshop 3.0 -- my first exposure to this software. It was absolutely mind-blowingly awesome what Photoshop enabled one to do with digital images. I was consumed. If I was awake, I was at the computer, creating something whacky from scratch or morphing something old into a new form. Though the work that I had done was focused on graphics for HTML and getting into the nascent field of "web design," some of my efforts ended up getting published as illustrations for a book.

I continued doing graphics as part of my work (in some form or another) from then on, usually borrowing a friend's machine in order to use Photoshop (which I could not afford). Despairing of not having my own copy of the software, I would often try out other drawing, painting, and image manipulations applications. Without fail, none would ever measure up to the power and even more, the usability, of Photoshop.

It wasn't too much after this that I got involved with Linux and open source software. As for many from that time, my life has not been the same since. But those early years were painful. My first distro (Slackware), I had to manually hack the ethernet drivers in C just to get connected to the network using my particular hardware. Regardless, Linux was all I could think about; it was all I wanted to use.

Desperately hungering for a Photoshop-like experience on Linux, I searched HotBot on a regular basis, and finally discovered GIMP. Based on a quick glance at the now-historic splash screens, it looks like my first use of the GIMP started after the 1.0 release, though I didn't start using it more regularly until 1.2 (which had my favorite GIMP splash screen to date).

Despite my hopes and the best efforts of an amazing development team, the GIMP infuriated me. What took one step in Photoshop usually took about 5 steps in the GIMP. The user interface was anti-intuitive, and seemed to be built by folks with an intimate knowledge of the underlying API and reflected this far too well.

With each major release of the GIMP, I would find myself downloading and installing it, but only actually using it a handful of times. Ultimately, I'd run crying back to Photoshop, borrowing a friend's computer or persuading someone to give me an unused copy. The very nature of graphic arts is visual and intuitive... I felt that the GIMP was blocking that natural flow. As a result, I couldn't get anything done in it (and what little I did looked bad).

Year later, while working at Canonical, I had several opportunities to create graphics for various fun projects, gags, or practical jokes. Given the nature of the company's mission, I made sure to do that work using the GIMP. Though the pain of the old days had faded considerably, and GIMP had, by 2008, reached a much higher degree of usability, I still found myself enduring awkward workflow moments on a regular basis and this continued throughout the 4 years of version 2.6's release. However, during that time, an impressive selection of GIMP plugins were created, some of which I came to adore so completely that I would switch from Photoshop to GIMP, just so I could use them on a project. This was a new -- and amazingly welcome -- change for me.

Then, earlier this year, the last few pieces fell into place.

With the release of GIMP 2.8 in May, I have not looked for nor pined over any other sophisticated graphics programs. At long last, the GIMP completely satisfies. The biggest single point of pain for me in 2.6 (given that I was running on Mac OS X most of the time) was the lack of single-window support. For every UI interaction, I had to click twice -- once to focus on the window, and once to actually perform the given action. It was crazy-making. 2.8 finally released the goodness of single-window mode for GIMP users around the globe. Futhermore, since I have been a layer-using maniac since Photoshop 3.0 and my graphics files tend to reach into the 200 and 300 MB range due to the numbers of layers (and resolution, of course), I have a desperate need to organize the chaos of all those layers. Prior to 2.8, my layer pallet on large projects was almost completely unnavigable. With 2.8, sanity and cleanliness abounds.

And there's more :-) Check out this short list:

single-window mode
multi-column dock windows
increased screen real-estate
layer groups
Cairo is used for all rendering
on-canvas text editing

For more details, you'll want to visit this page.

Oddly, this walk down memory lane and ultimate endorsement of the GIMP relates to MindPool. I used the GIMP to do all the complicated graphics work for our initial branding, logos, etc. Looking at the results, I'm sure if seems fairly silly and even trivial... but there's a LOT that went into that simplicity :-)

For instance, the shot of the moon is actually taken from a photograph of the moon. It has been passed through multiple filters in order to get just the level of abstract color and shape I was looking for. Similarly for the water, but the settings were actually quite different to achieve the result I was seeking. I think the layer count peaked around 30 in one version of the file.

Another example is the logo that we'll be using for the company's main site. This also took advantage of some plugins (new and old), though mostly it relied upon paths, brush strokes, and mask layers (the latter for the water pool reflection watermarked into the image). The new features in 2.8 made all of that work very easy to do. No longer is the tool interfering with my workflow; rather, it is facilitating it. Working with graphics using open source tools is now an absolute pleasure for me.

For giggles, I'm including a screenshot of GIMP 2.8 in action, with my single-window mode view stretched across two 27" monitors (full version is here). Graphical editing has never been so clean!

Thanks, GIMP team :-) It's nice to finally be home.

Fun link: For the historically inclined, you might enjoy this quick read about the beginnings of Photoshop and the family that made it all happen :-)

Duncan McGreggor: Getting Started with Steel Bank Common Lisp

Duncan McGreggor (noreply@blogger.com) — 2013-03-14T03:43:27+00:00

As some of you know, I've been a closet Lisp fan for several years. When I first joined Canonical in 2008, I was hacking on Lisp in Python, so that I could do genetic programming in Python. In fact, my first and only lightening talk at a Canonical sprint was on genetic algorithms and programming :-) (This was the same set of lightening talks that Vincenzo Di Somma gave a wonderful presentation on his photography; completely unrelated: this is one of my favorite pics of Vincenzo :-) ).

A few years later, I talked to Jim Baker about Python's AST, and how one might be able to do genetic programming by manipulating it directly, instead of running a Lisp in Python.

Throughout all this time, I've been touching in with various community projects, hacking on various Lispy Things, reading, etc., but generally doing so quite quietly. Over the past few months, however, I've really gotten into it, and Lisp has become a real force in my life, rapidly playing just as dominant a role as Python.

Similarly, MindPool has become active in several Lisp projects; as such, there are a great many things to share now. However, before I begin all that, I'd like to take an opportunity to get folks up and running with an example Lisp environment.

Future posts will explore various areas of Common Lisp, Scheme dialects, I/O loops, etc., but this one will provide a basis for all future posts that relate to Common Lisp and specifically the Steel Bank implementation.

Installing SBCL
If you don't have SBCL (Steel Bank Common Lisp; a pun on it's source parent, CMUCL), you need to install it:

For Ubuntu (12.04 LTS has 1.0.55): $ sudo apt-get install sbcl
Or you can go to the download page for everyone else.

apt-get for Lisp
Next, you'll need to install Quicklisp (as you might have surmised, it's like Debian apt-get for Common Lisp). The instructions on this page will get you up and running with Quicklisp.

I like having quicklisp available when I run SBCL, so I did the following after installing Quicklisp (and you might want to as well) from the sbcl prompt:
* (ql:add-to-init-file)

Readline Support
The default installation of SBCL doesn't have readline support for the REPL, so using your arrow keys won't give you the expected result (your command history). To remedy that, you can use a readline wrapper. First, install rlwrap:

Ubuntu: $ sudo apt-get install rlwrap
Mac OS X: $ brew install rlwrap

Then, create the chmoded 755 script ~/bin/start-sbcl with the following content (make sure that ~/bin is in your path):
rlwrap sbcl

At which point you can run the following and have access to a command history in SBCL:
$ start-sbcl
*

Why Steel Bank?
CMUCL gained an excellent reputation for being a highly performant, optimized implementation of Lisp. Based on CMUCL and continuing this tradition of excellent performance, SBCL's reputation preceded it. Over a range of different types of programs, SBCL not only compares favorably to other Lisp dialects, it seriously kicks ass all over.

SBCL comes in at 8th place in that benchmark ranking, beating out Go in 9th place. In all the languages that made it into the Top 10, I've only ever touched C, C++, Java, Scala, Lisp, and Go. In my list, SBCL made the Top 5 :-) Regardless, of all of them, Lisp has the syntax a find most pleasurable. Given my background in Python, this is not surprising ;-)

What's next?
Funny that you should ask... given my background with Twisted, I'll give you one guess ;-)

Duncan McGreggor: libevent for Lisp: A Signal Example

Duncan McGreggor (noreply@blogger.com) — 2013-03-14T03:42:51+00:00

At MindPool, there are several async I/O options we've been exploring for Lisp:

CMUCL/SBCL's SERVE-EVENT
IOlib
cl-event (libevent for Lisp, using cffi; three years old)
cl-async (also using cffi libevent wrapper, actively developed)

As luck would have it, I started with cl-event, and it was a fun little adventure (given the fact that it hasn't been maintained). I corresponded with the very nice original author a bit, asking if there were any updates in other locations, but sadly there weren't.

I was ready to dive in and get things current, when one last Google search turned up cl-async. This little bugger was hard to find, as at that point it had not been listed on CLiki. (But it is now :-)). Andrew Lyon has done a tremendous amount of work on cl-async, with a very complete set of bindings for libevent. This is just what I had been looking for, so I jumped in immediately.

As one might imagine from the topic of this post, there's a lot to be explored, uncovered, and developed further around async programming in Lisp. I'll start off slowly with a small example, and add more over the course of time.

I also hope to cover IOlib and SBCL's SERVE-EVENT in some future posts. Time will tell... For now, let's get started with cl-async in SBCL :-)

Dependencies
In a previous post, I discussed getting an environment set up with SBCL, I the rest of this post assumes that has been read and done :-)

Getting cl-async and Setting Up an SBCL Environment for Hacking
Now let's download cl-async and install the Libevent bindings :-)

With the Lisp Libevent bindings installed, we're now ready to create a Lisp image to assist us when exploring cl-async. A Lisp image saves the current state of the REPL, with all the loaded libaries, etc., allowing for rapid start-ups and script executions. Just the thing, when you're iterating on something :-)

Example: Adding a Signal Handler
Let's dive into some signal handling now! Here is some code I put together as part of an effort to beef up the examples in cl-async:

Note that the as: is a nickname for the package namespace cl-async:.

As one might expect, there is a function to start the event loop. However, what is a little different is that one doesn't initialize the event loop directly, but with a callback. As such, one cannot set up handlers, etc., except within the scope of this callback.

We've got the setup-handler function for that, which adds a callback for a SIGINT event. Let's try it out :-)

Once your script has finished loading the core, you should see output like the above, with no return to the shell prompt.

When we send a SIGINT with ^C, we can watch our callback get fired:

Next up, we'll take a look at other types of handlers in cl-async.

Duncan McGreggor: Async in Clojure: Playing with Agents, Part II

Duncan McGreggor (noreply@blogger.com) — 2013-03-14T03:40:32+00:00

In the last post, we took a look at basic usage of Clojure's agent function. In this post, we'll dive a little bit deeper

Validation
We glossed over the options that you can define when creating an agent; one of them is the validator which one can use to check before the agent is updated with the passed value.

If we want to make sure that our read-agent always gets a string value, this is all we have to do:

Similarly, any function that takes a single value as a parameter can be used here. As you can see, we had to change our default value for the agent from nil to "" since there is now a string validator. If we hadn't, any time we tried to use that agent, we'd get java.lang.IllegalStateException.

When Things Go Wrong
Another option you can set when defining an agent is the error handler. This will be used in the event of an error, including if a value fails to be validated by your validator function. Here's an example:

With both of these options, you don't have to set them when the agent is defined; you can do it later with a function call, if you so desire (or if needs demand it):

Watch This!
So, we've got an error handler but no event handler? Yup. However, you can actually get callback-like behavior using watches. Check this out:

Now, any time our agent's state changes, the function passed to the watch will fire. As described in the docs, the parameters are: the agent, a key of your devising (must be unique per agent), and the handler that you want to have fired upon state change. The handler takes as parameters: the key you defined, the agent, the agent's old value, and the agent's new value.

All Together Now
With all our example code in place, we can now exercise the whole thing at once. Here's the whole thing:

To simply demonstrate the async nature and the callbacks in action, let's run the following:

Eventually, our callback will render output very much like the following:

Do note, however, that if we called a series of send-offs with different times (using the same agent and watch), we wouldn't see the ones with shorter times come back first. We'd see the callback output in the same order we called send-off. This is because the watch function is called synchronously on the agent's thread before any pending send-offs (or sends) are called. In future posts, I'll cover ways around this (constructing agents on the fly as well as exploring alternative solutions with external libraries).

Regardless, with these primitives, there are all sorts of things one can do. For instance...

Dessert
To close, check out this neat little bit of code that sends 1,000,000 messages in a ring. This code creates a chain of agents, and then actions are relayed through it (taken from the agents doc page):

Kicking this puppy off, our million messages finish in about 1 second :-)