unethical blogger - Python

Unclog the tubes; blocking detection in Eventlet

R. Tyler Croy — Sat, 28 Aug 2010 22:12:07 +0000

Colleagues of mine are all very familiar with my admiration of Eventlet, a Python concurrency library, built on top of greenlet, that provides lightweight "greenthreads" that naturally yield around I/O points. For me, the biggest draw of Eventlet besides its maturity, is how well it integrates with standard Python code. Any code that uses the built-in socket module can be "monkey-patched" (i.e. modified at runtime) to use the "green" version of the socket module which allows Eventlet to turn regular ol' Python into code with asynchronous I/O.

The problem with using libraries like Eventlet, is that some Python code just blocks, meaning that code will hit an I/O point and not yield but instead block the entire process until that network operation completes.

In practical terms, imagine you have a web crawler that uses 10 "green threads", each crawling a different site. The first greenthread (GT1) will send an HTTP request to the first site, then it will yield to GT2 and so on. If each HTTP request blocks for 100ms, that means when crawling the 10 sites, you're going to block the whole process, preventing anything from running, for a whole second. Doesn't sound too terrible, but imagine you've got 1000 greenthreads, instead of everything smoothly yielding from one thread to another the process will lock up very often resulting in painful slowdowns.

Starting with Eventlet 0.9.10 "blocking detection" code has been incorporated into Eventlet to make it far easier for developers to find these portions of code that can block the entire process.

    import eventlet.debug
    eventlet.debug.hub_blocking_detection(True)

While using the blocking detection is fairly simple, its implementation is a bit "magical" in that it's not entirely obvious how it works. The detector is built around signals, inside of Eventlet a signal handler is set up prior to firing some code and then after said code has executed, if a certain time-threshhold has passed, an alarm is raised dumping a stack trace to the console. I'm not entirely convinced I'm explaining this appropriately so here's some pseudo-code:

    def runloop():
        while True:
            signal.alarm(handler, 1)
            execute_next_block()
            if (time.time() - start) < resolution:
                clear_signal() # Clear the signal if we're less than a second, otherwise it will alarm

The blocking detection is a bit crude and can raise false positives if you have bits of code that churn the CPU for longer than a second but it has been instrumental in incorporating non-blocking DNS support into Eventlet, which was also introduced in 0.9.10 (ported over from Slide's gogreen package).

If you are using Eventlet, I highly recommend running your code periodically with blocking detection enabled, it is an invaluable tool for determining whether you're running as fast and as asynchronous as possible. In my case, it has been the difference between web services that are fast in development but slow under heavy stress, and web services that are fast always regardless of load.

Being a Libor, Addendum

R. Tyler Croy — Tue, 18 May 2010 16:00:00 +0000

A couple of weeks ago I wrote a post on how to "Be a Libor", trying to codify a few points I feel like I learned about building a successful engineering team at Slide. Shortly after the post went live, I discovered that Libor had been promoted to CTO at Slide.

Over coffee today Libor offered up some finer points on the post in our discussion about building teams. It is important, according to Libor, to maintain a "mental framework" within which the stack fits; guiding decisions with a consistent world-view or ethos about building on top of the foundation laid. This is not to say that you should solve all problems with the same hammer, but rather if the standard operating procedure is to build small single-purpose utilities, you should not attack a new problem with a giant monolithic uber-application that does thirty different things (hyperbole alert!).

Libor also had a fantastic quote from the conversation with regards to approaching new problems:

Just because there are multiple right answers, doesn't mean there's no wrong answers

Depending on the complexity of the problems you're facing there are likely a number of solutions but you still can get it wrong, particularly if you don't remain consistent with your underlying mental framework for the project/organization.

As usual my discussions with Libor are interesting and enjoyable, he's one of the most capable, thoughtful engineers I know, so I'm interested to see the how Slide Engineering progresses under his careful hand as the new CTO. I hope you join me in wishing him the best of luck in his role, moving from wrangling coroutines, to herding cats.

God speed mooncat

Is programming with Twisted really as awful as it sounds?

R. Tyler Croy — Wed, 12 May 2010 16:45:00 +0000

Early this week Can forwarded this post on Quora to me, which asks the question:

Is programming with Twisted really as awful as it sounds?

Yes. Yes. YES IT IS. HOLY CRAP IT'S AWFUL

Here's some good alternatives:

Eventlet, my preference
gevent, an alternative to Eventlet tied to libevent
Java. because let's face it, if you're using Twisted, you've already decided not to write Python, so use something with proper threading support.

That is all.

How-to: Using Avro with Eventlet

R. Tyler Croy — Fri, 07 May 2010 16:45:00 +0000

Working on the plumbing behind a sufficiently large web application I find myself building services to meet my needs more often than not. Typically I try to build single-purpose services, following in the unix philosophy, cobbling together more complex tools based on a collection of distinct building blocks. In order to connect these services a solid, fast and easy-to-use RPC library is a requirement; enter Avro.

Note: You can skip ahead and just start reading some source code by cloning my eventlet-avro-example repository from GitHub.

Avro is part of the Hadoop project and has two primary components, data serialization and RPC support. Some time ago I chose Avro for serializing all of Apture's metrics and logging information, giving us a standardized framework for recording new events and processing them after the fact. It was not until recently I started to take advantage of Avro's RPC support when building services with Eventlet. I've talked about Eventlet before, but to recap:

Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it

What this means in practice is that you can write highly concurrent network-based services while keeping the code "synchronous" and easy to follow. Underneath Eventlet is the "greenlet" library which implements coroutines for Python, which allows Eventlet to switch between coroutines, or "green threads" whenever a network call blocks.

Eventlet meets Avro RPC in an unlikely (in my opinion) place: WSGI. Instead of building their own transport layer for RPC calls, Avro sits on top of HTTP for its transport layer, POST'ing binary data to the server and processing the response. Since Avro can sit on top of HTTP, we can use eventlet.wsgi for building a fast, simple RPC server.

Defining the Protocol

The first part of any Avro RPC project should be to define the protocol for RPC calls. With Avro this entails a JSON-formatted specification, for our echo server example, we have the following protocol:

{"protocol" : "AvroEcho",
"namespace" : "rpc.sample.echo",
"doc" : "Protocol for our AVRO echo server",
"types" : [],
"messages" : {
    "echo" : {
        "doc" : "Echo the string back",
        "request" : [
                {"name" : "query", "type" : "string"}
                ],
        "response"  : "string",
        "errors" : ["string"]
    },
    "split" : {
        "doc" : "Split the string in two and echo",
        "request" : [
                {"name" : "query", "type" : "string"}
                ],
        "response"  : "string",
        "errors" : ["string"]
    }
}}

The protocol can be deconstructed into two concrete portions, type definitions and a message enumeration. For our echo server we don't need any complex types, so the types entry is empty. We do have two different messages defined, echo and split. The message definition is a means of defining the actual remote-procedure-call, services supporting this defined protocol will need to send responses for both kinds of messages. For now, the messages are quite simple, they expect a query parameter which should be a string, and are expected to return a string. Simple.

(This is defined in protocol.py in the Git repo)

Implementing a Client

Implementing an Avro RPC client is simple, and the same whether you're building a service with Eventlet or any other Python library so I won't dwell on the subject. A client only needs to build two objects, an "HTTPTransceiver" which can be used for multiple RPC calls and grafts additional logic on top of httplib.HTTPConnection and a "Requestor".

client = avro.ipc.HTTPTransceiver(HOST, PORT)
requestor = avro.ipc.Requestor(protocol.EchoProtocol, client)
response = requestor.request('echo', {'query' : 'Hello World'})

You can also re-use for same Requestor object for multiple messages of the same protocol. The three-line snippet above will send an RPC message echo to the server and then return the response.

(This is elaborated more on in client.py in the Git repo)

Building the server

Building the server to service these Avro RPC messages is the most complicated piece of the puzzle, but it's still remarkably simple. Inside the server.py you will notice that we call eventlet.monkey_patch() at the top of the file. While not strictly necessary inside the server since we're relying on eventlet.wsgifor writing to the socket. Regardless it's a good habit to get into when working with Eventlet, and would be required if our Avro-server was also an Avro-client, sending requests to other services. Focusing on the simple use-case of returning responses from the "echo" and "split" messages, first the WSGI server needs to be created:

listener = eventlet.listen((HOST, PORT))
eventlet.wsgi.server(listener, wsgi_handler)

The wsgi_handler is a function which accepts the environment and start_response arguments (per the WSGI "standard"). For the actually processing of the message, you should refer to the wsgi_handler function in server.py in the example repository.

def wsgi_handler(env, start_response):
    ## Only allow POSTs, which is what Avro should be doing
    if not env['REQUEST_METHOD'] == 'POST':
        start_response('500 Error', [('Content-Type', 'text/plain')])
        return ['Invalid REQUEST_METHOD\r\n']

    ## Pull the avro rpc message off of the POST data in `wsgi.input`
    reader = avro.ipc.FramedReader(env['wsgi.input'])
    request = reader.read_framed_message()
    response = responder.respond(request)

    ## avro.ipc.FramedWriter really wants a file-like object to write out to
    ## but since we're in WSGI-land we'll write to a StringIO and then output the
    ## buffer in a "proper" WSGI manner
    out = StringIO.StringIO()
    writer = avro.ipc.FramedWriter(out)
    writer.write_framed_message(response)

    start_response('200 OK', [('Content-Type', 'avro/binary')])
    return [out.getvalue()]

The only notable quirk with using Avro with a WSGI framework like eventlet.wsgi is that some of Avro's "writer" code expects to be given a raw socket to write a response to, so we give it a StringIO object to write to and return that buffer's contents from wsgi_handler. The wsgi_handler function above is "dumb" insofar that it's simply passing the Avro request object into the "responder" which is responsible for doing the work:

class EchoResponder(avro.ipc.Responder):
    def invoke(self, message, request):
        handler = 'handle_%s' % message.name
        if not hasattr(self, handler):
            raise Exception('I can\'t handle this message! (%s)' % message.name)
        return getattr(self, handler)(message, request)

    def handle_split(self, message, request):
        query = request['query']
        halfway = len(query) / 2
        return query[:halfway]

    def handle_echo(self, message, request):
        return request['query']

All in all, minus comments the server code is around 40 lines and fairly easy to follow (refer to server.py for the complete version). I personally find Avro to be straight-forward enough and enjoyable to work with, being able to integrate it with my existing Eventlet-based stack is just icing on the cake after that.

If you're curious about some of the other work I've been up to with Eventlet, follow me on GitHub :)

Be a Libor

R. Tyler Croy — Fri, 30 Apr 2010 14:45:00 +0000

I reflect occasionally on how I've gotten to where I am right now, specifically to how I made the jump from "just some kid at a Piggly Wiggly in Texas" as Dave once said, to the guy who knows stuff about things. I often think about what pieces of the Slide engineering environment were influential to my personal growth and how I can carry those forward to build as solid an engineering organization at Apture.

The two pillars of engineering at Slide, at least in my naive world-view, were Dave and Libor. I joined Dave's team when I joined Slide, and I left Libor's team when I left Slide. Dave ran the client team, and did exceptionally well at filling a void that existed at Slide bridging engineering prowess with product management. Libor often furrowed his brow and built some of the large distributed systems that gave Slide an edge when dealing with incredible growth. In my first couple years I did my best to emulate Dave, engineers would always vie for Dave's time, asking questions and working through problems until they could return to their desk with the confidence that they understood the forces involved and solve the task at hand. Now that I'm at Apture, I'm trying to emulate Libor.

(Note: I do not intend to idolize either of them, but cite important characteristics)

To understand the Libor role, the phrase "the buck stops here" is useful. A Libor is the end of the line for engineering questions, unlike some organizations the "question-chain-of-command" is not the same as the org-chart. If a problem or question progressed up the stack to a Libor, and between an engineer and a Libor the pair cannot solve the problem, you're screwed.

What does it take to be a Libor you may be thinking: * No Guessing: When acting as a Libor, knowing is crucial. That is not to say you must understand everything about all the nooks and crannies of the code-base, but when you give an answer it is crucial you actually know what the hell you are talking about. The consequences of being wrong are far worst than the consequences of not knowing, if a fellow engineer builds on your guess, when that code ships live in a few days/weeks there is a serious risk of everything falling over.

Grok the stack: A Libor is expected to hold a wealth of information internally, much like a clock maker, a Libor should understand where every single gear and spring fit together in a large complex system. It is not necessary to understand how each component individually works but instead, understand how all the pieces operate in concert. Some amount of acting as a Libor requires direct discussions with the operations team as well as the rest of engineering, when all that JavaScript and Python rolls out to 10, 20, 100, or 1,000 machines, somebody should have at least considered the ramifications of adding 3 more database calls to every request, that's the Libor.
Maintenance and accountability: Typically working at the lower ends of the stack, a Libor has to relive and tolerate last month's and last year's short-sighted decisions over and over. A Libor should not let himself nor colleagues "fire and forget" code, poor judgement will haunt a Libor for much longer than most people's New Year's resolutions. Because of this mistake-longevity, a Libor should be quite concerned with how well thought-out and tested new changes, particularly drastic ones, are.
Focus on Engineering: Code quality and extendability are Libor's primary focus, that is not to say that a Libor's role is to impede product development, but rather ensure that it is properly framed. While a product manager's primary concern may be to get a feature deployed as soon as possible, the primary concern of a Libor is to ensure that once that feature is shipped it doesn't break or otherwise degrade the quality of service of the rest of the site. When interfacing with other engineers a Libor should be asking questions about code, intentions and implementation. Code review is as important as communication with the team, flatly rejecting code is unacceptable, but discussing with engineers the potential pitfalls of certain approaches ensures that the group moves forward.

Playing the Libor character at Apture has been interesting to say the least, I've done a lot of work getting a number of systems in place to help educate my decisions, particularly in our production environment. Focusing on the entire stack as a complex system has allowed us to make some adjustments here and there that have literally started to pay dividends the day after they ship.

Non-engineering also benefits from having a Libor character in the organization, at Apture the product development narrative has changed, I find myself emphasizing:

Tell me what you want, we'll find a way to do it

That's a breakthrough.

Pyrage: Static isn't just something on the radio

R. Tyler Croy — Fri, 26 Feb 2010 13:45:00 +0000

Dealing with statics in Python is something that has bitten me enough times that I have become quite pedantic about them when I see them. I'm sure you're thinking "But Dr. Tyler, Python is a dynamic language!", it is indeed, but that does not mean there aren't static variables.

The funny thing about static variables in Python, in my opinion, once you understand a bit about scoping and what you're dealing with, it makes far more sense. Let's take this static class variable for example:

>>> class Foo(object):
...   my_list = []
... 
>>> f = Foo()
>>> b = Foo()

You're trying to be clever, defining your class variables with their default variables outside of your __init__ function, understandable, unless you ever intend on mutating that variable.

>>> f.my_list.append('O HAI')
>>> print b.my_list
['O HAI']
>>>

Still feeling clever? If that's what you wanted, I bet you do, but if you wanted each class to have its own internal list you've inadvertantly introduced a bug where any and every time something mutates my_list, it will change for every single instance of Foo. The reason that this occurs is because my_list is tied to the class object Foo and not the instance of the Foo object (f or b). In effect f.__class__.my_list and b.__class__.my_list are the same object, in fact, the __class__ objects of both those instances is the same as well.

>>> id(f.__class__)
7680112
>>> id(b.__class__)
7680112

When using default/optional parameters for methods you can also run afoul of statics in Python, for example:

>>> def somefunc(data=[]):
...    data.append(1)
...    print ('data', data)
... 
>>> somefunc()
('data', [1])
>>> somefunc()
('data', [1, 1])
>>> somefunc()
('data', [1, 1, 1])
>>>

This comes down to a scoping issue as well, functions and methods in Python are first-class objects. In this case, you're adding the variable data to the somefunc.func_defaults tuple, which is being mutated when the function is being called. Bad programmer!

It all seems simple enough, but I still consistently see these mistakes in plenty of different Python projects (both pony-affiliated, and not). When these bugs strike they're difficult to spot, frustrating to deal with ("who the hell is changing my variable!") and most importantly, easily prevented with a little understanding of how Python scoping works.

PYRAGE!

If you want a viral license, use the GPL

R. Tyler Croy — Tue, 23 Feb 2010 14:45:00 +0000

My "roots" in the open source community come from the BSD side of the open source spectrum, my first major introduction being involvement with FreeBSD and OpenBSD. It is not surprising that my licensing preferences fall on the BSD (2 or 3 clause) or MIT licenses, the MIT license reading as follows:

Copyright (c) [year] [copyright holders]
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

I bring the subject up because I wanted to address a brief "kerfuffle" that occurred recently on the Eventlet mailing list with the maintainer of gevent, a fork/rewrite of Eventlet. Both projects are MIT licensed which gives anybody that would like to fork the source code of either project a great deal of leeway to hack about with the code, commercialize it, etc. Disclaimer: I personally am a fan of Eventlet, use it quite often, and have recently taking up maintaining Spawning, a WSGI server that supports multiple processes/threads, non-blocking I/O and graceful code reloading, built on top of Eventlet.

The "kerfuffle" occurred after Ryan, the maintainer of Eventlet, took a few good modules from gevent; shocking as it may seem, a developer working with liberally licensed code took liberally licensed code from a similar project. The issue that the maintainer of gevent took with the incorporation of his code was all about attribution:

I don't mind you borrowing code from gevent, the license permits it. However, please make it clear where are you getting the code from.

Upon first reading the email, I doubled over to the eventlet source on Bitbucket, checked the files that were incorporated into the codebase (timeout.py and queue.py)and sure enough the copyright attributing the original author were still in tact, surely this is a non-issue?

Unfortunately not, license pedantry is an open source community past-time, right up their with drinking and shouting. When I replied mentioning that the copyrights were correctly in place, mentioning that both projects were MIT licensed so both constraints of the license were met, that is, the MIT license notice was included with the code. In essence the disagreement revolves around what the phrase "this permission notice shall be included" entail, my interpretation of the license is such that the MIT license itself shall be included, not the specific file with additions from one project to another; after sending off my mail, I received the following reply:

Ok, it's acceptable to use one LICENSE file but only if the copyright notice from gevent is present unchanged.

That is, take the notice from here http://bitbucket.org/denis/gevent/src/tip/LICENSE (the line with the url), and put it into eventlet's LICENSE, on a separate line. (It's OK to add "Copyright (c) 2009-2010" before it to make it in line with others).

That would settle it.

Slightly pedantic in my opinion, the MIT license enumerates a line for copyright holders which has been hijacked for other information that the maintainer of gevent would like to propagate, I don't necessarily agree, but this is a mailing list not a court of law, so I'll allow it. The thread continues:

The license did not change. I've only updated the copyright notice to include the url of the project to protect against abusive borrowers, that's it.

This is where I draw the line, go all in, plant my flag in the sand and other unnecessary metaphors. Abusive borrowers? Analyzing the semantics of the phrase alone makes my head hurt, I have a mental image of two old ladies wherein one says to the other: "may I borrow a cup of sugar, you horse-faced hunch-backed bucket of moron?" The rest of the email is full of similarly head-hurting quotes, for brevity I won't include them here (you can read the thread in the archives).

I'm simply dumbfounded by the ignorance of what the MIT license actually means, unlike the LGPL or the GPL license which were specifically drafted to protect against "abusive borrowers", such as Cisco, the MIT license is so open it's almost public domain.

To a certain extent I can understand the emotions behind the thread on the mailing list, I don't agree with them. If you're seeking attribution past the copyright line in a header, than perhaps the "original" 4 clause BSD license is for you, or perhaps the LGPL or GPL which give you more control over what happens to the source code you originate? If this is what you're after, the MIT license is the wrong license.

Supporting Python 3 is a Ghetto

R. Tyler Croy — Sun, 21 Feb 2010 23:02:28 +0000

In my spurious free time I maintain a few Python modules (py-yajl, Cheetah, PyECC) and am semi-involved in a couple others (Django, Eventlet), only one of which properly supports Python 3. For the uninitiated, Python 3 is a backwards incompatible progression of the Python language and CPython implementation thereof, it's represented significant challenges for the Python community insofar that supporting Python 2.xx, which is in wide deployment, and Python 3.xx simultaneously is difficult.

As it stands now my primary development environment is Python 2.6 on Linux/amd64, which means I get to take advantage of some of the nice things that were added to Python 3 and then back-ported to Python 2.6/2.7. Regular readers know about my undying love for Hudson, a Java-based continuous integration server, which I use to test and build all of the Python projects that I work on. While working this weekend I noticed that one of my C-based projects (py-yajl) was failing to link properly on Python 2.4 and 2.5. It might be easy to cut-off support for Python 2.4, which was first released over four years ago, there are still a number of heavy users of 2.4 (such as Slide), in fact it's still the default /usr/bin/python on Red Hat Enterprise Linux 5. What makes this C-based module special, is that thanks to Travis, it runs properly on Python 3.1 as well. Since the Python C-API has been fairly stable through the 2 series into Python 3, maintaining a C-based module that supports multiple versions of Python.

In this case, it's as easy as some simple pre-processor definitions:

#if PY_MAJOR_VERSION >= 3
#define IS_PYTHON3
#endif

Which I can use further down the line to modify the handling some of the minor internal changes for Python 3:

#ifdef IS_PYTHON3
    result = _internal_decode((_YajlDecoder *)decoder, PyBytes_AsString(bufferstring),
                PyBytes_Size(bufferstring));
    Py_XDECREF(bufferstring);
#else
    result = _internal_decode((_YajlDecoder *)decoder, PyString_AsString(buffer),
                  PyString_Size(buffer));
#endif

Not particularly pretty but it gets the job done, supporting all major versions of Python.

Python on Python

Writing modules in C is fun, can give you pretty good performance, but is not something you would want to do with a large package like Django (for example). Python is the language we all know and love to work with, a much more pleasant language to work with than C. If you build packages in pure Python, those packages have a much better chance running on top of IronPython or Jython, and the entire Python ecosystem is better for it.

A few weeks ago when I started to look deeper into the possibility of Cheetah support for Python 3, I found a process riddled with faults. First a disclaimer, Cheetah is almost ten years old; it's one of the oldest Python projects I can think of that's still chugging along. This translates into some very old looking code, most people who are new to the language aren't familiar with some of the ways the language has changed in the past five years, let alone ten.

The current means of supporting Python 3 with pure Python packages is as follows:

Refactor the code enough such that 2to3 can process it
Run 2to3 over the codebase, with the -w option to literally write the changes to the files
Test your code on Python 3 (if it fails, go back to step 1)
Create a source tarball, post to PyPI, continue developing in Python 2.xx

I'm hoping you spotted the same problem with this model that I did, due to the reliance on 2to3 you are now trapped into always developing Python targeting Python 2. This model will never succeed in moving people to Python 3, regardless of what amazing improvements it contains (such as the Unladen Swallow work) because you cannot develop on a day-to-day basis with Python 3, it's a magic conversion tool away.

Unlike with a C module for Python, I cannot #ifdef certain segments of code in and out, which forces me to constantly use 2to3 or fork my code and maintain two separate branches of my project, duplicating the work for every change. With Python 2 sticking around on the scene for years to come (I don;t believe 2.7 will be the last release) I cannot imagine either of these workflows making sense long term.

At a fundamental level, supporting Python 3 does not make sense for anybody developing modules, particularly open source ones. Despite Python 3 being "the future", it is currently impossible to develop using Python 3, maintaining support for Python 2, which all of us have to do. With enterprise operating systems like Red Hat or SuSE only now starting to get on board with Python 2.5 and Python 2.6, you can be certain that we're more than five years away from seeing Python 3 installed by default on any production machines.

Building a game for ET. Day 1 with Pygame

R. Tyler Croy — Sun, 14 Feb 2010 09:35:24 +0000

Earlier this week I was checking out Pygame, pondering what I could possibly build with it that could keep me motivated enough to finish it. Motivation would like be the primary problem for me with any amount of game programming; I'm not a gamer, I don't harbor a dislike of games, they're just not something I typically spend time playing (I do like to play "haggard late night open-source hacker" though, that's a fun one). Friday night I stumbled across an idea, ET likes to play (casual) games, perhaps we could write a game together; ask any engineer at EA or Ubisoft, there's nothing more romantic than working on a game.

Talking over the idea with ET on the ride home from the office, we talked about creating a typing-oriented game and started to brainstorm. The tricky aspect of a typing-oriented game is you have to walk the fine line of "educational gaming", that is to say, the game's goal is not to teach the player how to type. That sucks. Contrasted to some other games where the means of progressing in some games is by solving a puzzle, killing noobs in others, in this game we wanted the player to progress through levels/situations with their typing ability (ET finds this fun, we do not have this in common).

Over pizza we discussed more about how the levels would work, I decided that I wanted to use stories/articles instead of random words for the "content" of the game. We settled on a couple fundamental concepts: the player would earn coins by correctly completing a words as the scrolled from right to left (similar to a ticker tape), they would lose coins if they made a mistake or could not keep up. After a player completed a level (i.e. a "story") they would find themselves in a "store" of sorts, where they could purchase "tools" for future levels with their coins. The tools we decided would be a very important, as the player reached their upper bound of typing speed the utility of these various tools would necessary as a means of strategically conquering the level. One of the few things we didn't particularly cover was the "end game", whether the player would simply play increasingly more difficult levels (a la Tetris) or if they could actually "beat" the game. With at least the basics of the concept sketched out, it was time to start writing some code.

Starting with Pygame

It's incredibly important to mention that I've never programmed a game before. Never-ever. From my work with network programming I was already familiar with the concept of the run-loop that's pretty core to Pygame, but I had never really made use of any to animate objects on the screen or deal with handling any kind of events from mouse movements to key presses, etc. Fortunately I'm already a professional Python developer, so writing code wasn't the difficult part so much as laying it out. Orienting things into classes to handle separate components such as animating text (which is a painful in Pygame, in my opinion) to keeping track of user-input.

Animating text across the screen wasn't particularly difficult, with Pygame you first create your primary "surface" (i.e. the window) and then you can render things onto that surface. With text, you end up rendering a surface which contains your text, "hello world" which you then place onto the primary surface. Easy peasy thus far:

    import pygame
    surface = pygame.display.set_mode((640, 680), pygame.HWSURFACE | pygame.DOUBLEBUF)
    font = pygame.font.SysFont('Courier', 42)
    ### render(text, antialias, rgb color tuple
    font_surface = font.render('hello world', 0, (0, 0, 0)) 
    ### draw `font_surface` onto `surface` at (x=0, y=0)
    start_x, start_y = 0, 0
    surface.blit(font_surface, (start_x, start_y))
    while True:
         ### Holy runloop batman
         pygame.display.update()

That was fun, I now have "hello world" rendered onto my screen, now to animate I suppose I'll just render font_surface a little further right every iteration of the runloop, i.e.

    while True:
         ### Holy runloop batman
         surface.blit(font_surface, (start_x, start_y))
         start_x += 0.5
         pygame.display.update()

This blurs the text however, so I then changed to:

    while True:
         ### Holy runloop batman
         surface.blit(font_surface, (start_x, start_y))
         surface.fill((0, 0, 0))
         start_x += 0.5
         pygame.display.update()

This will cause the (primary) surface to be repainted (washed over) every iteration of the runloop ensuring that the text will properly animate, drawing the text in one spot, wiping the surface then drawing it slightly further to the left, resulting in the scrolling animation. All's fine and good until you determine that you want to have other elements on the screen and you also don't want to redraw them every time around the carousel. I then discovered how to "fill" just one particular rectangle on the surface, i.e. the rectangle behind the text:

    text_w, text_h = font.size('hello world')
    while True:
         ### Holy runloop batman
         surface.blit(font_surface, (start_x, start_y))
         surface.fill((0, 0, 0), rect=pygame.Rect(start_x, start_y, text_w, text_h))
         start_x += 0.5
         pygame.display.update()

Once I was able to get text properly scrolling across the screen, the rest of the afternoon of hacking was far easier. My confidence in my ability to grok Pygame in order to do what I wanted. I then set forth organizing my code into some logical classes, for example I created a LetterSpool class which would record the user's progress through the current word, rendering it at the bottom-center of the screen and firing an event when the user hit the space bar (denoting the word "complete"), additionally I wrapped my text animation code into AnimatedWord so I could easily string words together to scroll across the screen in conjunction similar to a textual screensaver.

Not a whole lot more to write about with regards to my progress today, hooked up some music and basic sound effects which was trivial after looking at some sample code. Next I need to start addressing some more fundamentals for user-interaction: scoring and level-changing.

You can track the progress of the game "Typy" (pronounced: typey) on GitHub

Thread-safety assumptions in Django

R. Tyler Croy — Tue, 19 Jan 2010 05:23:26 +0000

These days, the majority of my day job revolves around working with Apture's Django-based code which, depending on the situation, can be a blessing or a curse. In some of my recent work to help improve our ability to scale effectively, I started swapping out Apache for Spawning web servers which can more efficiently handle large numbers of concurrent requests. One of the mechanisms by which Spawning accomplishes this task, is by using eventlet's tpool (thread pool) module in addition to some other clever tricks. With Apache, we used pre-forked workers to accomplish the work needed to be done and while still using forked child processes with Spawning, threading was also thrown into the mix, that's when "shit got real" (so to speak).

We started seeing sporadic, difficult to reproduce errors. Not a lot, a trickle of exception emails throughout the day. Digging deeper into some of the exceptions, careful stepping through Apture code, into Django code and back again, I started to realize I had thread-safety problems. Shock! Panic! Despair! Lunch! Disappointment! Shock! I felt all these things and more. I've long lamented the number of globals used in Django's code base but this is the icing on the cake.

Apparently Django's threading problems are sufficiently documented in a few places. Using a slightly older version of the Django framework certainly doesn't help but it doesn't appear that recent releases (1.1.1) can guarantee thread-safety anyways. I think it's safe to assume the majority of Django framework users are not using threaded web servers in any capacity, else this would have become a far larger issue (and hopefully of been fixed) by now. From NoReverseMatch exceptions, to curious middleware problems to thread-safety issues in the WSGI support layer, Django has potholes lying all along the road to multithreadedness.

Beware.

Virtual Hosting with HAProxy and WSGI

R. Tyler Croy — Sun, 17 Jan 2010 00:29:38 +0000

Lately I've fallen in love with a couple of fairly simple but powerful technologies: haproxy and WSGI (web server gateway interface). While the latter is more of a specification (PEP 333) the concepts it puts forth have made my life significantly easier. In combination, the two of them make for a powerful combination for serving web applications of all kinds and colors.

HAProxy is a robust, reliable piece of load balancing software that's very easy to get started with, For the uninitiated, load balancing is a common means of distributing the load of a number of inbound requests across a pool of processes, machines, clusters and so on. Whenever you hit any web site of non-trivial size, your HTTP requests are invariably transparently proxied through a load balancer to a pool of web machines.

I started looking into haproxy when I began to move Urlenco.de away from my franken-setup of Lighttpd/FastCGI/Mono/ASP.NET to a pure Python stack. After poking around some articles about haproxy I discovered it can be used for virtual hosts as well as simple load balancing. Using a haproxy's ACLs feature (see Section 7 in the configuration.txt), you can redirect requests to one backend or another. While my "virtual hosting" with haproxy is using the ability to inspect the HTTP headers of inbound requests, you can use a number of different criterion to determine the right backend for serving a request: url matching, request method matching (GET/POST), protocol matching (haproxy can load balance any kind of TCP connection) and so on.

WSGI (pronounced: whiskey) comes into play on the backend side of haproxy, using the eventlet.wsgi module which provides a WSGI interface I can build web applications very quickly, test them and deploy them. When deployed, I can run them as "nobody" in userspace on the server, binding to some higher numbered port (i.e. 8080) and haproxy will do the work routing to the appropriate WSGI process.

Below is a simple haproxy configuration that I'm using to run Urlenco.de and a site for my wedding and many more as soon as I finish them. The section to note is frontend http-in in which the ACLs are defined for the different virtually hosted domains and the conditionals for selecting a backend based on those ACLs.

global
    maxconn         20000
    ulimit-n        16384
    log             127.0.0.1 local0
    uid             200
    gid             200
    chroot          /var/empty
    nbproc          4
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    retries 3
    option redispatch
    maxconn 2000
    contimeout 5000
    clitimeout 50000
    srvtimeout 50000

frontend http-in
    bind *:80
    acl is_urlencode hdr_end(host) -i urlenco.de
    acl is_wedding hdr_end(host) -i erinandtylerswedding.com

    use_backend urlencode if is_urlencode
    use_backend wedding if is_wedding
    default_backend urlencode

backend urlencode
    balance roundrobin
    cookie SERVERID insert nocache indirect
    option httpchk HEAD /check.txt HTTP/1.0
    option httpclose
    option forwardfor
    server Local 127.0.0.1:8181 cookie Local

backend wedding
    balance roundrobin
    cookie SERVERID insert nocache indirect
    option httpchk HEAD /check.txt HTTP/1.0
    option httpclose
    option forwardfor
    server Local 127.0.0.1:8081 cookie Local

New Year's Python Meme

R. Tyler Croy — Mon, 04 Jan 2010 09:28:36 +0000

While I'm not aggregated into the Python Planet I wanted to join in the meme that's already going on.

What’s the coolest Python application, framework or library you have discovered in 2009?

While I didn't discover it until the latter half of 2009, I'd have to say eventlet is the coolest Python library I discovered in 2009. After leaving Slide, where I learned the joys of coroutines (a concept previously foreign to me) I briefly contemplated using greenlet to write a coroutines library similar to what is used at Slide. Fortunately I stumbled across eventlet in time, which shares common ancestry with Slide's proprietary library.

What new programming technique did you learn in 2009?

I'm not sure I really learned any new techniques over the past year, I started writing a lot more tests this past year but my habits don't quite qualify as Test Driven Development just yet. As far as Python goes, I've been introduced to the Python C API over the past year (written two entire modules in C PyECC and py-yajl) and while I wouldn't exactly call implementing Python modules in C a "technique" it's certainly a departure from regular Python (Py_XDECREF I'm looking at you)

What’s the name of the open source project you contributed the most in 2009? What did you do?

Regular readers of my blog can likely guess which open source project I contributed to most in 2009, Cheetah, of which I've become the maintainer. I also authored a number of new Python projects in 2009: PyECC a module implementing Elliptical Curve Cryptography (built on top of seccure), py-yajl a module utilizing yajl for fast JSON encoding/decoding, IronWatin an IronPython-based module for writing WatiN tests in Python (supporting screengrabs as well), PILServ an eventlet-based server to do server-side image transformations with PIL, TweepyDeck a PyGTK+ based Twitter client and MicroMVC a teeny-tiny MVC styled framework for Python and WSGI built on eventlet and Cheetah.

What was the Python blog or website you read the most in 2009?

The Python reddit was probably the most read Python-related "blog" I read in 2009, it generally supercedes the Python Planet with regards to content but also includes discussions as well as package release posts.

What are the top three things you want to learn in 2010?

Python 3. After spending a couple weekends trying to get Cheetah into good working order on Python 3, I must say, maintaining a Python-based module on both Python 2.xx and 3.xx really feels like a nightmare. py-yajl on the otherhand, being entirely C, was trivial to get compiling and executing perfectly for 2.xx and 3.xx
NoSQL. Earlier this very evening I dumped a boatload of data out of PostgreSQL into Redis and the resulting Python code for data access using redis-py is shockingly simple. I'm looking forward to finding more places where a relational database is overkill for certain types of stored data, and using Redis instead.
Optimizing Python. With py-yajl Lloyd and I had some fun optimizing the C code behind the module, but I'd love to learn some handy tricks to making pure-Python execute as fast as possible.

Using Cheetah templates with Django

R. Tyler Croy — Sat, 26 Dec 2009 20:31:11 +0000

Some time ago after reading a post on Eric Florenzano's blog about hacking together support for Cheetah with Django, I decided to add "proper" support for Cheetah/Django to Cheetah v2.2.1 (released June 1st, 2009). At the time I didn't use Django for anything, so I didn't really think about it too much more.

Now that I work at Apture, which uses Django as part of its stack, Cheetah and Django playing nicely together is more attractive to me and as such I wanted to jot down a quick example project for others to use for getting started with Cheetah and Django. You can find the django_cheetah_example project on GitHub, but the gist of how this works is as follows.

Requires

Django
Cheetah (>= v2.2.1)

Getting Started

For all intents and purposes, using Cheetah in place of Django's templating system is a trivial change in how you write your views.

After following the Django getting started documentation, you'll want to create a directory for your Cheetah templates, such as Cheetar/templates. Be sure to touch __init__.py in your template directory to ensure that templates can be imported if they need to.

Add your new template directory to the TEMPLATE_DIRS attribute in your project's settings.py.

Once that is all set up, utilizing Cheetah templates in Django is just a matter of a few lines in your view code:

    import Cheetah.Django
 
    def index(req):
        return Cheetah.Django.render('index.tmpl', greet=False)

Note: Any keyword-arguments you pass into the Cheetah.Django.render() function will be exposed in the template's "searchList", meaning you can then access them with $-placeholders. (i.e. $greet)

With the current release of Cheetah (v2.4.1), there isn't support for using pre-compiled Cheetah templates with Django (it'd be trivial to put together though) which means Cheetah.Django.render() uses Cheetah's dynamic compilation mode which can add a bit of overhead since templates are compiled at runtime (your mileage may vary).

Pyrage: from toolbox import hammer

R. Tyler Croy — Thu, 24 Dec 2009 08:23:26 +0000

Those that have worked with my directly know I'm a tad obsessive when it comes to imports in Python. Once upon a time I had to write some pretty disgusting import hooks to solve a problem and got to learn first-hand how gnarly Python's import subsystem can be. I have a couple coding conventions that I follow when I'm writing Python for my own personal projects that typically follows:

"strict" system imports first (i.e. import time)
"from" system imports second (i.e. from eventlet import api)
"local" imports (import mymodule)
local "from" imports (from mypackage import module)

In all of these sections, I like to list things alphabetically as well, just to make sure that at no point are modules ever doubley-imported. This results in code that looks clean (in my humblest of opinions):

    #!/usr/bin/env python
    import os
    import sys
    from eventlet import api
 
    import app.util
    from app.models import account
 
    ## Etc.

A module importing habit that absolutely drives me up the wall, I was introduced to and told "don't-do-that" by Dave: importing symbols from modules; in effect: from MySQLdb import IntegrityError. I have two major reasons for hating the importing of symbols, the first one is that it messes with your module's namespace. If the symbol import above were in a file called "foo.py", the foo module would then have the member foo.IntegrityError. Additionally, it makes the code more difficult to understand when you flatten the module's namespace out; 500 lines down in the file if you see acct_m = AccountManager() as a developer new to the file you'll have to go up to the top and figure out where the hell AccountManager is actually coming from to understand how it works.

As code with these sort of symbol-level imports ages, it becomes more and more frustrating to deal with, if I need OperationalError in my module now I have three options:

Update the line to say: from MySQLdb import IntegrityError, OperationalError
Add import MySQLdb and just refer to IntegrityError and MySQLdb.OperationalError
Add import MySQLdb and update all references to IntegrityError

I've seen code in open source projects that have abused the symbol imports so badly that an import statement look like: from mod import CONST1, CONST2, CONST3, SomeError, AnotherClass (ad infinium).

I think poor import style is a good indicator of how one can expect the rest of the Python code to look, I cannot recall a single instance where I've looked at a Python module with gross import statements and clean classes and functions.

    from MySQLdb import IntegrityError, OperationalError, MySQLError, ProgrammingError, \
    NotSupportedError, InternalError

PYRAGE!

One year of Cheetah

R. Tyler Croy — Sun, 20 Dec 2009 01:04:49 +0000

While working at Slide I had a tendency to self-assign major projects, not content with things being "good-enough" I tended to push and over-extend myself to improve the state of Slide Engineering. Sometimes these projects would fail and I would get uncomfortably close to burning myself out, other times, such as the migration from Subversion to Git, turned out to be incredibly rewarding and netted noticable improvements in our workflow as a company.

One of my very first major projects was upgrading our installation of Cheetah from 1.0 to 2.0, at the time I vigorously hated Cheetah. My distain of the templating system stemmed from using a three year old version (that sucked to begin with) and our usage of Cheetah which bordered between "hackish" and "vomitable." At this point in Slide's history, the growth of the Facebook applications meant there was going to be far less focus on the Slide.com codebase which is where some of the more egregious Cheetah code lived; worth noting that I never "officially" worked on the Slide.com codebase. When I successfully convinced Jeremiah and KB that it was worth my time and some of their time to upgrade to Cheetah 2.0 which offered a number of improvements that we could make use of, I still held some pretty vigorous hatred towards Cheetah. My attitude was simple though, temporary pain on my part would alleviate pain inflicted on the rest of the engineering team further down the line. Thanks to fantastic QA by Ruben and Sunil, the Cheetah upgrade went down relatively issue free, things were looking fine in production and everybody went back to their regularly scheduled work.

Months went by without me thinking of Cheetah too much until late 2008, Slide continued to write front-end code using Cheetah and developers continued to grumble about it. Frustrated by the lack of development on the project, I did the unthinkable, I started fixing it. Over the Christmas break, I used git-cvsimport(1) to create a git repository from the Cheetah CVS repo hosted with SourceForge and I started applying patches that had circulated on the mailing list. By mid-March I had a number of changes and improvements in my fork of Cheetah and I released "Community Cheetah". Without project administrator privileges on SourceForge, I didn't have much of a choice but to publish a fork on GitHub. Eventually I was able to get a hold of Tavis Rudd, the original author of Cheetah who had no problem allowing me to become the maintainer of Cheetah proper, in a matter of months I had gone from hating Cheetah to fulfilling the oft touted saying "it's open source, fix it!" What was I thinking.

Thanks in part to git and GitHub's collaborative/distributed development model patches started to come in and the Cheetah community for all intents and purposes "woke up." Over the course of the past year, Cheetah has seen an amazing number of improvements, bugfixes and releases. Cheetah now properly supports unicode throughout the system, supports @staticmethod and @classmethod decorators, supports use with Django and now supports Windows as a "first-class citizen". While I committed the majority of the fixes to Cheetah, five other developers contributed fixes:

Jean-Baptiste Quenot (unicode fixes)
Mike Bonnet (unicode fixes, test fixes)
James Abbatiello (Windows support)
Arun Kumar
Doug Knight (fixes for #raw directive)

In 2008, Cheetah saw 7 commits and 0 releases, while 2009 brought 342 commits and 10 releases; something I'm particularly proud of. Unforunately since I've left Slide, I no longer use Cheetah in a professional context but I still find it tremendously useful for some of my personal projects.

I am looking forward to what 2010 will bring for the Cheetah project, which started in mid-2001 and has seen continued development since thanks to a number of contributors over the years.

Pyrage: Generic Exceptions

R. Tyler Croy — Fri, 18 Dec 2009 06:10:13 +0000

Earlier while talking to Ryan I decided I'd try to coin the term "pyrage" referring to some frustrations I was having with some Python packages. The notion of "pyrage" can extend to anything from a constant irritation to a pure "WTF were you thinking!" kind of moment.

Not one to pass up a good opportunity to bitch publicly, I'll elaborate on some of my favorite sources of "pyrage", starting with generic exceptions. While at Slide, one of the better practices I picked up from Dave was the use of specifically typed exceptions to specific errors. In effect:

    class Connection(object):
        ## Pretend this object has "stuff"
        pass
 
    class InvalidConnectionError(Exception):
        pass
    class ConnectionConfigurationError(Exception):
        pass
 
    def configureConnection(conn):
        if not isinstance(conn, Connection):
            raise InvalidConnectionError('configureConnection requires a Connection object')
        if conn.connected:
            raise ConnectionConfigurationError('Connection (%s) is already connected' % conn)
        ## etc

Django, for example, is pretty stacked with generic exceptions, using builtin exceptions like ValueError and AttributeError for a myriad of different kinds of exceptions. urllib2's HTTPError is good example as well, overloading a large number of HTTP errors into one exception leaving a developer to catch them all, and check the code, a la:

    try:
        urllib2.urlopen('http://some/url')
    except urllib2.HTTPError, e:
        if e.code == 503:
            ## Handle 503's special
            pass
        else:
            raise

Argh. pyrage!

Python/JSON Eat/Drink-up in San Francisco

R. Tyler Croy — Wed, 09 Dec 2009 08:36:31 +0000

A few weeks ago when I started working more on py-yajl I discovered that there are actually a number of Python developers who work(ed) with JSON parsers in the Bay Area. As luck would have it Lloyd, the author of Yajl, is going to be in town next weekend for Add-on-Con, time to meet up and have some beers! I've invited the authors of: simplejson, jsonlib, jsonlib2 and a few other Python hackers in the Bay Area that I know of.

If you're in San Francisco this Saturday (Dec. 12th) and loves you some Python, don't hesitate to swing by 21st Amendment around 1pm-ish to join us!

Server-side image transforms in Python

R. Tyler Croy — Sat, 05 Dec 2009 06:51:33 +0000

While working at Slide, I became enamored with the concept of cooperative threads (coroutines) and the in-house library built around greenlet to implement coroutines for Python. As an engineer on the "server team" I had the joy of working in a coro-environment on a daily basis but now that I'm "out" I've had to find an alternative library to give me coroutines: eventlet. Interestingly enough, eventlet shares common ancestry with Slide's internal coroutine implementation like two different species separated thousands of years ago by continental drift (a story for another day).

A few weekends ago, I had a coroutine itch to scratch one afternoon: an eventlet-based image server for applying transforms/filters/etc. After playing around for a couple hours "PILServ" started to come together. One of the key features I wanted to have in my little image server project was the ability to not only pass the server a URL of an image instead of a local path but also to "chain" transforms in a jQuery-esque style. Using segments of the URL as arguments, a user can arbitrarily chain arguments into PILServ, i.e.:

http://localhost:8080/flip/filter(blur)/rotate(45)/resize(64x64)/<url to an image>

At the end of the evening I spent on PILServ, I had something going that likely shows off more of the skills of PIL rather than eventlet itself but I still think it's neat. Below is a sample of some images transformed by PILServ running locally:

On GitHub and how I came to write the fastest Python JSON module in town

R. Tyler Croy — Fri, 04 Dec 2009 09:30:09 +0000

Perhaps the title is a bit too much ego stroking, yes, I did write the fastest Python module for decoding JSON strings and encoding Python objects to JSON. I didn't however write the parser behind the scenes.

Over the summer I discovered "Yet Another JSON Library" on GitHub, written by Lloyd Hilaiel, jonesing for a Saturday afternoon project I started the "py-yajl" project to see if I could implement a Python C module atop Lloyd's marvelous parsing library. After tinkering with the project for a while I got a working prototype building (learning how to define custom types in Python along the way) and let the project stagnate as my weekend ended and the workweek resumed.

A little over a week ago "autodata", another GitHub user, sent me a "Pull Request" with some minor changes to make py-yajl build cleaner on amd64; my interest in the project was suddenly reignited, amazing what a little interest can do for motivation. Over the 10 days following autodata's pull request I discovered that a former colleague of mine and fellow GitHub user "teepark" had forked the project as well, working on Python 3 support. Going from zero to two people interested in the project, I quickly converted the code from a stagnant, borderline embarrassing, dump of C code into a leak-free, swift JSON library for Python. Not one to miss out on the fun, I pinged Lloyd who quickly became as enamored with making py-yajl the best Python JSON module available, he forked the project and almost immediately sent a number of pull requests my way with further optimizations to py-yajl such as:

Swapping out the use of Python lists to a custom pointer stack for maintaining internal state
Accelerating parsing and handling of Number objects
Pruning a few memory leaks here and there

Thanks to mikeal's JSON post and jsonperf.py script, Lloyd and I could both see how py-yajl was stacking up against cjson, jsonlib, jsonlib2 and simplejson; things got competitive. Below are the most recent jsonperf.py results with py-yajl v0.1.1:

json.loads:         6470.22037ms
simplejson.loads:   202.21063ms  
yajl.loads:         145.32621ms
cjson.decode:       102.44788ms

json.dumps:         2309.15286ms
cjson.encode:       276.49586ms   
simplejson.dumps:   201.59785ms
yajl.dumps:         161.00153ms

Over the coming days or weeks (as time permits) I'm planning on adding JSON stream parsing support, i.e. parsing a stream of data as it's coming in off a socket or file object, as well as a few other miscellaneous tasks.

Given the nature of GitHub's social coding dynamic, py-yajl got off the ground as a project but Yajl itself gained an IRC channel (#yajl on Freenode) and a mailing list ([email protected]). To date I have over 20 unique repositories on GitHub (i.e. authored by me) but the experience around Yajl has been the most exciting and finally proved the "social coding" concept beneficial to me.