unethical blogger - Hudson

Sometimes Software as a Service Sucks

R. Tyler Croy — Tue, 23 Mar 2010 14:00:00 +0000

Being a big fan of "continuous integration", particularly with Hudson, I've often thought about the possibilities of turning it into a business. It's no surprise really, my first commercial application as a rogue Mac software developer was a product called BuildFactory which, while fun to build, never sold all that many licenses. With the advent of Amazon's EC2 service and the transition of these cloud computing resources into a building block for many businesses, I've long thought about the idea of building "continuous integration as a service."

At face value the idea sounds incredibly fun to build, I'll build a service that integrates with GitHub, Google Code, SourceForge and private source control systems. The end (paying) user would "plug-in" to the "continuous integration grid", they'd work throughout the day, committing code and then the CI grid would pick up those changes, build releases and run tests against a number of different architecture, automatically detecting failures and reporting them back to the developers. It involves some of my favorite challenges in programming:

Scaling up
Efficiently using cycles, and only when needed
Building and testing cross-architecture and cross-platform

Unfortunately, it's a crap business idea, I now have second-hand confirmation from a group of guys who've attempted the concept. The folks behind RunCodeRun are shutting down the service. In the post outlining why they're shutting down, they've hit the nail on the head on why "continuous integration as a service" can never work:

Large scale hosted continuous integration is consumed as a commodity but built as a craft, and the rewards, both emotional and financial, are insufficient to support the effort.

Elaborating further on their point, continuous integration by itself is a relatively basic task: build, test, repeat. The biggest problem with continuous integration as a service however, is that no two projects are alike. My build targets or requirements might be vastly different from project to project, let alone customer to customer, making the amount of tweaking and customization per-job too large such that at some point the only benefit that one derives from such a service is the hosting of the machines to perform the task. If you're just taking care of that, why wouldn't your customers just use Hudson in "the cloud" themselves? The CI grid at that point offers no exceptional value.

As much as I regret letting a fun idea die, I think I'll have to file this one under "To do after becoming so rich I'll care about capital gains taxes."

Mourning Sun

R. Tyler Croy — Sun, 31 Jan 2010 03:51:52 +0000

Some users of Hudson have already started to notice a subtle addition to the latest release, 1.343, a new background watermark image.

The commit message (r26728) from Kohsuke, the incredibly talented founder and maintainer of the Hudson project, adds a bit of sadness to the whole affair:

In tribute to Sun Microsystems and all my colleagues who had to go today. I hope the community would forgive me for doing this.

Given the incredible speed at which the tech industry grows and moves, it's easy to forget that there are a number of talented engineers that have spent their careers at Sun building technologies that have helped change the face of modern computing, regardless of whether or not Sun could figure out how to sell them: SunOS/Solaris, Java, DTrace, SPARC 64-bit chips, Sun Grid Engine, JRuby, the W3C XML specification, ZFS, OpenOffice (acquisition), MySQL (acquisition), and VirtualBox (acquisition).

As a corporation, I personally think Sun was a failure, as a foundation of engineering in Silicon Valley, I think Sun has been quite successful.

To those that are being pushed out as part of the merger with Oracle, I want to sincerely thank you for your contributions to computing and wish you the best of luck. Here's the "full" version of the image, which I found via @jtnl's TwitPic stream:

Pre-tested commits with Hudson and Git

R. Tyler Croy — Thu, 31 Dec 2009 23:22:16 +0000

A few months ago Kohsuke, author of the Hudson continuous integration server, introduced me to the concept of the "pre-tested commit", a feature of the TeamCity build management and continuous integration system. The concept is simple, the build system stands as a roadblock between your commit entering trunk and only after the build system determines that your commit doesn't break things does it allow the commit to be introduced into version control, where other developers will sync and integrate that change into their local working copies. The reasoning and workflow put forth by TeamCity for "pre-tested commits" is very dependent on a centralized version control system, it is solving an issue Git or Mercurial users don't really run into. Those using Git can commit their hearts out all day long and it won't affect their colleagues until they merge their commits with others.

In some cases, allowing buggy or broken code to be merged in from another developer's Git repository can be worse than in a central version control system, since the recipient of the broken code might perform a knee-jerk git-revert(1) command on the merge! When you revert a merge commit in Git, what happens is you not only revert the merge, you revert the commits associated with that merge commit; in essence, you're reverting everything you just merged in when you likely just wanted to get the broken code out of your local tree so you could continue working without interruption. To solve for this problem-case, I utilize a "pre-tested commit" or "pre-tested merge" workflow with Hudson.

My workflow with Hudson for pre-tested commits involves three separate Git repositories: my local repo (local), the canonical/central repo (origin) and my "world-readable" (inside the firewall) repo (public). For pre-tested commits, I utilize a constantly changing branch called "pu" (potential updates) on the world-readable repo. Inside of Hudson I created a job that polls the world-readable repo (public) for changes in the "pu" branch and will kick off builds when updates are pushed. Since the content of public/pu is constantly changing, the git-push(1) commands to it must be "forced-updates" since I am effectively rewriting history every time I push to public/pu.

To help forcefully pushing updates from my current local branch to public/pu I use the following git alias:

% git config alias.pup "\!f() { branch=\$(git symbolic-ref HEAD | sed 's/refs\\/heads\\///g');\
      git push -f \$1 +\${branch}:pu;}; f"

While a little obfuscated, thie pup alias forcefully pushes the contents of the current branch to the specified remote repository's pu branch. I find this is easier than constantly typing out: git push -f public +topic:pu

In list form, my workflow for taking a change from inception to origin is:

hack, hack, hack
commit to local/topic
git pup public
Hudson polls public/pu
Hudson runs potential-updates job
Tests fail?
- Yes: Rework commit, try again
- No: Continue
Rebase onto local/master
Push to origin/master

Using this pre-tested commit workflow I can offload the majority of my testing requirements to the build system's cluster of machines instead of running them locally, meaning I can spend the majority of my time writing code instead of waiting for tests to complete on my own machine in between coding iterations.

IronWatin; mind the gap

R. Tyler Croy — Tue, 13 Oct 2009 21:57:49 +0000

Last week @admc, despite being a big proponent of Windmill, needed to use WatiN for a change. WatiN has the distinct capability of being able to work with Internet Explorer's HTTPS support as well as frames, a requirement for the task at hand. As adorable as it was to watch @admc, a child of the dynamic language revolution, struggle with writing in C# with Visual Studio and the daunting "Windows development stack," the prospect of a language shift at Slide towards C# on Windows is almost laughable. Since Slide is a Python shop, IronPython became the obvious choice.

Out of an hour or so of "extreme programming" which mostly entailed Adam watching as I wrote IronPython in his Windows VM, IronWatin was born. IronWatin itself is a very simple test runner that hooks into Python's "unittest" for creating integration tests with WatiN in a familiar environment.

I intended IronWatin to be as easy as possible for "native Python" developers, by abstracting out updates to sys.path to include the Python standard lib (adds the standard locations for Python 2.5/2.6 on Windows) as well as adding WatiN.Core.dll via clr.AddReference() so developers can simply import IronWatin; import WatiN.Core and they're ready to start writing integration tests. When using IronWatin, you create test classes that subclass from IronWatin.BrowserTest which takes care of setting up a browser (WatiN.Core.IE/WatiN.Core.FireFox) instance to a specified URL, this leaves your runTest() method to actually execute the core of your test case.

Another "feature"/design choice with IronWatin, was to implement a main() method specifically for running the tests on a per-file basis (similar to unittest.main()). This main method allows for passing in an optparse.OptionParser instance to add arguments to the script such as "--server" which are passed into your test classes themselves and exposed as "self.server" (for example). Which leaves you with a fairly straight-forward framework with which to start writing tests for the browser itself:

#!/usr/bin/env ipy
 
# The import of IronWatin will add a reference to WatiN.Core.dll
# and update `sys.path` to include C:\Python25\Lib and C:\Python26\Lib
# so you can import from the Python standard library
import IronWatin
 
import WatiN.Core as Watin
import optparse
 
class OptionTest(IronWatin.BrowserTest):
    url = 'http://www.github.com'
 
    def runTest(self):
        # Run some Watin commands
        assert self.testval
 
if __name__ == '__main__':
    opts = optparse.OptionParser()
    opts.add_option('--testval', dest='testval', help='Specify a value')
    IronWatin.main(options=opts)

Thanks to IronPython, we can make use of our developers' and QA engineers' Python knowledge to get the up and running with writing integration tests using WatiN rapidly instead of trying to overcome the hump of teaching/training with a new language.

Deployment Notes: We're using IronPython 2.6rc1 and building WatiN from trunk in order to take advantage of some recent advances in their Firefox/frame support. We've not tested IronWatin, or WatiN at all for that matter, anywhere other than Windows XP.

Doing more with less; very continuous integration

R. Tyler Croy — Wed, 02 Sep 2009 08:42:02 +0000

Once upon a time I was lucky enough to take an "Intro to C++" class taught by none other than Bjarne Stroustrop himself, while I learned a lot of things about what makes C++ good and sucky at the same time, he also taught a very important lesson: great engineers are lazy. It's fairly easy to enumerate functionality in tens of hundreds of lines of poorly organized, inefficient code, but (according to Bjarne) it's the great engineers that are capable of distilling that functionality into it's most succinct form. I've since taken this notion of being "ultimately lazy" into my professional career, making it the root answer for a lot of my design decisions and choices: "Why bother writing unit tests?" I'm too lazy to fire up the whole application and click mouse buttons, and I can only do that so fast; "Why do you only work with Vim in GNU/screen?" I can't be bothered to set up a new slew of terminals when I switch machines, and so on down the line.

Earlier this week I found another bit of manual work that I shouldn't be doing and should be lazy about: building. The local build is something that's common to every single software developer regardless of language, Slide being a Python shop, we have a bit more subtle of a "build", that is to say, developers implicitly run a "build" when they hit a page in Apache or a test/script. I found myself constantly switching between two terminal windows, one with my editor (Vim) and one for running tests and other scripts.

Being an avid Hudson user, I decided I'd give the File system SCM a try. Very quickly I was able to set up Hudson to poll my working directory and watch for files to change every minute, and then run a "build" with some tests to go with it. Now I can simply sit in Vim all day and write code, only context-switching to commit changes.

Setting up Hudson for local continuous integration is quite simple, by visiting hudson-ci.org you can download hudson.war which is a fully self contained runnable version of Hudson, you can start it up locally with java -jar hudson.war. Once it's started, visit http://localhost:8080 and you've find yourself smack-dab in the middle of a fresh installation of Hudson.

First things first, you'll need the File System SCM plugin from the Hudson Update Center (left side bar, "Manage Hudson" > "Manage Plugins" > "Available" tab)

After installing the plugin, you'll need to restart Hudson, then you can create your job, configuring the File System SCM to poll your working directory:

Of course, add the necessary build steps to build/test your software as well, and you should be set for some good local continuous integration. Once the job is saved, the job will poll your working directory for files to be modified and then copy things over to the job's workspace for execution.

After the job is building, you can hook up the RSS feed (http://localhost:8080/rssLatest) to Growl or some other form of desktop notifier so you don't even have to move your eyes to know whether your local build succeeded or not (I use the "hudsonnotify" script for Linux/libnotify below).

By automating this part of my local workflow with Hudson I can take advantage of a few things:

I no longer need to context switch to run my tests
I can make use of Hudson's nice UI for visually inspecting test results as they change over time
I have near-instant feedback on the validity of the changes I'm making

The only real downside I can think of is no longer having any excuse for checking in code that "breaks the build", but in the end that's probably a good thing.

Instead of relying on commits, you can get near-instant feedback on your changes before you even get things going far enough to check them in, tightening the feedback loop on your changes even further, very-very continuous integration. Your mileage may vary of course, but I recommend giving it a try.

hudsonnotify.py

Jython, JGit and co. in Hudson

R. Tyler Croy — Wed, 22 Jul 2009 06:16:38 +0000

At the Hudson Bay Area Meetup/Hackathon that Slide, Inc. hosted last weekend, I worked on the Jython plugin and released it just days after releasing a strikingly similar plugin, the Python plugin. I felt that an explanation might be warranted as to why I would do such a thing.

For those that don't know, Hudson is a Java-based continuous integration server, one of the best CI servers developed (in my humblest of opinions). What makes Hudson so great is a very solid plugin architecture allowing developers to extend Hudson to support a wide variety of scripting languages as well as notifiers, source control systems, and so on (related post on the growth of Hudson's plugin ecosystem). Additionally, Hudson supports slaves on any operating system that Java supports, allowing you to have a central manager (the "master" Hudson server/node) and a vast network of different machines performing tasks and executing jobs. Now that you're up to speed, back to the topic at hand.

Jython versus Python plugin. Why bother with either, as @gboissinot pointed out in this tweet? The interesting thing about the Jython plugin, particularly when you use a large number of slaves is that with the installation of the Jython plugin, suddenly you have the ability to execute Python script on every single slave, regardless of whether or not they actually have Python installed. The more "third party" that can be moved into Hudson by way of the plugin system means reduced dependencies and difficulty setting up slaves to help handle load.

Take the "git" versus the "git2" plugin, the git plugin was recently criticized on the #hudson channel because of it's use of the JGit library, versus "git2" which invokes git(1) on the command line. The latter approach is flawed for a number of reasons, particularly the reliance on the git command line executables and scripts to return consistent formatting is specious at best even if you aren't relying on "porcelain" (git community terminology for front-end-ish script and code sitting on top of the "plumbing", the breakdown is detailed here). The command-line approach also means you now have to ensure every one of your slaves that are likely to be executing builds have the appropriate packages installed. One the flipside however, with the JGit-based approach, the Hudson slave agent can transfer the appropriate bytecode to the machine in question and execute that without relying on system-dependencies.

The Hudson Subversion plugin takes a similar approach, being based on SVNKit.

Being a Python developer by trade, I am certainly not in the "Java Fanboy" camp, but the efficiencies gained by incorporating Java-based libraries in Hudson plugins and extensions is a no brainer, the reduction of dependencies on the systems incorporated in your build farm will save you plenty of time in maintenance and version woes alone. In my opinion, the benefits of JGit, Jython, SVNKit, and the other Java-based libraries that are running some of the most highly used plugins in the Hudson ecosystem continue to outweigh the costs, especially as we find ourselves bringing more and more slaves online.

Using Glib's gtester Results in Hudson

R. Tyler Croy — Thu, 18 Jun 2009 08:02:02 +0000

For one of the projects I've been working on lately, I've been making use of Glib's Testing functionality to write unit tests for a C-based project. I'm a fairly big fan of the Hudson Continuous Integration Server and I wanted to run tests for my C-based project.

Unfortunately, the gtester application generates XML in a custom format that Hudson cannot understand (i.e. JUnit formatted XML). In order to come up with some JUnit XML, I spent about an hour and a half toying with XSLT for transforming the XML gtester generates (see the snippet below).

I added a shell script build step to the end of the build process that runs:

gtester test_app --keep-going -o=Tests.xml
xsltproc -o tests.junit.xml gtester.xsl Tests.xml

Then I just specified "tests.junit.xml" in the Publish JUnit test result report section of the Post-build Actions and then Hudson would properly process and post the test results when the job was finished.

Do not fear continuous deployment

R. Tyler Croy — Fri, 13 Mar 2009 08:13:29 +0000

One of the nice things about living in Silicon Valley is that you have relatively easy access to a number of the developers you may work with through open source projects, mailing lists, IRC, etc. Today Kohsuke Kawaguchi of Sun Microsystems, the founder of the Hudson project, stopped by the Slide offices to discuss Hudson and the "cloud", continuous deployment and our workflow with Hudson here at Slide. Continuous deployment being the most interesting topic for me, and the most relevant in terms of the importance of Hudson in our current infrastructure.

Since reading Timothy Fitz's post on the setup for "continuous deployment" at IMVU, I've become obsessed to a certain degree with pushing Slide in that direction as an engineering organization. Currently we push a number of times a day as necessary, and it's almost as if we have manual-continuous-deployment as it is right now, there's just a lot of room for optimizations and automation to cut down on the tedium and allow for more beer drinking.

@agentdero continuous deployment = when build is green, autoship? sounds terrifying...

(@tlipcon)

As a concept, continuous deployment can be quite scary "wait, some robot is going to deploy code to my production site, wha!" It's important to remember that the concept of continuous deployment doesn't necessarily mean that no QA is involved in the release process, it is however ideal to have enough good test cases such that you can do a fully automated unit/integration/system test run. The biggest difficultly with the entire concept of "continuous deployment" however is not writing tests or actually implementing a system to deploy, it forces you to understand your releases and production environment; it's about eliminating the guess work from your process and reducing the amount of human error (or potential for human error) involved in deployments.

In my opinion, continuous deployment isn't about making a hard switch, firing your QA and writing boat-loads of tests to ensure that you can push the production site straight from "trunk" as much as humanly possible. Continuous deployment is far more about solidifying your understanding of your entire stack, evolving your code base to where it is both more testable and better covered by your tests, then putting your money where your mouth is and relying on those tests. If your codebase moves rapidly, unit/integration/system tests are only going to be up to date and valuable if you actually rely on them. If breaking a single unit test pre-deployment becomes a Big Deal™, then the developer responsible for the code being deployed will make sure that: (a) the test is valid and up to date and (b) the code that the test is covering does not contain any actual regressions.

Take the typical repository layout for most companies which is, as far as I've seen, made up of a volatile trunk, stable release branch and then a number of project branches. In an engineering department QA would be responsible for ensuring that projects are properly vetted before merging from project branches (also called "topic branches" in the Git community) into the more volatile trunk branch. Once the CI server (i.e. Hudson) picks up on changes in trunk, the testing process would begin at that particular revision. Provided the test suites passed with flying colors Hudson would start to kick up the process to do a slow/sampled deploy as Timothy describes in his post. If the tests failed however, alarms would start beeping, sirens would wail and there would be much gnashing of teeth, somebody has now broken trunk and is blocking any other deployments coming down the pipe. In this "disaster scenario" the QA involved in the process would be thoroughly shamed (obviously) but then given the choice to block future pushes while the developer(s) create a fix or revert their changes out of trunk and take them back to a project branch to correct the deficiencies. This attention to detail will have an larger benefit in that developers won't become numb to test failures to where they're no longer important.

What good is writing tests if there aren't have real-consequences for them failing? Releases shouldn't be a scary time of the day/week/month, you should certainly be nervous (keeps you sharp), but if you fear releases then it is probably an error in your release process that allows for too much uncertainty: inadequate test coverage, insufficient blackbox testing, poor release practices, etc. Continuous deployment might not be the magic solution to your woe of shipping software but the practice of moving towards continuous deployment will greatly improve your release process whether or not you ever actually make the switch over to a fairly automated deployment process as the engineers at IMVU have.

How confident are you in your test coverage?

Head in the clouds

R. Tyler Croy — Fri, 06 Mar 2009 07:42:03 +0000

I've spent the entire day thinking about "cloud computing", which is quite a twist for me. Seeing "impressive" conferences centered around "cloud computing" I've ridiculed the concept mercilessly, it has a phenomenally high buzzword/usefulness ratio, which makes it difficult to take seriously. It tends to have an air of idiocy attached to it of the same style that the re-invention of thin-clients did a few years back. That said, I think the concept is sound, and useful for a number of companies and uses (once distilled of the buzz).

Take Slide for example, we have a solid amount of hardware, hundreds of powerful machines constantly churning away on a number of tasks: serving web pages, providing back-end services, processing database requests, recording metrics, etc. If I start the work-week needing a new pool of machines either set up or allocated for a particular task, I can usually have hardware provisioned and live by the end of the week (depending on my Scotch offering to the Operations team, I can get it as early as the next day). If I can have the real thing I clearly have no need for cloud computing or virtualization.

That's what I thought, at least, until I started to think more about what would be required to get Slide closer to the lofty goal of continuous deployment. As I was involved in pushing for and setting up our Hudson CI server, I constantly check on the performance of the system and help make sure jobs are chugging along as they should be, I've become the defacto Hudson janitor.

Our current continuous integration setup involves one four-core machine running three separate instances of our server software as different users, processing jobs throughout the day. One "job" typically consists of a full restart of the server software (Python) and running literally every test case in the suite (we walk the entire tree aggregating tests). On average the completion of one job takes close to 15 minutes, and executes around 400+ test cases (and growing). Fortunately, and unfortunately, our Hudson machine is no longer able to service this capacity during development peak in the middle of the day, this is where the "cloud" comes in.

We have a few options at this point:

Setup another one or more machines
Rethink how we provision hardware for continuous integration

The fundamental problem with provisioning resources for continuous integration, at least at Slide, is that the requirements are bursty at best. We typically queue a job for a particular branch when a developer executes a git push (via the Hudson API and a post-receive hook). From around 9 p.m. until 9 a.m. we don't need but maybe two actual "executors" inside Hudson to handle the workload the night-owl developers tend to place on Hudson, from 12 p.m. until 7 p.m. however our needs fluctuate rapidly between needing 4 executors, and 10 executors. To exacerbate things further, due to "natural traffic patterns" in how we work, mid-afternoon on Wednesday and Thursday require even more resources as teams are preparing releases and finishing up milestones.

The only two possible solutions to solve the problem are to: build a continuous integration farm with full knowledge capacity will remain unused for large amounts of time, or look into "cloud computing" with service provides like Amazon EC2 which will allow for Hudson slaves to be provisioned on demand. The maintainer of Hudson, Kohsuke Kawaguchi has already started work on "cloud support" for Hudson via the EC2 plugin which makes this a real possibility. (Note: using EC2 for this at Slide was Dave's idea, not mine :))

Using Amazon EC2 isn't the only way to solve this "bursty" problem however, we could just as easily solve the problem in house with provisioning of Xen guests across a few machines. The downside of doing it yourself is amount of time between when you know you need more capacity and when you can actually add that capacity to your own little "cloud". Considering Amazon has an API for not only running instances but terminating them, it certainly provides a compelling reason to "outsource" the problem to Amazon's cloud.

I recommend following Kohsuke's development of the EC2 plugin for Hudson closely, as continuous integration and "the cloud" seem like a match made in heaven (alright, that pun was unnecessary, it sort of slipped out). At the end of the day the decision comes down to a very fundamental business decision: which is more cost effective, building my own farm of machines, or using somebody else's?

(footnote: I'll post a summary of how and what we eventually do to solve this problem)

But Who Will Write The Tests?

R. Tyler Croy — Mon, 12 Jan 2009 16:43:02 +0000

In addition to frothing at the mouth about Git, I've been really getting into the concept of automated unit tests lately (thus my interest in Hudson). Just like code comments however, tests are good, no tests is bad, wrong tests is worse. That means once you give in to the almighty power of unit testing, you are saddled with the curse of knowing that you will have to update them, forever.

Taking up Test-driven Development is like having a child, if you are at a point in your life where you're ready to accept that kind of responsibility, it can be wonderful, a lot of work, but ultimately you will feel satisfied with your new role as a Responsible Developer (tm). If you're not prepared to take on the burden that TDD will present you with, you will likely regret it or neglect your tests (Deadbeat Developer, I like this metaphor).

In the Top Friends Team at Slide, we practice the more "loose" definition of TDD; tests are not written before functionality is written, but rather functionality is written, and then as part of the QA and release process, the appropriate and accompanying tests are written. Our basic workflow is usually as follows:

Tickets are written and assigned to milestones and developers in Trac
Branch is created in central Git repository
General plan-of-action is discussed between developers
Hack-hack-hack
Code complete is reached, QA starts to test milestone
Developers write tests if needed for functionality
Once QA signs off, and tests look solid, code is shipped live

There are two primary flaws with this workflow, the first and most obvious one is that it is far to easy to "forget to write the tests." That is, the project scheduled to start development tends to "flow forward" into the allotted test-writing time. As important as test coverage is, at the end of the day Slide did not raise funding on having solid test coverage, and our priorities lie in shipping software, first and foremost. Solving the flow-forward of scheduled projects into any available space is something that can be worked on, but never solved, it really comes down to discipline between those in charge of setting up any given project's particular roadmap.

The second, more subtle flaw in this workflow, and I think all Test-driven Devleopment workflows, revolves around the writer of the tests. The fundamental nature of almost all bugs in software is human error, our natural tendency to make mistakes means that nothing we do will ever be perfect, including our tests. If Developer A is writing a couple new methods to handle data validation prior to that data going into the database. Chances are that Developer A's life is going to be made far easier by writing some test cases to run through some predefined user-input, and pass his validation code over it. Therein lies the problem, if the developer doesn't think of a particular edge case when he's writing the code to handle the data validation, the chances he'll remember and account for that particular edge case while he's working on the unit tests is nil.

How do you really ensure that tests are of high enough quality to actually catch errors and regressions?

I think a certain extent of intra-team test writing and code review, depending on the level of communication between developers, can really help. In this case less developer communication is better. If Developer A tells Developer B how his code works, Developer B is now going to have an unnecessary expectation when he starts to write tests for Developer A's code. If Developer B reviews the code for what it actually is, instead of what Developer A thinks it is, the tests that will ultimately be written will be more thorough than if Developer A had written the whole suite himself.

This still isn't sufficiently fool-proof to where I feel all that confident in test coverage, the tests being written are subject to the availability, thoroughness and understanding that Developer B brings to the table. Inside a small team like this one, one of those is almost always in short supply (usually availability).

One approach I'm anxious to try is the more active involvement of QA engineers in the test writing process, both in the pre-fail and post-fail scenarios. The pre-fail scenario being one like that which I detailed above, where new code is being written. In this case a QA engineer's experience can help guide the developer on what sets of user-input have typically caused issues in the past. The second case, post-fail, is actually already occuring at Slide; a live issue, data validity bug, or regression is caught by QA engineers who detail the reproduction case in Trac and as a result a regression test can be written for that specific issue.

This still is subject to the three things I cited above: availability, thoroughness and understanding of those involved. I still have a lot of unanswered questions about the ideal QA and Dev workflow however, how does this scale to a team of tens or hudnreds? Who writes the tests for large teams? What about a team of 1 Dev and a 1 QA, what about the lone-hacker? How do you write quality code, without getting bogged down in the mush of writing thousands of tests for everything you can imagine could go wrong?

Who writes the tests?

Reliable Locks in Hudson

R. Tyler Croy — Thu, 06 Nov 2008 05:23:41 +0000

There has been some amount of discussion on the Hudson user's list recently about the status of the "Locks and Latches" plugin. The plugin allows for one to create "locks" for Jobs in a similar manner to how "locks" work in a multithreaded programming environment. The need for such a plugin becomes very clear once you start to run multiple jobs that depend on some set of shared resources, take the following example:

Jobs A,B,C must run unit tests that fetch data from a test site
Slave #1 can only run one instance of Apache at a time

How one would accomplish this with the Locks and Latches plugin would be to create a lock like "Site Lock" in the Hudson configuration, and then bind Jobs A, B, C to that Lock. Making the (large) assumption that the plugin works correctly and locks properly in order to prevent A and B from running concurrently, this would be enough to satisfy the requirements we have for the scenario above. Unfortunately it seems the plugin is largely unmaintained and buggy; in the past couple weeks of experimenting with such a set up on a variety of different slaves we've noticed that the locks aren't always respected, causing some locked jobs to execute in parallel spewing bad test results and build failures (the crux of this issue seems ot have been reported by Sergio Fernandes in #2450).

The Loopback Slave
The easiest way I found to work around the short-comings of the Locks and Latches plugin was to "break up" the Locks. Locks are only really useful if you have more than one "executor" on a Hudson node, in order to allow Hudson to execute jobs simultaneously. In essence, if you only have one executor, the Hudson queueing system will technically perform your "lock" for you by default. And thus the "loopback slave" was born! When explaining this to a co-worker, I likened my workaround to the fork(2) call, whereas the Locks and Latches plugin is much more of a pthread_mutex_lock(2) call. According to the "Distributed Builds" page on the Hudson wiki, you can start slave agent headlessly on any machine, so why not the master node?

Above is the configuration of one such "loopback slave" that took the place of one of the executors on the master node.

After setting up the loopback slave, it's just a matter of tying the Job to that node for building.

In short our set up was before: Jobs A, B, C all use the Lock "Site Job" in order to queue properly. With this change, now there is no lock, and Jobs A, B, C are all bound to the loopback slave in place of the lock on the master node. While certainly not ideal, given the frustrations of the Locks and Latches plugin going unmaintained this is the best short-term solution I've come up with thus far.

Did you know! Slide is hiring! Looking for talented engineers to write some good Python and/or JavaScript, feel free to contact me at tyler[at]slide

Hudson Build Bookmarklet

R. Tyler Croy — Sat, 04 Oct 2008 09:36:06 +0000

During the usual friday-frenzy I sat down and wrote a quick 10 minute little bookmarklet to start a Hudson job. Unlike most bookmarklets that "do things" this one actually "does things" but doesn't take you away from your current page. Using the Hudson Remote Access API you can query information from Hudson programmatically, but you can also kick off builds remotely with nothing more than a simple HTTP request to the properly formed URL.

By dragging the link below to your bookmark bar, and updating the URL within ("http://hudson/") to the URL of your Hudson instance, you can queue a Hudson build from any page at any time (without leaving the page).

The Bookmarklet

Build Hudson Job

The Code

<a href="javascript:var b = prompt('Enter a job you would like to build:', 'main'); if (!b) { } else { var i = document.createElement('img'); i.src = 'http://hudson/job/' + b + '/build'; i.height = 1; i.width = 1; document.body.appendChild(i);}; alert('Build ' + b + ' queued');">Build Hudson Job</a>

How it actually works

After talking the concept of making cross-domain HTTP requests over with Sergio, he suggested just using an "IMG" tag (or "IFRAME") to accomplish the task. The bookmarklet doesn't actually have to send any form parameters or receive any data, Hudson just needs to receive any HTTP request to the right URL. By creating the IMG object in JavaScript, and appending it to the body of the current page the user is on, it'll effectively con the browser into making the HTTP request without needing to pull off any XmlHttpRequest hacks. One of the more interesting things that we found out when playing with the end of the bookmarklet, was that if we returned "false" or tried to wrap the whole thing in a closure, the link would still execute and the browser would change pages. However, if we stuck an "alert()" call into the tail end of the bookmarklet JavaScript, execution would stop and the link wouldn't change the page in the browser (tested this in Firefox 3).

Happy Hudsoning :)

Hudson notifications with libnotify

R. Tyler Croy — Fri, 19 Sep 2008 06:40:20 +0000

I've been using a Gnome-based desktop for about the past 8-9 months and one of the things I've come to really appreciate is that most Gnome applications integrate with "libnotify". Libnotify is a simple Windows taskbar-like notification system that presents status messages at the bottom of your screen. Like all great pieces of software, it has a solid Python interface which allows for incorporating it in those little 10-minutes scripts I find myself writing every now and again.

One of the things I wanted to script was the notification of the build status of the numerous jobs that we're running in our Hudson instance here at Slide. Using the Universal Feed Parser and pynotify (listed under "notify-python"), I had a good little Gnome Hudson Notifier running in less than 10 minutes.

Source code after the jump.

import feedparser
import pynotify
import time
 
BASE_TITLE = 'Hudson Update!'
 
def success(job):
        n = pynotify.Notification(BASE_TITLE, 
                                        '"%s" successfully built :)' % job,
                                        'file:///usr/share/pixmaps/gnome-suse.png')
        n.set_urgency(pynotify.URGENCY_LOW)
        return n
 
def unstable(job):
        return pynotify.Notification(BASE_TITLE, 
                                        '"%s" is unstable :-/' % job,
                                        'file:///usr/share/pixmaps/gnome-suse.png')
 
def failure(job):
        n = pynotify.Notification(BASE_TITLE, 
                                        '"%s" failed!' % job,
                                        'file:///usr/share/pixmaps/gnome-suse.png')
        n.set_urgency(pynotify.URGENCY_CRITICAL)
        return n
 
def main():
        pynotify.init('Hudson Notify')
        old_items = []
        while True:
                feed = feedparser.parse('http://hudson/rssLatest')
                items = [t['title'] for t in feed['entries']]
                new_items = list(set(old_items).difference(items))
 
                for i in new_items:
                        i = i.split(' ')
                        job, build, status = (i[0], i[1], i[2])
                        status = status.replace('(', '').replace(')','')
                        if status == 'SUCCESS':
                                success(job).show()
                        elif status == 'UNSTABLE':
                                unstable(job).show()
                        elif status == 'FAILURE':
                                failure(job).show()
 
                old_items = items
                time.sleep(60)
 
if __name__ == '__main__':
        main()

It's pretty basic right now, but does everything I really wanted it to do. I may add it into a public Git repository in the near future if I spend any more time on the project. Hope you like it :)

Don Quixote's new side-kick, Hudson

R. Tyler Croy — Sun, 07 Sep 2008 06:04:53 +0000

I recently wrote about "one-line automated testing" by way of Hudson, a Java-based tool that helps to automate building and test processes (akin to Cruise Control and Buildbot). If you were to read this blog regularly, you'd be well aware that I work primarily with Python these days, at a web company no less! What does a web company need with a continuous integration tool? Especially if they're not using a compiled language like Java or C# (heresy!).

As any engineering organization grows, it's bound to happen that you reach a critical mass of developers and either need to hire an equitable critical mass of QA engineers, or start to approach quality assurance from all sides. That is to say, automated unit testing and automated integration testing becomes a requirement for growing both as a engineering organization but as a web application provider (user's don't like broken web applications). With web products like Top Friends, SuperPoke! and Slide FunSpace we have a large amount of ever-changing code, that has been in a constant state of flux for the past 16-18 months. We can accomodate for ever-changing code on the backend for the past year and half with PyUnit and development discipline.

How do you deal with months of ever changing code for the aforementinoned products' front-ends? Your options are pretty slim, you can hire a legion of black-box QA engineers to manually go through regression tests and ensure your products are in tip-top shape, or you can hire a few talented black-box QA engineers to conscript a legion of robots to go through regression tests and ensure your products are in tip-top shape. Enter Windmill. Windmill is a web browser testing framework not entirely unlike Selenium or Watir with two major exceptions: Windmill is written in Python and Windmill has a great recorder (and lots of other features). One of my colleagues at Slide, Adam Christian has been working tirelessly to push Windmill further and prepare it for enterprise adoption, the first enterprise to use it, Slide.

Adam and I have been working on bringing the two ends of the testing world together with Hudson. About half of the jobs currently running inside of our Hudson installation are running PyUnit tests on various Subversion and Git branches. The other half of the jobs are running Windmill tests, and reporting back into Hudson by way of Adam's JUnit-compatible reporting code. Thanks to the innate flexibility of PyUnit and Windmill's reporting infrastructure we were able to tie all these loose ends together with a tool like Hudson that will handle Jabber-notifications or email notifications when test-runs fail and include details in it's reports.

We're still working out the kinks in the system, but to date this set up has helped us fix at least one critical issue a week (with a numerous other minor issues) since we've launched the Hudson system, more often than not before said issues reach the live site and real users. If you've got questions about Windmill or Hudson you can stop by the #windmill or the #hudson channels on Freenode.

Automated testing is like a really good blend of coffee, until you have it, you think "bah! I don't need that!" but after you start with it you can't help but wonder how you could tolerate the swill you used to drink.

Did you know! Slide is hiring! Looking for talented engineers to write some good Python and/or JavaScript, feel free to contact me at tyler[at]slide

One-line Automated Testing

R. Tyler Croy — Wed, 20 Aug 2008 08:16:26 +0000

For about as long as my development team has been a number larger than one, I've been on a relatively steady "unit test" kick. With the product I've worked on for over a year gaining more than one cook in the kitchen, it became time to start both writing tests to prevent basic regressions (and save our QA team tedious hours of blackbox testing), but also to automate those tests in order to quickly spot issues.

While I've been on this pretty steadily lately, I'm proud to say that automated testing was one of my first pet projects at Slide. If you ever crack into the Slide corporate network you can find my workstation under the name "ccnet" which is short for Cruise Control.NET, my first failed attempt at getting automated testing going on our now defunct Windows desktop client. As our development focus shifted away from desktop applications to social applications the ability to reliably test those systems plummeted; accordingly our test suite for these applications became paltry at best. As the organization started to scale, this simply could not stand much longer else we might not be able to efficiently push stable releases on a near-nightly schedule. As we've started to back-fill tests (test-after development?) the need to automate these tests has arisen to which I started digging aronud for something less painful to deal with than Cruise Control, enter Hudson.

Holy Hudson Batman!

I was absolutely astounded that I, nor anybody I knew, was aware of the Hudson project. Hudson is absolutely amazing as far as continuous integration systems go. The only major caveat is that the entire system is written in Java, meaning I had to beg one of our sysadmins to install Java 1.5 on the unit test machine. Once that was sorted out, starting the Hudson instance up was incredibly simple:
java -jar hudson.war
In our case the following to keep the JVM within manageable virtual memory limits:
java -Xmx128m -jar hudson.war --httpPort=8888

Once the Hudson instance was up and rnuning, I simply had to browse to http://unittestbox:8888/ and the entire rest of the configuration was set up from the web UI. Muy easy. Muy bueno.

Plug-it-in, plug-it-in!

One of the most wonderful aspects of Hudson is it's extensible plugin architecture. Adding plugins like "Git", "Trac" and "Jabber" means that our Hudson instance is now properly linking to Trac revisions, sending out Jabber notifications on "build" (read: test run) failures and monitoring both Subversion and Git branches for changes. From what I've seen from their plugin architecture, it would be absolutely trivial to extend Hudson with Slide-specific plugins as the needs arise.

With the integration of the PyUnit XMLTestRunner (found here) and working an XML output plugin into Windmill we can easily automate testing of both our back-end code and our front-end.

Hudson in action

And all with one simple java command :)

Did you know! Slide is hiring! Looking for talented engineers to write some good Python and/or JavaScript, feel free to contact me at tyler[at]slide