Which open source CI tool is best suited for your application’s environment?

The term “continuous integration,” commonly attributed to Martin Fowler, who first wrote about the software development practice back in 2000 (see Resources for a link to the article), is one of the pillars of modern programming techniques. CI involves automatically building and testing an application at frequent intervals on a dedicated integration server. Developers regularly commit small updates and are notified rapidly if their changes cause the build to fail. This practice is first and foremost a risk-reduction strategy, ensuring that the application integrates correctly at all times and reducing the risk of integration issues appearing late in the project. It has the additional benefits of improving communication between team members and reducing the time needed to prepare a release.

The most fundamental feature of any CI tool is the automatic build cycle. The CI tool, either at a scheduled point in time or whenever any changes are detected in the source code repository, fetches the source code’s latest version and attempts to build and test the project. If the build fails, developers can be notified in any number of ways: email, instant messaging, or through some other notification technique.

Many CI tools, both open source and commercial, are on the market, and deciphering which one would best suit your particular environment can prove difficult. Some CI tools also boast additional SDLC (software development lifecycle) features, such as release management and build artifact management, while others concentrate on core CI functionalities. Some try to provide a wide range of notification methods and support virtually every version control system under the sun, whereas others concentrate on delivering a small, light, easy-to-understand product. In this article, we will look at four of the better-known and more interesting continuous integration tools in the open source world and see how they measure up:

• Cruise Control
• Continuum
• Luntbuild
• Hudson

This list is obviously not exhaustive. There are many other CI tools out there, both open source and commercial. Of course, each tool has its particularities, and there is no one-size-fits-all solution. The aim of this article is to provide a brief survey of some of the more interesting tools available and to help readers decide which tool suits their particular environment best.

CruiseControl: The original Java CI tool

CruiseControl is a mature and robust Java continuous integration tool that enjoys a strong user base and a solid industry reputation. It supports virtually any type of project, be it Ant, Maven, Maven 2, make, or just a plain-old command line script, as well as a wide range of SCM (source configuration management) tools. Its notification techniques are second to none. And it also benefits from a large number of third-party plug-ins.

Setting up a project using CruiseControl involves a bit more work than with the other tools we look at in this article. CruiseControl is a highly configurable tool, and all CruiseControl configuration is done in a central XML configuration file. As of recent versions, there is a graphical interface written in Swing that helps you manipulate this file, but it is still a good idea to be able to configure this file directly. Thus, configuring a CruiseControl project can be a somewhat painful experience, with a fairly steep learning curve.

In addition to configuring your project in the XML configuration file, you also need to set up a copy of your project manually in the CruiseControl projects directory. As with all CI tools, SCM integration plays a crucial role in the CruiseControl architecture. CruiseControl supports a wide range of SCM products, including CVS, Subversion, BitKeeper, IBM Rational ClearCase, MKS, Perforce, Serena PVCS, Borland StarTeam, Visual SourceSafe, and more. However, CruiseControl simply polls these SCM tools to detect code changes: getting the code out of the code repository is largely your responsibility, and CruiseControl expects your project to be able to do this by itself. For this reason, developers using Ant often write a “wrapper” Ant build script (typically called “cc-build.xml”), which updates the source code from the repository and then calls the main build script to do the real work. Maven users can use the SCM plug-in to avoid this issue.

Once your project is correctly configured, you can monitor CruiseControl builds via the CruiseControl Website, which is actually a separate application built on top of the CruiseControl core. The latest version comes bundled with a Jetty server; this site is fairly easy to deploy.

The Website is simple and to the point. The latest build status of each project displays on the homepage. You can also view detailed build results, unit test results, and some simple build metrics, such as the percentage of successful builds over time.

One of CruiseControl’s strengths is its sophisticated and reactive build cycle. CruiseControl continuously monitors the SCM servers for any updates and triggers a build whenever it detects any new changes to the source code repository. This makes for a more reactive agile development environment, as developers don’t need to wait for scheduled builds to learn about integration issues. Builds can also be scheduled to be run at certain times or intervals.

Running your builds regularly is good, but it’s even better to let everyone know how they’re going. Notification is another of CruiseControl’s strong points. Out of the box, CruiseControl supports email, Jabber IM, RSS feeds, and even blogs. You can also perform various tasks such as implementing FTP or SCP file transfers, invoking an Ant target, running command line tasks, or even using an X10 interface to hook up a lava lamp, or any other X10-compatible device you can put your hands on. X10 is an industry standard for controlling electronic devices. CruiseControl has an X10 interface that lets you publish build status information through a variety of electronic devices. For example, a lava lamp may turn from green to red and bubble frenetically when a build fails.

One common issue in larger projects is that of inter-project dependencies. Basically, when you build a particular project, you may also need to build other libraries to avoid integration issues. This is not one of CruiseControl’s strengths. In the current version, CruiseControl offers only limited support for inter-project dependencies (via the plug-in). CruiseControl also offers little in the way of build management in the broader sense: it is difficult to define inter-project dependencies, for example.

Overall, CruiseControl is a powerful and flexible continuous integration tool, albeit with a non-trivial learning curve. It is a pure CI tool, with little in the way of inter-project dependency or build artifact management. However, its flexibility and sophisticated notification techniques can make it an excellent choice for experienced continuous integration practitioners.

Continuum: The official Maven 2 CI tool

Continuum is a simple, intuitive, and lightweight CI tool developed by the team who brought you Maven. It is easy to install and involves little initial configuration. Continuum comes with its own Jetty Web server and uses an Apache Derby embedded database, which makes the tool pretty self-reliant.

Continuum handles Maven 1 and 2, Ant, and even plain-old shell scripts. In fact, one of Continuum’s selling points is its strong integration with Maven 2. In Maven 2, you can define SCM configuration details and list team members all within your project definition file (or POM, project object model). If your Maven 2 project definition file contains enough information, all you need to do is provide the POM file to add your project to the Continuum server. If this information isn’t available in the POM file, you can always define it through the Continuum Web interface.

Continuum also handles Maven 1 and Ant projects well: you just need to supply the SCM repository details and the build script, and Continuum will do the rest. This proves more convenient than CruiseControl, where you need to write a special script to check out the latest source code from the repository.

Continuum uses the Maven SCM library, which supports a reasonable and growing number of SCM systems: Subversion, CVS, Borland StarTeam, Perforce, and Bazaar, as well as partial support for IBM Rational ClearCase and Visual Source Safe.

Continuum provides a Web-based administration tool that is simple and fairly intuitive (see Figure 2). From here, you can add new projects, monitor the status of your project builds, and examine the detailed build results when a build fails. You can also browse the working copy of the source code that Continuum is using for its builds. This is particularly useful when you are trying to figure out why the build that works on your machine fails on the Continuum server.

Continuum builds are scheduled using cron-type expressions. Cron expressions, well-known in Unix administration circles, provide a succinct, if cryptic, way of defining a schedule for a repeated task. For example, “0 0 * * * *” means “on the hour, every hour,” and “0 0 6-18 ? * MON-FRI” means “on the hour, every hour from 6 a.m. to 6 p.m., Monday through Friday.”

In the current version, however, Continuum does not actively query SCM servers to detect updated source code, which results in a slightly less reactive CI tool than one such as CruiseControl, which can do this.

Continuum provides the usual notification techniques: email and instant messaging (Internet Relay Chat, Jabber, MSN). Email notifications provide ample details of the changes made and the build results. Currently no built-in support for RSS notifications is available, though you can write your own fairly easily using the API. RSS notification is useful for project managers or quality assurance people who just need to monitor the build status and history, and don’t need to react quickly if a build fails.

Continuum is still a relatively young product, with a lot of room for new and improved features. A potential issue for big projects or programs is that, like CruiseControl, Continuum provides no simple way of handling dependencies between projects.

Nevertheless, Continuum is easy to install and easy to use, and will suffice in many situations. Its simplicity, and its intuitive Web administration console, make it a good candidate for introducing continuous integration into organizations that have had no previous experience with the subject.

Luntbuild: A graphical build management tool

Luntbuild is another open source CI tool written in Java. It is particularly easy to install and configure, and, like Continuum, all server administration tasks are done via a Web-based administration console. In fact, Luntbuild is designed to do more than just manage the continuous integration process: it also lets you store and manage generated artifacts, label and promote versions, and manage dependencies between builds. It supports a wide range of version control tools, and notifications can be diffused via email, IM, and even on a blog site. Indeed, it is one of the most feature-rich of the open source CI tools.

Luntbuild is produced and maintained by a company called PMEase. Luntbuild also has a commercial cousin, called QuickBuild, which is marketed by the same company.

Luntbuild comes with a real installer, which walks you through the installation process and proposes a number of configuration options. At the simplest level, you can use a fully self-contained embedded database and Jetty Web server. Or you can configure Luntbuild to work with an external database, and/or on an external Web server. You can even set up your user accounts from an external LDAP (lightweight directory access protocol) directory.

It is a powerful and flexible tool, and as such, the administration console is relatively complex (see Figure 3). Precisely because of numerous options that Luntbuild proposes, creating a new project is a fairly involved task. For a typical project, you need to set up the version control configuration, define user access rights and notification methods, create one or several build scripts that can be executed for this project, and set up schedules to define when these scripts are to be executed.

The notification methods are defined at a system level and can be shared across all projects. Luntbuild supports a limited range of notification methods, including email, MSN, Jabber, and some blogging software. On the other hand, it supports a wide range of version control products, including CVS, Subversion, Borland StarTeam, AccuRev, IBM Rational ClearCase, and even Visual Source Safe. Since each CVS product has its own particularities, each CVS adaptor has its own specific configuration screen. The Subversion screen, for example, recognizes the Subversion convention of storing the main development trunk, separate branches, and revision tags in separate specific directories.

One of the distinctive features of Luntbuild is the way it distinguishes between configuring the builds and scheduling them. Command line, Ant, Maven, and even Rake build scripts are supported. Setting up the build tasks is intuitive and flexible, and you can set up as many as you like for each project.

Once you’ve set up your build scripts, you define schedulers to indicate when to run the scripts. By separating the schedulers from the build scripts, you can easily write modular build scripts that can be used and reused in different schedules. Schedules can only be, well, scheduled. Luntbuild cannot actively poll the source code repository for changes as both CruiseControl and Hudson (see below) can.

Luntbuild does provide good support for project dependency management and build artifact management, and you can create different types of dependencies between projects. For example, you can require that all the dependent libraries of a project be built successfully before the project itself can be built.

Other than an arguably uncool name, there is little to find fault with Luntbuild. It is a solid, feature-rich tool with a clean user interface, support for a wide range of SCM tools, and a good variety of notification techniques. However, the scheduling SCM architecture used by Luntbuild is, as I have already pointed out, less reactive than the polling approach. Some of its complexity may nevertheless be overkill for users with little experience in continuous integration. Nevertheless, it is definitely a tool worth evaluating.

Hudson: The new kid in town

Hudson is a newcomer to the CI field and is a focused CI tool. In many regards, it has considerably fewer features then the other tools we’ve been looking at, concentrates more on Subversion and CVS-based projects, and provides only a limited number of notification techniques. But the features it does have are extremely well thought-out, with some being quite innovative.

Hudson is easy to install, though you do need to deploy it onto your own Java Web server (it is delivered as a deployable war file). Both system configuration details and projects can be managed from a slick Web console (see Figure 5). System-wide configuration is fairly simple: you just need to configure the paths to your system JDKs, and to Maven and Ant, as well as some configuration details for email notification.

Its SCM support is limited to CVS and Subversion, which are, admittedly, currently the two most popular version control tools on the market. Its notification capabilities are also limited to the strict minimum: only email and RSS notifications are supported out of the box. It would be nice to see a few more notification techniques, such as instant messaging support.

Like CruiseControl, in addition to cron-based scheduling, you can ask Hudson to poll the SCM server for updates (you can even configure the frequency of its polling). This is a powerful feature. This, combined with email notification and the ability to display unit test results, provides for a tight development/integration cycle.

Hudson boasts a particularly clear presentation of build results (see Figure 6). In addition to the build status and a summary of the changes made in the source code repository since the last build, the build results page can also display downloadable artifacts generated by the build. Hudson also provides excellent built-in support for JUnit test reports: test results, as well as a graph of historical test data can be viewed directly from within the build results page.

There is also a nifty “fingerprint” feature. Hudson records a unique “fingerprint” (actually, just the MD5 checksum) for each artifact generated during a build. So if you find yourself with an artifact, and you aren’t sure which build generated it, Hudson can work it out for you. This functionality can be useful when tracking down incompatibilities between library versions.

Hudson provides some interesting support for inter-project dependencies: you can specify that a given project wait until all other projects are built, and you can also specify that other projects be rebuilt after a project is successfully built.

The main issue developers will find with Hudson, other than its relatively modest feature set, is probably its lack of flexibility. For example, Hudson assumes you have your Ant build.xml file in your project’s root directory. Although this is a widely followed convention, some Ant projects have their build scripts in a subdirectory, which gives Hudson a bit of trouble. Maven projects are handled more consistently, probably because of the strong emphasis on following standards and conventions found in Maven projects.

Despite its modest feature set (which will probably improve in future versions), Hudson offers some powerful and efficient core CI functionalities, as well as some interesting innovations. It is definitely a tool to keep on your CI radar!

Conclusion

Many CI tools are available out there, and there is no one-size-fits-all solution. Different tools will suit different environments and different people. At the end of the day, to choose the right tool for you, you need to set up the tools and try them for yourself. Hopefully I have given you some tips to help you investigate further.