Sam Newman's site, a Consultant at ThoughtWorks

Posts from the ‘Build And Deployment’ category

When using a Continuous Integration build, before long you’ll break it. Breaking a build is not a bad thing, however it is typically the team’s top priority to have such a build fixed. Beyond the shame associated with having been named as the breaker, you then have the hassle of lots of people informing you you’ve broken it.

As a way of letting the development team know that:

  • You know the build is broken
  • You are fixing it

A simple broadcast mechanism can be highly useful.


Whilst I have seen high-tech solutions being recommended, the most effective example of a Build Fix Flag I’ve seen is simply using a giant paper flag. It was about 2 1/2 feet in height, and could be clearly seen above monitors. When a build failure was seen, a quick glance across the floor would indicate if someone was working on it.

What was nice was that before long, the same mechanism was used for notification of a number of development environments

Rules for the build fix flag

When using such a flag, we quickly decided on a set of rules as to how to use it:

  1. If you saw a CI build breakage, you looked for the flag
  2. If someone had the flag, you left them alone
  3. If you couldn’t see the flag, you tried to identify the person who made the last check in
  4. If you couldn’t find a likely culprit, you raised the flag and fixed it yourself

When running a suite of tests – either as part of a Continuous Integration build, or part of a check-in gate – speed is the enemy. You are always trying to find the balance between test coverage and time to complete the entire suite.

Both a check-in gate and continuous integration build have slightly different time pressures. The optimal duration for both will probably vary from team to team. A fish-eye suite is one way of formulating which tests should be included.

In a fish-eye suite, you focus on those tests which provide the best coverage for those areas of functionality currently being developed, whilst those areas not directly being affected have only minimal coverage. The logic goes that tests are there to pick up bugs, and therefore tests which cover the functionality currently being worked on are the most important.

Regularly changing the focus

In an iterative development process, working out where to change the focus of the suite can be easy. At the beginning of each iteration, the team as a whole decide which tests (or group of tests) should be run in each suite. During a development process where there is less segmentation, changing the suite on an ongoing basis may be more sensible

Reactive test suite

Rather than changing the focus of the suite at the start of each iteration, you can also decide to adapt the contents of the suite in certain situations – for example when the total time for the suite to run exceeds an agreed limit, or the number of bugs reported against a functional area increases.

For example, the team may decide that the check-in gate build should take 30 seconds, and the CI build should take no more than five minutes. The moment they take longer, the team as a whole should redefine what tests should be included. When removing tests, the team should look to remove those tests which cover areas of functionality furthest away from those currently being worked on.

Likewise if the number of bugs being raised against a certain functional area increase, the development team may decide that increasing the test coverage of a certain area makes sense. Again, the team may have to remove some tests in order to still be within the optimal build time. In which case, tests associated with areas of functionality with low defect rates make a good candidate for removal.

Source-Code Management

Being able to create and manage a fish-eye suite presupposes that you group your tests into functional areas. This can be an issue with typically packaging structures which are defined in horizontal terms, whereas the long running functional tests should cover a vertical slice of functionality.

Even if initially you aren’t going to use fish-eye suites, grouping your tests into functional (vertical) groupings rather than system (horizontal) groupings makes sense as it allows to to easily use this approach later. It also tends to make more sense to testers who see things in terms of usable functions rather than horizontal tiers.

It’s common practise within a team to define a set of tasks which should be run by each developer prior to checking in. The purpose of the Check-in Gate is to attempt to ensure that all code satisfies some basic level of quality.

Like all development standards, a check-in gate helps a team stay on the same page. It eases integration, and can help give confidence as to the quality of the code checked in.

Check-in Gate with Continuous Integration

A check-in gate is typically used prior to checking in to a system which uses Continuous Integration – where the tasks in run during the check-in gate are the same tasks used as part of the continuous integration build. The main source of embarrassment associated with breaking a CI build tends to come down to the fact that the shamed individual completely forgot to run their check-in gate build.

The importance of speed

Check-in Gates need to be fast. The longer they take to run, the less developers will want to run them – this either results in less frequent check-ins or in developers not running them at all. Fewer check-ins result in more complex (and more error prone) integrations. Not running the check-in gate at all can result in a breakdown of code quality and can be a slippery slope to the gate being abandoned altogether.


The simplest example of a check-in gate would probably be ensuring that the code compiles prior to check-in. More often, the team will decide to run either some or all of a test suite. Again the constraining factor as to what you’ll want to run as part of a check-in gate is typically time – deciding how and what to run should always be defined by the team.

I’ve seen the diagram below used to describe “Agile Development”. I found it to be quite a good overview of a typical agile development process.

Continue reading…

As described before, the build pipeline is a series of builds, each performing some specific task. The result of one build becomes the input of the next.

Many people see the pipeline describing only those parts which can be automated – as such, you’ll often see the pipeline end far short of production ready code – once it’s got past the final automated barrier, the code then plunges into a grey mass of manual, distributed, often ad-hoc processes. For the build pipeline to work at all, it has to be continued all the way to producing production ready code. That is not to say the whole pipeline should be automated.

Continue reading…

Update 1: Fixed arrows on two of the images – damn Visio…

Update 2: Tweaked the final diagram to show build artifacts being checked in and retrieved

Lets look at a fairly simple build.

Continue reading…

Update: Fixed typo


Slow builds are perhaps the most irritating thing for a developer. Having worked on two projects now with +30 minute continuous integration builds, it’s something of a personal bugbear for me. Before we look at some ways to speed up the build, I thought I’d start with my definition of what a build isÂ…

What is a build?

Quite simply a build is the process of taking source code and producing some artifacts. The artifacts could be a war file, some test output or even a full blown desktop application. A project might (and probably should) have several different builds for different purposes – one for developers, QA, production etc.

Speeding up the build

If you have a build which you want to go faster, there are two approaches you can take

  1. Speed up the creation of the artifacts themselves
  1. Reduce the number of artifacts being created

    I’m not really going to talk much about the first point – that is typically either trivial or very hard. If you have to compile 10,000 source files there is a relatively little you can do to speed that up. If you have some tests which are IO bound, you can speed them up by mocking out the IO if possible.

    The biggest wins come when you can reduce what the build is doing, either by reducing the number of artifacts it is trying to produce, or reducing the scope of the artifacts being produced.

    Option 1: Reduce the work

    You might consider splitting the build on horizontal lines, in order to create fewer artifacts. For example a typically developer build will compile and run the unit tests, and might produce a deployed application from exploratory testing. You could split this single build into three separate builds – one to compile the code, one to test it, the other to create the deployed app. This is only useful if not all the artifacts are required – in many circumstances they are. One technique that has worked in my experience is the idea of a tiered build – we had two continuous integration builds, one designed for fast developer feedback which ran just the unit tests, and a longer QA build which runs all the regression and functional tests. This is something I’ll go into at a later date.

    Option 2: Divide and conquer

    The other approach is to split things along vertical lines. In the case of the developer build, keep the build compiling, testing and deploying the code, but just have it compile, test and deploy less code, by splitting the code along functional boundaries. This will work as long as the developer is working completely within said functional boundary – subdivide your code base too finely and you’ll introduce too much complexity in terms of integrating your artifacts. Subdivide too broadly and you’ll end up compiling and testing code needlessly.

    Attempting to break up a code base after work has already started is not a simple job. It will involve disruption to development, and your development effort will need to be refocused to work as much as possible within your new components. Working out where to break code up before you start work is likewise not easy – it might not be clear initially where the divisions can be made, and you might find out later that you didn’t need to break up the code, in which case all you’ve done is add complexity to your build process. In either case, whilst the individual component builds will end up being simpler, and the builds for each component shorter, builds at the integration level will become more complex.