Recently, both Paul Julius and Chris Read pointed out that I was perhaps the first person to document the concept of build pipelines, at least in terms of how it relates to continuous integration and the like. As it turns out, the original posts on the subject are from further back than I remember:
- An Introduction To Build Pipelining, 7th January 2005
- Automating The Build Pipeline, 10th January 2005
- The Agile Release Process, 14th of February, 2005
I plan to pull together my previous posts on the subject and update them a little, but in the meantime thought I’d give a bit of background as to where much of this came from.
A Harsh Introduction
My first exposure to continuous integration was by being dropped in at the deep end during my first ThoughtWorks project. The project in question was for an electronic point of sale system, and at its peak had over 50 developers in three countries working on the project. During this time I started reading up on the topic, specifically Martin Fowler & Matt Foemmel’s paper on the topic (Martin has since created an updated version).
Much of the experiences at this first, large project were dominated by long, slow build times, caused in part by an inability to separate out activities being performed by individual teams. A full discussion as to things we learnt from that project can certainly wait for another time, but I came out of that experience liking the concept of Continuous Integration, but feeling incredibly constrained by the actual implementation.
Subsequently, I worked as what we used to call a ‘Build Monkey’ at a London-based ISP. My role (which we now tend to call an Environment Specialist) was typically to identify the causes of build failure, keep the build running smoothly, as well as manage deployments to a number of different environments. Throughout this time, discussions around the theory behind managing Continuous builds for larger software teams was continuing – primarily with colleagues like Julian Simpson, Jack Bolles & others.
The challenge we seemed to face, time and again, was how you balance the various activities associated with getting software from developers machines into production, all whilst providing the fastest feedback possible.
Typically, we came at the problem from two different directions – in the first instance from the point of view of how to hammer our tools into supporting the kind of processes involved, but the more important angle was understanding what the pipeline – from developer workstation to production – actually was. This thinking can now be best thought of in terms of Continuous Deployment – although that topic is far more nuanced that the often simplistic thinking regarding systems where 50 deployments a day is possible, or even desirable.
The Present Day
Since I wrote my original articles, many other people have done work in this area, to the extent that tools like ThoughtWork’s own Cruise builds support for build pipelining & visualisation directly into the tool.
Update 1: Corrected spelling of Paul’s surname – sorry Paul!
3 Responses to “A Brief And Incomplete History Of Build Pipelines”
[…] Newman has just published a A Brief And Incomplete History Of Build Pipelines which makes interesting […]
[…] for some time, and was first documented publicly in 2005 (in a blog entry by our colleague Sam Newman). Dave and I have both written papers on the subject. More recently, there has been a new variation […]
And now, Jenkins and Hudson have a plugin that supports this – http://www.centrumsystems.com.au/blog/?p=121
Worth a look for all you that use the Hudson and Jenkins continuous integration servers.