Development | magpiebrain

As described before, the build pipeline is a series of builds, each performing some specific task. The result of one build becomes the input of the next.

Many people see the pipeline describing only those parts which can be automated – as such, you’ll often see the pipeline end far short of production ready code – once it’s got past the final automated barrier, the code then plunges into a grey mass of manual, distributed, often ad-hoc processes. For the build pipeline to work at all, it has to be continued all the way to producing production ready code. That is not to say the whole pipeline should be automated.

January 10, 2005

Update 1: Fixed arrows on two of the images – damn Visio…

Update 2: Tweaked the final diagram to show build artifacts being checked in and retrieved

Lets look at a fairly simple build.

January 7, 2005

First off, I’d like to say that 100% test coverage does not mean that your code is bug free – it simply means you haven’t written a test good enough to find a bug. Your tests could even cause all of your code to be executed, but not have a single assert anywhere.

How much assurance you can gain from having unit tests depends on the quality of the test itself, and on how much of your code is covered by tests. Tools like Clover and Emma can tell you what code gets checked as part of a test run.

A build is considered passed if it meets certain criteria. Typically these criteria include the fact that it compiles, or that the unit tests pass. What is to stop the code coverage being used as one of these criteria? Beforehand you agree a level below which code coverage is not allowed to fall (in an existing project this is easier – you could set the existing code coverage percentage as the target). Any new code has to have at least the same coverage if not better for the build to pass. If you submit new code in a continuous integration environment and the code coverage is worse, your submission gets rejected. In practise this will hopefully result in improving code coverage, which will also result in the target level being increased.

January 6, 2005

Update: Fixed typo

Introduction

Slow builds are perhaps the most irritating thing for a developer. Having worked on two projects now with +30 minute continuous integration builds, it’s something of a personal bugbear for me. Before we look at some ways to speed up the build, I thought I’d start with my definition of what a build is

What is a build?

Quite simply a build is the process of taking source code and producing some artifacts. The artifacts could be a war file, some test output or even a full blown desktop application. A project might (and probably should) have several different builds for different purposes – one for developers, QA, production etc.

Speeding up the build

If you have a build which you want to go faster, there are two approaches you can take

Speed up the creation of the artifacts themselves

Reduce the number of artifacts being created
I’m not really going to talk much about the first point – that is typically either trivial or very hard. If you have to compile 10,000 source files there is a relatively little you can do to speed that up. If you have some tests which are IO bound, you can speed them up by mocking out the IO if possible.

The biggest wins come when you can reduce what the build is doing, either by reducing the number of artifacts it is trying to produce, or reducing the scope of the artifacts being produced.

Option 1: Reduce the work

You might consider splitting the build on horizontal lines, in order to create fewer artifacts. For example a typically developer build will compile and run the unit tests, and might produce a deployed application from exploratory testing. You could split this single build into three separate builds – one to compile the code, one to test it, the other to create the deployed app. This is only useful if not all the artifacts are required – in many circumstances they are. One technique that has worked in my experience is the idea of a tiered build – we had two continuous integration builds, one designed for fast developer feedback which ran just the unit tests, and a longer QA build which runs all the regression and functional tests. This is something I’ll go into at a later date.

Option 2: Divide and conquer

The other approach is to split things along vertical lines. In the case of the developer build, keep the build compiling, testing and deploying the code, but just have it compile, test and deploy less code, by splitting the code along functional boundaries. This will work as long as the developer is working completely within said functional boundary – subdivide your code base too finely and you’ll introduce too much complexity in terms of integrating your artifacts. Subdivide too broadly and you’ll end up compiling and testing code needlessly.

Attempting to break up a code base after work has already started is not a simple job. It will involve disruption to development, and your development effort will need to be refocused to work as much as possible within your new components. Working out where to break code up before you start work is likewise not easy – it might not be clear initially where the divisions can be made, and you might find out later that you didn’t need to break up the code, in which case all you’ve done is add complexity to your build process. In either case, whilst the individual component builds will end up being simpler, and the builds for each component shorter, builds at the integration level will become more complex.

January 4, 2005

A quick post those people reading this site via JavaBlogs. You might like to know that you’ll no longer be receiving my “del.ico.us(My del.icio.us links)”:http://del.icio.us/padark links spliced in with my feed. If you still want to get them, you’ll need to subscribe to my “FeedBurner feed(magpiebrain – summary posts with del.icio.us links spliced in, RSS 2.0)”:http://feeds.feedburner.com/Magpiebrain. All available feed-types are listed on my “feeds page(magpiebrain – available RSS feeds)”:http://www.magpiebrain.com/feeds, or are available via the normal “auto-discovery techniques(http://diveintomark.org/archives/2002/05/30/rss_autodiscovery)”:http://diveintomark.org/archives/2002/05/30/rss_autodiscovery.

December 22, 2004

Introduction

With my sojourn at my current client due to come to an end at Christmas, I’ve started a period of documenting some of the things I’ve learnt. As I’ve “mentioned before(magpiebrain – There are some things you just don’t want to learn)”:http://www.magpiebrain.com/archives/2004/11/18/lessons , not everything I’ve learnt has been pleasant, but it’s all be informative. One of the most valuable experiences has been my first exposure to Weblogic. Previously I’d worked at smaller clients, and as such had only really used JBoss, so a chance to use what is considered by many to be the best of the “big” J2EE containers was welcome.

In general, I was fairly impressed with Weblogic. Yes it costs a huge amount of money. No, it isn’t perfect. But it does have some excellent features, especially impressive being it’s cluster management. So, for posterity’s sake (and for the sake of my own poor short term memory) I thought I’d introduce some Weblogic clustering concepts – domains, admin servers, node managers, managed instances and of course, clusters.
Continue reading…

December 20, 2004

Introduction

As you may remember, I’ve been struggling with a tricky build at my current client. Whilst many of the problems have been around how we’ve used the tools available to us (well, Ant), I realised that Ant itself might just not be up to the job. Once a build becomes non-trivial, you inevitably want to start using it as a program, something Ant itself is not really good at.

“SCons”:http://www.scons.org/ is a build API form python. Being based on a proper programming API the promise of testable build processes, nice syntax (no more ghastly constructs just to do the build equivalent of an @if@) and above all a system more intuitive to the average programmer than Ant is.

When looking to adopt any open source tool it has to prove itself to fairly quickly – in the case of SCons I set it a few challenges and see how it stacked up.
Continue reading…

December 17, 2004

I’m currently working on a project with a relatively large (~2000 line) Ant build file, with several supplemental files. The first thing that’s struck me, is that something about the nature of Ant itself makes people treat it very differently from the code it’s building. Rules concerning consistency, testing, maintainability and even common sense seem to go out the window. Whilst it’s true that Ant doesn’t go out of its way to help us – there is little tool support and the XML syntax is fundamentally unsuited to a program (which is really what an Ant file is) – if developers engaged their brain a little more, they’d make life a little easier for everybody.

I’m going to revisit some of the problems with Ant at a later date, but today I’d like to focus on one particular annoyance – the use of the full stop in naming of targets and properties.
Continue reading…

December 15, 2004

“FreeMind”:http://freemind.sourceforge.net/ is a pretty good open source “mind mapping(Wikipedia – Mind mapping)”:http://en.wikipedia.org/wiki/Mind_mapping tool, which we’ve been using to capture the efforts of various bits of brainstorming, as well as keep track of ever evolving processes. One of the processes we’ve detailed using FreeMind has been our build process – during which I was struck by just how good FreeMind could be at actually modelling an “ant”:http://jakarta.apache.org/ant build structure. High level tasks (such as @deploy@) can be easily broken down into smaller dependent tasks (@compile@, @copyToServer@ etc).
Continue reading…

December 2, 2004

Act 1: In support of the pessimist

Fred:: So, I’ve made this thing for you
Bob:: Cool! What’s that tube for – and why the electrodes?
Fred:: Don’t worry about that, have a play…
Bob:: Cool, now – what does this do? Oh, and if I poke a stick there, put my finger there, then stick my tongue in the power socket…*ouch*!
Fred:: Dammit! That’s not how you’re supposed to use it! I’ll never get that out of the carpet now….

==
Continue reading…

October 21, 2004

magpiebrain

Sam Newman's site, a Consultant at ThoughtWorks

Posts from the ‘Development’ category

Introduction

What is a build?

Speeding up the build

Option 1: Reduce the work

Option 2: Divide and conquer

Introduction

Introduction

Act 1: In support of the pessimist